Complete Issue in PDF

Transcription

April 2006
Volume 9 Number 2
Educational Technology & Society
An International Journal
Aims and Scope
Educational Technology & Society is a quarterly journal published in January, April, July and October. Educational Technology &
Society seeks academic articles on the issues affecting the developers of educational systems and educators who implement and manage such
systems. The articles should discuss the perspectives of both communities and their relation to each other:
• Educators aim to use technology to enhance individual learning as well as to achieve widespread education and expect the technology to
blend with their individual approach to instruction. However, most educators are not fully aware of the benefits that may be obtained by
proactively harnessing the available technologies and how they might be able to influence further developments through systematic
feedback and suggestions.
• Educational system developers and artificial intelligence (AI) researchers are sometimes unaware of the needs and requirements of typical
teachers, with a possible exception of those in the computer science domain. In transferring the notion of a 'user' from the humancomputer interaction studies and assigning it to the 'student', the educator's role as the 'implementer/ manager/ user' of the technology has
been forgotten.
The aim of the journal is to help them better understand each other's role in the overall process of education and how they may support
each other. The articles should be original, unpublished, and not in consideration for publication elsewhere at the time of submission to
Educational Technology & Society and three months thereafter.
The scope of the journal is broad. Following list of topics is considered to be within the scope of the journal:
Architectures for Educational Technology Systems, Computer-Mediated Communication, Cooperative/ Collaborative Learning and
Environments, Cultural Issues in Educational System development, Didactic/ Pedagogical Issues and Teaching/Learning Strategies, Distance
Education/Learning, Distance Learning Systems, Distributed Learning Environments, Educational Multimedia, Evaluation, HumanComputer Interface (HCI) Issues, Hypermedia Systems/ Applications, Intelligent Learning/ Tutoring Environments, Interactive Learning
Environments, Learning by Doing, Methodologies for Development of Educational Technology Systems, Multimedia Systems/ Applications,
Network-Based Learning Environments, Online Education, Simulations for Learning, Web Based Instruction/ Training
Editors
Kinshuk, Massey University, New Zealand; Demetrios G Sampson, University of Piraeus & ITI-CERTH, Greece; Ashok Patel, CAL
Research & Software Engineering Centre, UK; Reinhard Oppermann, Fraunhofer Institut Angewandte Informationstechnik, Germany.
Associate editors
Alexandra I. Cristea, Technical University Eindhoven, The Netherlands; John Eklund, Access Australia Co-operative Multimedia
Centre, Australia; Vladimir A Fomichov, K. E. Tsiolkovsky Russian State Tech Univ, Russia; Olga S Fomichova, Studio "Culture,
Ecology, and Foreign Languages", Russia; Piet Kommers, University of Twente, The Netherlands; Chul-Hwan Lee, Inchon National
University of Education, Korea; Brent Muirhead, University of Phoenix Online, USA; Erkki Sutinen, University of Joensuu, Finland;
Vladimir Uskov, Bradley University, USA.
Advisory board
Ignacio Aedo, Universidad Carlos III de Madrid, Spain; Sherman Alpert, IBM T.J. Watson Research Center, USA; Alfred Bork,
University of California, Irvine, USA; Rosa Maria Bottino, Consiglio Nazionale delle Ricerche, Italy; Mark Bullen, University of
British Columbia, Canada; Tak-Wai Chan, National Central University, Taiwan; Nian-Shing Chen, National Sun Yat-sen University,
Taiwan; Darina Dicheva, Winston-Salem State University, USA; Brian Garner, Deakin University, Australia; Roger Hartley, Leeds
University, UK; Harald Haugen, Høgskolen Stord/Haugesund, Norway; J R Isaac, National Institute of Information Technology,
India; Paul Kirschner, Open University of the Netherlands, The Netherlands; Rob Koper, Open University of the Netherlands, The
Netherlands; Ruddy Lelouche, Universite Laval, Canada; Rory McGreal, Athabasca University, Canada; David Merrill, Brigham
Young University - Hawaii, USA; Marcelo Milrad, Växjö University, Sweden; Riichiro Mizoguchi, Osaka University, Japan; Hiroaki
Ogata, Tokushima University, Japan; Toshio Okamoto, The University of Electro-Communications, Japan; Gilly Salmon, University
of Leicester, United Kingdom; Timothy K. Shih, Tamkang University, Taiwan; Yoshiaki Shindo, Nippon Institute of Technology,
Japan; Brian K. Smith, Pennsylvania State University, USA; J. Michael Spector, Florida State University, USA.
Assistant Editors
Sheng-Wen Hsieh, National Sun Yat-sen University, Taiwan; Taiyu Lin, Massey University, New Zealand; Kathleen Luchini,
University of Michigan, USA; Dorota Mularczyk, Independent Researcher & Web Designer; Carmen Padrón Nápoles, Universidad
Carlos III de Madrid, Spain; Ali Fawaz Shareef, Massey University, New Zealand; Jarkko Suhonen, University of Joensuu, Finland.
Executive peer-reviewers
http://www.ifets.info/
Subscription Prices and Ordering Information
Institutions: NZ$ 120 (~ US$ 75) per year (four issues) including postage and handling.
Individuals (no school or libraries): NZ$ 100 (~ US$ 50) per year (four issues) including postage and handling.
Single issues (individuals only): NZ$ 35 (~ US$ 18) including postage and handling.
Subscription orders should be sent to The International Forum of Educational Technology & Society (IFETS), c/o Prof. Kinshuk,
Information Systems Department, Massey University, Private Bag 11-222, Palmerston North, New Zealand. Tel: +64 6 350 5799 ext 2090.
Fax: +64 6 350 5725. E-mail: [email protected].
Advertisements
Educational Technology & Society accepts advertisement of products and services of direct interest and usefulness to the readers of the
journal, those involved in education and educational technology. Contact the editors at [email protected].
Abstracting and Indexing
Educational Technology & Society is abstracted/indexed in Social Science Citation Index, Current Contents/Social & Behavioral Sciences,
ISI Alerting Services, Social Scisearch, ACM Guide to Computing Literature, Australian DEST Register of Refereed Journals, Computing
Reviews, DBLP, Educational Administration Abstracts, Educational Research Abstracts, Educational Technology Abstracts, Elsevier
Bibliographic Databases, ERIC, Inspec, Technical Education & Training Abstracts, and VOCED.
ISSN
ISSN1436-4522
1436-4522.
(online)
© International
and 1176-3647
Forum
(print).
of Educational
© International
Technology
Forum of
& Educational
Society (IFETS).
Technology
The authors
& Society
and (IFETS).
the forumThe
jointly
authors
retain
andthe
the copyright
forum jointly
of the
retain
articles.
the
copyright
Permission
of the
to make
articles.
digital
Permission
or hard copies
to make
of digital
part or or
allhard
of this
copies
workoffor
part
personal
or all of
or this
classroom
work for
usepersonal
is granted
or without
classroom
feeuse
provided
is granted
that without
copies are
feenot
provided
made orthat
distributed
copies
are
fornot
profit
made
or or
commercial
distributedadvantage
for profit and
or commercial
that copies advantage
bear the full
andcitation
that copies
on thebear
firstthe
page.
full Copyrights
citation on the
for components
first page. Copyrights
of this work
for owned
components
by others
of this
than
work
IFETS
owned
mustbybe
others
honoured.
than IFETS
Abstracting
must with
be honoured.
credit is permitted.
AbstractingTowith
copy
credit
otherwise,
is permitted.
to republish,
To copy
to otherwise,
post on servers,
to republish,
or to redistribute
to post ontoservers,
lists, requires
or to redistribute
prior specific
to lists,
permission
requiresand/or
prior a
specific
fee. Request
permission
permissions
and/or afrom
fee. the
Request
editors
permissions
at [email protected].
from the editors at [email protected].
i
Guidelines for authors
Submissions are invited in the following categories:
• Peer reviewed publications: a) Full length articles (4000 - 7000 words), b) Short articles, Critiques and Case studies (up to 3000 words)
• Book reviews
• Software reviews
• Website reviews
All peer review publications will be refereed in double-blind review process by at least two international reviewers with expertise in the
relevant subject area. Book, Software and Website Reviews will not be reviewed, but the editors reserve the right to refuse or edit review.
• Each peer review submission should have at least following items: title (up to 10 words), complete communication details of ALL
authors , an informative abstract (75-200 words) presenting the main points of the paper and the author's conclusions, four - five
descriptive keywords, main body of paper (in 10 point font), conclusion, references.
• Submissions should be single spaced.
• Footnotes and endnotes are not accepted, all such information should be included in main text.
• The paragraphs should not be indented. There should be one line space between consecutive paragraphs.
• There should be single space between full stop of previous sentence and first word of next sentence in a paragraph.
• The keywords (just after the abstract) should be separated by comma, and each keyword phrase should have initial caps (for example,
Internet based system, Distance learning).
• Do not use 'underline' to highlight text. Use 'italic' instead.
Headings
Articles should be subdivided into unnumbered sections, using short, meaningful sub-headings. Please use only two level headings as far
as possible. Use 'Heading 1' and 'Heading 2' styles of your word processor's template to indicate them. If that is not possible, use 12 point
bold for first level headings and 10 point bold for second level heading. If you must use third level headings, use 10 point italic for this
purpose. There should be one blank line after each heading and two blank lines before each heading (except when two headings are
consecutive, there should be one blank like between them).
Tables
Tables should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be
provided for every table (below the table) and must be referenced in the text.
Figures
Figures should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be
provided for every figure (below the figure) and must be referenced in the text. The figures must NOT be larger than 500 pixels in width.
Please also provide all figures separately (besides embedding them in the text).
References
• All references should be listed in alphabetical order at the end of the article under the heading 'References'.
• All references must be cited in the article using "authors (year)" style e.g. Merrill & Twitchell (1994) or
•
•
"(authors1, year1; authors2,
year2)" style e.g. (Merrill, 1999; Kommers et al., 1997).
Do not use numbering style to cite the reference in the text e.g. "this was done in this way and was found successful [23]."
It is important to provide complete information in references. Please follow the patterns below:
Journal article
Laszlo, A. & Castro, K. (1995). Technology and values: Interactive learning environments for future generations. Educational Technology,
35 (2), 7-13.
Newspaper article
Blunkett, D. (1998). Cash for Competence. Times Educational Supplement, July 24, 1998, 15.
Or
Clark, E. (1999). There'll never be enough bandwidth. Personal Computer World, July 26, 1999, retrieved July 7, 2004, from
http://www.vnunet.co.uk/News/88174.
Book (authored or edited)
Brown, S. & McIntyre, D. (1993). Making sense of Teaching, Buckingham: Open University.
Chapter in book/proceedings
Malone, T. W. (1984). Toward a theory of intrinsically motivating instruction. In Walker, D. F. & Hess, R. D. (Eds.), Instructional
software: principles and perspectives for design and use, California: Wadsworth Publishing Company, 68-95.
Internet reference
Fulton, J. C. (1996). Writing assignment as windows, not walls: enlivening unboundedness through boundaries, retrieved July 7, 2004,
from http://leahi.kcc.hawaii.edu/org/tcc-conf96/fulton.html.
Submission procedure
Authors, submitting articles for a particular special issue, should send their submissions directly to the appropriate Guest Editor. Guest
Editors will advise the authors regarding submission procedure for the final version.
All submissions should be in electronic form. The editors will acknowledge the receipt of submission as soon as possible.
The preferred formats for submission are Word document and RTF, but editors will try their best for other formats too. For figures, GIF
and JPEG (JPG) are the preferred formats. Authors must supply separate figures in one of these formats besides embedding in text.
Please provide following details with each submission: Author(s) full name(s) including title(s), Name of corresponding author, Job
title(s), Organisation(s), Full contact details of ALL authors including email address, postal address, telephone and fax numbers.
The submissions should be uploaded at http://www.ifets.info/ets_journal/upload.php. In case of difficulties, they can also be sent via
email to (Subject: Submission for Educational Technology & Society journal): [email protected]. In the email, please state clearly that the
manuscript is original material that has not been published, and is not being considered for publication elsewhere.
ISSN
ISSN1436-4522
1436-4522.
(online)
© International
and 1176-3647
Forum
(print).
of Educational
© International
Technology
Forum of
& Educational
Society (IFETS).
Technology
The authors
& Society
and (IFETS).
the forumThe
jointly
authors
retain
andthe
the copyright
forum jointly
of the
retain
articles.
the
copyright
Permission
of the
to make
articles.
digital
Permission
or hard copies
to make
of digital
part or or
allhard
of this
copies
workoffor
part
personal
or all of
or this
classroom
work for
usepersonal
is granted
or without
classroom
feeuse
provided
is granted
that without
copies are
feenot
provided
made orthat
distributed
copies
are
fornot
profit
made
or or
commercial
for profit and
or commercial
bear the full
andcitation
that copies
on thebear
firstthe
page.
full Copyrights
citation on the
for components
of this work
for owned
components
by others
of this
than
work
IFETS
owned
mustbybe
others
honoured.
than IFETS
Abstracting
must with
be honoured.
AbstractingTowith
copy
credit
otherwise,
is permitted.
to republish,
To copy
to otherwise,
post on servers,
to republish,
or to redistribute
lists, requires
or to redistribute
prior specific
to lists,
permission
requiresand/or
prior a
specific
fee. Request
permission
permissions
and/or afrom
fee. the
Request
editors
permissions
ii
Journal of Educational Technology & Society
Volume 9 Number 2 2006
Table of contents
Special issue articles
Theme: Interoperability of Educational Systems
Editorial: Interoperability of Educational Systems - Editorial of Special Issue
Daniel Olmedilla, Nobuo Saito and Bernd Simon
Interoperability in Personalized Adaptive Learning
Lora Aroyo, Peter Dolog, Geert-Jan Houben, Milos Kravcik, Ambjörn Naeve, Mikael Nilsson and
Fridolin Wild
1-3
4-18
Providing Author-Defined State Data Storage to Learning Objects
Ayalew Kassahun, Adrie Beulens and Rob Hartog
19-32
Supporting Interoperability and Reusability of Learning Objects: The Virtual Campus Approach
Elisabetta Di Nitto, Luca Mainetti, Mattia Monga, Licia Sbattella and Roberto Tedesco
33-50
Spinning Interoperable Applications for Teaching & Learning using the Simple Query Interface
Frans van Assche, Erik Duval, David Massart, Daniel Olmedilla, Bernd Simon, Stefan Sobernig,
Stefaan Ternier and Fridolin Wild
51-67
Full length articles
The Tablet PC For Faculty: A Pilot Project
Rob R. Weitz, Bert Wachsmuth and Danielle Mirliss
68-83
Teachers’ Collaborative Task Authoring to Help Students Learn a Science Unit
Yavuz Akpinar and Volkan Bal
84-95
Teaching ICT to Teacher Candidates Using PBL: A Qualitative and Quantitative Evaluation
Sevinç Gülseçen and Arif Kubat
96-106
Recency Effect on Problem Solving in Interactive Multimedia Learning
Robert Zheng and Bei Zhou
107-118
Can a Hypermedia Cooperative e-Learning Environment Stimulate Constructive Collaboration?
Mary Victoria Pragnell, Teresa Roselli and Veronica Rossano
119-132
Web-based Learning in a Geometry Course
Hsungrow Chan, Pengheng Tsai and Tien-Yu Huang
133-140
Logistic Regression Modeling for Predicting Task-Related ICT Use in Teaching
Petek Askar, Yasemin Kocak Usluel and Filiz Kuskaya Mumcu
141-151
The Relationship between Educational Ideologies and Technology Acceptance in Pre-service Teachers
Ercan Kiraz and Devrim Ozdemir
152-165
A Web-Based Synchronous Collaborative Review Tool: A Case Study of an On-line Graduate Course
Fatma Cemile Serce and Soner Yildirim
166-177
An empirical assessment of pedagogical usability criteria for digital learning material with elementary
school students
Petri Nokelainen
178-197
A framework and a methodology for developing authentic constructivist e-Learning environments
Imran A. Zualkernan
198-212
Technology Adoption of Medical Faculty in Teaching: Differentiating Factors in Adopter Categories
Nese Zayim, Soner Yildirim and Osman Saka
213-222
Inducing Fuzzy Models for Student Classification
Ossi Nykänen
223-234
ISSN 1436-4522
1436-4522.(online)
© International
and 1176-3647
Forum(print).
of Educational
© International
Technology
Forum&ofSociety
Educational
(IFETS).
Technology
The authors
& Society
and the
(IFETS).
forum The
jointly
authors
retainand
thethecopyright
forum jointly
of theretain
articles.
the
Permissionoftothe
copyright
make
articles.
digital
Permission
or hard copies
to make
of part
digital
or all
orof
hard
thiscopies
work for
of part
personal
or allorofclassroom
this work use
for is
personal
grantedorwithout
classroom
fee provided
use is granted
that copies
without
arefee
notprovided
made or that
distributed
copies
for
are profit
not made
or commercial
or distributed
advantage
for profitand
or that
commercial
copies bear
advantage
the fulland
citation
that copies
on the bear
first page.
the full
Copyrights
citation onfor
thecomponents
of this workfor
owned
components
by others
of than
this work
IFETS
owned
must by
be
honoured.
others thanAbstracting
IFETS mustwith
be honoured.
Abstracting
To with
copy credit
otherwise,
is permitted.
to republish,
To copy
to post
otherwise,
on servers,
to republish,
or to redistribute
to post on
to lists,
servers,
requires
or to prior
redistribute
specifictopermission
lists, requires
and/or
priora
specific
fee. Request
permission
permissions
and/orfrom
a fee.
theRequest
editors permissions
iii
Book review(s)
Using Technology in Teaching
Reviewer: Jarkko Suhonen
235-237
Unlock the genius within; neurobiological trauma, teaching, and transformative learning
Reviewer: Richard Malinski
238-240
ISSN
ISSN1436-4522
1436-4522.
(online)
© International
and 1176-3647
Forum
(print).
of Educational
© International
Technology
Forum of
& Educational
Society (IFETS).
Technology
The authors
& Society
and (IFETS).
the forumThe
jointly
authors
retain
andthe
the copyright
forum jointly
of the
retain
articles.
the
copyright
Permission
of the
to make
articles.
digital
Permission
or hard copies
to make
of digital
part or or
allhard
of this
copies
workoffor
part
personal
or all of
or this
classroom
work for
usepersonal
is granted
or without
classroom
feeuse
provided
is granted
that without
copies are
feenot
provided
made orthat
distributed
copies
are
fornot
profit
made
or or
commercial
for profit and
or commercial
bear the full
andcitation
that copies
on thebear
firstthe
page.
full Copyrights
citation on the
for components
of this work
for owned
components
by others
of this
than
work
IFETS
owned
mustbybe
others
honoured.
than IFETS
Abstracting
must with
be honoured.
AbstractingTowith
copy
credit
otherwise,
is permitted.
to republish,
To copy
to otherwise,
post on servers,
to republish,
or to redistribute
lists, requires
or to redistribute
prior specific
to lists,
permission
requiresand/or
prior a
specific
fee. Request
permission
permissions
and/or afrom
fee. the
Request
editors
permissions
iv
Olmedilla, D., Saito, N. & Simon. B. (2006). Interoperability of Educational Systems - Editorial of Special Issue.
Educational Technology & Society, 9 (2), 1-3.
Interoperability of Educational Systems - Editorial of Special Issue
Daniel Olmedilla
L3S Research Center and Hannover University, Deutscher Pavillon, Expo plaza 1, D-30539 Hannover, Germany
Tel: +49 511 762.9741
[email protected]
Nobuo Saito
Keio University - Shonan Fujisawa Campus, Keio Research Institute at SFC, 5322 Endo, Fujisawa, Japan
Tel: +81 466 47 5049
[email protected]
Bernd Simon
Vienna University of Economics and Business Administration, Institute of Information Systems & New Media
Augasse 2-6, A-1090 Vienna, Austria
Tel: +43 1 31336 4443
[email protected]
Why a Special Issue on Interoperability of Educational Systems?
Interoperability - defined as the capability of different systems to share functionalities or data - has become a hot
topic for educational technologists. From the educational point of view the increasing attention for
interoperability research has been driven, for example, by
¾ the desire to collaborate on the development of content (maybe stored in multiple systems),
¾ the need for making content accessible in or via various systems (re-use),
¾ cross-organisational, collaborative learning and teaching,
¾ sharing of assessment data for the purpose of effective personalization of learning environments.
Economical motivations for interoperability include:
¾ securing investments in content development,
¾ making designs of learning environments exchangeable (good practices),
¾ increasing the user value of the systems provided by integrating components of other systems,
¾ allowing specialisation in the field, so that vendors can focus on particular aspects of the educational
value chain (e.g., content creation, assessment, skill management).
From the information systems (IS) design point of view interoperability is required in order to
¾ break up the technological isolation of learning management systems and alike by fully integrating
them in a company’s IT infrastructure,
¾ get access to crucial data stored in legacy systems,
¾ integrate business process-driven solutions with learning management,
¾ reduce time for and costs of system integration by providing reference specifications.
National and international standardization consortia, such as ISO, IEEE, ADL, AACE, CEN/ISSS, HR-XML,
IMS, or DIN, sometimes accompanied by research projects have produced an overwhelming number of
standards and specifications. The sometimes even competing work is still heavily discussed in the field. At this
stage of a global discussion process, not all answers have been provided yet. Many of us working in the field of
the technology-enhanced learning have been frequently confronted with issues such as: what are the use cases or
pedagogical models standards such as SCORM, LOM, or QTI have been designed for? Can the specifications
help to fulfil the requirements of learning environments or system integration projects? What about industry
adoption - have they reached a critical mass and if so, is an outstanding ability to solve a particular problem
driving the adoption or are other forces driving the process?
Motivated by the pedagogical, technological, and IS design driven need for interoperability, on the one hand, and
the still ongoing discussion about specifications and standards, on the other, this special issue is devoted to give
an overview about and advance current state-of-the art of interoperability research.
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at [email protected].
1
Some Metrics on the Call for Contributions to this Special Issue
This special issue has followed up a workshop held in conjunction with the 14th International World Wide Web
Conference (WWW2005), May 10-14, 2005, in Chiba (Japan). Supported by the workshop the call for
contributions has attracted 38 papers in total, from which 33 of them went into the review process (5 had to be
rejected at an early stage, because of a lack of relevancy to the special issue). Each paper was reviewed by a
minimum of two reviewers. In total 85 reviews were collected in a double-blind review process. Finally, four
papers were accepted for leading with an acceptance rate of 11%. Each accepted paper was assigned to a
responsible editor in order to ensure that reviewer’s comments were interpreted in the most beneficial way.
In this Special Issue
The issue is composed of five articles covering interoperability-related topics including the authoring of learning
material using metadata and improvement of reusability, protocols for the exchange of such metadata, extensions
to standard specifications for inclusion of new features in Learning Management Systems in a common way, and
integration of cross-domain technologies and standards for learning.
Aroyo et al. claim in “Interoperability in Personalized Adaptive Learning” (accepting Editor: Nobuo Saito) that
interoperability plays a crucial role in adaptive technologies. They discuss state-of-the art and main challenges of
interoperability specifications. Based on an enhanced Adaptive Hypermedia Application Model they model
systems that support distributed user profiling in the context of interconnected educational repositories.
Kassahun et al. (accepting editor: Daniel Olmedilla) extends the current SCORM standard in order to support
author-defined data for learning objects which require the management of state information. “Providing AuthorDefined State Data Storage to Learning Objects” describes such an extension and a plug-in, DBLink, that
implements it.
Di Nitto et al. discuss in “Supporting Interoperability and Reusability of Learning Objects: The Virtual Campus
Approach” (accepting editor: Bernd Simon) strengths and weaknesses of the latest SCORM specification. Based
on their experiences in the Virtual Campus project they propose specific extensions to SCORM’s metadata and
sequencing specifications. They claim that reusability of learning objects can be increased by intruding attributes
such as supervision mode, access modality or learning objective. The relationships they propose for connecting
learning objects in a learning flow shall provide enhanced capabilities for instructional designers to model
effective learning environments.
“Spinning Interoperable Applications for Teaching & Learning using the Simple Query Interface”, by Van
Assche et al. (accepting Editor: Nobuo Saito), describe the Simple Query Interface, a protocol intended to
provide an universal interoperability layer for the search and exchange of resource’s metadata. Authors discuss
the technical details of the interface and provide several case studies as a proof of its applicability.
Conclusions
The papers received to this call for contributions gave a clear sign that interoperability will still remain to be a
hot topic of research in the forthcoming years. The standards and specifications currently available seem to solve
only a small piece of the big puzzle. Issues such as how to describe learning resources best for re-use are still not
fully resolved while completely new topics such as connecting business processes with learning, for example via
re-usable competency definitions, arise while technology enhanced learning disseminates in corporate
environments.
So it remains clear that the seamless development of high quality content and making it accessible in a smart
way is still a vision that requires a lot of research to be performed. However, the authors of the above papers
have made their contributions to bring this ultimate goal one step closer.
Acknowledgements
Special thanks to our reviewers who have provided relevant and detailed comments to the authors of all papers
submitted and helped us in the selection process:
2
Allyn Radford, Ambjörn Naeve, Andreas Pinterits, Arthur Stutt, Bill Blackmon, Daniel R. Rehak, David
Massart, Eric Roberts, Erik Duval, Frans Van Assche, Hiroaki Chiyokura, Hiroshi Komatsuwaga, Hiroshi
Makoshi, Jan Brase, John Toews, John Williams, Juan Quemada, Keiko Okawa, Kinshuk, Kinya Tamaki, Lora
Aroyo, Lorna M. Campbell, Makiko Miwa, Marek Hatala, Martin Dzbor, Massimo Marchiori, Matthew J.
Dovey, Michael Sintek, Mitsuru Ikeda, Nobuyau Makoshi, Norm Friesen, Oleg Liber, Peter Dolog, Peter Spyns,
Peter van Rosmalen, Pythagoras Karampiperis, Rob Koper, Ruimin Shen, Seok-Choon Lew, Simos Retalis,
Stefano Ceri, Uwe Zdun and Wayne Hodgins.
3
Aroyo, L., Dolog, P., Houben, G-J., Kravcik, M., Naeve, A., Nilsson, M. & Wild, F. (2006). Interoperability in
Personalized Adaptive Learning. Educational Technology & Society, 9 (2), 4-18.
Interoperability in Personalized Adaptive Learning
Lora Aroyo
Technische Universiteit Eindhoven, Computer Science – Information Systems
PO Box 513, 5600 MB Eindhoven, the Netherlands
[email protected]
Peter Dolog
L3S Research Center, University of Hannover, Expo Plaza 1, 30539 Hannover, Germany
[email protected]
Geert-Jan Houben
Vrije Universiteit Brussel, Computer Science, Pleinlaan 2, 1050 Brussels, Belgium
[email protected]
Milos Kravcik
Fraunhofer FIT, Institute for Applied Information Technology
Schloss Birlinghoven, 53754 Sankt Augustin, Germany
[email protected]
Ambjörn Naeve and Mikael Nilsson
Knowledge Management Research (KMR) Group, Centre for user oriented IT-Design (CID)
Royal Institute of Technology (KTH), 100 44 Stockholm, Sweden
[email protected]
[email protected]
Fridolin Wild
Vienna University of Economics, and Business Administration
Institute of Information Systems and New Media, Augasse 2-6, 1090 Vienna, Austria
[email protected]
ABSTRACT
Personalized adaptive learning requires semantic-based and context-aware systems to manage the Web
knowledge efficiently as well as to achieve semantic interoperability between heterogeneous information
resources and services. The technological and conceptual differences can be bridged either by means of
standards or via approaches based on the Semantic Web. This article deals with the issue of semantic
interoperability of educational contents on the Web by considering the integration of learning standards,
Semantic Web, and adaptive technologies to meet the requirements of learners. Discussion is m ade on the
state of the art and the main challenges in this field, including metadata access and design issues relating to
adaptive learning. Additionally, a way how to integrate several original approaches is proposed.
Keywords
Semantic Interoperability, Learning Standards, Personalized Adaptive Learning, Meta-Data
Introduction
Looking into the past, we can see that most ideas how to learn are not new. What is new, are the circumstances
and opportunities. The existing education and training system has been engineered in an industrial age to suit the
needs of routinely performed manufacturing processes. The knowledge age we live in, however, is demanding
higher skilled jobs with capabilities for critical thinking, creativity, self-regulation, collaboration, and
interpretation. Accordingly, the number of so called ‘knowledge workers’ is rapidly growing. Moreover,
economic success has become increasingly dependant on flexible and anticipatory adjustment of business
models, processes, and behaviours (Hamel & Välikangas, 2003). Again accordingly, studies show that 50% of
all employee skills become outdated within three to five years (Moe & Blodgett, 2000).
As a consequence, we need new pedagogical methods to cover the exigencies in education and training visible
today. For technology enhanced learning, this boils down to one of the contemporary grand challenges (CRA,
2003): to develop environments that effectively enable each learner to get individual support in filling everchanging skills and competence gaps – i.e. to create environments for personalised adaptive learning. More
4
specifically, this requires “developing semantic-based and context-aware systems to acquire, organise,
personalise, share and use the knowledge embedded in web and multimedia content, […] and to achieve
semantic interoperability between heterogeneous information resources and services” (IST, 2004).
If our educational solutions are to be cost effective they have to share and reuse distributed learning resources to
achieve a critical mass of material. Then efficient retrieval mechanisms providing access to relevant and accurate
information are needed, like federated searches over distributed learning repositories. To reduce the overload of
retrieved resources and to provide individualized educational experiences personalization of learning activities
and learning materials has to be considered. As learning repositories have distinct systems, semantic and
structural heterogeneity, various interoperability issues arise that must be solved in open environments. These
issues concern all the conceptual components in adaptive educational systems, like sharing of learning content,
concepts, learner profiles, context models, learning design, adaptation and presentation specifications. There are
already solutions; e.g. for the exchange of learning objects and learner profiles, but this is just a part of the whole
complex problem.
Interoperability can be circumscribed as “a condition that exists when the distinctions between information
systems are not a barrier to accomplishing a task that spans multiple systems”(Christian, 1994). To minimize
costs and raise quality, we are not as interested in the possibility of interfacing through human effort. Semantics
(the study of meaning) is usually defined to investigate the relation of signs to their corresponding objects. In our
context, semantic interoperability thus denotes the study of how to bridge differences between information
systems on two levels:
¾ on an access level, where system and organisational boundaries have to be crossed by creating
standardised interfaces that sharing of system-internal services in a loosely-coupled way
¾ on a meaning level, where agreements about transported data have to be made in order to permit their
correct interpretation
Semantic interoperability can be achieved when models are available that more or less resemble each other and
where not too many (semantic) differences are to be detected and resolved. Here, we define model as a formal
and explicit representation that can be used to precisely describe a part of the design, e.g. the content or a user's
knowledge state.
The rest of this article is structured as follows. In Section 2, we are going to discuss the implications put forth by
semantic interoperability when applied to adaptive systems. This encompasses types, architectures,
representations, and interoperability problems emerging from distributed adaptive personalisation in technology
enhanced learning. In Section 3, we are going to present a formal model for personalised adaptive learning and
give an overview on existing interoperability alternatives, related standards, and projects. In Section 4 we
investigate issues related to the access of meta-data in interoperable personalised adaptive learning solutions. We
will focus on querying learning object repositories at first to subsequently present an illustrative example of
querying learner profiles. The article will conclude by a summary of the status quo reached so far in personalised
adaptive learning interoperability and the identification of problems which we feel need to be tackled in the
future.
Semantic Interoperability of Adaptive Systems
The increasing demand for personalization in e-learning applications leads to a process of user (learner) profiling
which is inherently distributed. For multiple applications to effectively share and exchange user information
between them for the sake of adapting the content to the user, they need to know the semantics of this user
information and therefore resolve the issues related to semantic interoperability. In this part of the article, before
we go into details later on, we consider the general state of the art in semantic interoperability in relation to
distributed user profiles by introducing methods, techniques, tools, and issues related to distribution and
semantic interoperability of adaptive systems.
In the past decade we have witnessed a growing interest in applying adaptation and personalization in numerous
application domains. The process of engineering information systems has shown a considerable change and
adaptation has been a significant driver of this change. Concept-based systems are information systems that
represent content using concept structures, i.e. that a model of the content (often referred to as domain model or
content model) is a characteristic element of the design. This approach generally includes, as relevant aspects,
the user's knowledge (user model) and the adaptation knowledge (adaptation model).
5
These systems are distinguished from systems in which the adaptation is defined without an explicit model of the
content (e.g. because the content and structure are rather straightforward or small). Adaptive concept-based
systems are becoming especially accepted in application areas where the main goal is to tailor large amounts of
information to the characteristics of predefined groups, or to the individual preferences and knowledge state of
the individual users. In the case of educational applications, it has become more or less standard that the systems
express a behavior matching the individual user. The construction of concept-based systems is not a
straightforward issue, certainly when the challenge is combined with the desire to add adaptation, e.g. for
adaptive hypermedia systems, adaptive web information systems, and adaptive task-based systems.
When we talk about adaptation and personalization, the user necessarily plays a fundamental role in the system
and therefore in its design. The system might want to record a user's preferences (e.g. learning and cognitive
styles, language), but also its assumptions about the current user's (knowledge) state. The systems typically
maintain a model of the individual user as an overlay of the domain model in order to record the current state of
the user with respect to their knowledge of domain concepts. These application dependant user models with the
preferences and the state of the user are integrated in the user profile that in general comprises the available
information about a user, which is used as a basis for adaptation of the content presentation to the user. In the
rest of this section we focus on the user model sharing. Interoperability of adaptive systems generally includes
sharing of the content, concepts, context, learning design, and adaptivity.
Distributed User Profiles
The nature of interoperability between adaptive systems and applications implies that there is a distributed
process of sharing and exchanging user profiles. This process includes on the one hand providing profile
information and on the other hand consuming user models for personalization or adaptation. The issue of
semantic interoperability in the context of user profiling is a direct consequence of the distribution in user profile
information. The decentralized process of distributed user profile information management demands a control
that is essential for a successful application of user profiles.
For a long time it has been very difficult or even impossible to share and exchange user profile data. More
recently, suppliers and consumers of user profiles have shown an increased awareness of the need for standards
for representing and exchanging user profile data, especially in the e-learning domain significant progress, (as
we will see later in this article). At the same time we observe that the amount and diversity of profile-based
applications makes it practically impossible to easily create a unified user profile infrastructure. One important
aspect that should not be underestimated is that the meta-data for user profile data implies a lot of manual labor
before it can effectively be exploited in data exchange between applications. However, the technological
advances of the last few years, especially in the context of web and semantic web research, can come to the
rescue with tools and methods to combine available data, to annotate profile information (semi-)automatically,
and to provide applications with necessary profile metadata.
The (semi-)automatic generation of metadata is an essential prerequisite for the semantic interoperability of
profile-based applications such as e-learning applications. The creation of such metadata usually requires
considerable intellectual input of humans. Contemporary web technology may offer opportunities for semantic
interoperability between applications and their meta-data on a large scale basis, which could not be achieved by
uncoordinated human input alone. When we investigate the means of automatic creation of metadata, we observe
that ontologies provide an option for achieving semantic coherence between profile data items. Tools can help to
minimize user efforts required for creating and maintaining annotations – and could therefore help to increase
their overall quality.
Architecture and Representation
To effectively manage distributed user profiles in adaptive concept-based systems, an architecture for distributed
profile exchange and management is needed. Different types of systems use different kinds of architectural
solutions. There are differences in the way in which user profiles are used, and this has consequences for the
personalization. We want to differentiate three basic architectural types: adaptive web-based systems, adaptive
hypermedia systems, and adaptive task-based systems.
All these architectures share the facility to maintain a representation of assumptions about one or more
characteristic(s) in the models of individual users. In other words the system should maintain a model about the
6
user that for instance contains assumptions about their knowledge, misconceptions, goals, plans, preferences,
tasks, or abilities. There are numerous issues that need to be considered regarding the user representation in a
complex approach, including user environment (e.g. class, school, family, background), roles and stereotypes,
historic and sensor information, trust and acceptance, generality and domain independence, expressiveness,
inferential capabilities, import and export, privacy, and mobility.
Obviously we can also distinguish different ways to communicate user profile data, e.g. via a centralized server,
via peer-to-peer communication, using agent-based techniques, or using a constraint-based approach. For
distributed user profile architectures, we need data models and languages for profile meta-data, especially to
describe the semantics and semantic differences.
The languages and technologies designed for the development of the semantic web provide useful instruments
for the representation of semantics of profile data. We mention the concept of ontology as an explicit
specification of a conceptualization. This basically means that an ontology is a formal way of describing (some
aspects of) a possible real world. With this key concept, the semantic web research has given us languages that
are useful for the basic interoperability of user profile data. The semantic web provides a framework for
expressing and using ontologies through e.g. the use of RDF, RDF Schema and OWL (RDF, 2005). These
languages come also with relevant tool support, such as APIs (e.g. Jena and Sesame), browsers and editors (e.g.
Protege and KAON), as well as reasoners.
Interoperability Issues with Distributed Profiles
Representing user profile data is one step in the process, but with the distribution come several interoperability
problems and issues related to their semantic differences. As examples, we have incompatibility (both between
profiles and between profiles and applications), incompleteness in the sense of information missing from
profiles, and contradiction in (unified) profiles.
The need to consider these issues arise from the fact that a learner may attempt to use an application that requires
more information than the user’s profile can provide, or that responds with information that cannot be
accommodated in the user’s profile. The complementary case arises when an application cannot handle
parameters such as preferences, specified by the learner or provides a response that contains too little
information to enable the user to choose between alternate follow-up actions. Another class of problems arises
when the learner’s profile contains sufficient information but the application possesses information of its own
that disagrees with the information present in the learner’s profile, because of conflicting values or meanings.
When two e-learning applications are directly interacting to provide a learner with a certain service, but without
the direct involvement of the user, they may face the situation of having access only to partly overlapping,
incomplete profiles. The question now is how to resolve overlaps and fill existing gaps. When applications are
allowed to autonomously replenish missing data, it might happen that two applications create contradictory
information. Can and should this be prevented from happening? And if not, how can contradictory data be
corrected afterwards? How can co-existing, possibly conflicting data be dealt with?
Other sources of contradictory information are different versions of the same information (freshness). The issue
here is whether to trust the most recent version or to establish a procedure to validate information. When facts
are contradictory within one source (document), one speaks of an inconsistency. When information statements
from different sources are contradictory, one speaks of disagreement. A disagreement may turn up when two
sources are merged (e.g. in a data warehouse project or virtually as described above). These two situations
require different handling. In the context of interoperability, one may assume as a starting point that the sources
are consistent. The proper treatment of disagreement is the more relevant problem to tackle.
These examples illustrate the situations that have to be prepared for and dealt with. All these problems can be
discussed from different angles: on the level of schemas or ontologies, at the level of instances, within an
information source, or between information sources. The main question is how to identify and deal with missing
or incomplete information. Interoperability problems in distributing user profiles include imprecise information,
imprecise manipulation, uncertain information, schema and ontology mapping, data clearing, inconsistency,
mediation, data dissemination, data replication, conflict detection and reconciliation. Techniques and
architectures to be considered for solving these interoperability problems draw from different research areas –
such as classical databases, data warehouses, mobile information systems, and semantic web technologies.
7
A Formal Model and Related Standards
After giving the general perspective on interoperability and distribution of adaptive components, we now turn to
formal models and standards that play a role in the adaptive hypermedia systems that are at the basis of most
adaptive learning systems. To start with, we observe that the knowledge driving the adaptation process can be
represented in adaptive hypermedia systems as five complementary models (Figure 1) – the domain model
specifies what is to be adapted, the user and context models tell according to what parameters it can be adapted,
and the activity (instruction) and adaptation models express how the adaptation should be performed. This model
enhances the Adaptive Hypermedia Application Model (De Bra et al., 1999; Wu et al., 2001), which is based on
the Dexter Hypertext Reference Model (Halasz & Schwartz, 1994). We use an analytical approach to identify the
different design aspects in which the separation between various types of knowledge asks for interoperability as
well.
Figure 1: Enhanced Adaptive Hypermedia Application Model
Thus, in accordance with cognitive science results, instead of uniform rules that should control the whole
personalized adaptive leaning experience, several specialized parts take care of particular functions and interact
with each other. Note that individual models may be distributed in reality. In the following paragraphs we
discuss formal models and standards that apply to each of the particular models. As we see, the existing
standards do not really support interoperability as a common abstract model is missing. They can be used in
isolation, but that is not desirable.
Domain Model
The domain model specifies the conceptual design of an adaptive hypermedia application, i.e. what will be
adapted. The information structure of a domain model in a typical adaptive hypermedia system can be
considered as two interconnected networks of objects (Brusilovsky, 2003):
¾ Knowledge Space – a network of concepts
¾ Hyperspace – a network of hyperdocuments
Accordingly, the design of an adaptive hypermedia system involves three key sub-steps: structuring the
knowledge, structuring the hyperspace, and connecting the knowledge space and the hyperspace.
Modern adaptive hypermedia systems model the knowledge space of the domain as a semantic network
(Brusilovsky, 2003). They use network models with several kinds of links that represent different kinds of
relationships between concepts, e.g. prerequisite links between concepts which represent the fact that one of the
related concepts has to be learned before another, or classic semantic links “is-a” and “part-of”. These domain
8
ontologies represent the expert’s knowledge about the domain. The domain model offers also a natural
framework for goal modeling. An individual educational goal can be modeled as a structure (e.g. sequence, tree,
or stack) of subsets of domain concepts.
The hyperspace consists of learning objects. The Learning Object Metadata (LOM) standard defines a learning
object as any entity, digital or non-digital, that may be used for learning, education or training (LOM, 2002).
Content models identify different kinds of learning objects and their components. A comparative analysis of six
known content models (Verbert & Duval, 2004) led to the creation of a general model that includes the existing
standards and distinguishes between:
¾ Content fragments – learning content elements in their most basic form (text, audio, video),
representing individual resources uncombined with any other; instances
¾ Content objects – sets of content fragments; abstract types
¾ Learning objects – they aggregate instantiated content objects and add a learning objective
The standards that can be used at this level include
¾ IMS Content Packaging – description and packaging of learning material
¾ IMS Question and Test Interoperability – XML language for describing questions and tests
¾ IEEE Learning Object Matadata – description of learning resources
Learning objects distributed in various repositories with associated metadata can be nowadays retrieved by
means of federated search (Simon et al., 2005). Early adopters have already started using these services. These
users can be either learners using learning objects in a similar way like textbooks, or teachers that need suitable
materials to support their classes and possibly applying blended learning approaches.
User Model
A user model represents relevant user characteristics, like preferences, knowledge, competencies, tasks, or
objectives. The majority of educational adaptive hypermedia systems use an overlay model of user knowledge
(Brusilovsky, 2003). The key principle of the overlay model is that for each domain model concept, an
individual user knowledge model stores some data that is an estimation of the user knowledge level on this
concept. A weighted overlay model of user knowledge can be represented as a set of pairs “concept-value”, one
pair for each domain concept. Some systems store multiple evidences about the user level of knowledge
separately. Another alternative to model the user knowledge is provided by a historic model that keeps some
information about user visits to individual pages. Some systems use this model as a secondary source of
adaptation.
The learner’s goals can be modeled as a set of concepts (competencies) that can be represented similarly to the
overlay model. Additionally to these dynamic dimensions the learner model includes also a more static one –
user preferences. The most relevant ones are preferred cognitive and learning styles, as well as the language.
The following standards relate to user modeling:
¾ IEEE Public And Private Information – specifies both the syntax and semantics of a 'Learner Model,'
which will characterize a learner and his or her knowledge/abilities
¾ IMS Learner Information Package – learner information data exchange between systems that support
the Internet learning environment
New approaches for open-world user modeling able to elicit extended models of users and to deal with the
dynamics of a user’s conceptualization are required to effectively personalize the Semantic Web. The ultimate
goal is to pave the road towards utilizing the adaptive learning environments with an enhanced learner model
that will integrate different learner perspectives, such as knowledge, personal preferences, and interests,
browsing patterns, cognitive and physical state. Complementing to the open-world view is the fact that learner
can be also modelled by various peers and learning service providers in a distributed network. The key research
issues in this area are: interoperable learner model artefacts, techniques for describing such artefacts, methods for
extracting relevant learner model parts for particular learning situations or services, techniques for exchanging
and communicating such artefacts. Other major challenges and directions for further research include effective
learner modeling, addressing personalization for disabilities, considering the time and evolving user context(s),
including the user control, and relating them to the issues of privacy. New challenges are recognized with regard
to moving to open educational semantic world.
9
Context Model
The term context can be defined as “the circumstances in which an event occurs; a setting”. We are considering
it as the environment characteristic rather than the user ones that are represented in the user model. The user
(learner) and context models specify to what parameters the application should adapt. One of the primary aims is
to generate both objective and subjective metadata automatically, based on the current context and by means of
suitable sensors – physical as well as semantic ones. This will enable more precise retrieval of the data when
learning objects are processed or elaborated by students and teachers.
Context management deals with such issues as automatic acquisition of context metadata, contextualized
delivery of content, activities, and services. We observe that the current exchange formats for contextualization
of resources need to be extended for capturing and handling additional context data. There are no relevant
standards for the context model yet.
Modern context-adaptive systems employ generic and mobile user models to provide human centered and
ubiquitous services. As the learner models will also include situated learning in the context of work where
learning is an integrated part of working (learning on demand) user competence profile has to be taken into
account, as well as analyzing group modeling and pattern recognition in the user behavior. Metadata about the
learner and her context should enrich queries into learning object repositories to maximize the efficiency of
information retrieval.
Instruction Model
The instruction (pedagogical) and adaptation models specify the navigational design for an adaptive hypermedia
application. Together with the presentation specification they tell how the adaptation should be performed, so
they describe the dynamics (“flow”) of the system.
Learning design is a way of modeling learning activities and scenarios, as different types of learners prefer
different learning approaches – learning styles. A key axiom that is common to all major educational approaches
says that “learners perform activities in an environment with resources”. The IMS Learning Design (Koper &
Tattersall, 2005) uses the metaphor of a theatrical play to describe the workflow involved in learning and
teaching scenarios. It separates the design of the pedagogical model from the content. Main challenges include
encoding dynamic interactions between users and system, representing scenarios (objectives, tasks/activities),
and describing interactions between participating roles and system services.
Standards that are related to the design of pedagogical activities are:
¾ IMS Simple Sequencing – representing the intended behavior of an authored learning experience
¾ IMS Learning Design – defining diverse learning approaches (scenarios); it defines 3 levels of
implementation and compliance (IMS LDIM, 2003):
Level A – the core vocabulary needed to support pedagogical diversity
Level B – introduces properties and conditions, which enable implementation of adaptive strategies
Level C – notification that can support adaptive self-driven and collaborative learning
The primary aim of the learning design standard was to provide an explicit notation that would enable
interoperability on the level of systems. Thus the instructional knowledge does not have to be hardwired in the
learning environment, but authors can define it specifically for each learning application, representing an
appropriate pedagogical pattern. To allow personalization a method can contain conditions, i.e. If-Then-Else
rules that further refine the assignment of activities and environment entities for persons and roles. Conditions
can be used to personalize learning designs for specific users. The ‘If’ part of the condition uses Boolean
expressions on the properties that are defined for persons and roles in the learning design. Thus IMS Learning
Design can be used to model and annotate adaptive learning design, but designing more complex adaptivity
behavior can cause problems. Currently, it is not possible to annotate learning content or define student roles
considering their characteristics. We can say that a primary objective of this standard was interoperability
between various systems, rather than reusability of learning design methods in various courses or learning units.
10
Adaptation Model
This model specifies the specific adaptation semantics – seen, mastered, recommended objects, etc. Adaptation
specifications define the status of individual objects (e.g. content objects or fragments) based on their metadata
attributes and the current parameters of the user model and the context model. The adaptation effect is usually
achieved by adapting contents and links using suitable adaptation techniques that can be chosen on this level.
The taxonomy of adaptive hypermedia technologies (Brusilovsky, 2001) includes:
¾ Adaptive presentation (content level adaptation) to ensure for different classes of users that the (most)
relevant information is shown and the user can understand it, e.g. adaptive text presentation, adaptive
multimedia presentation, adaptation of modality
¾ Adaptive navigation support (link level adaptation) to guide the user towards the relevant, interesting
information, e.g. direct guidance, adaptive link sorting, adaptive link hiding, adaptive link annotation,
adaptive link generation, map adaptation
There can be also other adaptation dimensions, e.g. adaptive learning activity selection, adaptive
recommendation, or adaptive service provision.
The Adaptive Hypermedia Application Model or AHAM (mentioned earlier) uses Condition-Action rules and
due to their complexity, it is not supposed that authors will write all the rules by hand. Some other models build
upon AHAM identifying additional relevant layers, with the objective to enable reusability at various levels,
focusing mainly on adaptation strategies and techniques. LAOS (Cristea & de Mooij, 2003) is a generalized
model for generic adaptive hypermedia authoring, based on the AHAM model and on concept maps. It aims at
clear separation of primitive information (content) and presentation-goal related information (e.g. pedagogical
information). Previously they have defined a layered model for adaptive hypermedia authoring design
methodology for (WWW) courseware (Cristea & Aroyo, 2002). This model suggested the usage of the following
main three layers:
¾ Conceptual layer expressing the domain model (with sub-layers: atomic concepts and composite
concepts – with their respective attributes)
¾ Lesson layer (of multiple possible lessons for each concept map or combination of concept maps)
¾ Student adaptation and presentation layer (based on: adaptation model and presentation model)
All these layers should have been powered by the adaptation engine. Already they were using the lesson model
as an intermediate one between the domain model, the user and adaptation model. LAG (Cristea & Calvi, 2003)
is a generalized adaptation model for generic adaptive hypermedia authoring. The idea behind it was to let the
author of adaptive educational hypermedia work on a higher semantic level, instead of struggling with the
‘assembly language of adaptation’. Furthermore, these patterns should represent the first level of reusable
elements of adaptation. However, reusability can go further than that. Even this adaptation language might still
be difficult to handle for some authors (teachers). So reuse should be aimed for even at the level of adaptation
strategies (that correspond to cognitive/learning strategies).
On a higher level the presentation specification defines how to present the chosen adaptation techniques as well
as how the objects with a particular status should be presented to the user (e.g. hiding, sorting, emphasizing,
annotation techniques).
Accessing Meta-Data of Adaptive Learning Systems
Part of solving the semantic interoperability problem is that systems have to mutually understand their access
mechanisms to learning content objects and associated meta-data. More specifically, they must know
programming interfaces to connect to, retrieve, and manipulate all necessary meta-data. The Application
Program Interfaces (API) are either domain specific, i.e. they are based on specific metadata models, or they are
generic, usually suitable to query metadata based on multiple schemas by making use of general purpose query
languages like SQL. First we are dealing with the issues related to querying learning repositories and then we
present an illustrative example for querying learner profiles.
Generic APIs to Access Learning and Learner Repositories
One possibility to access learning content and learner profiles is to use generic APIs. Those APIs usually provide
functions to execute general purpose query languages independent of a domain. Examples of such APIs are
11
Simple Query Interface (Simon et al., 2005) used mostly exclusively for but not limited to querying metadata
about learning objects or APIs to access user profiles at User Modeling Servers (Fink & Kobsa, 2002).
Interoperability among learning object repositories requires a common communication framework for querying
and retrieving references to the stored objects. In the following we present different query APIs in the learning
domain.
The Learning Object Interoperability (LORI) Framework is an abstract model for interoperability issues in the
context of learning technology (Simon et al., 2005). LORI is a layered integration architecture, which defines
services in order to achieve interoperability between independent learning object repositories. These services can
be differentiated into core services (e.g. authentication services, session management services) and application
services (e.g. retrieval services, provision services). Part of the LORI framework is the Simple Query Interface
(SQI 2005). SQI is an API that provides method support for asynchronous and synchronous queries. Albeit SQI
is in principle not bound to a specific schema and is thus only dependant on a jointly defined canonical schema
of the specific community where it shall be applied, several examples like the HCD-Suite (www.hcdonline.com) can be found that use application profiles to mix standardized concepts from IEEE LOM, Dublin
Core, and other standards simultaneously in order to satisfy the needs inherent in their community. One of its
major design objectives of SQI was to keep the specification simple and easy to implement.
The Content Object Repository Discovery and Resolution Architecture (CORDRA) is supposed to develop into
an abstract, formal model for repository federations and their interoperability (CORDRA, 2004). Currently,
CORDRA encompasses similarly to OpenURL an identifier infrastructure and is currently not bound to any
specific search interface implementation (Rehak, et al 2005). OpenURL (OpenURL, 2004) is an initiative to
investigate the problem of uniquely identifying resources.
Z39.50-International: Next Generation (ZING) covers a number of initiatives by Z39.50 implementers to make
Z39.50 (ZING, 2001) more broadly available and to make Z39.50 more attractive to information providers,
developers, vendors, and users. SRW is the Search/Retrieve Web Service protocol, which is developed within
ZING and aims to integrate access to various networked resources, and to promote interoperability between
distributed databases, by providing a common utilization framework. SRW is a web-service-based protocol
(SRW, 2004). SRW takes advantage of CQL ("Common Query Language"), a powerful yet human-readable
query language. SRU, the Search and Retrieve URL Service, is a companion service to SRW. Its primary
difference is its access mechanism: SRU is a simple HTTP GET form of the service (SRU, 2005). SRW
encourages the use of Dublin Core, but is in general schema-neutral (like SQI).
The purpose of the IMS Digital Repository Interoperability (DRI) Specification (IMS DRI, 2003) is to provide
recommendations for the interoperation of the most common repository functions. The DRI specification
presents five core commands, i.e. search/expose, gather/expose, alert/expose, submit/store, and request/deliver,
on a highly abstract level. The specification leaves many design choices for implementers. For example, while
recommending Z39.50 (with its own query language) it also recommends XQuery as a query language.
OKI (Open Knowledge Initiative) is a development project for a flexible and open system to support on-line
training on Internet (OKI, 2004). OKI has issued specifications for a system architecture adapted to learning
management functions. One of the main characteristics of the project is its commitment to the open source
approach for software component development. OKI supplies specifications for a model of functional
architecture and an API called Open Service Interface Definition (OSID). OSID is directed towards
specifications for a flexible and open source model of functional architecture. Service Interface Definitions
(SIDs) organize a hierarchy of packages, classes and agents and propose Java versions of these SIDs for use in
Java-based systems and also as models for other object-oriented and service-based implementations.
Components developed by OKI are compliant with specifications issued by IMS and ADL SCORM.
Edutella is an RDF-based Peer-to-Peer infrastructure for querying distributed learning object repositories that
comes with its own query language QEL (QEL, 2004). The Resource Description Framework (RDF) is one of
the key pillars of the Semantic Web (RDF, 2005). RDF is an extensible way to represent information about
(learning) resources. One of RDF’s design assumptions is that resources are identified by a Unique Resource
Identifier (URI) allowing various users and agents to make assertions about uniquely identified objects. The API
in Edutella was used both to query learning resource metadata and learner profiles in Elena project.
12
Domain Specific APIs to Access Learning and Learner Repositories
Another possibility how to access learning and learner repositories is to use domain specific APIs. The learning
repositories nowadays conform to Learning Object Metadata standard by applying its profiles. The concepts
from LOM can be used to design APIs and libraries to query the repositories by using the concepts from the
domain of interest. Similarly, several open specifications have emerged to provide shared information models to
represent learner profiles like IEEE PAPI (IEEE PAPI, 2003) or IMS LIP (IMS LIP, 2001). The concepts from
those information models can be used to build APIs for accessing learner profiles. An example of learning
repositories domain specific API is EduSource. EduSource project (Hatala et al., 2004) aims to implement a
holistic approach to building a network for learning repositories. As part of its communication protocol –
referred to as the EduSource Communication Language (ECL) – the IMS Digital Repository Specification was
bound and implemented. A gateway for connecting between EduSource and the NSDL initiative, as well as a
federated search connecting EduSource, EdNA and Smete serve as a first showcase.
A different example of domain specific API is the API for accessing learner profiles. The Lerner API (Dolog &
Schäfer, 2005) was developed in the context of FP5 EU/IST project Elena – Creating Smart Spaces for Learning.
The API is based on a learner ontology. Figure 2 depicts an excerpt of a learner profile ontology configured from
fragments based on three specifications (the Elena project web site http://www.elena-project.org and its
personalization section provide the complete ontology in RDFS). The abbreviated syntax for namespaces is used
in concept and relation labels (e.g. qti stands for Question and Test Interoperability namespace at
http://www.elena-project.org/images/other/qtilite.rdfs). The default namespace is http://www.elenaproject.org/images/other/learner.rdfs. The conceptual model describes a situation where a learning performance
(IEEE PAPI is used to model performance and portfolio, http://ltsc.ieee.org/archive/harvested-200310/working_groups/wg2.zip) of a student is exchanged in terms of his achieved competencies (IMS RDCEO –
Reusable Definition of Competency and Educational Objectives, http://www.imsglobal.org). The competencies
have been evaluated by learner assessment (e.g. tests) and were derived from learning objectives of tests (IMS
QTI). Furthermore, all other educational activities, further materials, and projects created within the activities are
reported within the portfolio of the performance. Additional information which is reported under preferences
(IMS LIP) comprises language, device, resource and learning style preferences. The standards and open
specifications guarantee wider acceptance between e-learning systems and as such can be seen as good
candidates for the learner exchange models. Currently, none of the referenced standards present their metadata in
a way that makes it possible to use them in combination as depicted above. Therefore, an RDF translation of
these standards had to be developed, which made it possible to use them in combination. This RDF translation is
‘unofficial’, and we therefore view it as an important direction for future standardization work that the standards
use a common framework such as RDF and the Semantic Web, to enable the added value of using the standards
together.
Figure 2: Conceptual model for learner profile – an excerpt
Figure 3 depicts several possible scenarios of how to access and exchange learner profile fragments. The
fragments can be accessed programmatically by the use of a Java API, the web service which exports the learner
model through the API and acts as a learner model server, and through a query infrastructure for RDF
repositories like Edutella (Nejdl et al. 2002).
13
Figure 3: Use of API in several scenarios – an example
A Java API has been developed. It is structured according to the learner ontology fragments mentioned above.
The API is meant to be used to retrieve, insert, and update the learner profiles stored in the structures described
above. The API defines a class and properties for each class from the RDFS for the learner model. The interface
provides access functions for getting, deleting and updating a model of the fragment. It provides further
functions to derive additional information or to process more complex manipulations over referenced
information types as well. The API is implemented for the RDF representation (instances of the RDFS described
above). The API is easily extensible by providing further specializations if additional extensions and interface
implementations for local repositories and data models are needed.
The second implementation is provided through web services where several clients can access one model which
is persistent on one server. The server holds the main model, i.e. the data of a learner profile gathered from
several sources, and handles all requests from the clients. Each client is uniquely identified at the server and can
be used by a browsing or assessment system. Furthermore, a client can be used by other learning systems which
want to make use of the learner profiles or which want to contribute to them. The model can be accessed directly
by invoking functions of a web service or in a synchronized replicated way; i.e. each client has its own
repository which is synchronized with the main server every time a change occurs. The web services framework
can be used in a distributed way as well (several servers exchanging learner models between each other).
The learner profiles are created in RDF. Therefore, a query infrastructure for RDF data is another access option.
Edutella provides a Datalog-based language to query RDF data provided in a distributed P2P environment. This
option enables to collect various fragments by utilizing for example the algorithm from (Dolog, 2004). Another
advantage of the P2P sharing infrastructure used with the learner profiles is that it can facilitate an expert finding
based on the provided profile which can be queried by people who need a help in learning.
Summary and Conclusion
Complex problems require more knowledge than any single person possesses, therefore it is necessary that all
involved stakeholders communicate, collaborate, and learn from each other. The process of arranging
personalized adaptive learning experiences is a very complex one and usually people with different expertise
have to collaborate to achieve a good quality solution. The complexity of this problem comes from the difficulty
to formalize all the knowledge necessary in the pedagogical process. The authoring process can be simplified if
at various levels of the application reusable components are constructed that can be assigned to the formal
models mentioned earlier in this paper – learning objects, domain ontologies, pedagogical methods, and
adaptation specifications. Following standards requires an increased initial investment, but has a higher potential
for the future.
14
It has been observed that the more context is assigned to learning objects the lower is their reusability (Hodgins,
2005). The validity of this statement can be enhanced also for specifications of learning activities and adaptivity.
As stated by R. Koper (Koper & Tattersall, 2005) “the notation must make it possible to identify, isolate, decontextualise and exchange useful parts of a learning design so as to stimulate their reuse in other contexts”.
Therefore it would be beneficial to distinguish well-defined learning layers with clear interfaces, so that each
object of a given layer can be substituted with other objects of the same layer and combined with other objects
from different layers in order to build a comprehensive solution.
There is a lack of support for adaptive behavior in existing learning standards that leads to higher costs and lower
reusability of personalized learning solutions. B. Towle and M. Halm claim that IMS LD provides a way to
implement simple adaptive learning strategies, but not complex forms of adaptive learning, like multiple rules
interactions or enforced ordering (Koper & Tattersall, 2005). The aLFanet system (van Rosmalen et al., 2006)
was built according to a standard-based model for adaptive e-learning. They have found out that learning
standards are not harmonized to work with each other. Additionally, available tools are too complex for nonspecialized authors. It is necessary to improve usability and minimize complexity of the authoring tools. Another
approach (Zarraonandia et al., 2006) has focused on reusability at the level of learning design. In this case an
architecture is being developed that will automatically adapt units of learning to their actual context of execution
via runtime interpretation of small adaptive actions that are specified separately from the Learning Design
definition.
In the WINDS project (Kravcik et al., 2004; Kravcik & Specht, 2004) authors without programming skills could
produce adaptive courses by specifying declarative knowledge for adaptation by means of pedagogical metadata.
This together with procedural knowledge encoded in the course player generated adaptive delivery of courses. A
generalization of this approach (Kravcik, 2004) aimed at more flexibility, reusability and interoperability of
partial learning resources via separation of different kinds of knowledge and their interaction, taking into account
a typical learning design process as well as content object preferences for various learner profiles and contexts.
A challenge is the creation and use of ontologies to represent various types of knowledge relevant for
personalized adaptive learning (Knight et al., 2006). Such ontologies could be used by software agents to assist
authors in the design of individualized learning or even to directly generate such experiences themselves.
This article is aiming to map the current situation in the area of interoperability for adaptive learning
components. We have focused on general aspects of semantic interoperability of adaptive systems, formal
models and standards, as well as access to metadata, and have given examples of concrete tools, applications,
and suggestions how to integrate different approaches. Interoperability demands can be recognized both at the
horizontal level (between various systems) and at the vertical one (between formal models). In none of these two
cases we can be satisfied with the existing solutions. There exist standard based solutions supporting
interoperability of learning objects and learner models. Standardized learning design enables interoperability
between systems, but is not reusable in general. Interoperability of domain ontologies is an open issue, for the
context and adaptation models standards are still missing. We can state that in this field we are still far from
achieving general interoperability, since the different standards are not enough to realize it and therefore a
mediation based or Semantic Web based approach is still to be devised to reach reasonable results. This puts also
the impressive looking list of standards and tools in the field in a realistic perspective.
Acknowledgement
This paper originates in the work done in the PROLEARN project – Network of Excellence in Professional
Learning, which is an EU funded project #507310 in the IST programme. We would like to express our gratitude
to our colleagues, especially to Stefano Ceri for his valuable internal review comments and to Bernd Simon for
his support.
References
Brusilovsky, P. (2001). Adaptive Hypermedia. User Modeling and User-Adapted Interaction, 11 (1/2), 87-110.
Brusilovsky, P. (2003). Developing adaptive educational hypermedia systems: From design models to authoring
tools. In Murray. T., Blessing, S. & Ainsworth, S. (Eds.). Authoring tools for advanced technology learning
environment. Dordrecht: Kluwer Academic Publishers, 377-409.
15
Christian, E. (1994). The Government Information Locator Service (GILS): Report to the Information
Infrastructure Task Force. IITF Committee on Information Policy, retrieved March 16, 2006 from
http://www.gils.net/gils.doc.
CORDRA (2004). CORDRA: Technical Introduction and Overview, retrieved March, 16, 2006 from
http://www.lsal.cmu.edu/lsal/expertise/projects/cordra/intro/intro-v1p00.html.
CRA (2003). Grand Research Challenges in Information Systems, Washington, DC, USA: Computing Research
Association.
Cristea, A. I. & Aroyo, L. (2002). Adaptive Authoring of Adaptive Educational Hypermedia, In Proceedings of
AH 2002, Adaptive Hypermedia and Adaptive Web-Based Systems, Springer, 122-132.
Cristea, A. & Calvi, L. (2003). The three Layers of Adaptation Granularity. In Proceedings of the International
Conference on User Modelling. Springer, 4-14.
Cristea, A. & de Mooij, A. (2003). LAOS: Layered WWW AHS Authoring Model and their corresponding
Algebraic Operators. In Proceedings of the WWW 2003 Conference, ACM, retrieved March 16, 2006 from
http://www2003.org/cdrom/papers/alternate/P301/p301-cristea.pdf.
De Bra, P., Houben, G. J. & Wu, H. (1999). AHAM: A Dexter-based Reference Model for Adaptive
Hypermedia. In Proceedings of the ACM Conference on Hypertext and Hypermedia. ACM, 147-156.
Dolog, P. (2004). Identifying relevant fragments of learner profile on the semantic web. In Proceedings of
SWEL’2004 — Intl. Workshop on Semantic Web for eLearning, Intl. Semantic Web Conference 2004, Hiroshima,
retrieved March 16, 2006 from http://www.l3s.de/~dolog/pub/swelatiswc2004.pdf.
Dolog, P. & Schäfer, M. (2005). A Framework for Browsing, Manipulating and Maintaining Interoperable
Learner Profiles. In Proceedings of UM’2005, Springer LNAI, 397-401.
Fink, J., & Kobsa, A. (2002). User modeling in personalized city tours. Artificial Intelligence Review, 18 (1), 3374.
Halasz, F., & Schwartz, M. (1994). The Dexter hypertext reference model. Communications of the ACM, 2, 3039.
Hamel, G., & Välikangas, L. (2003). The Quest for Resilience. Harvard Business Review, 81 (9), 52–63.
Hatala, M., Richards, G., Eap, T., & Willms, J. (2004). The Interoperability of Learning Object Repositories and
Services: Standards, Implementations and Lessons Learned. In Proceedings of the 13th World Wide Web
Conference. New York City, retrieved March 16, 2006 from http://www2004.org/proceedings/docs/2p19.pdf.
Hodgins, W. (2005). Grand Challenges for Learning Objects, Learntec, Karlsruhe.
IEEE PAPI (2003). IEEE P1484.2 Learner Model Working Group: PAPI Learner, Draft 7 Specification, IEEE
Standards Committee on Learning Technology.
IMS LIP (2001). IMS Global Learning Consortium: IMS Learner Information Package Specification v. 1.0,
retrieved March 16, 2006 from http://www.imsglobal.org/profiles/lipinfo01.html.
IMS LDIM (2003). IMS Global Learning Consortium: IMS Learning Design Information Model, retrieved
March 16, 2006 from http://www.imsglobal.org/learningdesign/ldv1p0/imsld_infov1p0.html
IMS DRI (2003). IMS Digital Repositories Interoperability - Core Functions Information Model, retrieved
March 16, 2006 from http://www.imsglobal.org/digitalrepositories/driv1p0/imsdri_infov1p0.html
IST (2004). Information Society Technologies - 2005-06 Work Programme, retrieved March 16, 2006 from
ftp://ftp.cordis.lu/pub/ist/docs/ist_wp-2005-06_final_en.pdf.
16
Koper, R., & Tattersall, C. (2005). Learning Design. A Handbook on Modelling and Delivering Networked
Education and Training, Springer.
Knight, C., Gasevic, D., & Richards, G. (2006). An Ontology-Based Framework for Bridging Learning Design
and Learning Content. Educational Technology & Society, 9 (1), 23-37.
Kravcik, M. (2004). Specification of Adaptation Strategy by FOSP Method. In AH 2004: Workshop
Proceedings, TU/e, 429-435.
Kravcik, M., & Specht, M. (2004). Authoring Adaptive Courses – ALE Approach. Advanced Technology for
Learning, 1 (4), 215-220.
Kravcik, M., Specht, M., & Oppermann, R. (2004). Evaluation of WINDS Authoring Environment. In: De Bra,
P. & Nejdl, W. (eds.) Proceedings of Adaptive Hypermedia and Adaptive Web-Based Systems, Springer, 166175.
LOM (2002). IEEE Standard for Learning Object Metadata, IEEE.
Moe, M., & Blodgett, H. (2000). The knowledge web, Merrill Lynch & Co., Global Securities Research &
Economics Group, Global Fundamental Equity Research Department.
Nejdl, W., Wolf, B., Qu, C., Decker, S., Sintek, M., Naeve, A., Nilsson, M., Palmer, M., & Risch, T. (2002).
EDUTELLA: a P2P Networking Infrastructure based on RDF. In Proceedings of the 11th World Wide Web
Conference, Hawaii, 604–615.
OKI (2004). Open Knowledge Initiative, retrieved March 16, 2006 from http://web.mit.edu/oki/.
OpenURL (2004). OpenURL, retrieved March 16, 2006 from http://library.caltech.edu/openurl/.
QEL (2004). Edutella Query Exchange
http://edutella.jxta.org/spec/qel.html.
Language
(QEL),
retrieved
March
16,
2006
from
RDF (2005). RDF, retrieved March 16, 2006 from http://www.w3.org/RDF.
Rehak, D., Dodds, P., & Lannom, L. (2005). A Model and Infrastructure for Federated Learning Content
Repositories. In Proceedings of the WWW 2005 Conference, retrieved March 16, 2006 from
http://cordra.net/information/publications/2005/www2005/cordrawww2005.pdf.
Simon, B., Massart, D., van Assche, F., Ternier, S., Duval, E., Brantner, S., Olmedilla, D., & Miklos, Z. (2005).
A Simple Query Interface for Interoperable Learning Repositories. In Proceedings of the WWW 2005
Conference, retrieved March 16, 2006 from http://nm.wu-wien.ac.at/e-learning/interoperability/www2005workshop-sqi-2005-04-14.pdf.
SQI (2005). Simple Query Interface Specification, retrieved March 16, 2006 from http://www.prolearnproject.org/lori.
SRU (2005). Search and Retrieve URL
http://www.loc.gov/z3950/agency/zing/srw/sru.html.
SRW (2004). ZING, Search/Retrieve Web
http://lcweb.loc.gov/z3950/agency/zing/srw/.
Service,
Service
(SRW),
retrieved
retrieved
March
February
16,
15,
2006
2006
from
from
van Rosmalen, P., Vogten, H., van Es, R., Passier, H., Poelmans, P., & Koper, R. (2006). Authoring a full life
cycle model in standard-based, adaptive e-learning. Educational Technology & Society, 9 (1), 72-83.
Verbert, K., & Duval, E. (2004). Towards a global architecture for learning objects: a comparative analysis of
learning object content models. In Proceedings of the ED-MEDIA 2004 World Conference on Educational
Multimedia, Hypermedia and Telecommunications, AACE, 202–209.
17
Wu, H., de Kort E., & De Bra, P. (2001). Design Issues for General-Purpose Adaptive Hypermedia Systems. In
Proceedings of the ACM Conference on Hypertext and Hypermedia, 141-150.
Zarraonandia, T., Dodero, J. M., & Fernandez C. (2006). Crosscutting runtime adaptations of LD execution.
ZING (2001). Z39.50: Part 1 - An Overview, retrieved March 16, 2006 from http://www.bibliotech.com/html/z39_50.html.
18
Kassahun, A., Beulens, A. & Hartog, R. (2006). Providing Author-Defined State Data Storage to Learning Objects,
Educational Technology & Society 9 (2), 19-32.
Providing Author-Defined State Data Storage to Learning Objects
Ayalew Kassahun
Wageningen Multimedia Research Center, Wageningen University
Dreijenplein 2, 6703 HB Wageningen. The Netherlands
Tel. +31 317 484723
Fax: +31 317 483158
[email protected]
Adrie Beulens
Information Technology Group, Wageningen University
Dreijenplein 2, 6703 HB Wageningen. The Netherlands
Tel. +31 317 488460
Fax: +31 317 483158
[email protected]
Rob Hartog
Wageningen Multimedia Research Center, Wageningen University
Dreijenplein 2, 6703 HB Wageningen. The Netherlands.
Tel. +31 317 483408
Fax: +31 317 483158
[email protected]
ABSTRACT
Two major trends in eLearning are the shift from presentational towards activating learning objects and the
shift from proprietary towards SCORM conformant delivery systems. In a large program on the design,
development and use of digital learning material for food and biotechnology in higher education, a large
amount of experience has been gained with regard to the possibilities of learning objects that induce
students to be active. These learning objects are highly appreciated by both students and instructors. An
important requirement for these learning objects is the need to support the storage, retrieval and sharing of
state and history information that is defined by the author of the learning object. However, neither the
current learning management systems nor the current SCORM standard provides adequate support for
storing author-defined data. In this article, we discuss some of the problems related to the current learning
management systems and the SCORM standard in supporting activating learning objects, and we propose a
data model and a simple HTTP-based communication protocol as a solution to these problems. This article
describes DBLink, a plugin that implements the proposed extensions.
Keywords
Learning Management System, Author-defined Data, State Information, SCORM, Activating Learning Objects.
Introduction
Currently, most institutions of higher education operate a learning management system (LMS). Learning
management systems are used to control access to digital learning objects, to support communication between
students and their peers as well as between students and their teachers and to support assessments. With respect
to learning objects, there is a trend in eLearning not only from presentational learning material towards learning
material that induces the student into actions and activities but also from proprietary to standards based authoring
tools and delivery platforms.
The use of learning objects that strongly induce students to perform actions is based on the general belief that
learning, understanding and retention benefit from opportunities to become active with and to elaborate on
presented information (Merriënboer, 1997; Biggs, 1999; Anderson, 2000). We call interactive learning objects
that enable and induce actions and activities activating learning objects.
Over the years, the Food and Biotechnology (FBT) program of Wageningen University has produced a number
of activating learning objects, which have usually been implemented as Flash movies or Java applets. Moreover,
these learning objects are often based on client-server architecture in which the learning objects rely on
dedicated servers to store data in any format and structure. Most of these learning objects are also large learning
objects that store the state of student’s interaction and even provide internal navigation. Both the required level
of user interaction as well as the data requirements make it difficult to break up these large learning objects into
19
smaller ones. In this article, we will deal with such learning objects and restrict the term activating learning
object to those activating learning objects that are based on client-server architecture.
Author-defined data storage (Sessink et al., 2003) is the ability of the learning object (LO) to store and retrieve
data that are defined by the author and are stored in some data store that is not a part of nor managed by means
of the learning management system. Usually, the author of the learning object has defined the data structure of
the data source and the instructor who uses the learning object in his course has to manage the data source. We
call this type of data source external data store.
Within the FBT program, we have developed activating LOs for activities such as interactive exercises in Food
Chemistry (Diederen et al., 2002, Aegerter-Wilmsen et al., 2003, Diederen et al., 2003), experimental design
(Aegerter-Wilmsen et al., 2003) and design of downstream processing (Schaaf et al., 2003). These interactive
exercises and design teaching aids are large activating LOs of simulations or virtual laboratories in which
students take part in a simulation or in an experiment. State information – the state of a student’s interaction with
an LO – is stored so that students can do experiments that take longer than a single session. Stored state
information also allows the instructor to monitor the progress of students in an instructor-led training.
The ability to store and retrieve author-defined data enables content authors and instructors to support activating
pedagogical models that need functionalities such as adaptivity, collaborative learning and retrieval of state and
shared data (Sessink et al. 2003). Adaptivity refers to a learning model that takes into account the learner’s
competence, goals, and preferences. State information refers to the ‘state’ of the learning experience at a given
time and is essentially based on tracking of a student’s progress. By shared data, we mean data that can be
accessed by more than one learning object or by more than one student. Shared data are especially important in
collaborative learning. In the Hygienic design LO (see section 4), for instance, the learning object stores state
history and the highest scores of all students. Students can compare their current results with their results from
previous attempts. A number of high scores from all students taking the LO are flagged as shared data so that
students can compare their results with the highest scores and be encouraged to perform better on the next trial.
Due to the lack of standards for supporting author-defined data in the current learning management systems,
different ad hoc methods have been used for storing data in external data stores. One of the widely used methods
to incorporate a large activating learning object in a learning management system is to host the actual learning
object on a separate server and use a proxy learning object that is managed by the LMS. The proxy learning
object contains a hyperlink to the actual learning object and when the student launches such a learning object, the
proxy learning object forwards him to an external site. To actually use the learning object, the student is then
either asked to authenticate himself using password authentication or he is automatically logged in using single
sign on. However, the data source on which the actual learning object is based on can only allow or deny access
to data based on the validity of the user and not the context in which the user interacts with the LMS. Another
method to incorporate an activating learning object is to host the learning object partially in the LMS and store
the data on a separate server. However, the same problem that was mentioned above still remains to be solved.
These configurations do not enable LO authors to benefit from the facilities of LMSs in managing access to LOs.
This shortcoming is unlikely to change in the near future since the recent Sharable Content Object Reference
Model (SCORM) (ADL, 2004) does not have sufficient support for author-defined data storage.
In this article, we discuss the problems related to the current learning management systems and the SCORM
standard in supporting activating learning objects. We propose a number of data model elements as an extension
to the SCORM 2004 data model. We describe how LOs can access data from an external data source through
LMSs based on DBLink, an abstract plugin. A prototype implementation of DBLink has been built for the
Blackboard LMS. For supporting non-SCORM LMSs, we define an HTTP based communication protocol to be
used instead of the SCORM run-time API. However, we strongly suggest that the SCORM run-time API should
be used over other non-standard methods.
Some of the other issues related to the SCORM standard are lack of support for sharing data across learning
objects, learners and LMSs; difficulty in migrating large learning objects to SCORM LMSs; scalability issues
related to data size restrictions; and lack of support for instructor-led training. Although the scope of this article
is restricted to the issue of supporting activating learning objects, some of these other issues are discussed in
relation to activating learning objects.
The rest of the article is organised into four sections. The next section, section two, discusses the requirements
for DBLink for both SCORM as well as non-SCORM learning management systems. Section three covers
design considerations for DBLink and explains the DBLink data model as an extension to the SCORM data
20
model; it also defines a communication protocol based on HTTP. Section four presents a use case. In the last
section, section 5, concluding remarks are made.
Requirements for DBLink
The main requirement for DBLink is to support author-defined data storage so that authors of learning objects
will be able to support activating learning objects. This requirement has a number of consequences for LMSs,
LOs and for the role instructors and LMS administrators play in deploying learning objects.
Learning objects that are student activating rely on the ability of storing and retrieving state and history
information. Support for data storage for learning objects is very limited or non-existent in most of the current
learning management systems. Some of the most widely used LMSs do provide application programming
interfaces (API’s) that enable users to provide the necessary functionalities by extending the learning
management system. However, without a common data model to support student activation, authors of activating
LOs need to develop their own plugins for each LMS on which they plan to deploy their LOs. By providing a
standard data model and communication protocol, DBLink should avoid the need for developing plugins by
individual authors.
In the case of SCORM 2004 conformant LMSs, the SCORM data model provides elements such as
cmi.launch_data and cmi.suspend_data, for storing state information, but the use of these elements is very
limited for a number of reasons. Among the most important reasons for these limitations are that data cannot be
shared among LOs or among students, and the size of data that can be stored is severely limited. The IMS
Shareable State Persistence specification (SSP) is proposed to solve these problems; however, it still relies on the
LMS for data storage (IMS, 2004). The applicability of SSP is also limited to those LMSs that are SCORM
conformant. DBLink should be a lightweight component that does not require conformance to the SCORM
standard. However, we suggest that the SCORM run-time API should be used when both the LO and the LMS
are already SCORM conformant.
Activating LOs can generate large amounts of data; therefore, it is highly desirable to setup a separate server for
data storage and to access it through the LMS. In addition, many existing activating LOs are mostly web-based
applications and have their own data storage. The new approach should not require a substantial reengineering of
already existing LOs. As a result, the way the LOs communicate with the server needs to be supported by
DBLink and data storage requirements should not be restricted. DBLink should also support LOs that are
SCORM conformant and use the SCORM Runtime API. In addition, because some activating LOs are meant for
instructor-led training (ILT), DBLink needs to support ILT by providing data model elements for the most
common instructor activities.
The requirement that data needs to be stored externally affects the task of the LMS administrator or the instructor
in the case of ILT. In addition to deploying learning objects, the instructor or LMS administrator also need to set
up data sources. They do this by setting up a data source for each individual activating learning object or by
setting up data source at a higher level of aggregation such as a module or course. In many cases, it is sufficient
to setup the location of the data source and some access credentials. DBLink should provide data model elements
for this purpose.
DBLink tries to solve the above problems by providing an extension to the SCORM data model and HTTP based
communication protocol. The purpose of this article is to propose a method for supporting accessing and
persisting author-defined data that may be considered in future standards.
Design and implementation
The following sequence of events clarifies the essence of the approach that is widely used at present to host the
activating LOs in current LMSs.
1.
2.
3.
4.
5.
The student starts an activating learning object.
The learning object locates the address of the data source that is included in the learning object.
The student authenticates to the data source using either password authentication or single sign-on.
The student begins interacting with the learning object.
The learning object interacts with data source to retrieve or store data.
21
The data source is an external database or a web server that is usually specifically set up to serve one or more
activating learning objects. Due to a lack of support for author-defined data by most of the current LMSs,
authors and instructors devise workarounds to host their activating LOs that use external data along the lines of
the approach outlined above. Figure 1 shows the schematic representation of the approach. For ease of clarity in
this and subsequent figures, terms from the SCORM standard such as SCO and assets are used whether the LMS
depicted is SCORM-based or not.
There are a few problems associated with this scenario. First, over the years, institutions have made large
investments and created several activating learning objects that are generally not designed for modern LMSs.
These learning objects are usually large, student activating LOs based on a client-server model. Workarounds for
hosting these learning objects in current LMSs can be implemented as outlined above. However, a student who
accesses these learning objects via a LMS may still need to authenticate on the external data source. Moreover,
in the absence of a standard data model to access data and a standard communication protocol or runtime API,
instructional designers should implement custom made solutions.
LM S
User(Browser)
SCO
Asset
Author-defined data server
Author-defined data
Figure 1. In the present LMSs, activating LOs access author-defined data directly instead of through the
LMS. Students should usually authenticate themselves on the data server.
The DBLink plugin
The following is the essence of the approach proposed in this article.
1.
2.
3.
4.
The student starts an activating learning object.
The learning object interacts with the LMS requesting the address of DBLink.
The student begins interacting with the learning object.
The learning object requests data from DBLink. DBLink seamlessly interacts with the LMS to
determine the student access rights and accesses data source on behalf of the student.
We propose an extension to the SCORM data model that should be implemented in DBLink. In LOs for
SCORM conformant LMSs, the values of the data model elements can be set and get using the SCORM runtime
API. In LOs for non-SCORM LMSs, the data model elements will still be used. Since non-SCORM LMSs do
not have a (standard) runtime API, we propose a simple HTTP based protocol to access the values of the data
model elements. Activating learning objects built for both SCORM-based and non-SCORM LMSs should first
try to use SCORM API. Only upon failure to use SCORM runtime API should the learning object revert to the
DBLink communication protocol. By adopting HTTP based communication for non-SCORM systems as a
means of communication between the learning object and DBLink, we not only use the most common
communication protocol of accessing data over the web but also avoid a substantial reengineering of legacy
activating learning objects. Most legacy student activating learning objects are web-based and use HTTP
protocol.
Figure 2 shows a LMS that uses DBLink to access author-defined data. The solid lines (lines 1a, 2a) represent
how activating LOs communicate with the LMS server in a non-SCORM LMS. The dotted lines (lines 1b, 2b)
represent how an SCO uses the facilities of DBLink. Line 3 shows how DBLink accesses the data source, which
is the same in both SCORM and non-SCORM LMSs.
22
LMS
Browser
2b
DB Link
Service
2a
API Adapter
1a
SCO/LO
1b
Asset
3
Author-defined data storage
Author-defined data
Figure 2. Proposed way of providing access to author-defined data. DBLink seamlessly communicates with the
learning management system to determine the student access rights and retrieve data on the student’s behalf.
Figure 3 depicts the second approach as a UML sequence diagram (not to be confused with IMS sequencing).
The figure shows how the data source for an activating learning object is configured and how the learning object
ultimately uses DBLink. First, the instructor or the LMS administrator deploys a course or a specific learning
object. In many cases, instructors are responsible for preparing their learning material, but the actual deployment
can be done either by the instructor himself or by the LMS administrator. Data source connection parameters are
then configured. Data source configuration involves mainly specifying the location of data source and
authentication credentials. The authentication credentials are used by DBLink to connect to the data source.
Students do not need to authenticate on the data source separately since access to data is controlled by DBLink
based on the student’s access rights as specified in the LMS. Once the data source configuration has been set up,
the data source is accessible to students who are authorised for the specific learning material for which the data
source is configured. The learning material can be the learning object, the whole course or any other aggregation
level used by the LMS. When the student interacts with the learning object, the learning object requests data
from DBLink (through the LMS), DBLink checks with the LMS if the student is authorised to use the learning
object and subsequently serves the requested data. How DBLink and the LMS work together to authenticate the
student for accessing data is dependent on the implementation of the specific DBLink and LMS and thus cannot
be specified here.
To implement the DBLink plugin, the LMS needs to make an API available to access learners’ information, to
access LOs information, and to check if the user is authorised to use the specific learning object. The LMS
should have an instructor user type so that instructor type users can manage the external data. In cases where
this is not possible, the LMS administrator user type can be associated with DBLink instructor user type.
The data model
The data model elements provided fall into two groups: data elements for the purpose of accessing data and data
elements for data source configuration. Data elements to read and modify data are used by both students and
LMS Administrators. In the case of ILT, instructors have the same data access rights as LMS administrators.
However, instructors can only access those learning materials for which they are responsible, while LMS
administrators can access all learning materials. Data source configuration and the associated data elements are
required because the data is stored on a separate server and the location of the data source and access credentials
are unknown before deployment of the learning object. Therefore, the data source has to be configured by the
instructor or the administrator after the learning objects that make use of the data source have been deployed.
DBLink defines a very limited set of data types that provide a basic functionality sufficient for most purposes.
These data elements are not dependent on other SCORM data elements and there are no requirements on the type
of data fields of the data source. In order to demonstrate our approach, we chose data sources to be SQL database
applications (in contrast to an arbitrary type of application). Furthermore, we assumed that the parameterised
SQL statements (in contrast to more general methods such as web-services) are, for the most part, sufficient as a
way of accessing data – thus, the name DBLink (Database Link).
23
Instructor or
LMS Administrator
(Browser)
Student
(Browser)
LMS
DBLink
Data Store
Add student-activating learning object
Set up author-defined data
Student activating learning
object requests data
DBLink interacts with the
LMS
to determine student
authorisation
Request for data access
or to store data
Return data
DBLink services author-defined data
Figure 3. A UML sequence diagram showing the interactions among the LMS, the activating learning object
(student), DBLink and the instructor. Data source configuration can be done either by the LMS administrator
or the instructor.
The elements of the data model shown in Table 1 and Table 2 resemble, in format, the data model elements of
SCORM; therefore, we will explain only those elements that are new and will not discuss items such as _count
or id.
Table 1 Data model elements for commands
Details
Number of commands
Student and LMS administrators/instructors can get this value
Id of the nth command
LMS administrators/instructors can get and set this value
Student can get this value
dblink.author_defined_data.
The command string (SQL statement) of the nth command
commands.n.command
LMS administrators/instructors can get and set this value
The result of executing the nth command.
commands.n.value
Number of parameters for the nth command
commands.n.parameters._count
Id of the nth parameter.
commands.n.parameters.n.id
The value of the nth parameter.
commands.n.parameters.n.value
Student and LMS administrators/instructors can get and set this
value
Dot-Notation Binding
commands._count
commands.n.id
Table 1 shows the data elements required to execute commands for the purpose of accessing data. The
dblink.author_defined_data.commands.n.command data model element refers to the command used to access
data from a database. In DBLink access to data is restricted to SQL databases; thus, the commands are expressed
as SQL statements. Since some of the values used in SQL statements can only be determined at runtime,
unknown values at the time of LO creation are represented as parameters. We call these SQL statements
parameterised SQL statements. At runtime, the parameter values are substituted by the values supplied by the
student or the LMS. The dblink.author_defined_data.commands.n.value data model element represents the
results of executing the command represented by dblink.author_defined_data.commands.n.command.
The dblink.author_defined_data.commands.n.command.n.parameters.n.value data model element defines the
value of the nth parameter. The values are set by the learning object before executing the corresponding
command.
24
In the parameterized SQL statement shown in Figure 4 the parameters (shown in italics) are preceded by a
question mark. At runtime, the parameters (in the example the parameters are colParam and keyParam) are
substituted by actual values supplied by the student or the LMS. In non-SCORM LMS, The learning object
supplies the values to these parameters as HTTP parameter-value pairs using either the GET or POST method. In
SCORM conformant LMS, all parameters should be set by the learning object before a getValue method call on
dblink.author_defined_data.commands.n.value.
UPDATE aTable SET colName=?colParam WHERE primaryKey=?keyParam;
Figure 4. A parameterized SQL Command.
Data source configuration
The method we propose assumes that the data source, which henceforth will be known as database, is not
necessarily one central database for the LMS that stores data for all LOs. Therefore, instructors or LMS
administrators should be able to set-up the database connection information either for the LMS as a whole, for
each course or for each learning object independently. Database configuration consists of specifying the address
of the database server and any required credentials for authentication on the database.
Table 2 shows the data elements defined for data source configuration. The address (URL and port number) of
the database server, the username, the password and optionally a database instance name are usually sufficient to
specify database connection information. We, therefore, limit the data elements for configuration to these four
elements.
Dot-Notation Binding
dblink.dblink_address
server_address
database
username
password
Table 2. Data elements for database configuration
Details
The
URL
DBLink.
Students and LMS administrators/instructors can get this value
The URL of the author-defined data store
Only LMS administrators/instructors can get and set this value
Database used as author-defined data store
User name to be used to logon to the database
Password to be used to logon to the database
Only LMS administrators/instructors can set this value
Communication protocol
The learning object may interact with DBLink using HTTP protocol. DBLink depends on the LMS to forward
commands that are destined for DBLink. For instance, if the learning management system is running on
http://www.the-lms.com/, then all calls to http://www.the-lms.com/<dblink> will be
forwarded to DBLink. Here <dblink> refers to the relative path to which all calls are forwarded to DBLink by
the LMS.
Figure 5 shows the protocol used in DB Link. In this figure, variables are shown in italics and need to be
substituted with actual values. The URL of the DBLink is the value of dblink_base_url. The command
handler in DBLink that processes the HTTP request is represented by command_handler. One of the
command handlers is executesql. This handler processes SQL data storage and retrieval from databases.
Another command handler is getuserdata, which retrieves user information. The value of content_id is
the id of the LO or course for which the database is configured. The values of parameter_id and
parameter_value are the optional ID and value of the parameters. Table 3 gives a full description of the
protocol elements.
25
dblink_base_url/[command_handler]?
[content_id=content_id]
[command_id=command_id]
[&parameter_id=parameter_value]
Figure 5. Communication protocol between learning objects and DBLink.
Variable
dblink_url
content_id
command_id
parameter_id
parameter_value
Table 3. Communication protocol parameters
Details
The URL of DBLink. This is a value returned by getValue(dblink.
dblink_address) in a SCORM conformant system.
The unique id of the learning object.
The id of the command. The learning object should check by calling
getValue(dblink.author_defined_data. commands. id) in a SCORM conformant
system to make sure that this parameter id is set.
The id of the parameter. The learning object should check by calling
getValue(dblink.author_defined_data. commands.n.parameters.id) in a
SCORM conformant system to make sure that this parameter id is set.
The parameter value to be set SCO.
A use case
Three different implementations of DBLink are satisfactorily in use: one for the Blackboard LMS (Blackboard,
2006); one for the Moodle LMS (Moodle, 2006); and one for the TopShare knowledge management system
(TopShare, 2006). In this section, we present a use case based on a LO for a virtual ice-cream factory and
DBLink implementation for the Blackboard LMS. The learning object is used in the course Hygienic Design at
Wageningen University.
A learning object for a virtual ice-cream factory
During MSc courses in Food Technology and Food Safety, students learn about hygienic design. For this course,
a learning object depicting a virtual factory for the production of ice cream has been built (
Figure 6). In this virtual production facility, the student, who plays the role of a quality manager, enters the
factory and has to scrutinise production and storage facilities with respect to hygienic design criteria.
Upon entering the virtual factory, the student is personally greeted by and introduced to a virtual employee of the
factory. Next the overview of the factory is displayed, which the student can walk through asking critical
questions (
Figure 6-1). To proceed through the factory, the student has to click on the individual factory sections and
identify items that need to be inspected with respect to hygienic risks (Figure 6-2 and Figure 6-3).
When the student interacts with the learning object, such as entering the virtual factory or identifying an item for
inspection, the learning object executes a command and provides the necessary variables and variable values. A
typical command and the corresponding SQL statement are shown in Table 4. This command gets the location of
the student in the learning object. The learning object launches the command, and DBlink translates the
command into a SQL statement. Note that in the example shown in Table 4, the values of content_id and
user_id have either been previously obtained or have been specified using special place holder variables
which DBLink can expand at run time.
After the student has answered a number of questions about a specific component of the production facility, he
receives feedback on all the questions he answered (Figure 6-4). When the student has finally completed the
walk through of the factory, he is presented with his score and the five top scores of all participants in the
exercise. In this way, he can compare his own score with those of other students (Figure 6-5).
The described functionalities require the learning object to retrieve student information, store selections made by
the student, and retrieve the history of student selections to proceed to the next step or to compute the score. To
26
retrieve student user information, which is used throughout the learning object, the learning object uses
getuserdata command. In subsequent steps, i.e.
Figure 6-1 through Figure 6-5, the learning object launches executesql command to retrieve or store data in
a database. Table 5 shows a list of commands used by the ice cream virtual production facility.
Table 4. An example of parameterized SQL statement and the corresponding DBLink command.
SELECT attempt, allowed_attempts, location FROM
SQL (ID=10)
command
HD_Sate_Table case WHERE UserID=?userId
http://<dblink_address>/executesql?content_id=1000&
command=10&userId=james
Table 5. DBLink URL’s for retrieving student and state information and to store state information.
Retrieve student information http://<dblink_address>/getuserdata
http://<dblink_address>/executesql?content_id=
Retrieve state information
Store state information
<content_id>&command=<command_id>
http://<dblink_address>/executesql?content_id=
<content_id>&command=<command_id>?[<parameter_id>=
<parameter_value>]
Figure 6-1. The student who plays the role of Quality Manager is presented with an overview of the virtual
ice cream factory. The student clicks on a section of the factory to perform quality inspection.
27
Figure 6-2. The student has to identify items that need to be inspected.
Figure 6-3. If the student identifies an item that needs inspection, the LO presents a predefined question
about the item.
28
Figure 6-4. Once the student completes the inspection of a section of the facility, he receives feedback on
his performance.
Figure 6-5. Upon completion of the exercise, the student’s score and the five top scores in the exercise are
presented to the student.
Figure 6. A virtual ice cream production facility.
29
DBLink plugin for Blackboard
Blackboard is one of the most widely used LMS that manages content and learning processes (Blackboard,
2006). Blackboard is SCORM conformant. It also supports integration of external software tools through its
extended “building block” API. We used the building block API to develop a DBLink prototype for the
Blackboard LMS (Figure 7). The SCORM component in Blackboard is implemented as a building block and its
functionalities are not accessible through the building block API. We therefore use the DBLink HTTP based
communication protocol to access the data model elements.
DBLink for the Blackboard LMS is built as a course tool. Instructors have access to a configuration interface,
provided by DBLink in the course control panel. Through the configuration interface, instructors can specify the
values of database connection and command data model elements. To use DBLink, the instructor uploads the
learning object and subsequently uploads or configures the database connection parameters and commands.
These values can also be set at course or at LMS level in which case there will be no need for the instructor to
enter the values of the data model elements for individual LOs.
Figure 7. Database configuration of DBLink in Blackboard LMS.
Conclusion
This article is concerned with providing a way of hosting student activating learning objects in current and future
(SCORM-based) learning management systems. An important requirement for activating learning objects is the
ability to read and write data to and from an external data source. Using an external data source makes it possible
to store arbitrary data types with virtually no size limitation. However, both the current learning management
systems and the SCORM standard (ADL, 2004) do not provide functionalities or specifications that enable us to
achieve our goals.
We proposed an abstract plugin that enables us to access data from an external data source and an extension to
the SCORM data model to be used by this abstract plugin. To access data for LMSs that are not based on
SCORM, we proposed a communication protocol that the learning object can use to communicate with the
30
plugin based on HTTP. Our method works irrespective of the LMS and implementation of the plugin as long as
the DBLink data elements can be set and get with SCORM runtime API. The same applies to non-SCORM
LMS; however, in this case, the HTTP protocol we proposed is used instead of SCORM API. We have
implemented DBLink for Blackboard, Moodle and TopShare.
The data model has two parts: data model elements for accessing external data, which are usually used by
students, and data model elements for configuring the data source. Data model elements for data source
configuration allow the instructor, or the LMS administrator, to setup the data source and specify the data access
commands and parameters. To fully exploit the possibilities of data sharing provided by DBLink, an extended
set of metadata about the data stored is required. Developing such a metadata specification is a large task in itself
and does not fit in the scope of this article.
We described the functionality and the interface based on a plugin called DBLink for accessing database tables
that is implemented for the Blackboard learning management system. The purpose of DBLink is, on the one
hand, to bring a standardized solution for hosting a number of disparate learning objects developed in the FBT
program of Wageningen University (FBT, 2006) and the European Nutrigenomics Organisation (Nugo, 2006).
On the other hand, DBLink also shows how the abstract plugin can be implemented. Therefore, DBLink is
limited to accessing data from databases using parameterised SQL statements and does not implement all the
requirements stated in section 2.
Another consideration in the design and implementation of DBLink is to support a step-by-step migration of
legacy activating learning objects that are not made to be hosted in any LMS or are made to be hosted only for a
specific proprietary LMS. We support that by providing a HTTP based communication protocol, besides the
SCORM runtime API, since most learning objects already use HTTP to access data over the web. We believe the
communication protocol can easily be extended to support learning objects that rely on web services protocol.
We present the method provided in this article as a concept for consideration in future updates of SCORM.
Acknowledgements
The ice cream factory was developed under the auspices of the European Chair in Food Safety Microbiology by
Esther van Asselt, Marc Boncz, Martine Reij, and Leon Gorris, Laboratory of Food Microbiology, Wageningen
University.
References
Aegerter-Wilmsen, T., Hartog, R., & Bisseling, T. (2003). Web Based Learning Support for Experimental
Design in Molecular Biology: a Top-Down Approach. Journal of Interactive Learning Research, 14 (3), 301314.
ADL (2004), Sharable Content Object Reference Model (SCORM) version 1.3, retrieved March, 21 2006 from
http://www.adlnet.org.
Anderson, J. R. (2000). Cognitive Psychology and Its Implications, USA: Worth Publishers.
Biggs, J. B. (1999). Teaching for quality learning at university, Buckingham: Open University Press.
Blackboard. (2006). Blackboard
http://www.blackboard.com.
learning
management
system,
retrieved
March,
21
2006
from
Diederen, J.; Gruppen, H.; Hartog, R.; Moerland, G., & Voragen A. G. J. (2003). Design of activating digital
learning material for food chemistry education. Chemistry Education: Research and Practice, 4 (3), 353-371.
Diederen, J., Gruppen, H., Voragen, A. G. J., Hartog, R., Mulder, M., & Biemans, H. (2002). Design guidelines
for digital learning material for food chemistry education. In Barker, P. & Rebelsky, S. (Eds.), Proceedings EdMedia 2002, Denver, Colorado, USA: AACE, 402-407.
FBT (2006). Food and Biotechnology Program, retrieved March, 21, 2006 from http://fbt.wur.nl.
31
IMS. (2004). IMS Shareable State Persistence Information Model. Version 1.0 Final Specification, retrieved
March, 21, 2006 from http://www.imsglobal.org.
Merriënboer, J. J. G. (1997). Training Complex Cognitive Skills: A four-component instructional design model
for technical training, Englewood Cliffs, NJ, USA: Educational Technology Publications.
Moodle (2006). Moodle course management system (CMS), retrieved March 21, 2006 from http://moodle.org/.
NuGO (2006). The European Nutrigenomics Organisation, retrieved March 21, 2006 from http://www.nugo.org.
Sessink, O., Beeftink, R., Tramper, J. & Hartog, R. 2003. Author-Defined Storage in the Next Generation
Learning Management Systems. In V. Devedzic, J. M. Spector, D. G. Sampson & Kinshuk (Eds.), The 3rd IEEE
International Conference on Advanced Learning Technologies Conference Proceedings, Los Alamitos, USA:
IEEE Computer Society, 57-61.
Schaaf, H. V. D., Vermue, M. Tramper, J., & Hartog, R. (2003). A design environment for downstream
processes for bioprocess-engineering students. European Journal of Engineering Education, 28 (4), 507-521.
TopShare (2006). TopShare
http://www.topshare.com.
Knowledge
Management
Solution, retrieved March 21, 2006 from
32
Di Nitto, E., Mainetti, L., Monga, M., Sbattella, L. & Tedesco, R. (2006). Supporting Interoperability and Reusability of
Learning Objects: The Virtual Campus Approach. Educational Technology & Society, 9 (2), 33-50.
Supporting Interoperability and Reusability of Learning Objects: The
Virtual Campus Approach
Elisabetta Di Nitto
Politecnico di Milano - DEI, via Ponzio 34/5, I-20133 Milano, Italy
[email protected]
Luca Mainetti
Università degli Studi di Lecce - DII, via per Monteroni, I-73100, Lecce, Italy
[email protected]
Mattia Monga
Università degli Studi di Milano - DICo, via Comelico 39/41, I-20135 Milano, Italy
[email protected]
Licia Sbattella and Roberto Tedesco
Politecnico di Milano - DEI, via Ponzio 34/5, I-20133 Milano, Italy
[email protected]
[email protected]
Abstract
E-learning has the potential to offer significant advantages over traditional classroom learning. However, it
requires a complete redefinition of the dynamics of interaction between the various actors of a classroom.
Moreover, in this context, the authoring of instructional material requires much more time than in
traditional learning. Therefore, special care has to be posed to the definition of proper authoring approaches
where educators can reuse and easily assemble existing materials. In this scenario, a comprehensive
learning platform addressing the various interrelated aspects of authoring and fruition of instructional
material is needed. Such a platform should enable reusability of materials, so that it is possible to make
efficient use of preexisting experiences, and interoperability with existing platforms so that it is possible to
take advantage of their strengths. The SCORM standard offers, among other features, a rich data model that
can be used to define and share Learning Objects through different e-learning platforms. We argue that,
despite the fact that it is the most emerging and promising standard, SCORM does not address some key
issues properly, such as specification of metadata and LO composition. In this paper we focus on these
issues and propose some extensions to SCORM that aim to address the above issues.
Keywords
SCORM, Learning Object, Metadata, Aggregation, Sequencing.
Introduction
Pervasiveness of computing in modern societies enables computer-supported learning tools to be available both
in the classroom, and at home. In the classroom, such learning tools facilitate specific – often collaborative –
activities, while at home they support students in self-study, in taking part in virtual classes, etc.
As pointed out in (Dongsong et al. 2004), e-learning has the potential to deliver significant advantages with
respect to traditional classroom learning. As a drawback, by exploiting e-learning, the natural dynamic of
interaction between the various actors in a real classroom cannot be recreated completely and new dynamics
have to be enabled and driven. Moreover, the preparation of instructional material requires much more time than
in traditional learning, therefore, a special care has to be posed to the definition of proper authoring approaches
where educators can reuse and easily assemble existing materials. In this scenario, a comprehensive learning
platform addressing the various complex and interrelated aspects of authoring and fruition of instructional
material is needed.
Such a platform should address the needs of three main classes of actors: Authors, Teachers, and Learners.
Authors design and build courses, possibly modifying and composing LOs; Teachers enact and manage courses,
exploiting available LOs; finally, Learners attend courses by consuming LOs, possibly with the supervision of
Teachers. Moreover, the platform should enable reusability of materials, so that it is possible to make efficient
33
use of preexisting experiences, and interoperability with existing platforms, so that it is possible to take
advantage of the functionality they offer.
In general, there are at least two possible strategies that can be adopted and combined for interoperability
between heterogeneous systems: one interface-oriented and the other model-oriented. The former refers to the
idea of defining well-known interfaces that systems should expose. By exploiting such interfaces, systems can
call each other services thus enabling exchange of data, execution of queries on remote data, etc. The Simple
Query Interface (SQI) (Simon et al., 2004) is an example of such an interface for the e-learning domain. The
model-oriented strategy refers to the idea of having standard, semantically-rich data models shared by various
systems. Sharing the same data model enables the possibility of reusing the same data in different systems,
dramatically increasing the interoperability among them.
Within the context of e-learning, the SCORM standard (ADL, 2004a) supports both strategies. Restricting our
analysis to the model-oriented aspect (which is the focus of our paper), SCORM provides a rich data model that
can be used to define and share Learning Objects (LOs). LOs are defined not only in terms of their content, but
also in terms of the technological platform needed to exploit them and in terms of the way they need to be
sequenced for the purpose of a particular course. While experimenting with this standard in our Virtual Campus
project (Cesarini et al., 2004), we have realized that, despite the fact that it is the most emerging and promising
standard, SCORM does not properly address some key issues, which concern the specification of metadata
describing LOs, and the composition of LOs. As we discuss in this paper, such issues directly affect the
possibility to effectively perform two main activities related to reuse of instructional materials both within the
same e-learning system and, even more, across different systems: reuse for authoring and reuse for teaching.
Reuse for authoring, during creation of instructional materials, requires a data model for LOs that is rich enough
to support reuse of parts (independently of their level of granularity), creation of LOs that reassemble existing
ones, modification of the workflow that defines the way LOs will be executed (sequencing, in the SCORM
terminology). Reuse for teaching, at the time a course is given, requires mechanisms and metadata that simplify
the publication of the LOs and their enactment.
In this paper, based on the experiences we gained within the Virtual Campus project, we propose some
extensions to SCORM aiming at supporting reuse for authoring by empowering the mechanisms for LO
composition and reuse for teaching through the definition of proper metadata for LOs.
The paper is structured as follows. In Section 2 we present the SCORM approach and highlight its advantages
and disadvantages. In Section 3 we present the core contribution of this paper, that is, our extensions to SCORM.
In section 4 we present the high level architecture of the Virtual Campus runtime platform. In Section 5 we
evaluate our approach, comparing it against SCORM through an example. Finally, in Section 6 we present some
related work, and in Section 7 we draw some conclusions.
The SCORM Approach: Pros and Cons
SCORM (Sharable Content Object Reference Model) has been defined by the Advanced Distributed Learning
initiative established in 1997 by the US DoD. The aim was to develop an overall standardization strategy to
modernize education and training, and to promote cooperation between government, academia and business. To
address these goals, the initiative has defined high-level requirements for learning contents, such as content
reusability, accessibility, durability, and interoperability, and has defined three main standards aiming at
addressing the aforementioned requirements: The Content Aggregation Model, the Run-Time Environment, and
the Sequencing and Navigation.
The Content Aggregation Model (ADL, 2004b) contains guidance for identifying and aggregating resources into
structured learning contents. The model is based on the IMS Content Packaging Information Model (IMS, 2004).
It is based on three components Assets, Sharable Content Objects (SCOs), and Content Organizations that are
enacted by means of the Run-Time Environment (see next point). Assets are electronic representations of media,
text, images, sound, web pages, assessment objects or other pieces of data that can be delivered to a Web client.
Assets can be grouped to produce complex Assets. A SCO is a collection of one or more Assets and represents a
single executable learning resource. Since SCOs represent the lowest level of granularity of a learning resource
that communicates with the Run-Time Environment, they can be launched by the Learner, while Assets cannot.
Finally, a Content Organization is a tree composed of so-called Activity items, which can be mapped on SCOs or
Assets (see Figure 1, extracted from (ADL, 2004b)). All of these components can be tagged with metadata. As a
metadata standard, SCORM defines a variation of the IEEE Learning Object Model (LOM) (IEEE LTSC, 2002).
34
LOM envelops instructional material in metadata that describe it. Examples of metadata are the Language of the
instructional material, the Description associated to the instructional material, etc. LOM defines a Learning
Object (LO) as “any entity -digital or non-digital- that may be used for learning, education or training”.
Figure 1 - SCORM Content Organization
The Run-Time Environment (ADL, 2004c) describes a content object launch mechanism, a communication
mechanism between content objects and the SCORM engine on the server, as well as a data model for tracking
Learners’ experience with content objects. It is derived from the run-time environment functionality defined in
(AICC, 2004) CMI001 Guidelines for Interoperability.
The Sequencing and Navigation (ADL, 2004d) describes how SCORM-conformant contents may be sequenced
through a set of Learner-initiated or system-initiated navigation events. It allows the Author to define a path
through Activities. This path is used to guide the Learner in the way she/he takes the instructional material.
Sequencing and Navigation is based on the IMS Simple Sequencing standard (IMS, 2005). It introduces new
structures to aggregate Activities. An Activity Tree describes the branching and flow of Activities. Notice that
the Activity Tree coexists with the Content Organization structure, as they represent two different "views" on the
same objects. As in the Content Organization structure, Activities in the Activity Tree need to be mapped on the
previously defined SCOs and Assets. Within the tree, a single parent Activity and its first-level children are
considered as a new entity, called Cluster (see Figure 2, extracted from (ADL, 2004d)). Each Cluster has
associated a Sequencing Definition Model (SDM) to define sequencing behaviors of its first-level children.
SDM permits to define sequencing behavior in several ways. In particular, Sequencing Control Modes define the
instructional path in terms of workflow-like statements; Sequencing Rules represent a set of conditions that are
evaluated in the context of the Activities for which they are defined; Rollup Rules define how to evaluate
tracking information collected during the fruition; Objectives define how to evaluate Activities’ progress
information, etc.
Figure 2 – Simple Sequencing Activity Tree and Clusters
SCORM can certainly be considered a step towards the establishment of a comprehensive framework for the
definition and execution of LOs. If it will be actually accepted by the various vendors working in the area,
interoperability and reuse of existing assets will be dramatically enhanced. However, while experimenting with it
we have identified some weaknesses that are summarized in the following of this section.
35
The Content Aggregation Model defines components (i.e., SCOs and Assets) that seem to be slightly different
implementations of the same concept (the Learning Object) under different reusability properties and limitations.
As SCOs and Assets are both tagged with metadata and can be composed by means of the Content Organization
tree, the only difference seems to be how they are handled at runtime. Only SCOs are able to fully exploit the
functionalities provided by the Run-Time Environment in order, for example, to keep track of the interaction of
Learners with the system.
The Content Organization tree permits to aggregate SCOs and Assets. However, it actually aggregates Activity
items, which, in turn, need to be mapped onto previously defined SCOs and Assets. This choice maximizes the
flexibility of the standard but leads to a very complicated and redundant definition, as Authors need to define
Assets and SCOs, build the Content Organization tree based on Activities, and then provide a mapping scheme.
The Sequencing and Navigation specification further complicates the situation, as it adds new aggregation
structures - Activity Trees and Clusters - which seem to partially replicate the Content Organization definition.
The aim of allowing their coexistence is to separate simple aggregation (provided by the Content Aggregation
Model) from sequencing (defined via Sequencing and Navigation). The drawback is that the overall specification
becomes very complex and odd, since several concepts overlap and do not seem to be clearly defined.
Sequencing and Navigation provides several ways to define a learning path. This is interesting as it enables the
definition of very complex learning paths. However, the interactions among these sequencing statements can be
very hard to understand and lead to unexpected run-time behaviors. In fact, an Author can use the Sequencing
Control Mode to define a path through a workflow-like specification. Then, she/he can add Sequencing Rules
and Limit Conditions to assert when Learners are allowed to start a given Activity. These rules and conditions
are completely independent of the workflow specification and can easily result in contradictory definitions.
Authors are likely to commit such errors, as the complexity of the instructional material increases. Thus, they
need a verification tool able to detect such inconsistencies. This however makes the authoring cycle more
complex, and increases the difficulty in developing and using authoring tools.
Another aspect that needs to be analyzed concerns the purpose and potential of LOM. Shortcomings of LOM are
well known in literature. In particular (Duval & Hodgins 2003) proposes an LOM research agenda to improve
the standard in areas such as LO taxonomy, LO composition, automatic metadata generation, search, etc. In
(Quemada & Simon, 2003) it is argued that the fundamental problem with LOM is the broad definition of the
term “learning object” and the consequent inadequacy of the metadata set, when trying to apply it to a particular
scenario. In our opinion, LOM has been mainly defined to support search and discovery of instructional material,
but it could be extended to provide support to many other activities such as:
a) Automatic configuration of required software. Following the metadata specification, the platform could
automatically configure pieces of software required by the LO. As an example, whenever a given LO
exports a video content, a video streaming server could be automatically installed/configured.
b) Automatic configuration of supporting software. By supporting software we mean tools that are not
mandatory for LO execution, but that could improve the way the LO is exploited. As an example, an LO
requiring asynchronous communication among Learners and Tutors could be supported by a forum, one
requiring synchronous communication could exploit a chat, while one requiring the cooperative production
of some homework could take advantage of a shared, versioned repository. In summary, information about
the kind of communication and cooperation required by an LO could trigger the configuration of proper
software supporting these activities.
c) Tutoring. Metadata expressing instructional requirements could be useful to provide Learners with
personalized automatic tutoring. In fact, they could support the selection of the most appropriate LO for a
Learner, depending on his personal preferences and attitudes.
d) Evaluation. Metadata could be used by Teachers to analyze and evaluate the effectiveness of LOs.
These activities cannot take advantage of LOM metadata in their current form, because of the following
weaknesses. First of all, the exact meaning of some metadata is not clearly specified (e.g., Semantic Density,
Difficulty, etc.). Moreover, some important characteristics of LOs cannot be expressed. As an example, there is
no way to say whether a given LO has been designed to support group study or individual study. Finally, the
defined metadata are not fully machine-processable: Some of them are defined as free-text (e.g. Installation
Remarks that could be relevant to the configuration of required software) while others rely on vocabularies that
are not precise enough to enable a fully automatic processing.
36
The LOM extensions we propose in Section 3.1 try to address the points from A to D, while the LO composition
model we present in Section 3.2 integrates the composition and the sequencing aspect thus reducing the effort of
Authors and avoiding inconsistencies at runtime.
The Virtual Campus conceptual model as an extension to SCORM
The Virtual Campus project aims at offering a clean and simple conceptual model, called VC-LOM, that
supports both LO definition and fruition in an interoperable and reusable way. Moreover, it offers a software
platform supporting the entire life cycle of LO and their usage by Learners and Instructors. For the purpose of
our approach, we define a Learning Object (LO) as the union of instructional materials and the corresponding
metadata. In this section we focus on the conceptual model. In the following paragraphs, we present, in more
detail, the extensions we propose to the LOM standard, and then discuss our composition approach that supports
both aggregation and sequencing of LOs.
Metadata specification
Our extensions to the LOM mainly aim at supporting the activities of automatic configuration, tutoring, and
evaluation by explicitly expressing some LO properties the LOM does not consider. Of course other extensions
aiming at improving other aspects, such as discovery, could be included, but these are not our focus. The
extensions we have defined are summarized as follows (see Table 1):
¾ A new set of metadata providing details on how an LO should be used. These concern the level of
supervision a given LO requires (e.g., if a tutor should be available during fruition of the LO), whether
the LO requires group (cooperative) study, whether some artifacts have to be created at the end of
fruition, whether the LO requires communication facilities among Learners, and whether the
communication or cooperation have to be synchronous or asynchronous. Such metadata can have
various uses. In particular, they can help address issues presented in point B of Section 2. For instance,
if the “Cooperation Attribute” and “Artifact Attribute” are set, the runtime platform could automatically
configure a version-control server to facilitate the learners in working on shared documents. In addition,
if the “Supervision Mode” is set to tutored, an upload facility could be configured in order to allow
Learners to release the created documents, and Teachers to manage and evaluate them. These data can
also provide information useful for automating the tutoring of Learners and the evaluation of LOs
(points C and D in Section 2). For example, the fact that the "Cooperation Attribute" of an LO is set to
true triggers the monitoring of the cooperative abilities of Learners taking that LO (for instance, the
runtime environment could check the number of communications among Learners, the number of
check-in/out operations they perform on shared documents, etc.). The collected data are added to the
Learners’ profile and could be used by a tutoring system to push shy Learners toward interaction with
the others, or to infer, through automatic evaluation, other data both about the performance of LOs and
of Learners.
¾ Pre-conditions and Learning Objectives constraining the start and end of LO fruition. Both can
predicate on Time and on data available from the Learner’s profile.
¾ The possibility of considering traditional situated didactical activities and digital e-learning material in a
uniform way. Both of them in fact can be represented as LOs. A LO representing a situated activity will
have the Access Modality attribute set to “Situated”, and Date and Place will provide information on the
schedule and location where the activity will take place.
Table 1 - Examples of Virtual Campus additional metadata
Field description
The level of supervision on Learners' activities: “none” (no supervision); “tutored”
(a tutor is available, during fruition Learners can explicitly request his/her
supervision);
“supervised” (the supervisor is always present during the instructional process);
“driven” (Learners act in a passive way, by strictly following the Teacher's
instructions.)
Cooperation Attribute
Whether Learners should take the LO in cooperation.
Communication
Whether Learners will be provided with communication facilities, while exploiting
Attribute
the LO.
Synchronism Attribute
In case of a cooperative or communicative LO, it specifies if the LO must be taken
synchronously by all Learners or asynchronously.
Group Cardinality
The cardinality of the group involved into the fruition of the LO. Meaningful group
Field name
Supervision Mode
37
Artifact Attribute
Access Modality
Place
Date
Precondition on time
Precondition on user
profiles
Learning objective on
time
Learning objective on
user profiles
cardinalities are “1” (self-study), “2” (pair study, both Learner-Learner and
Learner-Teacher), “m” (group study). Note: 2 is the minimum group cardinality for
cooperative LOs and the maximum one for non-cooperative LOs.
Whether the LO requires Lerner(s) to produce an artifact.
“Situated” (in a specific physical location, e.g. a live lecture held in a classroom),
“Digital”.
The physical location name. If Modality is “Digital”, this field is ignored.
The date (including starting and ending time) the LO is given. It is used for
“Situated” LOs, but also for some kinds of “Digital” LOs that can be taken only at
certain points in time.
Constraints on time that have to hold before the LO is taken.
The skills and knowledge a Learner must have in order to exploit the LO. It can
also predicate on administrative constraints that have to be fulfilled by the user
before exploiting the LO (e.g., he/she must have paid the enrollment fee.)
The min/max amount of time required/allowed to complete the LO.
Educational objectives of the LO in terms of skills a Learner can obtain by
exploiting it. It can also express objectives that are not strictly educational (e.g. the
fact that the Learner achieves some kind of degree by completing the LO).
In order to support automatic installation of software required by LOs (see point A in Section 2), we are working
at a more precise specification of the LOM metadata 4.4 Requirements. In particular, we are currently
investigating on the possibility to express requirements and capabilities of software, by using the CC/PP (W3C,
2004) (Composite Capabilities/Preference Profiles) standard. The CC/PP is a standard aiming at describing
device capabilities and user preferences. It introduces the structure for representing delivery-context information
by means of profiles. Profiles consists of attribute/value pairs, which characterize in detail a certain feature of the
delivery context. A vocabulary defines a specific set of attributes and component classes, designed for bringing
together and describing the characteristics of a certain category of devices. The CC/PP itself does not depend on
specific vocabularies, it just provides a structure for constructing and representing them. Although CC/PP’s
initial goal is to describe devices and user-preferences, we argue its flexibility allows for the implementation of
an LO ad-hoc profile. This profile will be able to express LO requirements for software in a far more precise way
than the one permitted by LOM metadata.
We also plan to investigate the IMS Learning Design specification (IMS, 2003), as it explicitly models the
concept of “service facilities”. Service facilities are resources that cannot be given a URL at design time. They
have to be instantiated by a local runtime service. Current specification allows the following services: send-mail,
conference, monitor, and index search. This seems an interesting approach to support automatic configuration of
supporting software (see point B in Section 2).
Other approaches for the design of LOM extensions can be found in literature. In (Rivera et al., 2004), the LOM
Application Profiles is used to adapt semantics of LOM elements, and make LOM-conformant extension to
represent usage conditions and learning activities, in the context of the Elena project. In (Brase et al., 2003),
constraints on LOM’s fields are defined by means of inference rules. RDF is exploited to define metadata, while
an inference language explicitly developed for RDF (TRIPLE) represents axioms.
LO composition
We overcome the limits imposed on LO reusability by the Content Aggregation Model by defining a unique
model for aggregated and simple Learning Objects and by enabling a powerful recursive composition
mechanism. Moreover, we coherently integrate it with proper sequencing mechanisms.
As shown in Figure 3, we define an Atomic LO (ALO) as the smallest unit of reuse for LOs that may or may not
be associated to one or more multimedia contents. A Complex LO (CLO) is defined as an LO whose
instructional material is an aggregation of Learning Objects. Being an LO, a Complex LO can be treated exactly
as any other LO. Moreover, our composition mechanism defines the way some metadata of a CLO are
automatically derived from the metadata of the component LOs (e.g., the Size of a CLO can be computed from
the size of its components).
38
Figure 3 - The Virtual Campus LO model
Of course, LOs defined within a CLO can be sequenced in a way that depends on their content, on some external
conditions such as the availability of Teachers and Tutors, on the purpose of the specific CLO in which they are
contextualized, etc. As it can be noticed, the information that constrain such sequencing are not all known at the
same time (e.g., A Teacher’s appointment can be known few days before a seminar, while the need for taking
Mathematics before Physics could be intrinsic in some specific curricula). Indeed, they do not have the same
scope (again, the Teacher’s appointment can enforce certain sequencing only on a specific edition of a course,
while the need for having Mathematics before Physics should be always enforced). Based on these
considerations, we distinguish between three levels of abstraction that offer different but coherently related
facilities to define sequencing constraints LOs:
¾ At the Reusable Level, Authors define each CLO in terms of a graph where nodes univocally represent
LOs (either Atomic or Complex) while edges represent relationships between LOs. For instance, in
Figure 4, History of Mathematics is referred into Basic Concepts LO and, therefore, can be seen as an
optional, in-dept instructional material.
¾ At the Didactical Level, each CLO is seen in terms of a workflow (see Figure 5) that can be
automatically generated starting from the Reusable Level definition. Such a workflow shows all
possible learning paths for a CLO and it can be customized by the Teacher for specific purposes. For
instance, an available path can be disabled for some specific reasons (see Figure 6).
¾ At the Fruition Level, some details needed for fruition of LOs are introduced (see later).
More in detail, at the Reusable Level, rounded-corner rectangles inside a CLO represent particular CLOs called
Inner CLOs. They provide a mechanism to aggregate LOs, but, differently from other CLOs, they do not have an
identity and cannot be reused outside of the context of the CLO in which they are defined. They can however
participate in any relationship connecting two generic LOs. LOs inside an Inner CLO can be mutually exclusive
or not: The Inner CLO is labeled with two values indicating the minimum and the maximum number of LOs the
Learner has to exploit. The syntax is (n,m) where n represents the minimum and m the maximum. Values as (1,1)
indicates that LOs are mutually exclusive, while (0,1) states that Learners are allowed to skip the fruition of LOs.
Finally, if the label is not indicated, all of the LOs have to be exploited. LOs labeled with T are called Test-LOs.
They are used to model assessments Learners have to take.
Relationships indicate the presence of instructional constraints between two LOs in the context of a containing
CLO (outside that CLO, the relationship is no longer valid). A generic relationship from x to y in the context of
z, with x,y being LOs (either Atomic or Complex) and z a CLO (on Inner CLO), is represented by an arrow from
x to y inside z, labeled with the relationship name. The relationships can be named IsRequiredBy, References,
and RequiresOnFailure. Their meaning is summarized in Table 2, where, for the sake of brevity, we omit the
indication of the CLO where the relationship takes place.
Relationship
IsRequiredBy
References
Table 2 - Relationships between Reusable-Level LOs
Description
A IsRequiredBy B indicates that LO A must be completed before starting LO B; i.e., the
Learner has to possess A-related knowledge in order to achieve a correct understanding of
B. However, the IsRequiredBy relationship does not mean that Learners must complete A
immediately before B: Learners are allowed to make use of other LOs after A and before
B's fruition.
A References B indicates that A cites B as a source of more details on a topic related to A
itself. Taking B at fruition time is not compulsory but, in case it is taken, it has to be
entered after A and before taking any other LO not connected to A through a References
relationship. Many References can depart from the same referencing LO. In this case,
39
RequiresOnFailure
Learners can make use of one or more of the referenced LOs.
The RequiresOnFailure relationship always connects a Test-LO with some other LO. If
the Test-LO is failed, then the LO at the other end of the RequiresOnFailure relationship
has to be taken by the Learner. If no RequiresOnFailure is specified, Learners failing a
Test-LO have to re-start the fruition of the whole CLO.
The example shown in Figure 4 defines two different CLOs. The first one, “Mathematics”, is composed of
several LOs. “Basic concepts” and “Algebra” are both required by the Inner CLO enclosing “Calculus”,
“Geometry”, and “Limits”, so they should be taken in the first place. Notice that the relative order between
“Basic concepts” and “Algebra” is not constrained. All the LOs inside the Inner CLO have to be exploited, since
no label is present. The “History of mathematics” is left as an optional activity and, in case it is taken, it must
immediately follow “Basic concepts” that references to it. “Exam” is a Test-LO. In this example, since no
RequiresOnFailure relationship is defined, if “Exam” is failed the whole CLO have to be repeated.
The second CLO, “Engineering first year”, is composed by reusing “Mathematics” as well as some other LOs.
Learners have to complete “Mathematics” before entering “Physics”. “Chemistry A” and “Chemistry B” are
alternative. This is defined by the label (1,1) that states the Learner must exploit only one of the two. Notice that
Learners can take one of the “Chemistry” LOs either before “Mathematics”, or after “Physics”, or in between. It
is interesting to note that “Mathematics”, being reused in this context, appears as a black-box. Its internal
complexity is hidden thus allowing for an easy composition. Even more interesting is the fact that, in such a
representation, less arcs are drawn means that more freedom is left to Learners. As an extreme example, a
simple collection of LOs, with no arcs at all, permits the fruition of a course in which all possible paths are
allowed.
Figure 4 - CLO definitions
LOs can be (re)used either to define other LOs or to provide them to the Learners. In the second case, they need
to be translated into their didactical level representation. For instance, Figure 5 shows the workflow
representations of the two CLOs defined in Figure 4. In this representation, LOs are mapped into activities that
represent the fruition of the corresponding LOs. The syntax is derived from the UML activity diagram. However,
it has been adapted to our particular case. Simple arrows connecting activities represent a sequence. Vertical bars
encode the fork/join semantics (i.e. parallel execution or, more properly in our case, absence of constraints on the
relative fruition order), as well as the multiple switch semantics (i.e. execution of one or more of the involved
activities). A label (n,m) permits to define the behavior of the vertical bar (the meaning of (n,m) is the same as in
the Reusable Level language). The dashed, bidirectional arc denotes the fact that the corresponding path is not
mandatory (i.e. there is an optional activity). Finally, the double-arrow arc indicates the path Learners must
follow whenever they fail a Test-LO. If such an arc is not indicated, Learners have to take again the whole CLO
(in the example, the arc is drawn in order to permit a complete explanation of the workflow syntax and it could
be removed).
40
Figure 5 - Workflow description of CLOs
By using the editing tools provided by Virtual Campus (see Section 4), such a workflow representation is
automatically derived from the Reusable Level model of an LO. If needed, the Teacher can customize this
representation by performing the following operations:
¾ elimination of alternative paths by selecting a single path or a subset of the available ones;
¾ elimination/enforcement of optional activities;
¾ enforcement of the order of fruition, in case of parallel activities.
All these operations preserve the consistency between the resulting workflow and the corresponding high-level
description since they further constrain the way LOs are used by Learners. Figure 6 shows a possible
customization of the workflows depicted in Figure 5.
It is important to notice that if the Teacher does not need to customize the generated workflow, this step can be
left completely hidden. Thus, in the simplest case, the teacher does not have to deal with the workflow
representation.
Figure 6 – Workflow customization
At the Didactical Level an LO is not ready for fruition yet. Some additional details need to be added concerning
the course edition, the enrollment method, start and end dates, the course calendar, announcements, the Teacher's
name, the list of already enrolled students, etc. Such information is defined at the Fruition Level of abstraction
and cause the creation of a Course which is ready for fruition.
41
The Virtual Campus runtime platform
The Virtual Campus platform implements the VC-LOM and supports the design, deployment, fruition, and
evaluation of learning materials. It is composed of two main subsystems: The Authoring Environment and the
Fruition Environment (see Figure 7).
Figure 7 - Virtual Campus high-level architecture
The Virtual Campus authoring environment provides several editors to define Atomic LOs and Complex LOs at
the Reusable, the Didactical, and the Fruition Levels. The editor used at the Reusable Level is shown in Figure 8.
It is provided as an extended and personalized version of Microsoft Visio, to take full advantage of its drawing
capabilities. The tool permits the definition of a new Atomic LO, specifying its content (if any) and its metadata.
Complex LOs can be created, reusing and aggregating other LOs. In order to find LOs within the Virtual
Campus repositories, a dialog box permits the specification of searching criteria, using metadata. It is also
possible for authors to import SCORM LOs from external systems. Such LOs can be either standard SCORM
compliant, or extended SCORM compliant (incorporating the Virtual Campus extensions).
Figure 8 - The CLO editor (Reusable Level)
42
The application also features a Didactical Level Complex LO generator that automatically generates a first
version of the workflow associated to a Complex LO and then supports Teachers in customizing it. Finally, a
Fruition Level LO tailoring tool supports the insertion of all the necessary Fruition Level details (e.g. class start
time).
A LO can be serialized in the SCORM format at any of the abstraction levels defined in the Virtual Campus
conceptual model. In such a way, Authors and Teachers are free to choose the preferred trade-off between
abstract, highly reusable LOs (needing the final tailoring phase by the Teacher), and less reusable, ready-to-use
LOs (no effort by the Teacher is needed). Reusable Level, Didactical Level, and Fruition Level specifications
can all be exported at the same time in the same SCORM packet. The serialization algorithm maps both the
Reusable Level and the Didactical Level languages on a subset of the SCORM Simple Sequencing tags.
Moreover, the VC-LOM metadata are stored into the package as extensions to the standard LOM. When
SCORM packages, generated with our platform, are exported to a Virtual Campus system, the whole
specification is extracted. Otherwise, standard SCORM systems would simply ignore our extensions.
The Fruition Environment is based on the SCORM Run-Time specifications; it allows Learners to attend
courses, and Teachers to guide fruition (when required). The environment is Web-based. Learners and Teachers
only need a Web browser (possibly augmented with plug-ins, such as Flash, the Java Virtual Machine, etc.) to
exploit the contents of LOs.
Our SCORM-compatible Engine incorporates both the standard SCORM Run-Time engine, and a Sequencing
Engine. The Sequencing Engine enacts the fruition workflow associated to an LO, guiding Learners and
Teachers in the execution of the activities related to the usage of the LO. It is composed of several modules.
Besides the Workflow Engine, the other components are: a Precondition Resolution module which takes into
account constraints added to LO metadata (e.g. class date and time); a Learning Objectives Evaluation module
which evaluates the Learning Objectives specified in LO metadata. Whenever a given Learner finishes an LO,
the Learning Objectives Evaluation module is invoked in order to apply the specified rules. Then, the Workflow
Engine is asked for the list of LOs the Learner is allowed to enter as a next step. The resulting list is then filtered
by the Precondition Resolution module, leading to the final list that is returned to the SCORM Engine. Finally, a
Web page showing the list is presented to the involved Learner who can communicate her/his decision by
selecting one of the proposed alternatives. Once the confirm button is pressed, an event is generated and notified
to the Sequencing Engine.
The Engine activates the tools that are required for the execution of LOs. These tools can be of the following
types:
¾ Stand-alone applications (e.g., Microsoft PowerPoint or Flash), in this case off-line study is enabled by
the platform.
¾ Server-based applications (e.g., tools supporting, collaboration among Learners).
As an example, WebTalk (Barbieri et al., 1999) is an application, developed at Politecnico of Milano, providing
a 3D metaphor to browse through teaching material. The tool allows the user to maintain awareness of presence
of other users who browse through the same material, thus enabling unstructured interaction and sharing of
information among Learners. WebTalk exploits the Macromedia Shockwave Multiuser Server to permit
information exchange among Learner, while a Web browser (enhanced with the Macromedia Shockwave plugin) is required as a client. In this case fruition requires Learners to be on-line, connected to the Virtual Campus
platform.
The Fruition Environment (Sbattella et al., 2004) is instrumented with a monitoring tool that collects data on
actions performed by users. Profiles are then generated and exploited, on one side, to provide feedbacks to
Teachers on the validity of specific LOs and applications, and on the behaviour of Learners. On the other side,
they are exploited to instruct automated tutoring agents, enabling them to suggest and guide Learners throughout
the fruition process.
Virtual Campus Vs. Scorm – Comparative Evaluation
We have tested the Virtual Campus platform by experiments. The experimentation aimed at assessing the
possibility of arranging in complex structures heterogeneous LOs (lectures, studying activities, cooperative
sessions of work, and exams), and at testing the whole platform covering all the phases of course development
and fruition. A first experiment focused on evaluating the behavior of students while working on a group project,
43
exploiting collaborative LOs. Then, we developed a complete on-line version of a course on “Web Design”,
taught at the Politecnico di Milano. Finally, the course on “Software Architectures”, depicted below, has been
given to some undergraduate students of a software engineering course at Politecnico di Milano.
The results are still under evaluation. In general, students enjoyed the user-friendly interface and the easy
interaction with the system and their colleagues. Moreover, they appreciated the support to collaboration offered
by the Virtual Campus platform. All of them took the LOs defined as optional in the course, probably because of
the novelty of being able to autonomously make choices. We also found that the added value of contents based
on virtual environments was not perceived as determinant for the learning objectives. We expect that virtual
environments could be more deeply exploited and appreciated in presence of students with disabilities, for whom
traditional media are difficult to use. From the teacher’s perspective the system has been helpful to integrate
existing materials and build Complex LOs in a few hours. Moreover, the monitoring facilities enabled him and
the tutors to profile students’ behavior and to evaluate the effectiveness of the proposed Complex and Atomic
LOs.
As a complementary way to test Virtual Campus, we have evaluated the expressiveness of the model against the
one of SCORM, referring to the example of the “Software Architectures” course. In the following we discuss
this comparison.
The Virtual Campus model
The LO we have realized is composed of four main modules: “Introduction”, “Software Design”, “Design
Patterns”, and “Architectural Styles”. The optional “Seminary” focuses on presenting advanced mechanisms for
late-binding of software components. Finally, students are evaluated on both theoretical aspects (through a quiz)
and practical abilities (by means of a laboratory session).
Figure 9 shows the Reusable Complex LO associated to the course. The module on “Design Patterns” is offered
by two alternative Atomic LOs. One of them is based on slides while the other provides a video. Note that the
fact that such LOs are alternative is established by the Author of the course (considering the goals of the course),
and the scope of such relationship is limited to the Complex LO “Software Architectures” itself. The quiz is
implemented by the Test LO. Learners failing the test are required to take again the course starting from the
“Software Design” LO.
Figure 9 - The Reusable Complex LO of the course on Software Architectures
The laboratory session is a Complex LO whose components are “Meeting Point” and “Software Engineering
Project”. The former requires that Learners create groups, download the requirements of the simple event based
application to be developed and exchange comments on them. The latter requires Learners implement the
44
application. Since they have to work in teams, they are requested to coordinate themselves by using a
configuration management application.
Notice the usage of a nested Inner CLO asserting that “Design Patterns 1” and “Design Patterns 2” are
alternatives. It is enclosed in another Inner CLO that requires all its LOs to be exploited. In this specific case this
means that one of the two “Design Patterns” LOs, and the “Architectural Styles” LO must be exploited. The
optional LO “Seminary” can be exploited after “Architectural Styles”.
All the LOs depicted in Figure 9 are decorated with metadata. For example, "Software Engineering Project" is a
collaborative LO in which groups of Learners have to build a simple software application. Since it is tagged with
"Supervision Mode = tutored", the Virtual Campus platform enables a Tutor to access all the “Software
Engineering Project” instances exploited by the Learner groups. The tag "Artifact Attribute = true" tells the
system to configure an up-load page to let the groups deliver their artifacts. The tags "Cooperation attribute =
true" and "Group Cardinality = 2" allow for the creation of student pairs and for the installation of a versioning
server to support the cooperative creation of the artifact. The tags "Communication Attribute = true" and
"Synchronism = asynchronous" instruct the system to provide a forum. Finally, "Access Modality = Digital"
specifies that the LO does not need a classroom.
Figure 10 shows the Didactical Level Complex LO that is automatically derived from the Reusable LO in Figure
9. It can be customized depending on the needs of the specific course and teacher. For instance, Figure 11 shows
a customization where the video version of “Design Patterns” has been eliminated and a fruition order between
“Design Patterns” and “Architectural Styles” has been forced. This customized version of the course was then
translated into a Fruition level LO and made ready for fruition.
Figure 10 - The Didactical Level Complex LO of the course on Software Architectures
Figure 11 - The customized version of the course on Software Architectures
The SCORM model
The aforementioned example can be described using the SCORM Sequencing and Navigation. Figure 12 depicts
the SCORM Activity Tree representing our “Software Architectures” LO. Boxes labeled as “Anonim-n“
represent Activities we need to introduce, in order to reproduce the CLO aggregation structure shown in Figure
9. For example, “Anonym-4” represents the fact that “Architectural Styles” and “Seminary” can be seen as a
single entity (see the semantics of the References relationship).
45
Figure 13 depicts the sequencing information needed to reproduce the behavior of the LO “Test”. For the sake of
brevity, the description of the whole example is not depicted. It is easy to see that such a specification tends to
become very complex, as the number of Activities and/or the number of allowed instructional paths increases.
By contrast, the Reusable Level specification tends to be less and less complicated as the number of allowed
instructional paths increases (the less the number of arcs drawn, the more freedom is left to Learners).
Standard LOM metadata do not allow the specification of the aforementioned “configure up-load page” or
“install versioning server” requirements, in a structured, machine-processable manner. Authors can add such
requirements as textual notes, but machines cannot easily process such kind of unstructured specification.
Figure 12 – Activity Tree
Figure 13 - Sequencing information for the LO “Test”
SCORM graphic representations
In (Jun-Ming Su et al., 2005) a workflow-like, graphic representation of sequencing information is proposed, so
we think it is interesting for us to compare our approach against this. SCORM Clusters and the associated set of
SDM rules define a so-called Sequencing Object (SO). The model defines six kinds of SOs, each one
representing a workflow construct: Linear SO (a simple, linear sequence), Conditional Linear SO (adds
conditions αi on arcs, permitting to modify the strict linear behavior), Choice SO (selection among two or more
objects), Conditional Choice SO (adds conditions αi, permitting to describe more complex selection behaviors),
Loop SO (uses two conditions; useful to describe tests), and Exit SO (adds conditions αi that stop the
instructional process).
Figure 14 depicts how the SO-based language models the “Software Architectures” example. Gray boxes
represent SOs, while white boxes model elementary activities (the leafs in the Activity Tree). The special nodes
“Nin” and “Nout” do not represent any actual instructional content. They just act as input and output gates for
Choice and Conditional SOs. Notice that, in case of condition failure, the standard behavior of the Loop SO lets
learners return only to the SO that immediately precedes the current one. Therefore, a Linear SO is needed in
order to group all the previous LOs.
This specification resembles our Didactical Level language, while does not provide features equivalent to our
Reusable Level language. Thus, the main lack we found is the absence of a composition scheme that permits to
46
decompose Complex LOs into simpler objects, describing them as separate entities (“Software Architectures”
and “Laboratory”, in this example).
Figure 14 – SO-based description of Software Architectures
Related work
Besides SCORM, and the approaches directly related to it, in the literature there are various platforms that
support authoring and/or fruition of LOs. In this section we focus on the approaches supporting authoring, and
classify them into three categories: relationship-based, workflow-based, and rule-based.
Relationships, workflows, and rules
Relationship-based systems allow teachers to define a course structure by means of logic relationships among the
course components. MediBook (Knolmayer, 2001) is an example of such systems. In MediBook the important
domain concepts are formalized and related to each other by semantic relationships. In turn, LOs are associated
with concepts and are connected through so-called rhetorical relationships (e.g. LO-A deepens LO-B, LO-C ispart-of LO-D). MediBook uses the LOM standard to define LO metadata and to store rhetorical relationships.
Learners can navigate through both the rhetorical relationships structure and the semantic relationships structure.
In the latter case, they discover LOs starting from the associated concepts.
As an alternative approach (Steinacker et al., 2002) uses a sort of “direct prerequisite” relationship to order LOs
(e.g. LO-A is a direct prerequisite for LO-B). The matrix associated to the resulting graph shows the total
number of direct and indirect prerequisites between two LOs. When Learners choose an LO to exploit, it is
possible to calculate the list of required LOs. An integer-programming model is then built, taking into account
further constraints (e.g. the time effort required by a given LO). By minimizing the model’s target function,
some LOs are removed from the list. A sequencing procedure determining the “best” schedule for the remaining
LOs is then executed.
A similar approach, described in (Carchiolo et al., 2001), uses the same relationship, but adds weights in order to
represent how hard is to access a given topic, coming from a previous one. To choose a path, Learners select it
from the whole graph provided by the system. Each route is associated with a numeric index weighting the
“effort to learn” the target topic.
Ontologies are proposed in (Jin-Tan David Yang et al., 2004) as a mean to create relationships among
instructional material. The paper makes use of RDF/RDFS to define the ontology of a given instructional
domain. Sharing the ontology among authors, it is possible to generate SCORM-compliant Learning Objects, all
of which provide an outline using precisely defined terms.
47
Workflow-based systems allow teachers to define a course structure as a workflow. In (Padrón et al., 2004)
workflow languages such as BPEL4WS are exploited for the composition of learning web services and their
adaptation to the needs of a Learner or group of Learners. Once composed and packaged as Learning Objects,
these composite processes can be executed, instantiated and adapted to the Learner's particular needs. These
adaptations can be realized, either by predefined rules implemented into the process description and driven by
the Learner behavior, or in a supervised manner.
Rule-based systems customize the fruition of LOs by mean of rules. Preconditions and postconditions rules are
exploited in (Sicilia, M.A. et al., 2004). Preconditions encode prerequisites required for an instructional process
to take place, while postconditions states the expected outcomes. Relying on such rules it is possible to calculate
the list of LOs a given Learner is allowed to enter.
In (Quarati, 2003), weights are associated to LOs. Weights represent the cognitive contribution of objects within
a given context. If an LO has subcomponents, its weight is defined as the maximum weight of its
subcomponents. The knowledge already acquired by Learners, together with the weights of LOs, is used to
identify the part of the content domain available to an individual user at a certain point in time.
Discussion
Relationship-based, workflow-based, and rule-based systems offer both advantages and disadvantages.
Relationship-based systems focus mainly on the definition of complex association structures between LOs, while
the other classes of systems provide run-time mechanisms to check that such relationships are properly satisfied
and, in some case, enforced during the fruition of courses.
Workflow-based systems define a strict temporal order among activities. In some cases this forces teachers to
impose unnecessary constraints. On the contrary, rule-based systems provide support to the definition of
constraints on fruition paths, but do not offer mechanisms to let teachers define precise paths if this is needed.
Based on our practical experience in giving courses and preparing the related material we have noticed that
educators working on LOs at different levels of abstraction may need to exploit features offered by all the three
kinds of approaches. In particular, educators in charge of defining the structure of courses want to compose LOs
just by aggregating them and imposing some instructional constraints. For instance, should they need to build a
math course using LOs “Limits”, “Derivatives”, and “Geometry”, they would probably impose the constraint
“Limits” is required by “Derivatives”, while they would not constrain any sequencing between “Geometry” and
the other two LOs. To achieve this goal they would probably prefer a relationship-based system. Vice versa,
teachers willing to adapt LOs to a specific course instance (e.g., “Mathematics” for Mechanical Engineering)
wish to impose more restrictive constraints on the LOs in order to define the structure of a course. In particular,
teachers wish to have complete control over the fruition paths, for instance, imposing that “Geometry” has to be
taken before “Limits” when the organization of that specific course instance requires it, e.g., because of
problems related to the calendar structure. Thus, they would prefer to exploit a workflow-based system. Finally,
teachers willing to precisely define some preconditions to the LO fruition (e.g., administrative constraints that
have to be fulfilled), would prefer a rule-based system.
For the above reasons, we have chosen to exploit all these approaches at the various levels of abstraction. At the
Reusable Level, the composition of LOs is expressed according to a relationship-based approach, while at the
Didactical Level, workflow-based and rule-based supports are provided. SCORM supports all the three
approaches as well, however --as it should be clear from what we presented in the previous sections-- the lack of
a clear definition of different levels of abstraction in which these approaches are exploited, and the presence of
various overlapping concepts, makes it difficult to be used.
Conclusions
We see SCORM as a good opportunity to support interoperability among e-learning tools since it enables the
definition of a data model that can be shared among them. However, we have noticed some weaknesses in such a
data model. These weaknesses mainly concern the way learning resources can be structured and made available
for reuse.
48
In our vision all the learning resources have to be thought as LOs, that is, entities described by proper metadata
that can be recursively composed. Thanks to the recursive composition mechanisms, reuse both within a single
platform and among platforms can be greatly enhanced: A LO at any level of composition can be reused and
composed in another context. The definition of proper metadata can support not only browsing and reuse of LOs,
but also their installation and execution, and also enable tutoring and evaluation.
The Virtual Campus project provides an implementation of the aforementioned concepts. Moreover, it tries to
enhance the SCORM run-time environment, exploiting a workflow engine to guide Learners through the
instructional paths.
We rely on model-oriented strategies to improve interoperability. However, we are aware that adopting standard
interfaces is of course a crucial aspect. Therefore, as a future work, we plan to include the Simple Query
Interface specification into Virtual Campus. Another aspect that merits further investigation is the definition of
proper guidelines to support Authors and Teachers in the design of LOs. Clearly, the more their LOs correspond
to fine granularity learning materials, the more such materials are reusable and applicable in various contexts.
Indeed, the mechanisms to compose fine granularity LOs are essential in this case in order to avoid the
difficulties of having a huge, non-organized collection of LOs.
Acknowledgements
The Virtual Campus project has been funded by Microsoft Research, Cambridge (UK). We thank Sam Guinea
who participated to the implementation of the Virtual Campus authoring environment, and helped us reviewing
the paper. We also thank the editor and the anonymous reviewers for their helpful feedbacks and suggestions.
References
ADL (2004a). SCORM 2004 2nd
http://www.adlnet.org/downloads/70.cfm.
edition
-
overview,
retrieved
March
22,
2006
from
ADL (2004b). SCORM content aggregation model - version 1.3.1, retrieved March 22, 2006 from
ADL (2004c). SCORM run-time environment
-
version 1.3, retrieved March
22,
2006
from
ADL (2004d). SCORM sequencing and navigation - version 1.3.1, retrieved March 22, 2006 from
AICC (2004), CMI001 - AICC/CMI Guidelines For Interoperability v4, retrieved March 22, 2006 from
http://www.aicc.org/pages/down-docs-index.htm.
Barbieri, T., & Paolini, P. (1999). WebTalk: a 3D Collaborative Environment to Access the Web. In Proceedings
of EUROGRAPHICS ’99, 111-113.
Brase, J., Painter, M. & Nejdl, W. (2003). Completing LOM - How Additional Axioms Increase the Utility of
Learning Object Metadata. In Proceedings of the 3rd IEEE International Conference on Advanced Learning
Technologies (ICALT'03), Athens, Greece.
Carchiolo, V., Longheu, A. & Malgeri, M. (2001). Learning through ad-hoc formative paths. In Proceedings of
the International Conference on Advanced Learning Technologies (ICALT), Madison, Wisconsin, USA.
Cesarini, M., Guinea, S., Sbattella, L. & Tedesco, R. (2004). Innovative learning and teaching scenarios in
Virtual Campus. In Proceedings of the World Conference on Educational Multimedia, Hypermedia and
Telecommunications (ED-MEDIA), Lugano, Switzerland.
Dongsong, Z. L., Zhao, J., Zhou, L. & Nunamaker, J. F. (2004). Can e-learning replace classroom learning?
Communications of the ACM, 47 (5), 75-79.
49
Duval, E., & Hodgins, W. (2003). A LOM Research Agenda. In Proceedings of the 12th World Wide Web
Conference. Budapest, Hungary.
IEEE LTSC (2002). IEEE Draft Standard for Learning Object Metadata, retrieved March 22, 2006 from
http://ltsc.ieee.org/wg12/files/LOM_1484_12_1_v1_Final_Draft.pdf.
IMS (2004). Content Packaging Information Model, version 1.1.4 Final Specification, retrieved March 22, 2006
from http://www.imsglobal.org/content/packaging/#version1.1.4.
IMS (2003). Learning Design Information Model, Version 1.0 Final Specification, retrieved March 22, 2006
from http://www.imsglobal.org/learningdesign.
IMS
(2005).
Simple
Sequencing
v1,
http://www.imsglobal.org/simplesequencing/index.html.
retrieved
March
22,
2006
from
Knolmayer, G. F. (2002). Decision support models for composing and navigating through e-learning objects. In
Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS), Big Island, Hawaii.
Padrón, C. L., Torres, J., Dodero, J. M., Díaz, P., & Aedo, I. (2004). Learning Web Services Composition and
Learner Communities Support for the Deployment of Complex Learning Processes. In Proceedings of the IEEE
International Conference on Advanced Learning Technologies (ICALT’04), Joensuu, Finland.
Quarati, A. (2003). Designing shareable and personalisable e-learning paths. In Proceedings of the International
Conference on Information Technology: Computer and Communications (ITCC), Las Vegas, Nevada, USA.
Quemada, J. & Simon, B. (2003). A Use-Case Based Model for Learning Resources in Educational Mediators.
Rivera, G. M., Simon, B., Quemada, J., & Salvachúa, J. (2004). Improving LOM-Based Interoperability of
Learning Repositories. In Proceedings of the Workshop on Ontologies, Semantics, and E-learning (WOSE),
Larnaca, Cyprus.
Sbattella, L. & Tedesco, R. (2004). Profiling and tutoring users in virtual campus. In Proceedings of the 5th
International Conference on Information Technology Based Higher Education and Training (ITHET ’04),
Istanbul, Turkey.
Sicilia, M.-A., García, E., Pagés, C., Sánchez-Alonso, S., & Rius, A. (2004). Specifying semantic conformance
profiles in reusable learning object metadata. In Proceedings of the 5th International Conference on Information
Technology Based Higher Education and Training (ITHET 2004). Istanbul, Turkey.
Simon, B., Massart, D., & Duval, E. (2004). Simple query interface specification, retrieved February 15, 2006
from http://nm.wu-wien.ac.at/e-learning/interoperability/query.pdf.
Steinacker, A., Faatz, A., Seeberg, C., Rimac, I., Hrmann, S., El Saddik, A., & Steinmetz, R. (2001). Medibook:
Combining semantic networks with metadata for learning resources to build a web based learning system. In
Proceedings of the World Conference on Educational Multimedia, Hypermedia and Telecommunication (EDMEDIA), Tampere, Finland.
Su, J.-M., Tseng, S.-S., Weng, J.-F., Chen, K.-T., Liu, Y.-L., & Tsai, Y.-T. (2005). An Object based Authoring
Tool for Creating SCORM Compliant Corse. In Proceedings of the 19th International Conference on Advanced
Information Networking and Applications (AINA’05), Tamkang University, Taiwan.
W3C (2004). Composite Capability/Preference Profiles (CC/PP): Structure and vocabularies 1.0, retrieved
March 22, 2006 from http://www.w3.org/TR/2004/REC-CCPP-struct-vocab-20040115.
Yang J.-T., Yu, P.-T., & Chen, W. (2004). An Ontology-Based Course Editor (OBCE) for SCORM-compliant
Learning Objects. In Proceedings of the IEEE International Conference on Advanced Learning Technologies
(ICALT’04), Joensuu, Finland.
50
Van Assche, F., Duval, E., Massart, D., Olmedilla, D., Simon, B., Sobernig, S., Ternier, S. & Wild, F. (2006). Spinning
Interoperable Applications for Teaching & Learning using the Simple Query Interface. Educational Technology & Society, 9
(2), 51-67.
Spinning Interoperable Applications for Teaching & Learning using the
Simple Query Interface
Frans van Assche
European Schoolnet, Rue de Trèves 61, B-1040 Brussels, Belgium
Tel: +32 2 790 7575
[email protected]
Erik Duval
Katholieke Universiteit Leuven, Dept. Computerwetenschappen
Celestijnenlaan 200 A, B-3001 Leuven, Belgium
Tel: +32 16 327066
[email protected]
David Massart
European Schoolnet, Rue de Trèves 61, B-1040 Brussels, Belgium
Tel: +32 2 790 7575
[email protected]
Daniel Olmedilla
L3S Research Center and Hannover University, Deutscher Pavillon, Expo plaza 1, D-30539 Hannover, Germany
Tel: +49 511 762.9741
[email protected]
Bernd Simon and Stefan Sobernig
Vienna University of Economics and Business Administration
Institute of Information Systems & New Media, Augasse 2-6, A-1090 Vienna, Austria
Tel: +43 1 31336 4443
[email protected]
[email protected]
Stefaan Ternier
Katholieke Universiteit Leuven, Dept. Computerwetenschappen
Celestijnenlaan 200 A, B-3001 Leuven, Belgium
Tel: +32 16 327066
[email protected]
Fridolin Wild
Vienna University of Economics and Business Administration
Institute of Information Systems & New Media, Augasse 2-6, A-1090 Vienna, Austria
Tel: +43 1 31336 4443
[email protected]
Abstract
The Web puts a huge number of learning resources within reach of anyone with Internet access. However,
many valuable resources are difficult to find due to the lack of interoperability among learning repositories.
In order to achieve interoperability, implementers require a common query framework. This paper discusses
a set of methods referred to as Simple Query Interface (SQI) as a universal interoperability layer for
educational networks. The methods proposed can be used by a source for configuring and submitting
queries to a target system and retrieving results from it. The SQI interface can be implemented in a
synchronous or an asynchronous manner and is independent of query languages and metadata schemas. In
this paper SQI’s universal applicability has been evaluated by more than a dozen implementations
demonstrated in three different case studies. SQI has been finalized as a standard in the CEN/ISSS Learning
Technologies Workshop. Latest developments of SQI can be followed at http://www.prolearnproject.org/lori/.
Keywords
Educational mediators, Learning object metadata, Standards, Interoperability, Brokerage, Application Program
Interface, Learning Repositories, Querying, Web Services
51
Introduction
Performing a search constitutes a typical process for everyone involved in teaching and learning. For example,
pupils complement things they have learned in school by looking for learning resources when working on a
homework assignment. Teachers aim at enriching their courses with external resources in order to reduce their
own workload through re-use while increasing the effectiveness of their course at the same time. Employees
responsible for personnel development try to find a cost-efficient but effective learning arrangement that fill a
particular competence gap.
The Web puts a huge number of resources within reach of anyone with Internet access. However, many valuable
resources are difficult to find, because they are hidden in the closed and proprietary worlds of brokerage
platforms, learning (content) management systems, streaming media servers and online collaboration tools.
These systems - commonly referred to as learning object repositories - hold information on learning objects such
as courses, online tutorials, lecture notes, electronic textbooks, tutoring sessions, quizzes, etc.
Learning object repositories lacking interoperability create three major drawbacks for its users:
1. The number of learning objects accessible is limited to those held in the local repository. Teachers, learners,
and learning managers might miss the latest critical developments due to the fact that the existence of an
external learning object is hidden from them.
2. The way users can search for learning objects is restricted by proprietary semantics or by document-centred
interfaces offered by today’s search engines. While the proprietary semantics can lead to effective search on
the local repository, the lack of semantic alignment with the outside world reduces the usability. On the
other hand general search engines hardly provide means for effective searches. For example, it is difficult to
satisfy a search such as ‘List me all courses that improve my negotiation skills and are offered in Brussels
between 15th of October and 30th of November and that are frequently visited by employees of the
pharmaceutical industry’ using state-of-the-art technology.
3. Context information on the learning object is limited to the local environment hereby reducing the
possibilities for deploying effective recommendation mechanisms. The evolution of web search engines has
largely benefited from re-using context information (Page et al., 1998), e.g. by investigating the hyperlink
structure of web documents (‘who links to whom?’). This kind of information is not accessible for closed
educational systems, which is especially a drawback for making a highly complex decision such as selecting
the right learning object. The context of learning objects – especially information on its usage (‘who has
successfully used which learning object?’) – constitutes highly valuable information that is hidden in the
closed and proprietary worlds of today’s repositories.
Beyond the individual level, institutions are increasingly demanding interoperability in the educational domain
in order to realize scenarios, such as
¾ exchanging learner performance records between applications,
¾ integrating course descriptions from heterogeneous providers or sharing them among multiple
applications (e.g., learning management systems and performance measuring tools),
¾ delivering learning using a multi-application learning environment (content management systems and
video conferencing tools), or
¾ comparing course evaluation data between different sites using different evaluation tools.
On the institution level, interoperability is a driving force for integrating organisations, improving decision
making processes, or expanding electronic distribution channels.
In this paper we propose a common query interface as one part of the solution for making educational systems
interoperable and exploring the hidden educational web (Raghavan & Garcia-Molina, 2001). The remainder of
this paper starts with an introduction to the various aspects of interoperability. The sub-sequent section is
devoted to the specification of a query service, but also reviews related work in this area. We continue by
presenting three case studies established in different educational sub-domains that demonstrate interoperability.
The paper concludes with a discussion of the status quo and outlines future work in the area.
Within its focus on interoperability for querying, this paper targets architects of educational networks, managers
of learning object repositories, stakeholders in learning object re-use, as well as researches in web services and
system interoperability.
52
Aspects of Interoperability
Interoperability can be defined as ‘the ability of two or more systems or components to exchange information
and to use the information that has been exchanged’ (IEEE, 1990). This definition comprises three primary
aspects that constitute interoperability, namely, exchange (i.e., communication) and use (i.e., correct
interpretation) of information (i.e., data plus meaning).
Information emerges from data through processing and utilisation. To ensure the correct interpretation in
communication, as denoted in the definition, a common semantic model is required. This semantic model – also
referred to as ontology – should specify the properties of the learning object accessible within the repository
(Wiederhold et al., 1992). Each declaration of a learning object property constitutes an ontological commitment
to use the defined term in interactions with the repository.
The sum of all ontological commitments is reflected in a common schema. In the last couple of years institutions
and associations such as ISO, IEEE, IMS, ADL, CEN/ISSS, or the HR-XML consortium have given rise to an
inflationary number of standards and specifications that can be used as the basis for a common schema. In the
case of learning object descriptions the IEEE LOM standard is the predominant reference for developing
schemas. While in some cases LOM (and its standard XML binding) can be used as such, in other cases
application specific adaptations are necessary (Quemada & Simon, 2003, Najjar et al., 2004; Rivera et al., 2004).
These specific instantiations of the LOM model are referred to as LOM Application Profiles.
On a lower layer, data model & format choices need to be specified, as different options apply for data
representation format (i.e., the data model) and character set encodings. Typically, XML and relational databases
with UTF-8 support are to be found here.
Figure 1. Interoperability Framework
In order to realize system to system communication a common messaging service, enabling repositories to
interact, is required. XML RPC, Java RMI, and WSDL/SOAP are examples for such a messaging infrastructure.
The messaging service is based on lower level network protocols such as TCP/IP, or HTTP.
Based on the two pillars ‘communication’ and ‘representation’, interoperable applications can be built (see
Figure 1). These applications might take advantage of interoperable atomic services that are aggregated in
composite services. Atomic services might be needed, for example, to agree on a common procedure for
uniquely identifying learning objects. Another example for an atomic service is related with authenticating users
and repositories, or with creating and managing access control rights. Higher level services that create
interoperable applications are, for example, an indexing service, a harvesting service or a query service. An
indexing service is a kind of replication service that allows repository A to push learning object metadata to
repository B. It supports distributed maintenance of metadata through insert, delete or update operations. A
harvesting service is a service, where repository A pulls metadata from a repository B. The query service allows
repository A to search repository B for suitable learning objects, so the metadata transferred matches a specific
query. A contracting service assigns access rights to a learning object stored at a remote repository. A delivery
53
service interacts with the repository where the learning object is stored and delivers an electronic learning object
to the end user.
The design choices made when implementing the applications are driven by the use cases the applications are
designed for. Examples of use cases are ‘Course selection for the purpose of providing certain qualifications’ or
‘Provision of a federated search in order to satisfy teachers in their need for new course material’. Use cases take
place in a particular context. The context can be abstracted via the domain the application is designed for.
Educational sub-domains such as higher education, vocational training or schools influence interoperability
along all aspects. Technological choices made in the context also influence the concrete realization of
interoperable application services.
The Simple Query Interface – An Interoperable Application Service for Querying
While for many aspects of the Interoperability Framework wide-spread solutions do exist, the educational
domain is still lacking interoperable application services that take advantage of standards such as IEEE LOM,
WSDL/SOAP, and XML. In this section we propose an application service for querying that can be used for
federating heterogeneous learning object repositories.
An application service for querying needs to specify a number of methods a repository can make available in
order to receive and answer queries from other applications. To distinguish the requestor from the answering
system in our scenarios, the term ‘source’ is introduced in order to label a system which issues a search (the
source of the query). The term ‘target’ labels the system which is queried (the target of the query). Alternatively,
the ‘source’ can also be referred to as ‘requestor’ and ‘target’ as ‘provider’.
Metadata can be stored using different means, such as file-based repositories, (possibly distributed) relational
databases, XML databases, or RDF repositories. In order to make learning repositories interoperable, not only a
common interface needs to be defined, but also a common query language together with a common results
format for learning object descriptions needs to be agreed on.
The query service is used to send a query in the common query language to the target. Next, the query results,
represented in the common results format, are transported to the source. On the implementation level, wrappers
may need to be created to convert a query from a common query language X to a local query language Y and
transform the query and the query results from a proprietary format to a common one and vice-versa.
Figure 2 illustrates an exchange process, where Learning Repository A (the source) submits a query to Learning
Repository B (the target). It is assumed that both systems have agreed upon a common query language
beforehand. The concepts used in the query statement are part of a common (query) schema. At Repository B,
the interface component might need to transfer the query from the common query language to the local one. Also
some mappings from the common to the proprietary schema might be required before submitting the search. This
task is performed by a wrapper component. Once the search has yielded results, the results set is forwarded to the
source, formatted according to a common results format.
The collaborative effort of combining highly heterogeneous repositories has led to the following requirements:
¾ The application service needs to be neutral in terms of results format, query schema and query
language. The repositories connecting can be of highly heterogeneous nature: therefore, no assumptions
about these components of the interoperability framework can be made.
¾ The application service needs to support synchronous and asynchronous queries in order to allow the
application to be deployed in various use cases and heterogeneous network architectures.
¾ The application service needs to support, both, a stateful and a stateless implementation.
¾ The application service shall be based on a session management concept in order to separate
authentication issues from query management, but also for providing an anchor point for implementing
simple business models for access control.
54
Figure 2. Communication between two Repositories
In addition, the design of the application service itself is based on the following design principles:
¾ Command-Query Separation Principle,
¾ Simple Command Set and Extensibility.
The following sub-sections describe each of the above mentioned requirements and design principles of the
query service in more detail. Since one design objective of the query service is to keep the specification simple
and easy to implement, we decided to call the application service ‘Simple Query Interface (SQI)'.
Query Language and Results Format
In order to make use of SQI to implement full query functionality, the application service needs to be
complemented with agreements about:
¾ the set of attributes and vocabularies that can be used in the query,
¾ the query language and its representation, and
¾ the representation of (a list of) learning object metadata that satisfy the query.
Driven by the Interoperability Framework of Section 2, SQI is agnostic on these issues: Any agreement between
two or more repositories is valid for SQI. Such agreements can, for example, be expressed via XML schemas or
RDF schemas. Although SQI does not directly contribute to overcome the differences of the various paradigms
in metadata management (Z39.50, XML-based approaches, RDF community), it aims to become an independent
specification for all open educational repositories.
Synchronous and Asynchronous Queries
SQI can be deployed in two different scenarios: In the synchronous scenario, the source, after having sent a
query, keeps waiting for an answer from the target or a time-out. Results retrieval is therefore initiated by the
source through the submission of the query and through other methods allowing the source to access the query
results.
In the asynchronous scenario, results transmission is target-initiated. Whenever a significant amount of matching
results is found, these results are forwarded to the source by the target. To support this communication the source
must implement a results listener. The source must be able to uniquely identify a query sent to a particular target
(even if the same query is sent to multiple targets). Otherwise the source is not able to distinguish the search
results retrieved from various targets and/or queries previously submitted to a target.
Please note that the asynchronous query mode does not require an asynchronous handling on the messaging
layer. It can also be implemented by two synchronous functions at the source and the target, respectively.
A query interface operated in synchronous mode can perform multiple queries per session (even simultaneously).
In case of an asynchronously operated query interface, the source provides a query ID that allows it to link
55
incoming results to a submitted query (the source might query many targets and each target might answer to a
query by returning more than one result to the source). Multiple queries can also be active within a session in
asynchronous query mode.
Session Management
The application interfaces make abstraction from authentication and access control issues. However, there is a
need to authenticate the source in order to allow a target, for example, to link query policies to a source
repository. For instance:
¾ Repository A is allowed to query Repository B without any limitations,
¾ Repository C is only allowed to retrieve 1000 query results per day from Repository D at a maximum.
Ideally, authentication is performed only once for a series of interactions. To accomplish this, a session token
needs to be returned after successful authentication that can be used to identify the system in the subsequent
communication.
Session management needs to be understood as a higher-layer management of configuration settings and
authentication. The session ID serves as a mandatory element in the application interfaces in order to identify the
requestor/source in all query commands.
Therefore, the SQI is based on a simple session management concept. A session has to be established before any
further communication can take place. This specification separates query management and processing from
authentication and authorization.
The source establishes a session at the target and uses the Session ID, which is obtained from the target, to
identify itself during communication. Authentication does not need to be based on username/password
declarations or credentials, since also anonymous sessions can be created. The specification introduces an
incomplete list of possible means for establishing a session for the communication between two systems. Once a
session has been established, the source has the right to communicate with the target. In order to establish a
session, a user name and password or any other credential may be required. The identification of a source
repository can prevent candidate target repositories from opening up their systems to unknown partners, and
enables query policies.
A session is valid until it is destroyed. Hence, it continues to be active after a query has been executed.
Alternatively, a session times out when no communication takes place during e.g. 30 minutes. However, a
session might be valid much longer than 30 minutes and sometimes might even require manual destruction.
The specification assumes the use of secure authentication, and encryption mechanisms such as those provided
by state-of-the-art technology (e.g., SSL).
Stateful and Stateless Communication
Stateful and stateless are attributes that describe whether repositories are designed to keep track of one or more
preceding events in a given sequence of interactions. Stateful means that the target repository keeps track of the
state of interaction, for example, by storing the results of a previously submitted query in a cache. Stateless
means that there is no record of previous interactions and that each interaction request can only be handled on
the basis of the information that comes with it. The SQI specification allows implementers to opt for a stateful or
a stateless approach.
Command-Query Separation Principle
SQI design adheres to the principle of Command/Query Separation, a design convention originally devised for
the Eiffel programming language. It states that every method should either be a command that performs an
action (also referred to as ‘procedure’), or a query that returns data to the caller (a ‘function’), but not both. More
formally, methods should return a value only if they are referentially transparent, i.e. they can be expected to
leave an object’s state unchanged and hence cause no side-effects. This convention is valuable for assertion-
56
based programming, a building block of the design-by-contract approach, the latter being essential with respect
to service-oriented architectures in general (Henney, 2000).
Simple Command Set and Extensibility
In order to make the interface easily extensible an approach, minimizing the number of parameters of the various
methods rather than the number of methods is adopted. Variations of the interface (e.g., a separation between
common query schema and common results format), can easily be introduced by adding a new function (e.g.,
setSupportedQuerySchema) while no change in the already implemented methods is needed. Hereby, backwards
compatibility can be more easily maintained.
As a result, additional methods for setting query parameters like maximum duration and maximum number of
returned search results were introduced. This design choice leads to simpler methods, but the number of
interdependent methods is higher. However, default values can be used for many of these query parameter
configuration methods.
Overview of SQI Methods
Table 1 provides an overview of the various methods that are described below from a workflow perspective. A
detailed description of the methods is provided in the specification (CEN/ISSS, 2005).
First, the source needs to create a connection with the target, for example by using createAnonymousSession.
Once a session has been established, the query interface at the target awaits the submission of a search request.
In addition, a number of methods allow for the configuration of the interface at the target. Query parameters such
as:
¾ the query language (setQueryLanguage),
¾ the number of results returned within one results set (setResultsSetSize),
¾ the maximum number of query results (setMaxQueryResults),
¾ the maximum duration of query execution (setMaxDuration),
¾ and the results format (setResultsFormat)
can be set with the respective methods. The parameters set via these methods remain valid throughout the whole
session or until they are set otherwise. If none of the methods is used before the first query is submitted, default
values are assumed. The specification provides default values for MaxQueryResults, MaxDuration, and
ResultsSetSize.
Table 1. SQI Methods
Session Management
createSession
createAnonymousSession
destroySession
Query Parameter Configuration
setResultsFormat
setMaxQueryResults
setMaxDuration
Synchronous Query Interface
setResultsSetSize
synchronousQuery
getTotalResultsCount
Asynchronous Query Interface
asynchronousQuery
setSourceLocation
queryResultsListener
Next, the source submits a query, using either the asynchronousQuery or the synchronousQuery method. The
query is then processed by the target and produces a set of records, referred to as results set. The query is
expressed in a query language identified through a query parameter. In the query, reference to a common schema
57
might be made. In synchronous mode the query results are directly returned by the synchronousQuery method.
The getTotalResultsCount method returns the total number for matching metadata records found by the target
operating. In case of an asynchronously operated query interface the queryResultsListener method is called by
the target to forward the query results to the source.
In order to report abnormal situations (e.g., erroneous parameters or inability to carry out an operation), an
SQIFault is provided, which can be thrown by all the SQI methods. A system of fault codes permits to document
those abnormal situations.
Related Work
In recent years, researchers from various fields have become increasingly interested in interoperability standards
for information systems in the (wider) context of Technology Enhanced Learning. In order to dissociate the
efforts leading to the shaping and the adoption of SQI from comparable work, we provide a comparison on
several dimensions: motivation and constitutive uses cases of the projects under consideration, application
services offered, the respective extent of schema and representation handling, and messaging services and
networking protocols used. The following account is inspired by the Interoperability Framework presented in
Section 2.
Interoperability research is driven by initiatives of different levels of granularity. We consider most of these
orthogonal to our efforts, therefore differences manifests primarily at the levels of motivation, use case and
scope. OpenURL (NISO, 2002; http://library.caltech.edu/openurl/), for example, and the underlying framework
emerged from the field of digital libraries, or scholarly journal repositories to be more specific. The OpenURL
standard aims at realizing decentralized resource resolution architectures that are adaptive to the human seekers’
organisational context and open to be extended in terms of repositories published and resolution services offered
across organizational boundaries. OpenURL adopts a strict separation of referencing resources, i.e. creating
annotated and globally valid resource identifiers, and resolving these references by organization-specific resolver
facilities into, for instance, seeker-tailored library services (Van de Sompel & Beit-Arie, 2001). This highlights
the major difference between OpenURL and SQI: While OpenURL targets identifier creation and user-centric
identifier resolution across heterogeneous repositories, SQI realizes lookup infrastructures. An initiative of
similar granularity, the Content Object Repository Discovery and Resolution Architecture (Rehak, 2005) is
interested in providing a reference model for federating community- or organization-specific repositories and for
pyramids of repository federations (‘Federated CORDRA’). The model describes centralized services for
discovery (e.g., registries for content, affiliated repositories, system specifics) and resolving resource references.
The latter refers to a unique identifier infrastructure based on namespace hierarchies in and across federations
(Rehak, 2005). Efforts such as OpenURL and CORDRA are complementary to SQI as the latter – for instance –
might be re-used as discovery standard in a (federated) CORDRA network or for querying repositories that refer
to their learning objects by OpenURL identifiers (a more in-depth investigation of the applicability of SQI to
CORDRA including a comparison of SQI-related approaches has recently been conducted by researchers
involved in the CORDRA project (CORDRA, 2004).
As for lookup standards or other initiatives at a comparable granularity level, Google Web API
(http://www.google.com/apis/) and Amazon E-Commerce Service (www.amzaon.com/gp/aws/landing.html)
have recently gained momentum. In principle, while designed for the specific needs of exclusively publishing the
respective set of services in a (web) service-oriented manner their interaction standard models could be re-used
partly for more generic scenarios. Reacting to their emergence and in continuation of the Z39.50 family of
standards, the development of a Search/Retrieve Web Service (SRW) and a Search/Retrieve URL (SRU) for
learning object repositories was initiated by the Z39.50-International Next Generation (ZING) community
(Board, 2004). The domain of application of SRW/U-compliant services is still centred around digital libraries
and research repositories, mainly by providing backward or gateway compatibility to existing Z39.50 nodes,
although the protocols and available tool kits are adaptable to other contexts as well. SQI differs in this respect
as it has already been adopted in contexts involving metadata about learning activities, as contrasted to learning
material. SQI also meets the requirements for cooperating with providers other than digital libraries and
scholarly repositories such as corporate providers of professional training (see Section 4.3).
Another relevant initiative is the IMS Digital Repository Interoperability (DRI) Specification (IMS, 2003). It
provides recommendations for realizing interoperability with respect to most common repository functions
identified at a high level of abstraction, delegating many design choices to implementers. In contrast to
CORDRA, it emphasizes types of interaction or lookup functions while CORDRA provides an architectural
58
perspective. The IMS DRI specification therefore served as a skeleton, for example, for the EduSource project
(Hatala et al., 2004) and its underlying communication protocol – referred to as EduSource Communication
Layer (ECL) protocol. The Open Knowledge Initiative (OKI) develops an open system to support on-line
training. It provides a collection of Open Service Interface Descriptions (OSIDs) that are conceptualized as
contracts in service-oriented environments and range from an OSID for repository interoperability to workflow
and various learning service OSIDs. The concept of OSID is – like SQI – at a lower level of abstraction than the
IMS DRI specification and is closely tied to ready-made and publicly available software components.
The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) has been adopted beyond the digital
library community and integrates with resolution standards like OpenURL (Dolog et al., 2003). OAI-PMH, in
contrast to SQI or SRW/U, serves as facilitator for actual lookup services, also denoted as “Service Providers” in
OAI terminology. A lookup standard that did not originate from the digital library or learning object community
is OpenSearch (http://opensearch.a9.com/). Its main objective is the aggregation of lookup results of web-based
search engines, therefore the focus lies – in contrast to SQI – on standardizing repository or rather search engine
descriptions and result schemas for further processing. More recently, the issue of lookup has been discussed in
connection with versatile web-based query languages independent from traditional data base management
systems and their query exchanging and processing facilities: The Simple Protocol And RDF Query Language
(SPARQL) specification hosts both a query language definition for RDF and a query exchange protocol
(“SPARQL Protocol” in short) that is closely interwoven with the latter (DAWG, 2005b). This allows for
specific query language constructs to be directly converted into required exchange patterns between a query
source and a query target. The SPARQL protocol builds on concepts of (web) service-oriented architectures and
targets a large set of use cases including the discovery of RDF-based descriptions of learning resources (DAWG,
2005a). The tight coupling of query language and query exchange / lookup protocol is a clear distinction
between SPARQL protocol and SQI. The latter delivers arbitrary languages that are entirely independent from
SQI as shuttle protocol.
SQI comprises two central atomic services as expanded in more detail in Section 3: a lookup or query shuttle
service and a combined session, authentication and authorization management service. As for lookup, this allows
for two basic modes of operation, stateless and stateful, and therefore synchronous or asynchronous exchanges of
queries and corresponding result sets. In this respect, SRW/U shares similarities with SQI, but also shows major
differences. SRW/U is purely synchronous (source-initiated), i.e. query results are returned within a single
transaction. Query results, however, can be cached optionally and referred back to be based on a result set
identifier up to a configurable time-to-live. While limited in its basic lookup service, SRW/U offers differently
flavoured lookup refinements in addition, i.e. an index browsing (‘scan’) and advertising (‘explain’) service that
exposes repository capabilities on demand (Board, 2004). In the same way, OAI-PMH takes in the query
primitive ‘Identify’ that requests basic repository blueprints (Dolog et al., 2003). The OpenSearch standard, on
the other hand, requires depositing repository descriptions with an aggregator’s registry. Nevertheless, all
mentioned specifications including the EduSource ECL protocol are limited to synchronicity of their basic
lookup services, apart from SQI.
The IMS DRI specification outlines five core commands in terms of pre-defined conversational styles, i.e.
Search/Expose, Gather/Expose, Alert/Expose, Submit/Store, and Request/Deliver, on a highly abstract level. SQI
therefore conforms to the Search/Expose conversation pattern while EduSource, that explicitly implements IMS
DRI specification to a great extent in a service-oriented manner, provides a wider spectrum of lookup options,
i.e. Search/Expose, Submit/Store, Request/Deliver and Gather/Expose. The SPARQL Protocol itself does not
distinguish different types of lookup as they are imposed by the hosted RDF Query Language. The latter
comprises four distinctive ‘query forms’ for requesting only variable bound responses (‘select’), for generating
custom return RDF graphs (‘construct’), for requesting context information about resources (‘describe’) and for
limiting results to binary truth statements (‘ask’) (DAWG, 2004).
Most related interoperability standards such as SRW/U, OpenSearch, OAI-PMH or the SPARQL Protocol do not
explicitly include a formally conceptualized session or access management service, though they can be
complemented by opting for an application profile or by adopting an appropriate specification (CORDRA,
2004). Apart from SQI, the OKI project offers closely coupled Authentication and Authorization OSIDs that
form a generic access framework (OKI, 2005.). Moreover, there are some OSIDs defining basic services such as
lookup routing.
As far as composite services are concerned, EduSource provides the Splash Federated Search facility over a
registry of ECL-compliant repositories (EduSource, 2003; Hatala et al., 2004). SQI, however, has proven its
applicability for derived or enhanced services such as network for federated search, end-user software and LMS
59
plug-in facilities, and higher level location and storage resolution (see Section 4). OpenSearch defines an
aggregation service for result sets conforming to RSS as existing syndication standard and therefore raises the
issue of post-processing of lookup results as a composite service. More tailored composite services are devised
by OKI by providing OSIDs for a variety of learning administration scenarios, such as assessment, course
management, grading and the like (OKI, 2005).
The various approaches subject to this comparison also differ with respect to their specificity or neutrality
towards underlying representation models and metadata schemas, i.e. prescribed specifications and/or
application profiles. The degree of dependency varies widely for the levels of application services, messaging
and transport protocols. SRW/U is entirely bound to XML as representation model at all of these levels: The
SRW/U protocol is represented in XML, its SOAP flavor (SRW) in particular. SRW/U does not prescribe a
specific metadata specification and therefore operates over various XML bindings of metadata standards such as
Dublin Core, MARC and LOM. Furthermore, SRW/U was designed in mutual dependence for a specific query
language, i.e. the Common Query Language (CQL) that provides both expressive power and readability to
human seekers. As for shaping query results, SRW/U offers support for user selection of the schema used. In
OAI-PMH, XML is also used at the messaging layer yet exclusively as representation form of responses to OAIPMH requests. Similarly, OAI-PMH allows to use arbitrary XML bindings of various metadata standard with
Dublin Core, MARC and RFC1807 being provided off-the-shelf. In contrast, SQI is only partly dependent on
XML as representation model by design, provided that SQI is realized as SOAP-based web services. SQI equally
supports the selection of result schema although the generic character of SQI would not require it per se: Some
query languages offer constructs to achieve schema specificity of result sets, e.g. QEL or SPARQL. The IMS
DRI specification contains explicit recommendations for the usage of XML (and therefore XQuery) at the
mediating representation model and at the messaging layer, i.e. SOAP. As a consequence, the ECL protocol
relies exclusively on XQuery and enforces compatibility of affiliated repositories. The latter supports various
metadata schemas, including XML bindings of LOM, Cancore and Dublin Core. The SPARQL language and
protocol are closely tight to RDF as representation model with XML solely being used for some query forms
(apart from RDF/XML as notational form). At the messaging layer, XML is integral part of some proposed
protocol bindings such as SOAP. OpenSearch adopts XML in the shape of RSS as response format.
The interoperability standards can be realized differently at the messaging and transport layer. We distinguish
between different architectural styles that underlie interaction scenarios, i.e. Remote Procedure Call (RPC),
Representational State Transfer (REST) and Document / Message Exchange (D/ME), and embodiments of these
styles in terms of protocols (XML-RPC, SOAP, HTTP). SQI and OKI OSIDs reflect the RPC style as a large
proportion of the comparable work does, e.g. OAI-PMH, SRW/U, and SOAP is predominantly used as binding
protocol. In contrast to SQI and OKI, OAI-PMH, SRW/U (i.e. its SRU flavour) offer alternatives conforming to
the REST that are therefore realized as HTTP transactions. OpenSearch, however, builds on REST exclusively.
Implementation Case Studies
Since the first stable version of the specification was made available in March 2004 many learning repositories
have taken advantage of SQI to connect them to the outside world. Under the auspices of the CEN/ISSS
Learning Technologies Workshop the following projects took advantage of the SQI specification.
Ariadne
The basic mission of Ariadne is to enable better quality learning through the development of learning objects,
tools and methodologies that enable a "share and reuse" approach for education and training. The Ariadne
Learning Object Repository is called the “Knowledge Pool System” (KPS): the basic idea is that the KPS holds
learning objects and metadata that describe them, so that they can be more easily managed and made available in
appropriate contexts. Typical Ariadne user communities include higher education as well as corporate training.
The core of the Ariadne Knowledge Pool System (KPS), (Duval et al., 2001) is a distributed network of
Learning Object Repositories that replicate both (the publicly available subset of) content and metadata. On top
of this core infrastructure, Ariadne provides its members with a set of tools that are loosely coupled with the
KPS (Ternier & Duval, 2003). Through these tools, the user community can transparently manage learning
objects.
60
Examples of such applications that were coupled with the Ariadne KPS are Learning Management Systems (like
Blackboard, Moodle and Ines) or authoring tools (like Phoenix or Powerpoint). Integrating SQI into these
applications provides learners and teachers with transparent access to Learning Objects. Using e.g. the
PowerPoint plug-in, a teacher can easily reuse pictures, slides that are already stored in the Ariadne knowledge
pool, while the LMS plug-in enables reuse of complete learning objects like courses or presentations.
More generally, a distinction is made between two search scenarios that pose different requirements to a search
service: searching within the Ariadne KPS only and searching beyond the borders of Ariadne.
While implementing an SQI target on the Ariadne KPS, we experienced that a synchronous target lowers the
threshold for integrating search functionality into an application such as Microsoft PowerPoint. In this scenario,
an application sends queries to a synchronous target and only fetches additional results when they are requested
by the user. As a single Ariadne repository acts in this context as if it were a centralized system and as searching
these repositories is optimized with search indices, there is no need to cache results for a query. This motivated
the use of a stateless scenario. In a first phase, an SQI target was constructed that implements all SQI methods.
Next, persistent sessions were introduced. As some applications only need a subset of the capabilities of a
repository, we created persistent sessions that store a profile on the server. As an example, the PowerPoint plugin uses a session identifier that is pre-configured to only search within graphical objects using Lucene queries.
The Ariadne SQI target comes with three query languages. A simple query language enables a source to send a
list of search terms that are then used by the target to conduct a full metadata search. Next, a proprietary query
format provides Ariadne tools with high level advanced queries. Finally, the Ariadne target also supports the
Lucene query language.
In order to provide Ariadne members access to other repositories, a federated search engine (Ternier et al., 2005)
has been developed. This federated search engine acts as an intermediary service and makes it possible to
transparently search into different repositories. The service exposes itself as a complete SQI target and acts as
virtual repository that can be queried by a source. This facilitated the integration of the federated search engine
into Ariadne clients as the interface of the federated search engine is the same as the Ariadne SQI Target. In the
back-end, the federated search engine asynchronously distributes the query to different SQI enabled repositories.
In this scenario, all repositories register their results using the results listener that is implemented on the
federated search engine, which makes it easy for this engine to manage results that come from a large set of
repositories. When a source invokes a query on this target, a state is being maintained for 15 minutes on the
server. During this time, repositories can keep storing results for a query and the client can poll for additional
results. In this stateful implementation, invoking the getTotalResultsCount method will result in different results
as over time the amount of results that get registered will increase. Currently, searches are distributed into the
following repositories: Ariadne, EdNA Online, EducaNext, Merlot, Pond, RDN, Smete, Voced, Citeseer and
LionShare. As all these repositories support different query languages, which do not always easily map into one
another, we started with an approach where a least common denominator of all query languages was
implemented. In this approach, the query only consists of search terms which are translated by each repository
into a query it can process.
Currently, Ariadne requires each partner to return a minimum of metadata fields, encoded as LOM XML: a
URL, an identifier, a title and an identifier of the originating repository. The URL should resolve to the learning
object. If access to the learning object is prohibited, the URL resolves to contact information. A repository
identifier is necessary to give credits in a proper way to repository that yielded the results. Apart from the data
elements mentioned above, all other LOM metadata fields are optional in the results.
Celebrate and iClass
The iClass project aims at exploiting the potential of ICT to support a personalized, flexible and learner-centric
approach. This pedagogical approach strives to facilitate empowerment of both pupils and school teachers, while
producing personalized learning experiences. Based on this, the iClass project seeks to establish a framework to
deliver a personalized, adaptable, and adaptive learning experiences in a collaborative environment for learners.
The framework is based on an advanced learning system, named ‘Intelligent distributed Cognitive-based
Learning System for Schools’ (iClass). The latter takes advantage of an ontology-based architecture to sequence
knowledge and adapt it to the learner's level of understanding and learning style by dynamically creating
individualized learning objects.
61
The cost necessary to acquire, deploy, and integrate an e-learning infrastructure such as iClass makes it an
operation that cannot be renewed, especially by schools, each time that new tools are available. Moreover the
cost of acquiring and/or producing learning contents adapted to such an infrastructure is often more important
than the cost of the infrastructure itself. In this context, interoperability between learning systems, by allowing
legacy systems to get access to content developed in new systems (and conversely), makes it easier for schools to
accommodate to the technological evolution.
The iClass adapter is a component of the iClass system. It enables the end-users of ‘non-iClass’ systems, such as
the learning management systems and learning content management systems that are members of the Celebrate
federation (van Assche & Massart, 2004), to search and get access to iClass contents (i.e., metadata and learning
objects).
Figure 3. iClass and its adapter.
The diagram of Figure 3 depicts the main components of iClass that interacts with the adapter. The adapter is the
entry point to the iClass system for external systems (e.g., system members of Celebrate). The adapter is also
connected to:
¾ A content access service (CAS), which manages the access to metadata,
¾ A conductor service, which is in charge of coordinating the different tasks necessary to compose
learning objects that are adapted to the user that requested them, and
¾ A rights management service, which in charge of authenticating and authorizing users and requests.
Metadata are stored in a peer-to-peer network of metadata repositories accessible thanks to the CAS. Usually,
obtaining a learning object is a two-step process:
1. Searching and evaluating metadata: Selecting a learning object that satisfies user needs on the basis of the
description provided in the metadata;
2. Consuming the learning object: Getting the selected learning object at the location (usually a URL) provided
in the metadata.
The current version of the adapter only supports queries expressed in an ad hoc query language referred as ‘very
simple query language’. This language consists of a set of keywords that are searched against the field title,
description, and keywords of the learning object descriptions --For the second release of the adapter, it is planned
to add support to XQUERY. The queries return results in a format based on the Celebrate application profile, an
application profile of the IEEE Learning Object Metadata standard, which was especially developed to take into
account the requirements of schools in Europe (Nirhamo & Van Assche, 2003). This metadata format provides
an identifier of the described learning object rather than its location. Actually, the adaptive and multimedia
nature of the iClass learning objects makes it difficult to get access to learning objects directly. This is why an
extra step is required to resolve the location of a learning object identified in the metadata. This ‘resolvelocation’ step is used to send a request to the conductor service that takes care of generating the requested object.
The newly assembled learning object is then moved to a streaming server (called iClass presenter) and a URL
from which, the object can be consumed is returned by the conductor. Since this process has potentially a certain
duration, the conductor answers to these requests asynchronously in order to ensure adequate performance of the
caller, in this case the iClass adapter. It is only when a learning object is available at the presenter that its
location is known and can be returned.
62
Since there does not exist an open interface for performing this step asynchronously and rather than creating an
ad hoc interface, it was decided to use SQI for this task as well by taking advantage of SQI independence in
terms of query languages and results formats. This is achieved by adding a new ‘query language’ (for requesting
a location) and a new ‘results format’ (for returning a location) to the list of languages and formats supported by
the iClass adapter (Massart, 2005). The new query language is named ‘ICLASS-LO-ID’. A query in this ad hoc
language consists of the requested learning object's identifier as found in the metadata. The results format is
named ‘URL’. A result in this format consists of a URL pointing to the requested learning object.
This solution permits the minimization of the cost of implementing a ‘resolve-learning-object-location’ step for
those learning object repositories that already use SQI for searching metadata. It is currently implemented as an
extension of the SQI gateway of Celebrate.
Elena’s Smart Space for Learning
The third case study is instantiated in the context of personnel development. In order to achieve interoperability
among heterogeneous systems for corporate personnel development, the Elena project (Simon et al., 2004)
realised a novel search and retrieval infrastructure for learning resources called Smart Space for Learning. This
infrastructure allows for the integration of heterogeneous educational nodes in a semantic network and provides
‘smart’ access technology for it (Simon et al., 2003). Integrated into a process-support for learning goal
definition, personalized search, and feedback tools, the educational semantic network (the ‘space’) plays a
crucial role for supporting corporate personnel development.
A driving force for the design of the Elena Smart Space for Learning has been an extensive requirements
analysis emphasising the need for integrating divergent resource providers into one common access tool
(Gunnarsdottir et al., 2004). The broad variety of learning resources made available and accessible extends the
scope of learning resources available. Hereby, potential learners are not dependent on offerings by a particular
provider or are restricted to a specific learning format, e.g. a costly classroom-based course. On the contrary,
they can expand their search to several types of learning formats and providers, e.g. books from Amazon media
store.
Another driving force for setting up a smart space for learning is the provider’s interest in disseminating their
educational services and products via electronic distribution channels. The project team has evidenced a trend
that these providers are increasingly offering electronic distribution channels in machine-processable formats
such as XML while technology leaders even offer web services that allow for querying.
The infrastructure of Elena’s Smart Space for Learning is built on three corner stones:
1. The Simple Query Interface (SQI) as interaction standard for both providing and consuming learning
resources’ metadata within SQI-compliant networks. Elena opted for a service-oriented architecture and
therefore makes use of web service bindings (SOAP) for SQI.
2. A global metadata schema (‘Elena schema’) against which queries are issued and in which search results are
expressed. This schema is realised as LOM application profile that enriches the LOM element battery by
mixing in other metadata standards such as OpenQ, a metadata standard specifically designed for
exchanging course information (Rivera et al., 2004). The rational behind developing a LOM application
profile is to design a schema according to Elena’s use case requirements while preserving LOM
compatibility. As for representation model, Elena employs RDF bindings for the global schema. Users’
queries against this global schema are expressed in the Query Exchange Language (QEL, see Nilsson &
Siberski, 2003), initially devised as a building block of Edutella, and semantic mappings are provided to
convert from global to local schemas (Olmedilla & Palmèr, 2005).
3. Re-usable software components for integrating existing systems with minimum effort. They are applied for
aligning the global and various local representation models (representation management and matching) and
for mediating between query languages involved (query transformation).
So far, we have connected a number of providers to our network that can all be accessed from the EducaNext
Portal for Exchanging Learning Resources (http://www.educanext.org/networksearch/) and from the personnel
development portal Human Capital Development (HCD) Online (http://www.hcd-online.org/networksearch/).
The recurrent pattern of linking a provider’s system to a SQI-compliant network comprises five steps: In the first
two steps, a SQI Target and a SQI Session Management object are to be created and registered with the
underlying web service framework. Taking a white box perspective, the SQI target object is then to be populated
with a mapping that is established between the respective local metadata representation and schema and the
63
global ones. Next, we devise a query translation facility that makes use of this mapping. Finally, the binding of
the local results format to the global schema is realised, again in accordance with the previously defined
mapping. The diversity that we experienced when applying this pattern in various concrete integration scenarios
is considerable.
The quality of partnerships, ranging from highly protective and restrictive to more open ones – open refers to
service hosting, disclosure of local metadata schemas and provided interfaces – required differing decisions at
the architectural level: In restraining scenarios (e.g., Amazon media store and Seminarshop.com) gateway
configurations appear as appropriate, in less restrictive cases (e.g,. Moodle, ULI, etc.) intermediaries are not
required, therefore reducing the complexity of implementation, maintenance and guaranteeing service quality.
Characteristics of the local architectures affect architectural decisions similarly. A case in point is the integration
of the Edutella peer-to-peer network that operates inherently in an asynchronous manner while SQI consumers
might expect synchronous interaction (Olmedilla & Palmèr, 2005).
As for heterogeneity in terms of metadata representations and schemas, academic and commercial learning
(content) management systems (e.g., EducaNext, CLIX) or course databases (course catalogue of the Vienna
Executive Academy) predominantly rely on relational data base systems. Data base systems of different vendors,
in our case Oracle, Postgresql, MySQL, and Firebird, cause, for instance, discrepancies in the encoding of result
sets. Bridging and translating between representation models entail the risk of losses in terms of expressive
power with respect to semantics for instance. Proprietary and/or non-standards compliant metadata schemas
increase the complexity of mapping to the global schema (e.g., Seminarshop.com). We resolved these issues
with respect to heterogeneity both by means of materialised or persistent integration and virtual or non-persistent
integration. The former involves the periodic replication and transformation of a targeted metadata repository
into a persistent and global-schema-compliant replica. Examples are the Executive Academy of the Vienna
University of Economics whose relational repository is replicated into a single triple relation. Other examples
include Knowledgebay and LASON whose replication is based on a native XML database. Virtual integration
does not involve persistent storage of replicated, intermediate or transformed metadata but a recurring on-the-fly
mapping process. This is the case for most integration cases in the current network setting (e.g., Amazon media
store, BFI). In rare cases metadata repositories are already aligned to a certain extent both with respect to the
representation model and the schema used (ULI).
In either case, whether considering materialised or virtual integration, query translation is an issue. The case of
Amazon media stores demonstrates how to translate between a formal and highly powerful query language such
as QEL and a parameterized canned query format as the Amazon PowerSearch syntax that cannot be considered
a fully featured query language. Re-formulating queries expressed in RDF query languages such as QEL and
RDQL into SQL is facilitated by re-usable software components provided by Edutella or similar efforts, such as
D2RQ (Bizer & Seaborne, 2004). We developed and matured a comparable component for translating between
QEL and XQuery that turned out to be highly re-usable in various, virtual and materialised integration contexts
(Miklós & Sobernig, 2005).
Discussion and Concluding Remarks
In order to achieve interoperability among learning repositories, implementers require a common communication
framework for querying. This paper discussed a set of methods referred to as SQI as a universal interoperability
layer for educational networks. Although the effectiveness of the specification has been proven by several
implementations (see previous section), lessons drawn from these and the discussion of comparable research
efforts (see Section 3.8) suggest that some issues need to be further investigated. They equally relate to usage
scenarios and application services provided.
Scope for application: We consider the adoption of SQI in different contexts, such as exchanging (language
versions of) vocabularies, evaluation data about learning / training service providers and as building block for an
interaction standard for collaboration services in various learning scenarios.
Advertise or Explain primitive: Specifications such as SRW/U or OAI-PMH provide such primitives while
OpenSearch adopted some sort of capability schema for describing affiliated repositories (Board, 2004; Dolog et
al., 2003; OpenSearch, 2005). SQI currently lacks such functionality and should therefore be extended
accordingly, for instance, by adding an interface method that returns a SQI profile record that holds information
on the query languages and results formats supported. This might alternatively be distributed among a set of
64
interrelated
interface
methods
getSupportedResultsFormats.
such
as
getSupportedQueryMode,
getSupportedQueryLanguages,
State Inference / Modifier primitives: We evaluate the possibility of supporting search status management, for
example, for cancellation of search, or query status reporting. This would allow a user at a query source to infer
from and interfere with an active search process at the target.
Coupling result processing: OpenSearch primarily focuses on providing syndication of search results retrieved
(OpenSearch, 2005). In the context of SQI it is important to find means for leveraging ranking mechanisms to
the configuration of the actual lookup. This allows for reducing network traffic, since metadata that is anticipated
not to be of high interest to the query issuing user can be omitted. At the same time, the quality of results subject
to a process of pre-ranking is improved and the need for post-retrieval processing is considerably reduced.
Consolidated interaction model: In its current shape, the SQI specification involves some transaction overhead
with respect to passing necessary arguments to the lookup procedure calls. In order to be able to set fundamental
sets of arguments at once, a unifying method (setQueryParameters) could be introduced. Moreover, the
conceptualization of a session management service at the level of SQI might be questioned: Currently, the HTTP
specification and web service standards built on top of it do not explicitly provide session handling. Nonstandardized practices based on cookie transactions, session-specific HTTP parameters and extensions to the
SOAP Header element are proposed and realized. From this angle, a basic session management facility as
described by SQI can be considered justified.
Acknowledgements
The SQI specification has been developed and financially supported under the auspices of the CEN/ISSS
Workshop on Learning Technologies. This work is supported by European Commission via the IST projects
CELEBRATE (http://celebrate.eun.org/), Elena (http://www.elena-project.org/), ICLASS (http://www.iclass.info/)
and PROLEARN (http://www.prolearn-project.org/). We acknowledge contributions and comments to the SQI
specification from Christian Werner (L3S Research Center), Dan Rehak (Carnegie Mellon University), Griff
Richards (Simon Fraser University), Gerhard Müller (IMC), Julien Tane (Universität Karlsruhe), Marek Hatala
(Simon Fraser University), Matthew J. Dovey (Oxford University), Michel Arnaud (Université de Paris X
Nanterre), Nikos Papazis (NCSR), Peter Dolog (L3S Research Center), Sascha Markus (IMC), Stefan Brantner
(BearingPoint Infonova), Stefano Ceri (Politecnico Milano), Simos Retalis (University of Piraeus), and Teo van
Veen (Koninklijke Bibliotheek), Zoltán Miklós (Wirtschaftsuniversität Wien, Technische Universität Wien).
References
Bizer, C., & Seaborne, A. (2004). D2RQ - Treating Non-RDF Databases as Virtual RDF Graphs. In Short
Paper/Poster Proceedings of ISWC2004, retrieved March 23, 2006 from http://www.wiwiss.fuberlin.de/suhl/bizer/d2rq/.
Board, S. E. (2004). SRW: Search/Retrieve Webservice Version 1.1 (Consolidated Specifications): retrieved
March 23, 2006 from http://srw.cheshire3.org/SRW-1.1.pdf.
CEN/ISSS (2005). A Simple Query Interface Specification for Learning Repositories (CEN Workshop
Agreement #15454). Brussels, Belgium, retrieved 23March 23, 2006 from ftp://ftp.cenorm.be/PUBLIC/CWAs/eEurope/WS-LT/cwa15454-00-2005-Nov.pdf.
CORDRA (2004). CORDRA: Technical Introduction and Overview: retrieved March 23, 2006 from
http://www.lsal.cmu.edu/lsal/expertise/projects/cordra/intro/intro-v1p00.html.
DAWG (2004). SPARQL Query Language for RDF,
http://www.w3.org/TR/2004/WD-rdf-sparql-query-20041012/
retrieved
March
23,
2006
from
DAWG (2005a). RDF Data Access Use Cases and Requirements: W3C.
DAWG (2005b). SPARQL Protocol for RDF: W3C.
65
Dolog, P., Gavriloaie, R., Nejdl, W., & Brase, J. (2003). Integrating Adaptive Hypermedia Techniques and Open
RDF-based Environments. In Proceedings of 12th International World Wide Web Conference. Budapest,
Hungary: ACM.
Duval, E., Forte, E., Cardinaels, K., Verhoeven, B., Van Durm, R., Hendrikx, K., Wentland Forte, M., Ebel, N.,
Macowicz, M., Warkentyne, K., & Haenni, F. (2001). The Ariadne Knowledge Pool System. Communications of
the ACM, 44 (5), 72-78.
EduSource
(2003).
ECL
Protocol
(Version
0.3).
http://www.edusplash.net/technical/ecl/ECL_Protocol_0.3.03.pdf
retrieved
March
23,
2006
from
Gunnarsdottir, S., Kieslinger, B., Küchler, T., & Simon, B. (2004). From e-Learning to Learning Management:
Results of an International Survey. In Proceedings of 4th International Conference on Knowledge Management.
Graz, Austria.
Hatala, M., Richards, G., Eap, T., & Willms, J. (2004). The Interoperability of Learning Object Repositories and
Services: Standards, Implementations and Lessons Learned. In Proceedings of the 13th World Wide Web
Conference. New York City, USA.
Henney, K. (2000, December). A tale of two patterns. Java Report, 84-88.
IEEE (1990). IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries.
New York, NY: IEEE.
IMS (2003). IMS Digital Repositories Interoperability - Core Functions Information Model: retrieved March 23,
2006 from http://www.imsglobal.org/digitalrepositories/driv1p0/imsdri_infov1p0.html.
Massart, D. (2005). Accessing Learning Contents Using a Simple Query Interface Adapter. In Proceedings of the
ProLearn-iClass Thematic Workshop on Learning Objects in Context. Leuven, Belgium.
Miklós, Z., & Sobernig, S. (2005). Query Translation between RDF and XML - A Case Study in the Educational
Domain. In Olmedilla, D., Saito, N. & Simon, B. (Eds.), WWW Workshop on Interoperability of Web-Based
Educational Systems, Chiba, Japan: CEUR, 27-35.
Najjar, J., Ternier, S., & Duval, E. (2004). Interoperability of Learning Object Repositories: Complications and
Guidelines. IADIS International Journal of WWW/Internet, 2, 1-16.
Nilsson, M., & Siberski, W. (2003). RDF Query Exchange Language (QEL) - Concepts, Semantics and RDF
Syntax: retrieved March 23, 2006 from http://edutella.jxta.org/spec/qel.html.
Nirhamo, L., & Van Assche, F. (2003). The CELEBRATE Metadata Application Profile. Celebrate Deliverable
D 3.2 (IST-2001-35188): Available March 23, 2006 at http://celebrate.eun.org/docs/D3_2.pdf.
NISO (2002). Information Retrieval (Z39.50): Application Service Definition and Protocol Specification: NISO
Press, retrieved March 23, 2006 from http://www.niso.org/standards/resources/Z39-50-200x.pdf.
OKI (2005). Open Knowledge Initiative: http://www.okiproject.org/.
Olmedilla, D., & Palmèr, M. (2005). Interoperability for Peer-to-Peer Networks: Opening P2P to the rest of the
World. In Olmdeilla, D., Saito, N. & Simon, B. (Eds.), Workshop on Interoperability of Web-Based Educational
Systems. Chiba, Japan.
Page, L., Brin, S., Motwani, R., & Winograd, T. (1998). The PageRank Citation Ranking: Bringing Order to the
Web. Stanford Digital Library Technologies Project.
Quemada, J., & Simon, B. (2003). A Use-Case Based Model for Learning Resources in Educational Mediators.
Raghavan, S., & Garcia-Molina, H. (2001). Crawling the Hidden Web. In Proceedings of the 27th International
Conference on Very Large Databases.
66
Rehak, D. R., Dodds, P., & Lannom, L. (2005). A model and Infrastructure for Federated Learning Content
Repositories. Paper presented at the WWW Workshop on Interoperability of Web-based Educational Systems,
Chiba,
Japan,
retrieved
May
1,
2006
from
http://cordra.net/information/publications/2005/www2005/cordrawww2005.pdf.
Rivera, G., Simon, B., Quemada, J., & Salvachua, J. (2004). Improving LOM-Based Interoperability of Learning
Repositories. In Duval, E. & Spyns, P. (Eds.), Proceedings of the OTM 2004 Workshop on Ontologies,
Semantics and E-learning, October, 2004: Springer LNCS (3292).
Simon, B., Dolog, P., Miklós, Z., Olmedilla, D., & Sintek, M. (2004). Conceptualizing Smart Spaces for
Learning. Journal of Interactive Media in Education, 2004 (9).
Simon, B., Retalis, S., & Brantner, S. (2003). Building Interoperability among Learning Content Management
Systems. In Proceedings of the 12th World Wide Web Conference. Budapest.
Ternier, S., & Duval, E. (2003). Web services for the ARIADNE Knowledge Pool System. In Proceedings of the
3rd Annual Ariadne Conference. Leuven, Belgium.
Ternier, S., Olmedilla, D., & Duval, E. (2005). Peer-to-Peer versus Federated Search: Towards more
Interoperable Learning Object Repositories. In Proceedings of ED-MEDIA 2005.
van Assche, F., & Massart, D. (2004). Federation and brokerage of learning objects and their metadata. In
Kinshuk,. Looi, C. K., Sultinen, E., Sampson, D., Aedo, I., Uden, L. & Kähkönen, E. (Eds.), Proceedings of the
4th IEEE International Conference on Advanced Learning Technologies, Los Alamitos, CA: IEEE Computer
Society, 316-320.
Van de Sompel, H., & Beit-Arie, O. (2001). Open Linking in the Scholarly Information Environment Using the
OpenURL
Framework.
D-Lib
Magazine,
7
(3),
retrieved
May
1,
2006
from
http://www.dlib.org/dlib/march01/vandesompel/03vandesompel.html.
Wiederhold, G., Wegner, P. & Ceri, S. (1992). Toward Megaprogramming. Communications of the ACM, 35
(11), 89-99.
67
Weitz, R. R., Wachsmuth, B. & Mirliss, D. (2006). The Tablet PC For Faculty: A Pilot Project. Educational Technology &
Society, 9 (2), 68-83.
The Tablet PC For Faculty: A Pilot Project
Rob R. Weitz
Associate Professor, Department of Computing and Decision Sciences, Stillman School of Business
Seton Hall University, South Orange, NJ 07079, USA
[email protected]
Tel: 1-973-761-9540
Fax: 1-973-761-9217
Bert Wachsmuth
Associate Professor, Department of Mathematics and Computer Science, Seton Hall University
[email protected]
Tel: 1-973-761-9467
Fax: 1-973-275-2366
Danielle Mirliss
Instructional Designer, Teaching, Learning and Technology Center, Seton Hall University
[email protected]
Tel: 1-973-761-6021
ABSTRACT
This paper describes a pilot project with the purpose of evaluating the usefulness of tablet PCs for
university professors. The focus is on the value of tablets primarily with respect to teaching and learning
(and not for research or administrative work). Sixty-four professors, distributed across the various schools
of a university, were provided with tablet PCs and were trained in their use. A survey was distributed to the
participants at the end of the semester. There were 59 respondents, and of these 45 used the tablet in at least
one of their classes. This paper describes the pilot project and the survey results. We observed that a) only a
fraction of faculty are motivated to use tablet technology: roughly a third of faculty expressed an interest in
replacing their notebook computer with a tablet computer and b) generally, participating faculty did indeed
use tablet functionality in their classes and were convinced that this use resulted in a meaningful impact on
teaching and learning.
Keywords
Tablet PC, Instructional technology, Higher education, Electronic whiteboard
Introduction
A tablet PC may be defined as “a type of notebook computer that has an LCD screen on which the user can write
using a special-purpose pen, or stylus. The handwriting is digitized and can be converted to standard text through
handwriting recognition, or it can remain as handwritten text.” (Webopedia 2004). There are two basic versions
of tablet PCs, one that includes a keyboard and one that doesn’t. The keyboard variety doubles as a standard
notebook computer, with the screen swiveling and being laid flat over the keyboard when utilized in tablet mode.
The model without a keyboard, also termed a “slate”, sacrifices functionality for lighter weight and smaller size.
Generally speaking, tablet PCs command prices some several hundred dollars higher than comparably equipped
(in terms of processing power, RAM, hard drive capacity, etc.) standard notebooks. Targeted commercial
markets for tablet PCs include healthcare, insurance, sales force automation, finance and manufacturing/design
(Himmelsbach 2004, Niccolai 2003).
Several recent projects have focused on student usage of tablet PCs. On the elementary school level, Kravcik et
al. (2004) describe the use of a tablet PC in the design of a system enabling “virtual field trips” by elementary
school students. Crossland et al. (2000) provided tablet PCs to students in their distance tutoring system for
mathematics. Cooper and Brna (2001) discuss the use of tablet PCs in their project that “allows children of age
5-6 years to co-construct a multi-frame cartoon to help story-writing with the help of an empathic agent.”
At the higher education level, Scheele et al. (2004) discuss a lecture environment in which students are equipped
with a variety of mobile, wireless devices, including tablet PCs. McFall and Dahm (2004) evaluate a prototype
electronic textbook application running on the tablet PC platform. Yamasaki and Inami (2004) describe a tablet
PC-based system for helping students acquire the skills and knowledge for proper handwriting of Japanese
characters.
68
The use of tablet PCs, principally by students, in higher education was tested during the summer and fall of 2002
by Microsoft and its partners through their “Rapid Adoption Program.” Three universities- MIT, Bentley
College (in Waltham, Massachusetts), and the University of Texas at Austin - were chosen to participate in this
program (Foster 2003, Microsoft 2002, ITRC 2002, Bentley College 2003, Condon 2003).
Though details have not been released, Bentley College’s initial pilot, which ended in May 2003, indicated
positive responses to the use of the tablets and Bentley reported plans to extend their pilot through July 2004.
MIT utilized tablet PCs during its participation in the annual International Design Competition. Tablets
(connected via wireless networking) allowed students to write math equations as well as to sketch, modify and
share their designs easily. This approach was a significant improvement over paper-based or non-tablet
technology.
The participants of the pilot at the University of Texas at Austin were students in The School of Architecture,
who used the tablet in four courses. In one they collected field data using the tablets, and in the others they
reported positive results when note-taking in class and using the tablets with design and drawing software. Users
reported that the tablet’s pen allowed a more natural interaction with the software than that available via standard
computers. (However, though the pen allowed a more natural interaction with the electronic design environment,
students complained that it lacked the pressure sensitivity required of a drawing tool.) Participants also reported
that the tablet allowed greater mobility and creativity. They could use the tablet in almost any setting, enabling
users to spontaneously create designs that can then be refined at a later time.
It should be noted that users have also commented on some perceived shortcomings in tablets. In addition to the
lack of pressure sensitivity mentioned above, these deficiencies include short battery life and excessive weight
(Asay, 2002), the premium cost (Bentley College 2003), and difficulty in seeing the screens when outdoors in
bright light (ITRC 2002).
The aforementioned projects, reasonably enough, have been student-centered, with either explicit or implicit
constructivist philosophies (Duffy and Jonassen 1992, Wilson 1996). However, the functionality of tablet PCs
provide for some potentially valuable opportunities for faculty in their role as educators in a lecture style format.
For example, tablets may be used in the classroom (along with a projector) as an electronic blackboard. Supply
and demand curves, drawings of the human body, musical scores, electronic circuit diagrams, mathematical
expressions (e.g., summation and integral signs), along with text (in any alphabet) may be captured and
displayed. The tablet allows for multiple colors and widths of “ink”, and a virtually unlimited number of pages
over which to write without erasing. Further, class notes created in this way may be saved and posted to the
Web, where students can retrieve them and view them using freely available software (tablets not required).
There are instructional opportunities outside of the classroom as well. For example, tablets have the potential to
be used for electronic grading; assignments may be emailed to faculty or dropped off in digital drop boxes
whereupon faculty may write their comments on the work and return it to the students the same way.
There have been some initial reports of faculty use of tablets in the classroom. Wassgren and Krousgrill (2003)
used a tablet-based system to record their in-class, pen annotated, PowerPoint based lectures for (later) storage
and dissemination via the Web. Condon (2003) summarized their findings as follows: “(1) taking advantage of
the Tablet PC's Journal software, using partially prepared slides and annotating them in class, was the most
effective teaching style; (2) posting the recorded lectures on the web had no effect on attendance; (3) the students
liked the clearer diagrams with colors and highlighting; (4) class preparation took a little longer; (5) setup and
breakdown time for lecture delivery was only 2 to 5 minutes; and (6) few students viewed the online lectures if
they had come to class”. Anderson (2004) describes the use of a similar system. In this application, one faculty
member, responding to the increased functionality said, “Being able to diagram and spontaneously work
examples instead of having to use a pre-scripted PowerPoint slide deck—felt like teaching a real class.”
Logic, and these initial studies indicate that tablet PCs have something to offer faculty in academia. The purpose
of the pilot project described here was to test and evaluate faculty applications of tablet PCs apropos their
contribution to teaching and learning. Put another way, how would real faculty teaching actual classes use
tablets, and how would they evaluate the utility of doing so? The pilot would help us answer such questions as:
¾ How do faculty perceive the value of tablet PCs with regard to teaching and learning?
¾ Are tablet PCs actually useful for teaching and learning?
¾ If so, how valuable are they and in what ways?
¾ What challenges/pitfalls are there in using this technology for teaching and learning?
69
Tablets cost from $200 - $400 more than comparable notebooks (Einhorn 2004). As faculty at Seton Hall
University, where this study was conducted, were already provided with notebook computers, perhaps the
overarching question to be resolved by the study was, do the added benefits of tablet technology indicate that all
or some faculty should move from the notebook platform to the tablet?
Background
Seton Hall University (SHU) is a research and teaching-oriented university with some 10,000 graduate and
undergraduate students. It has had a major focus on instructional technology for almost a decade. In the fall of
1995 SHU instituted a pilot project for evaluating a notebook/mobile computing environment (Weitz 1997); this
pilot project resulted in notebook computers being distributed to all full-time students and faculty, and the
implementation of an infrastructure – technical, administrative and instructional – to support them. In 2003, for
example, more than 2,000 notebooks were distributed to freshmen, transfer students, faculty, and administrators,
and almost 1,000 notebooks of sophomore students were “refreshed”. (Notebooks are replaced with current
models every two years.) The campus has extensive wireless coverage, Internet2 connectivity, active use of
standard course management software, and a center supporting numerous faculty instructional technology
initiatives. SHU has ranked among the top 15 most wired universities in the United States in Yahoo! Internet
Life's survey of technology on college campuses.
The university formed a Teaching, Learning and Technology Roundtable (TLTR) after participating in the
American Association for Higher Education (AAHE) Summer Institute in 1995. A TLTR has been defined as “a
diverse group that meets regularly, considers issues, and (usually) provides advice to the chief academic office,
other leaders, and many constituencies” (TLT Group, n.d.).
The TLTR at SHU consists of the following action teams:
¾ Administrative Computing
¾ Best Practices
¾ Emerging Technologies
¾ Financial Resources Development
¾ Steering Committee
¾ Technology Assessment
The Emerging Technologies team, composed of faculty and administrators, is charged with 1) staying informed
of national trends in technology integration in higher education and 2) communicating its findings to the
university community. The team submits a yearly report with general recommendations and in early 2003 it was
recommended that the university explore the use of tablet PC technology by faculty and, to a lesser extent, by
administration and staff. To this end, a small “pre-pilot” study was approved.
The “Pre-Pilot” Study – Fall 2003
While open to exploiting new technologies for teaching and learning, SHU is, at the same time, naturally
cautious in this type of endeavor. With this in mind, prior to authorizing a full pilot study, in the spring of 2003 a
small (16 participant) “pre-pilot” was begun. Faculty, staff, and administrators were invited to participate in this
small experiment with tablet computing by filling out a simple form and returning it via email. They were asked,
in particular, to complete the following two statements.
¾ I think tablet PCs are more useful than regular laptops because:
¾ I would like to test my tablet PC for the following purpose(s):
Fifty-eight valid applications were received from all areas of the university. A subcommittee of the Emerging
Technology group met to evaluate the ideas presented and to award machines to the most promising applications.
Generally speaking the subcommittee was looking for applicants who were best able to describe the advantages
of a tablet PC over a regular laptop, and who had the most interesting suggestions for its use. The kind of things
looked upon favorably by the committee included:
¾ Demonstration of real pedagogical payoff; for example, the need to compose formulas, write in foreign
alphabets, draw diagrams, etc.
¾ Indication that the faculty member had thought about electronic storage and retrieval of tablet-created
documents.
70
¾
Ideas more significant than mundane usage; e.g., use of the tablet for note-taking at meetings or a
preference for a lighter-weight platform than the current notebook.
The faculty responses are available at http://www.cs.shu.edu/tabletpc/2003-ideas.html. They have been
summarized by Condon (2003) as follows:
¾
¾
¾
¾
¾
¾
¾
¾
Several faculty proposed using the tablets in place of writing on the chalkboard or on overheads. One
professor who alternates between using a laptop and the chalkboard in lectures has had a problem
getting chalk dust all over the computer keyboard. He hopes switching to a Tablet PC might allow him
to eliminate the use of the chalkboard altogether.
An English professor envisioned projecting a literary passage on the screen and being able to make
notes on the printed material.
Several professors who teach finance, statistics, and physics look forward to being able to write
equations quickly during lectures without resorting to the chalkboard and easily answer e-mailed
questions with appropriate equations.
A professor of clinical medicine was enthusiastic about teaching anatomy and physiology by annotating
anatomical slides.
A professor who will be teaching freshmen to use Excel and Access as management information
systems would like to be able to project the operating software while answering student questions by
drawing on the screen.
The biology department plans to issue the Tablet PC to a student who will compare the use of the Tablet
PC to her standard laboratory notebook to see which is more efficient as a way to record research.
One physics professor sees the Tablet PC as a way to seamlessly incorporate Excel, Java, web
animations, and Interactive Physics simulations in his lectures.
A professor who teaches computer graphics normally draws sketches on the chalkboard and then
teaches how to implement the design in the software. If he could draw the sketch on the Tablet PC, he
could make the sketch electronically available to his students outside of class.
Ideally the committee hoped for a distribution across schools and departments, with some administrative use.
Fortunately the applications broke down naturally along these lines.
Sixteen applicants were approved that represented a distribution across schools and departments, with some
administrative use. The sixteen participants self-rated their experience-comfort level with computers as follows:
Novice/Beginner: 2, Intermediate: 6 and Advanced/Expert: 8.
Participants responded to detailed surveys prior to the start of the semester and at its end. The surveys indicated
that the tablets were used productively, both inside and outside the classroom, resulting in a positive impact on
teaching and learning. A working paper detailing the pre-pilot and its evaluation are available from the authors.
With the encouraging results of the pre-pilot in mind a full pilot program was initiated for the fall of 2004. Given
financial and technical-support constraints, the pilot would be limited to about 60 participants.
The Pilot Study – Fall 2004
As stated previously, SHU provides all faculty and undergraduate students with notebook computers and these
computers are replaced with new models every two years. In the spring of 2004 faculty eligible for notebook
replacement were asked to express one of the following preferences: they could 1) exchange their current
notebook computer for the new model notebook, 2) keep their current notebook for another year, or 3) apply for
a tablet computer. If the number of faculty requesting tablets exceeded the maximum, about 60, an application
procedure would be established as it had been for the pre-pilot.
The specifications of the two machines were made available to the faculty, both via TLTR events and via the
Web. Aside from tablet functionality, the essential differences were that the tablet had a smaller screen than the
notebook, and an external CD/DVD-RW drive as opposed to an internal one for the notebook.
Faculty responses are provided in table 1.
71
Table 1: Faculty Preferences Regarding Notebook Computer Replacement
Option
Number of Faculty
Percentage of Faculty
Replace with the new model notebook
113
51.36%
Keep current notebook for an extra year
43
19.55%
Apply for a tablet PC
64
29.09%
Total
220
This data in itself is of interest. First, approximately 20% preferred keeping their two year-old computer as
opposed to upgrading. We surmise that the effort of transferring files and settings (even with the aid of technical
staff) is not worth the bother to these individuals. Further, it seems that (only) approximately 30% of faculty
eligible for hardware replacement felt that a tablet computer would be a better choice than a standard notebook,
or a sufficiently better choice to warrant the effort of learning how to use the technology. This figure provides a
measure of the perception faculty have of the benefits of tablet technology. We shall return to this data later in
the results section. In any case, as all faculty interested in a tablet could be accommodated, no further application
process for the pilot was required.
Pilot Participants
The pilot, with 64 faculty members participating, took place during the fall semester of 2004. A survey was
distributed at the end of the semester to participating faculty, and 59 responses were obtained. Of the 59
respondents, 45 utilized the special functionality of their tablet for teaching and learning in at least one of their
classes.
Table 2 provides the distribution across disciplines of respondents to the survey.
Table 2: Distribution of Tablet PCs for Pilot
School
Arts and Sciences
Business
Education
Nursing
Graduate Medical Education
Theology
Library
Total
All Survey Respondents
35
7
5
6
3
1
2
59
Respondents Who Used Tablet
Functionality for Teaching/
Learning
25
5
5
5
3
1
1
45
Applicants were asked to self-rate themselves in terms of their experience-comfort level with computers. Table 3
summarizes the responses.
Table 3: Participant’s Self-Rated Experience/Comfort Level with Computers
Respondents Who Used Tablet
Experience/Comfort
All Respondents
Percent
Functionality for Teaching/
Level With
Learning in At Least One Class
Computers
Novice/Beginner
8
13.6%
5
Intermediate
30
50.8%
20
Advanced/Expert
21
35.6%
20
Total
59
100.0%
45
Percent
11.1%
44.4%
44.4%
100.0%
We see no particular pattern among faculty who did not use tablet functionality in each least one course during
the semester. The reasons faculty gave for non-use are compiled below, in table 4.
72
Table 4: Reasons Tablet Functionality Not Used For Teaching and Learning
Reason
Number of Faculty
Sabbatical/Not Teaching
4
Received Tablet Too Late
2
Technical Problems
1
Just Didn’t Use
6
Was a Mistake to Get Tablet
1
Total
14
We see that four faculty simply were not teaching during the semester, and two received the tablet “too late” to
incorporate it into their teaching. The individual with technical problems indicated that the tablet was “out for
repairs” for most of the semester. This leaves seven who didn’t use tablet functionality other than for some
exogenous factor. The six faculty who “just didn’t use” the tablet functionality indicated that due to competing
work and time pressures they never got around to learning how to use the tablet or how to integrate it into their
course(s). The last individual simply indicated that, in retrospect he didn’t need or want tablet functionality.
For the remainder of this paper, except where noted otherwise, we utilize data only from the 45 respondents who
utilized tablet functionality in the service of teaching and learning in at least one of their classes.
In-Class Use of Tablets
This section discusses survey results related to faculty’s in-class use of the tablets.
Survey Data
Table 5 summarizes faculty responses with respect to the frequency of use of tablet applications. Table 6 focuses
on the respondent’s assessment of the impact on teaching and learning of each application used. We recognize
that these are different measures (and, in particular, that a low frequency of usage of a particular application does
not necessarily mean a low impact of that application).
The first column of table 5 provides a range of in-class tablet applications. The next five columns indicate
faculty responses regarding the frequency with which they used each application. So, for example, nine faculty
(20% of respondents) indicated that they used their tablet for writing mathematics in every or almost every class.
Three (6.67%) frequently did so. The “total” column indicates the number of respondents for each application.
Table 5: Frequency of use of tablet functionality: in-class applications
For this course, please indicate below how frequently you used the unique functionality of the tablet PC (e.g.
handwriting/drawing) in class as part of the lecture (e.g. in conjunction with a projector) for each of the
applications listed.
Always/Almost
Never/Not
In-class Applications
Always
Frequently Occasionally Applicable Total
9
3
8
25
45
Writing Mathematics
20.00%
6.67%
17.78%
55.56%
Writing non-English text; e.g.,
2
0
2
41
45
Hebrew, Russian, Greek (in a non4.44%
0.00%
4.44%
91.11%
mathematical context)
8
4
19
14
45
Drawing diagrams, charts, and/or
graphs
17.78%
8.89%
42.22%
31.11%
Correcting assignments/ projects
6
6
10
23
45
(e.g. writing on documents during
13.33%
13.33%
22.22%
51.11%
in-class demonstration)
Using the handwriting functionality
and a new (blank) Windows Journal
or Word file to create lecture notes.
(That is, using the tablet in-class as
11
7
10
15
25.58%
16.28%
23.26%
34.88%
43
73
an electronic whiteboard.)
to draw or write over prepared
(already has content) Windows
Journal, PowerPoint, Word or Excel
files.
Electronically distributing in-class,
tablet-created handouts/lecture
notes to students.
8
7
16
14
45
17.78%
15.56%
35.56%
31.11%
9
5
7
23
20.45%
11.36%
15.91%
52.27%
44
(Note that the “always” and “almost always categories”, and the “never” and “not applicable” categories were
separate responses in the questionnaires. They have been collapsed into single categories here for clarity.)
Respondents’ impressions of the impact of their use of each application are summarized in table 6. (Note that the
categories in the first column have been abbreviated in the interest of saving space.) Ten faculty felt that their
use of the tablet for writing mathematics had a “very positive” impact on teaching and learning. Percentages in
this table are based on faculty who used the application at least occasionally. So, for example, these ten faculty
make up 50.00% of all (20) respondents who used their tablet for writing mathematics in class at least
occasionally. If faculty did not use an application at all, they were (of course) not able to estimate the impact of
their use of that application; these responses are captured in the second-to-last column.
Table 6: Perceived impact of tablet functionality: in-class applications
(Percentages based on respondents who used the application at least occasionally.)
Now please consider the in-class applications listed above *that you have used at least occasionally*. Please
rate below the impact you feel YOUR use of that application has had on teaching and learning in this class.
neutral/no
don't
did not use
In-class applications
very
added
very
know/no
at least
value
positive positive
negative negative opinion occasionally total
10
8
1
0
0
1
25
45
Writing Mathematics
50.00% 40.00%
5.00%
0.00%
0.00%
5.00%
Writing non-English
3
0
0
0
0
1
41
45
text
75.00%
0.00%
0.00%
0.00%
0.00%
25.00%
Drawing diagrams,
charts, and/or graphs
Correcting
assign’ts/projects
Using the
handwriting
functionality to
create lecture notes.
Using the
handwriting
functionality to draw
or write over
prepared content.
Electronically
distributing in-class,
tablet-created
handouts/lecture
notes to students.
11
17
1
1
0
0
14
44
36.67%
7
31.82%
56.67%
6
27.27%
3.33%
5
22.73%
3.33%
0
0.00%
0.00%
0
0.00%
0.00%
4
18.18%
23
45
10
13
2
0
0
3
15
43
35.71%
46.43%
7.14%
0.00%
0.00%
10.71%
12
11
6
0
0
2
14
45
38.71%
35.48%
19.35%
0.00%
0.00%
6.45%
9
5
4
0
0
3
23
44
42.86%
23.81%
19.05%
0.00%
0.00%
14.29%
Discussion
Figure 1 provides a summary of the usage of in-class applications and the perceived impact of that usage. We
see, for example, that 45 faculty responded to the question regarding the usage of the tablet for writing
74
mathematics in the classroom. Twenty of these respondents indicated that they used this application at least
occasionally. Eighteen of these 20 rated the impact of their use of this application as positive or very positive.
50
40
30
20
10
0
Total Respondents
Distributing
Lecture Notes
Whiteboard Create Lecture
Notes
Whiteboard Writing Over
Documents
Correcting
Assignments
Drawing
Diagrams/Charts
Writing NonEnglish
Frequency: Used at Least
Occasionally
Writing Math
Number of Responses
Summary of Tablet In-Class Usage and Perceived
Impact By Application
Impact: Positive or Very
Positive
Application
Figure 1: Summary of in-class results
The number of faculty who used an application on at least an occasional basis provides a snapshot of the extent
of diffusion of that application. We see that approximately two-thirds of the responding faculty used their tablets
as a whiteboard – creating notes from scratch (65.12%) and writing over materials prepared before class
(68.89%). Not surprisingly, faculty used the tablets in domains where the drawing functionality is most useful:
writing mathematics (44.44%) and drawing diagrams, charts and graphs (68.89%). Almost half of respondents
(48.89%) used the tablets to correct assignments/projects as an in-class demonstration and almost half (47.73%)
electronically distributed in-class, tablet-created materials to their students. Only four faculty used the tablet for
writing non-English characters, though this small number reflects the few participants in the study likely to need
this functionality.
Faculty felt positively about the impact on teaching and learning of the applications they used. Approximately
90% felt that their use of a tablet for writing mathematics and drawing diagrams, charts and/or graphs had a
positive or very positive impact. Some 80% responded that the impact of using the tablet as a whiteboard
without prepared materials, was positive or very positive. Approximately 75% thought using the tablet to write
over prepared material had a positive or very positive impact. Over 60% felt that using the tablet to correct
assignments/projects as part of in-class demonstrations had a positive or very positive impact. Likewise over
60% of faculty who distributing the notes they created in-class felt that doing so had a positive or very positive
impact. Three of the four faculty who used their tablet to write non-English text felt that this application had a
positive or very positive impact.
Out-of-Class Use of Tablets
This section discusses survey results related to faculty’s use of the tablets outside of the classroom.
Survey Data
Again, we separately analyzed frequency of use and perceived impact data. Table 7 summarizes faculty
responses with respect to the frequency of use of tablet applications outside of the classroom. Table 8 focuses on
the respondent’s assessment of the impact on teaching and learning of each of these applications.
75
The first column of table 7 specifies the outside-of-classroom applications. The next five columns indicate
faculty responses regarding the frequency with which they used each application.
So, for example, eight faculty (17.78% of respondents) indicated that they used their tablet for grading
assignments and/or projects for every or almost every class. Five (11.11%) frequently did so. The “total” column
indicates the number of respondents for each application.
Table 7: Frequency of use of tablet functionality: outside-of-class applications
always/almost
never/not
Outside-of-class applications
always
frequently
occasionally
applicable
8
5
13
19
of your tablet to grade
17.78%
11.11%
28.89%
42.22%
assignments/projects.
of your tablet to prepare
handouts/lecture notes before class.
Electronically distributing
handouts/lecture notes (created
outside of class) to students.
5
7
13
20
11.11%
15.56%
28.89%
44.44%
8
4
12
21
17.78%
8.89%
26.67%
46.67%
45
45
45
total
Table 8: Perceived impact of tablet functionality: outside-of-class applications
(Percentages based on respondents who used the application at least occasionally.)
did not use
Outside-ofvery
positive neutral/no negative
very
don't
at least
class
positive
added value
negative
know/
occasionally
applications
no
opinion
Using
8
12
2
2
0
2
19
handwriting
30.77% 46.15%
7.69%
7.69%
0.00%
7.69%
functionality
to grade
Using
9
10
2
0
0
4
20
handwriting
36.00% 40.00%
8.00%
0.00%
0.00%
16.00%
functionality
to prepare
handouts/
lecture notes
Electronically
9
10
2
0
0
3
21
distributing
37.50% 41.67%
8.33%
0.00%
0.00%
12.50%
handouts/
lecture notes
total
45
45
45
Discussion
Figure 2 provides a summary of the usage of applications outside the classroom and the perceived impact of that
usage. Forty-five faculty responded to each question. We see, for example, that 26 indicated that they used the
handwriting functionality of their tablet for grading assignments/projects outside of class; of these 26, 20 rated
the impact of their use of this application as positive or very positive.
This data is fairly consistent across applications: a little more than half of the respondents used each outside-ofclass application, and about three-quarters of those who used an application felt that that usage resulted in a
positive or very positive impact.
76
50
Total Respondents
40
30
Frequency: Used at Least
Occasionally
20
10
Impact: Positive or Very
Positive
Electronically
Distributing
Handouts/Lecture
Notes
Preparing
Handouts/Lecture
Notes
0
Grading
Assignments/
Projects
Number of Responses
Summary of Tablet Outside-of-Class Usage and
Perceived Impact
Application
Figure 2: Summary of outside-of-class results
Qualitative Responses
We also polled participating faculty with open-ended questions. When asked to, “Indicate what seems to be the
most valuable attributes of the tablet for teaching and learning in this course,” faculty responded as follows.
¾ Ability to write and save lecture notes in class, especially involving graphs and equations
¾ Ability to write on my prepared documents, especially PowerPoint slides! (Gave students a much better
sense of what I was describing.)
¾ Because I teach courses about students with special needs, the entire instrument serves as a model for
what can be done to enhance opportunities for learning.
¾ Being able to create lecture notes inside of class and being able to distribute them electronically
¾ Being able to write over prepared lecture tools like PowerPoint.
¾ Distributing solutions to problem sets and lecture notes.
¾ Drawing diagrams.
¾ Easier to grade papers than track changes [using standard MS Word] which sometimes does not leave
space for me to type in the corrections or comments I want to make
¾ Flexibility; replaces chalk and allows an ongoing discussion without the usual chalkboard distractions.
¾ For enhancing lecture notes during class.
¾ Grading papers -- the light weight of the tablet and my ability to download papers into Word and grade
using ink annotation lightens my load, literally and figuratively. Hand writing my comments (including
drawing circles, arrows, etc.) makes my grading go faster.
¾ Hand grading papers.
¾ Hand writing for non-English.
¾ Hand writing for grading.
¾ I can truly write equations in little time. Easy to send to Blackboard and student can pay more attention
in class.
¾ I used the handwriting/annotating feature frequently in lectures using PowerPoint. However, this often
crashed, which diminished its usefulness. Still, I found the ability to annotate very helpful and the
students also seemed to feel it was helpful.
¾ If it worked well all the time - I would like the writing ability on my PowerPoint presentations and the
ability to compose new slides - technical problems have made this difficult.
¾ It is great not to have to turn away from my students to write, and it’s nice to have the many color and
pen options on digital ink, which I’ve used to create helpful diagrams (without carrying a bag of colored
markers to class).
¾ It is light and facilitates instruction.
77
¾
¾
¾
¾
¾
¾
¾
¾
¾
¾
¾
¾
¾
¾
¾
¾
[I] liked the handwriting component.
Posting in-class lecture on line.
Teaching statistics -- writing on the tablet instead of on the whiteboard enables me to face the students
while teaching. Also, it’s quicker as there’s no need to erase. I just pull up a new page. Another benefit
is that I’m not blocking what the students are looking at.
The ability to draw graphs freehand and to add content to prepared materials will be a useful attribute. I
also plan to make digital recordings of lecture material for in-class and online deliver for next
semester’s classes.
The ability to write and edit papers while on line.
The portability, added features of handwriting to emphasize points in the moment.
The use of the tablet is invaluable for grading papers and providing timely feedback to students.
This is a great tool, to be able to write mathematics on the computer, save it and send it to my students.
Using Microsoft Equation [Editor] or any of the other software used to type mathematics, is very time
consuming and not something I would use on a daily basis. Being able to write solutions to problems
using the Tablet PC has enhanced the way I can use the computer in the classroom.
Using it as a white-board.
Using note pad
Visual learners benefit by seeing instructor create real-time (diagrams, charts, etc.)
Writing and displaying Greek and Hebrew and distributing transcripts of class work.
Writing and displaying that writing directly while I FACE the class
Writing on my already prepped PowerPoints so that I don’t have to futz with the projector-in-front-ofthe-whiteboard.
Writing on PowerPoints.
Writing on the screen - I can get back to my chalkboard!!
It appears that faculty have used tablet functionality as expected, and generally have a positive response.
We asked, “What difficulties, if any, did you experience in using the tablet?” Responses are summarized below
in table 9.
Table 9: Difficulties Experienced by Participants with the Tablet
Difficulty Experienced
No. Respondents
System hangs/need to reboot
Oversensitive mouse -- cursor jumping associated with mouse/touchpad
Problems with the external DVD drive
Problems with wireless connectivity
Screen orientation
Don’t like external DVD drive
System instability/crashing when annotating in PowerPoint
Battery life
Problems with network connectivity
Problem with USB port
Insufficient hard drive capacity
11
7
7
3
2
2
2
1
1
1
1
The difficulties with the external DVD drive appear to be specific to this particular model. The screen orientation
difficulty refers to the fact that the image projected on the LCD projectors in the classroom was, by default,
inverted; this meant that the instructor had to rotate the image on the tablet 180 degrees – not a difficult process,
but one that requires several steps – and turn the tablet upside down. We don’t believe the connectivity issues are
distinct from those found using the standard notebooks at SHU. On the other hand, the problems associated with
the cursor skipping, and with the system hanging/crashing are in our view more troubling. Certainly
encountering these problems during a lecture can be harmful to continuity and focus.
We were interested in overall issues relating to the performance of the tablet. We asked the participants to rate
the hardware and technical aspects of the tablet along the dimensions outlined in table 10 below.
78
Table 10: Faculty Satisfaction with Hardware and Technical Aspects of the Tablet
very satisfied
unsatisfied or
not
neutral
or satisfied
very unsatisfied
applicable
27
11
7
0
memory/hard drive/ processing speed
60.00%
24.44%
15.56%
0.00%
35
3
7
0
size of keyboard
77.78%
6.67%
15.56%
0.00%
34
2
9
0
size of monitor
75.56%
4.44%
20.00%
0.00%
7
6
28
4
external DVD drive
15.56%
13.33%
62.22%
8.89%
41
4
0
0
weight/portability
91.11%
8.89%
0.00%
0.00%
33
7
4
1
wireless access
73.33%
15.56%
8.89%
2.22%
31
6
7
1
battery life
68.89%
13.33%
15.56%
2.22%
5
7
0
33
speech recognition
11.11%
15.56%
0.00%
73.33%
34
7
3
0
digital ink functionality (handwriting)
77.27%
15.91%
6.82%
0.00%
23
10
6
6
converting handwriting to text
51.11%
22.22%
13.33%
13.33%
total
45
45
45
45
45
45
45
45
44
45
Dissatisfaction with the external drive shows up here too, and in fact seems to be the only attribute of the tablet
about which there was significant unhappiness. Some 90% of respondents seemed pleased with the
weight/portability characteristics of the tablet. We suspect this is partly a function of the fact that the standard
notebook provided faculty is at the heavy end of the portable computer spectrum. Roughly 75% of respondents
were satisfied or very satisfied with the size of the keyboard, size of the monitor, wireless access and digital ink
functionality. Converting handwriting to text seems to get a more mixed picture with 23 respondents (roughly
50%) satisfied or very satisfied, and six unsatisfied or very unsatisfied, and six who didn’t use this functionality.
Only a few faculty used the speech recognition capability.
Overall, are you satisfied with your Tablet PC or would
you prefer to switch to the "regular" notebook if that
were possible?
No. Responses
60
50
All Respondents
40
30
Respondents Who Used
Tablet for Teaching and
Learning
20
10
0
satisfied does not switch missing
with tablet matter to back to
PC
me
notebook
Figure 3: Tablet vs. Notebook Preferences
Our view is that these results are very positive. This is reinforced by the results outlined in figure 3; it appears
that participants overwhelming preferred their tablet to a standard note book.
79
We see, for example, that some 85% of those who used tablet computing in at least one of their classes would
prefer to keep their tablet as opposed to returning to a standard notebook. It is worth noting that three
respondents of those who actually used tablet technology preferred to switch back to a notebook. These
individuals cited the technical problems noted earlier: failure of the external drive, repeated occurrences of the
system hanging, and network difficulties. With regard to all respondents, some 80% would stay with a tablet and
seven would prefer to go back to the standard notebook. Of these additional four who preferred a standard
notebook, one was the respondent whose tablet was “out for repair”, another indicated that he never used the
machine and that “it was a mistake” for him to request a tablet, the third simply didn’t use it and the fourth was a
novice user on sabbatical who got frustrated by problems with the external drive and a faulty printer cable.
We note that tablet PCs are more expensive than conventional notebook computers. Figure 4 below reproduces
the question we posed to the participants that brings this fact into decision-making, and provides a summary of
the responses.
A tablet pc costs about $300 more than a comparable
laptop. Do you think the University should continue to
provide a tablet pc option for faculty?
definitely yes
probably yes
no opinion
probably not
definitely not
not applicable
no response
Figure 4: Preferences Regarding the Tablet Option Given the Additional Cost
Approximately 85% (38 out of 45) responded that SHU should definitely or probably continue to provide a
tablet PC option to faculty. Five said the university probably or definitely should not, and two expressed no
opinion.
The results in figure 5 highlight perhaps the most important issue here: the value, as perceived by faculty who
used their tablet in at least one course, of this technology vis-à-vis teaching and learning.
Overall, how valuable was the tablet in terms of
teaching and learning?
very positive
somewhat positive
no measurable value
somewhat detrimental
very detrimental
don't know/no opinion
not applicable
Figure 5: Value of tablet in terms of teaching and learning
80
Forty of the 45 faculty (approximately 90%) who used the tablet in at least one course responded that their use
had a very or somewhat positive value with regard to teaching and learning. Two indicated that they observed no
measurable impact, one indicated a “somewhat detrimental” effect. This individual cited system crashes and
difficulties with cursor jumping associated with the mouse/touchpad.
Results
We feel that the results of this study may be summarized as follows.
There are tablet applications that will be used by some faculty with good results
Some faculty did use tablet functionality and their perception of the utility of that use vis-à-vis teaching and
learning was positive. Aside from writing non-English characters, the applications considered here were used at
least occasionally by 50% - 75% of respondents; further, on average 75% of these faculty rated the impact of this
use as positive or very positive. The percentage of faculty who used the applications considered here at least
frequently (as opposed to at least occasionally), ranged from 25% - 40%.
The tablet applications actually used by faculty are those predicted.
We see a good correspondence between the expectations of faculty and their actual usage of tablet applications.
Put another way, generally faculty were not disappointed; they were able to do what they planned on doing with
the tablets.
Negative feelings seem to be most strongly associated with technical problems.
The strongly negative feelings regarding the tablet felt by a few of the participating faculty seemed to result from
technical problems they experienced. In particular difficulties with the external DVD/CD drive, system
instability and cursor jumping were noted. Our feeling is that these are surmountable difficulties, with better
support and training, and are not particular associated with tablet technology.
Faculty who use tablets are generally pleased with the results and strongly prefer to continue using
tablets.
We measured faculty satisfaction with tablet computing via several questions, including one that explicitly took
into account (as much as was possible) the added cost of tablet technology. Faculty who used tablets were
favorably disposed to their usage, felt that the impact of that usage was positive, wanted to continue using tablet
technology, and felt all faculty at the university should have a tablet option.
The majority of faculty are not “demanding” tablets
We see that (only) 30% of the 220 faculty members at SHU eligible for notebook replacement in the fall of 2004
opted to try a tablet. Data for the fall of 2005 reflect an almost identical breakdown, with again about 30% of
faculty selecting the tablet option. Of the 64 who received a tablet in the fall of 2004, 59 replied to our survey,
and (only) 45 actually used their tablet in at least one class. Using this last figure, it seems that approximately
one-quarter to one-third of faculty perceive tablets to be something beneficial for their usage.
Conclusions
Despite well-intentioned efforts to limit class sizes at many colleges, the lecture remains in place
as the primary teaching format. The economics of higher education are such that few institutions
can afford to discard lecture courses altogether, however much a small seminar is preferred by
students and faculty members alike. With only one paid teacher and a large roomful of tuitionpaying students, how can you beat the numbers?
In my view, this is not necessarily a bad thing.
Jay Parini
“The Well-Tempered Lecturer”
The Chronicle of Higher Education
Volume 50, Issue 19, Page B5
January 16, 2004
We believe the above results indicate that tablet technology for a faculty should be pursued for those faculties
with an interest in an application for which the tablet is well-suited. In particular, it seems that those fields that
81
require freestyle drawing of diagrams, pictures and charts, as well as fields requiring the use of mathematical
symbols are good areas for the use of tablets. These fields include many areas in the sciences and the social
sciences, including, for example, physics, biology, chemistry, psychology and economics.
We believe that the results are consistent and provide reliable pointers towards the usefulness of tablet
computing for faculty at the university level. In academia as elsewhere, there are always technology leaders and
followers. The ultimate goal here is not to force all faculty members to use a technology, but rather to provide
individual faculty with the tools they believe will assist them in their educational mission.
With the results of this study in mind, Seton Hall University is pursuing tablet technology, both continuing the
option for faculty (as mentioned previously) and equipping students in our honor programs with tablets, in the
fall semester of 2005.
References
Anderson, R. (2004). Beyond PowerPoint: Building a New Classroom Presenter. Syllabus, Retrieved November
22, 2005 from http://www.syllabus.com/article.asp?id=9537.
Asay, P. (2002). The Killer App for Higher Education, Syllabus, Retrieved November 22, 2005 from
http://www.syllabus.com/article.asp?id=6246.
Bentley College (2003). Will Tablet PCs replace laptops on college campuses? Bentley College news release,
February
10,
2003.
Retrieved
November
22,
2005
from
http://www.bentley.edu/newsevents/pr_view.cfm?id=910
Condon, R. (2003). Table PCs in Education, National Institute for Technology and Liberal Education, Retrieved
November 22, 2005 from http://www.nitle.org/rsrc_issues_tabletpc.php
Cooper, B., & Brna, P. (2001). Fostering cartoon-style creativity with sensitive agent support in tomorrow's
classroom. Educational Technology & Society, 4 (2), 32-40.
Crossland, C. L, Monteith, L. K., Miller, T. K., Brock, L. B., Cuejilo, R., Wheeler, M. A., Viersen, R., &
Zielinski, L. (2000). The FORESEE Project: Connecting Communities to Create Competence. Educational
Technology & Society, 3 (3), 267-282.
Duffy, T. M., & Jonassen, D. H. (Eds.) (1992). Constructivism and The Technology of Instruction, Lawrence
Erlbaum Associates, Hillsdale, NJ, USA.
Einhorn, B., & Green, J. (2004) Headaches For Tablet PCs. BusinessWeek, March 1, 14.
Foster, A. L. (2003). Tablets Sneak Up on Laptops: The Chronicle of Higher Education, 49 (30), A33.
Himmelsbach, V. (2004). Nurturing a fledgling market. Computer Dealer News, 20 (5), Retrieved November 22,
2005 from http://www.looksmartcompanies.com/p/articles/mi_m3563/is_5_20/ai_n6129772.
ITRC (2002). Microsoft Tablet PC Rapid Adoption Project at UT Austin, University of Texas at Austin,
Information Technology Resource Central (ITRC), November 2002. Retrieved November 22, 2005 from
http://www.utexas.edu/computer/tabletpc-rap2002/.
Kravcik, M., Kaibel, A., Specht, M., & Terrenghi, L. (2004). Mobile Collector for Field Trips. Educational
McFall, R., & Dahm, E. (2004). Evaluation of a Prototype of an Electronic Textbook Application. In
Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004,
1530-1535.
Microsoft (2002.) Massachusetts Institute of Technology Changes the Face of Education Using Tablet PCs,
Retrieved
November
22,
2005
from
http://www.microsoft.com/resources/casestudies/
CaseStudy.asp?CaseStudyID=13513.
82
Niccolai, J. (2003). ViewSonic, Toshiba Refresh Tablet PC Lines. ComputerWorld, November 19, 2003,
Retrieved
November
22,
2005
from
http://www.computerworld.com/mobiletopics/mobile/story/
0,10801,87329,00.html.
Scheele, N., Seitz, C., Effelsberg, W., & Wessels, A. (2004). Mobile Devices in Interactive Lectures. In
Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004, 154161.
TLT Group (n.d.). Definitions and Characteristics of a TLT Roundtable, Retrieved November 22, 2005 from
http://www.tltgroup.org/programs/TLTR/Characteristics.htm.
Wassgren, C., & Krousgrill, C. M. (2003). Pen-based PCs for the classroom, Electronic file, April 3, 2003,
Retrieved November 22, 2005 from http://widget.ecn.purdue.edu/~wassgren/2003_04_03_TLTWorkshop/.
Webopedia Computer Dictionary (2004).
http://webopedia.com/TERM/t/tablet_PC.html.
tablet
PC,
Retrieved
November
22,
2005
from
Weitz, R. (1997). Lessons from Notebook/Mobile Computing. Information and Management, 32, 103-111.
Wilson, B. G. (1996). Constructivist Learning Environments, Englewood Cliffs, NJ, USA: Educational
Technology Publications.
Yamasaki, T., & Inami, N. (2004). Training System for Fair Handwriting of Japanese Characters Based on OnLine Character Recognition. In Proceedings of World Conference on Educational Multimedia, Hypermedia and
Telecommunications 2004, 1692-1698.
83
Akpinar, Y. & Bal, V. (2006). Teachers’ Collaborative Task Authoring to Help Students Learn a Science Unit.
Teachers’ Collaborative Task Authoring to Help Students Learn a Science
Unit
Yavuz Akpinar and Volkan Bal
Boğaziçi University, Faculty of Education, Bebek, Istanbul, Turkey
Tel: (90) 212 3594497
Fax: (90) 212 2575036
[email protected]
[email protected]
ABSTRACT
Currently available courseware packages for teaching the work unit may not meet different students’ needs.
Also, a single teacher, even with tools, is likely to have difficulties and may need cooperation of other
teachers in dealing with students’ problems in the learning unit. This research aims to devise a set of
computer based tools to meet the diverse needs of learners to comprehend a science learning unit. A model
of computer based tools on the learning unit for developing procedural knowledge to solve work problems
was developed together with a set of teacher customization and collaboration tools. The main components,
developed and implemented in an integrated manner for both students and teachers, are Student Activity
Environment, Curriculum Authoring Center, Global Activity Center and Teacher Collaboration Tools. The
framework of supporting students through teachers’ collaborative course authoring, considering the
different backgrounds of the students and preferred teaching/learning style of teachers/students, was
evaluated with students using two different task regimes. The evaluation studies presented encouraging and
promising results.
Keywords
Teacher collaboration, Collaborative task authoring, Learning environment design, Task regime, Work
Introduction
Science education should teach students to critically evaluate new information in order to interpret and test their
hypotheses. Students have to make correspondences between data related to the context and a formal
representation of scientific phenomena. They have to organize data and contexts through relationships e.g. if an
object travels faster or slower what kind of change occurs in time? What about the force applied to the object?
Reasoning and deduction about situations are needed. That requires students to grasp the operational definition
of prior concepts and to correctly superimpose existing knowledge about a physical phenomenon on its formal
representation.
Students arrive in the science classroom with preconceptions that are often contradictory to accepted science
thinking. These naive theories may lead to misconceptions and thus may interfere with accepted concept
development (Welmar, 1996). Misconceptions are troubling issues for students and teachers in science. This is
especially true in school science and physics due to its often abstract nature. Students may have two distinct
perspectives of science: One is reserved for the formal learning setting in the classroom while the other is used
outside that setting in everyday life (Cadmus, 1990; Tsai & Chou, 2002). For example, students’ misconceptions
in a work learning unit, which this study focuses, are identified as follows (Amasci, 2004): (1) Failing to identify
the direction in which a force is acting; (2) Believing that any force times any distance is work; (3) Believing
that machines put out more work than we put in it: Not realizing that machines simply transform the form of the
work we do; (4) Believing that the mass effects the work done under every condition.
Effective science instruction helps children distinguish between errors and misconceptions, and attempts to
confront learners with situations such as classroom discussion or computer-based visualization in which
everyday life explanations aren’t successful and offers them better explanations from science. Visualization of
scientific phenomena and laboratory experiences have been important components of the reinforcing and
understanding science concepts: Visualizing phenomena through demonstrations, simulations, models, real-time
graphs, and video can contribute to students' understanding by attaching mental images to these concepts
(Escalade et al, 1996). These visualization techniques not only allow students to observe how objects behave and
interact, but also provide students with visual associations that they may capture, and preserve the essence of
physical phenomena more effectively than do verbal descriptions (Cadmus, 1990).
Computer simulations can help students to understand invisible conceptual worlds of science through animation,
which can lead to more abstract understanding of scientific concepts (Hwang & Esquembre, 2003). Quantitative
84
data can be manipulated and visualized to help students form a qualitative mental picture. Such complex
experiences can help students to identify patterns within simulations, and formulate explanations for phenomena
in terms of models and theories. Simulations must not only allow learners to construct and manipulate screen
“objects” to explore underlying concepts, but they must also provide learners with the observation and
manipulation tools necessary to explore and to test hypotheses in the simulated world (Jonassen, 1996).
These support measures typically guide science learners to (1) focus on particular variables of the underlying
model, (2) generate hypotheses about relationships between these variables, (3) conduct simulated experiments
to test the hypotheses, and (4) evaluate the hypotheses in the light of the observed results. These tasks are
demanding but Chandler (2004) points out that interactive activities in science will be useful if they are
specifically related to the learning unit and if the knowledge base of the learner is also taken into account.
Teachers may customize and relate interactive computer based activities for the learner by considering the
knowledge base of the learner.
Computer Based Activities and Science Teachers
A study focusing on students’ learning problems, should also consider the teachers’ role in overcoming them.
When teachers are able to design and alter applications, they will then be better able to address learning
problems. According to Dunlop et al (2000; p.15) teachers need (1) to derive information, monitor the use, and
utilize the special resources that computer network technology makes available; (2) to have quick and easy
access to the information salient to their planning; (3) notification of salient information and feedback about
student progress and problems as well as teaching changes, and (4) ways of managing computer group work that
allow for a better understanding of the actions of computer-mediated groups and their instructors and a better
ability for teachers to make adjustments. Moreover, planning for remote collaborative work needs to be more
flexible and relaxed to allow for instructional differences.
Computerized learning environments should not be built only by expert programmers. It is necessary to continue
developing new types of cognitive environment tools, so that all teachers can participate in the construction of
technology rich learning environments. For teachers, a number of sophisticated tools, including commercial
products, have emerged to create interactive multimedia software. However, those tools base the interaction
around general models of instruction, which unfortunately are too general to serve as a specification for an
educational software (Bell, 1999). To better support the needs of the student and author/teacher, Murray (2003:
500) recommends authoring tools tailored for specific tasks or instructional situations. Further, to adapt to the
learners' demands, abilities and knowledge many researchers (Hasebrook & Gremm, 1999; Welch & Brownell,
2000) aim at making their tutoring systems more effective using "intelligent" software technologies. Recently,
there have been new efforts to transfer the technology of intelligent tutoring systems and authoring tools over the
Internet which needs much effort from domain and computer experts. We argue that efforts are best placed in the
middle ground between intelligent systems and the conventional, and should be used to develop a procedural
learning environment. That extends the conceptions of simulations to the environment descriptions and typically
gives a wider range of instructional functions to the user-system communication language.
Purpose of the Study
Courseware packages should continuously incorporate teachers’ observations and assessment of student
understanding in overcoming students’ difficulties and provide teachers with collaboration tools to make
courseware facilities fit individual students’ needs. This study presents a new courseware design framework to
overcome students’ learning difficulties in a science learning unit. The courseware framework will then be tried
out with students and teachers: (1) to evaluate effects of the system facilities for students’ learning, with
particular attention to the new interface and the effect of different task regimes, (2) to study classroom teachers’
reactions about a web based knowledge and experience sharing platform as well as collaborative activity
development.
Design of Procedural Learning Environment: Workers’ World
The review of literature both in science teaching and learning and use of technology suggest several
requirements to be considered in the design of learning environment for the learning unit, work, which is not a
commonly studied subject area in the software research. A variety of teaching and learning approaches should be
85
supported by learning environments (i) to overcome learners’ difficulties in this area, (ii) to avoid developing
misconceptions and (iii) to aid the learner in proceduralising his/her interaction with the system and eventually
be less prone to errors. A learning environment for studying abstractions should have a concrete object space
(Thomson, 1992) which can accommodate activities. The activities reveal real world facts in order to represent
the study domain in the environment. The environment should also provide two sets of mechanisms: One is for
checking the validity of students’ methods on the activities and the other is for providing feedback on the
appropriateness of students’ actions in relation to the presented task. Students can operate the object world in
different ways by a student-environment interaction language. Their operations show the effects of their actions
and connect to representations of the object space and its relations at higher levels of abstraction. This requires
the environment to be able to move its presentation modes from concrete to symbolic as the learners gain
competence. Hence the environment may enable a seamless transition (Draper et al., 1991; Kong & Kwok, 2005)
from facts to formalism.
The learning environment should allow science students to experiment with concepts and procedures in ways
that relate to the students’ experience to support guided discovery (Tait, 1994; Smeets, 2005) and directed
methods of instruction. Learning in such an environment should be contextualized and procedural through
different type of tasks. The teachers may manage task specifications in a curriculum management office
(Akpinar & Hartley, 1996). Hence an extension and integration of the students’ learning environment to the
teachers’ customization environment is necessary. Because students may need different tasks, teachers may share
students’ experiences and task regimes and may cooperate in developing new task regimes. Such collaboration
may be realized through web based data manipulation tools which can send data to a students’ environment.
Teacher
Global Activity Center
Collaboration
Global Task
Learner Record
Teacher Experience
Pool
Repository
Repository
Teacher
Activity Environment
Activity Center
Curriculum Authoring Center
Activity Specification Tools
Activity Space
Activity Set
Activity Handling Tools
Screen Design
Records
Course Authoring
Student
Figure 1: The architecture of Workers’ World and Teacher Collaboration Environment
The learning environment this study focuses upon should regulate the control between the student and the
system, accommodate real-life tasks and their solution methods which are rich in feedback and provide
interactive illustrations supporting conceptual understanding, learner controlled inspections and problem solving.
Collins and Brown (1988) and Thompson (1992) suggest that through the structured procedure capturing
systems which offer simple devices, perceptually reflective learning on concrete items can be achieved. Through
the students’ manipulation and inspection, aspects of the device structure can then be explicitly represented,
annotated and become the subject of didactic discussion. On the basis of a firm conceptual understanding, the
student’s actions, as part of a procedure, are to be evaluated and responded to by the computer program in order
86
to provide feedback about the effects the action would have in real world. The student then takes successive
action and each time explores more information. Further, because developing a firm conceptual understanding of
science domain depends heavily upon the constructive work with real world objects in a science community and
because the students need building materials, tools, patterns and sound work habits to learn operational
relationships, the functional components of such “procedural environments” should have the architecture given
in Figure 1. The learning/teaching environment, Workers’ World, proposed here, consists of mainly four
components to provide teachers and students with pedagogical tools in studying the learning unit. These
components are (1) Student Activity Environment, (2) Curriculum Authoring Center for Teachers, (3) Teacher
Collaboration Tools and (4) Global Activity Center.
Student Activity Environment
In a procedural learning environment, it is necessary to provide learners with an object-world whose properties
represent the study domain. The object world should be associated with operator(s) carrying out procedure(s) in
the study domain. The procedures must be completed by a learner on the operators. The environment is to
monitor and evaluate these operations. The represented object world should respond in clearly observable ways
that are easy to interpret by the students, which will help them to relate previous experiences and informal
knowledge to a semiformal representation of the knowledge covered in the environment.
To facilitate intensive procedural activities, the Workers’ World provides an activity environment for students.
The activity environment has basically five components: Activity Center; Activity Set; Activity Handling Tools
(objects and operators); Student Records; and Interface. The Activity Center consists of tasks authored either by
the classroom teacher or by other teachers but customized by the class teacher. The class teacher will prepare a
set of activities and pre-store them for students’ use in accordance with students’ learning problems and
requirements and these activities will be entirely tailored to students’ needs in number and type. The student
activity environment also provides tools to handle and complete activity sets. Activity handling tools in the
Workers’ World takes representation of the study domain into account. Hence it will be a domain representation
environment which represents two basic components as screen objects, namely objects and operators. The
objects, prescribed in a given activity, (workers and/or trolley and masses), are represented graphically. The
operators are work-meter and road, and are intended for proceduralization of the given task. Once a trolley with
or without a specified mass is pushed by a worker on a given road (distance to be specified), a particular amount
of work would be produced, measured and represented in the work-meter.
All these arrangements will support “making the goal of the user and representation of procedure by the learning
environment close” (Schneiderman, 1998). In turn, the user will make a ready transition from his/her planned
action into his/her instructions to the environment. Further, the environment makes the results of each user action
visible and interpretable. It will also be able to control the users’ attention in order to accommodate exploratory
and directed mode of interaction.
Figure 2: A task and learner tools in Workers’-World Learning Environment
87
In Workers’-World, the interface consists of three different segments (A task presentation segment, Task objects
and operators, A Work-Meter) for students to check activities (see Figure 2): The task objects together with the
operator will require students to analyze the given task, to find out what to do with the provided objects and to
proceduralize the given task. A presented task may be accomplished typically by the following procedures: (1)
Selecting workers and/or a weight; (2) Dragging and dropping selected objects one-by-one at the back (or top) of
the trolley on the distance, (3) Executing the push action to the trolley; (4) Observing the push action and reading
and noting the work-meter values. The procedure that students may wish to undertake is to apply a force on an
object and to measure the quantity of the force. The sequence of operations would be monitored by the
environment. They function as components of procedural language which is controlling interactions between the
student and the Workers’ World environment.
Curriculum Authoring Center for Teachers
To enhance active learning, the learning environment should provide different types of tasks. These tasks can be
specified and managed through a Curriculum Authoring Center (CAC). The CAC, lesson office for teachers, will
allow the learning to be contextualized and procedural in its instructional approach. It will also be capable of
setting to the task-needs of a teacher who will utilize his/her experiences with students’ learning of the domain
and from previous student records to author a lesson for a particular group of students. Such data exchange is
needed to provide students with a set of learning activities and teachers with students’ data. To take more student
data (records) into account in the task presentation in activity space and in course authoring, the CAC will
exchange data with other facilities. A teacher can use the provided activity specification tools to prepare eight
different modes of studying work. The number and order of activities would depend on the teachers. Hence, the
teachers will need a set of easy-to-use activity specification tools. The Workers’-World Learning Environment
enables a dynamic and visible authoring center with the following Activity Specification Tools (See Figure 3):
A text editor
1) A task ordering unit
2) A screen object management unit
2.1) Worker management
2.2) Masses & Weights management
2.3) Labeling management
2.4) Work management
2.5) Feedback management
2.6) Screen layout management
3) A task set management
3.1) Task selection
3.2) Task browsing (downloading)
3.3) Task landing (uploading)
Teacher-tool interaction in the environment is managed through a direct manipulation type of interface. A
teacher can write up tasks in contexts and select order of tasks, workers, weight-objects, force and students’
screen layout through drag and drop activities. Insertion, deletion, extraction and saving of text, numbers and
screen objects may be handled in the CAC. A single task or a set of tasks can be created in the lesson office or
imported from the global activity center. Each instructional screen object in the CAC has the same fundamental
properties. When an activity (task) is viewed by a learner, certain objects will be visible or invisible: Visibility
conditions are set by the teachers. For example, whether a student should be allowed to modify the property of
the road traveled by the worker in a given situation is adjusted prior to task presentation. Activities prescribed in
the lesson office form the instructional curriculum for a particular group of students. The learners must manage
this curriculum in an intensive interactive manner. The task (activity) specification managed through the
interface of the lesson office by the teacher must be able to support tasks that are exploratory in nature or more
directive in their presentation. The sequence of different mode of activities is to provide scaffolding where
subsequent task relates to the students’ previous experience. Further, help or procedural hints and feedback
should be attached to a particular activity. For each activity, the lesson office places its prescription on a screen
that permits tagging of the problem, including text describing the problem context and guidance to the learner.
The answers are also given to the system and the teacher can provide interpretation and overall task feedback by
88
using the response component. Hence, such flexibility in the environment will allow for use with individual
learners, small group demonstrations, or paired studying as determined by the classroom teacher.
Figure 3: Activity Specification Tools in GTP
Global Activity Center and Teacher Collaboration Tools
In conventional learning environments, a teacher plays an important role in determining what and how students
learn through activities. Teachers are responsible for monitoring the flow of each student’s activities, playing a
meta-cognitive function for the students by probing their knowledge and reasoning, monitoring participation and
student engagement. Student activities must be rich and need-based so that teachers make their educational
diagnosis and intervention accordingly. As student needs vary and those needs may be fulfilled with different
task regimes, it would be important for teachers to have access to a large activity pool which is constructed and
enriched by teachers. The Workers’ World accommodates such an activity pool managed within a Global
Activity Center (GAC).
Because the Internet provides a means of easy communication and information exchange platform, teachers can
collaborate to construct and tailor tasks, based upon their students’ performance records. A set of online tools
and resources support teachers as they, in turn, support the students through activities. This set of online
resources outline a suggested sequence of activities based on what has worked in the past; each activity listed is
linked to additional information regarding the purpose of the given activity, an elaboration of what the activity
entails, and tips for when to intervene. It also provides a teacher with practical strategies for how to guide
students’ work. To accomplish these functionalities and to make them an Internet communication medium, this
study constructed a three-module collaborative work platform for teachers: Global Task Pool (GTP), Experience
Repository and Learner Record Repository.
Global Task Pool: It is a database of tasks prepared by teachers and uploaded to the system. It contains tasks,
objects of tasks, operators to be used for tackling each task, prompt for each task and level of students for whom
tasks could be used. Each submitted task may be associated with a series of other tasks, hence allowing the
prescription of a task space for a (group of) student(s). Teachers can comment on tasks and share those
comments. The task pool (see Figure 4) would be enriched by the contributions of other teachers. Critique of and
suggestions about a particular task or task regime would enable teachers to work with better quality tasks,
constructly validated by colleagues. An important characteristic of study tasks which are used in this type of
89
learning and teaching environment is the authenticity of the activity. Teachers’ collaboration will help produce
more authentic activities and GTP will store them for further usage.
Figure 4: Access to Global Task Pool
Learner Record Repository: One of the benefits of computational learning environment is to keep record of
students’ performances in a given learning activity. When those performance records are taken into account in
developing new or altered activities, the learning environment is more likely to respond to students’ needs. When
teachers customize learning activities, they can refer to students’ records provided by other teachers from
different school(s) in the same or different regions, and have an idea of similar students’ behaviors over a series
of tasks. Such a learners’ record repository would be more helpful to teachers when the records contain
information about students’ learning difficulties and teachers’ experiences about overcoming those difficulties
and information about student reactions.
Experience Repository: The Global Task Pool provides teachers with a set of tools in order to share their
experiences about student reactions to a task or task regime. Once a teacher uploads his/her tasks used in a
classroom to the GTP, the teacher may also share her experiences and observations about the students’
performances and learning outcomes. These could be the result of comprehensive analyses of particular students’
learning process of work or an informal description of learner-task interaction. Further, the Experience
Repository can store information about students’ task manipulation and learning styles, actions after an activity,
the type of intervention they needed, and the type of additional help they required.
Fast communication means of Internet may help to form an activity building community. Then, colleagues can
share a rich learners’ profile and a large number of authentic tasks. The success of GAC to a large extent
depends on the quality of teachers’ communication. The design of the GAC should encourage participant
teachers to engage in developing meaningful practices through collaborative processes. The GAC will develop a
climate where commenting on each others’ work and giving and receiving feedback are integrated and routine
part of the collaborators’ work.
Implementation and Evaluation of the Workers’-World Environment
The entire teaching and learning environment was developed in different applications: The Student Activity
Environment and the Curriculum Authoring Center were constructed using Macromedia Flash. The Global
Activity Center and teacher collaboration tools were constructed with Microsoft .net platform because the GAC
needs database relations and other server-side requirements. The first two systems consist of main functions and
objects which control their operations. The system also works with an XML database. All the data shown to the
user is extracted from the xml file with “isle” function, the file will then be parsed to the data resource, and the
whole data are copied to an array written in ActionScript code. Flash platform aids with the main Keyframe
construction and its definition of functions. “Execution_OK”, “Ask_Task”, “Event_Time”, and “Save_Act” are
the main functions of the learning environment and are defined on the main Keyframe. For example, each task is
derived from the data array through Ask_Task function.
90
Sample
To evaluate the effects of the system facilities for students’ learning, a pilot evaluation was carried out with 69
seventh grade students (35 male and 34 female; average age is 14) from four different secondary schools. Four
different groups of students were randomly assigned to the two task regimes. The first task regime group
consisted of 39 students (20 male and 19 female) and the second task regime group consisted of 30 students (15
male and 15 female). In addition, four science teachers from four different schools participated in the study.
Procedure
In the evaluation study, particular attention was paid to the effect of different task regimes. Prior to the
experiment with the students, the teachers met in the researchers’ office and received a two hour training about
the study on how to use the Workers’ World environment online and offline, CAC and other facilities of the
software set. They were also asked to select an online partner among other teachers to prepare a task regime for
the Workers’ World. They were then instructed to prepare tasks in all modes of the software in use. One pair of
teachers was instructed to author a task regime with items requiring qualitative thinking and worksheet support.
The other pair was instructed to author a task regime with different tasks. Two different task regimes were
prepared in the CAC by the teachers in two weeks time and checked by the researchers. The first task regime
used only the courseware facilities, the second task regime with more qualitative items used a worksheet as a
support material to the courseware facilities. In the first task regime, there were ten tasks; six of them were in
essay format, three were in multiple-choice format and one was in fill-in-the blank format. The first nine tasks
required employing the given software facilities for the students to solve and to proceed. The final task, fill-inthe blank, required to form the formula of work problems using the given variables. The second task regime,
however, had tasks more qualitative in nature. This task regime had nine tasks; six in essay format, two in
multiple-choice format and the final one was in fill-in-the blank format.
A pretest and a posttest, both in multiple choice format, were prepared to measure student progress after using
the learning unit. Each test contained items on the basis of the lesson objectives as to describe the relationships
between; force and work, distance and work; force, distance and work; masses and work; and as to derive the
formula for solving work problems. The two task regimes in the Workers’ World environment were then
administered to the assigned groups. The pre and post tests and the treatment took place in computer laboratories
of the schools from which the sample was drawn. Four sessions (two for each task regime) were organised. In
both task regime treatments, prior to the treatment, the pretest was administered in ten minutes. Just after the
pretest, the researchers explained the sample how the software facilities to be handled. Then the students were
allowed to study the task regimes for an hour. Although an hour was allocated to the study of task regimes, the
second task regime was studied in about fourty minutes in average. The students had to finish all of the tasks
correctly by using the software facilities, even when a student predicted the answer. Therefore, all students
completed all tasks in both task regimes. During the study of task regimes, no help to find the correct answer was
provided. One of the researchers and one of the teachers (class teacher who was a member of task authoring
group) were present during the treatments. Following the task regimes, all students were given a paper and pencil
posttest similar to the pretest.
In addition to the teachers’ observations of the treatments, the data for the study was the pre and post test results
shown in Table 1. To see whether there is any significant difference between the students who studied the first
and second task regimes, a t-test was conducted (Table 2).
39
Table 1: Pre and post test data
Pretest
Pretest Standard
Mean
Deviation
1,690
1,303
30
2,135
n
First task regime: Without
worksheet
Second task regime: With
Worksheet
1,084
Posttest
Mean
2,370
Posttest Standard
Deviation
1,238
2,200
1,288
The data representing the samples’ pre/post-test scores differences was indicated as mean difference in Table 2.
It was found that there was a significant difference (t(2.084, 67) = 0.043, p<0.05) between seventh graders’
achievement in “work learning unit” studied through Workers-World CBL environment with worksheet support
and without worksheet support. It seems that both task regimes interacted with students’ learning; however the
91
first task regime helped students more than the second task regime with a worksheet. This should not be
interpreted as simply the CBL facilities are sufficient to learn the work unit, but it points to the fact that the
varied use of facilities and customization of facilities to the students are important.
Table 2: Pre and post test mean differences
Mean
Standard
n
Difference
Deviation
First task regime: Without worksheet
39
,680
1,280
Second task regime: With Worksheet
30
,065
1,201
Standard Error Mean
,205
,219
Following the study, the researchers met the four teachers who collaboratively authored the task regimes, and the
results of the study were shared with them. They were asked to be critical to provide information on the system
so that the revisions can be made. The teachers’ comments on the sample students’ progress were positive but
they pointed out that the students should be exposed to more number of both qualitative and quantitative learning
tasks in a period of time longer than the study allocated. The idea of collaboration with other teachers was found
interesting and useful. Information exchange and discussion specific to a learning task was found more useful
than discussion lists based on general domains such as science education, where many teachers state their
opinions and raise issues that generally result in superficial discussions. It was noted that thorough discussions
specific to a learning task may be systematized through tools similar to the Workers’ World. However, the study
with teachers should be verified with systematic tools in more comprehensive studies.
Discussion and Conclusions
The findings demonstrate that both task regimes have effects on students’ learning; however, the first task
regime helped students more. In the first task regime, the students studied the first and second tasks to relate a
force to “work” in a given construction-site work by measuring the workers’ forces. The third task aimed to
reveal direct proportionality between force and work concepts; the students observed that workers’ forces are
directly related to their work. The ratio relation between workers and their forces was studied in the fourth task
where one to one correspondence between the variables was established by the students. The fifth task focused
on work and distance relationship; to simplify the world in which only one worker, force, was given and the
students had to fill out a distance-work table. Similarly, the sixth task required comparing different work values
on different distances and the students completed a distance-time table. Further, the seventh task enhanced
matching force, work and distance relations. The eighth and ninth task focused on the effect of mass on work,
showing that machines simply transform the form of the work we do. In the ninth task, a worker had to carry 3, 5
and 12 kg masses with the help of the trolley. The students measured the work, and then had to fill in the table
cells with these measurements. All required cell results were 350 Joule. Converting the relationship between the
variables into an abstraction, building the equation, was studied in the last task where force, work, distance and
arithmetic symbols of addition, subtraction, multiplication, and division as well as the equal sign were given. In
this task, if students requested, the work mechanism was also visible.
Although the sample’s progress from pretest to posttest is small, the software facilities seem to, to a certain
extent, help students overcome learning difficulties and contribute to students’ learning. The Workers’ World
facilities allowed students to visualize the relations between force and work, also helped most students to see that
machines simply transform the work that we do. The students identified patterns of distance, force, work and
mass within the environment. They further formulated the relationship between the variables. If the teachers
benefited from an experience repository, a learners’ record repository and a task pool, they may have prepared
different task regimes, which may help students more.
The Workers’ World facilities were used by the students on an individual basis and on a limited time period.
This is why the students’ communication and disposition of inquiry in classroom discourse was not possible. The
only interaction students participated in was with the courseware facilities for about ten tasks. This may be a
drawback of the study, however, it seems that the software facilities used even in a short period provided
students with new ways of reasoning and various sense making tools. The progress in students’ scores from the
pretest to the posttest and the two different results for the two different task regimes show that varying task
regimes should be employed to help learning. In addition, the tools such as student-record repository and activity
center enable developing the most appropriate task regimes for particular group of students.
92
The collaborative communities provide opportunities for teachers to reflect deeply and critically upon their own
teaching practice on the content they teach and on the experiences and backgrounds of the learners in their
classrooms (Putnam and Borko, 1997; 1247). Briscoe and Peters’ (1997: 61) findings further indicate that
“collaboration facilitates change on teachers because it provides opportunities for teachers to learn both content
and pedagogical knowledge from one another, encourages teachers to be risk takers in implementing new ideas,
and supports and sustains the processes of individual change in science teaching”. The teacher tools designed in
this study may serve the purpose as pointed out by Jauhiainen et al (2002) who reported that the courses and
activities that were most valuable for day-to-day teaching were those where teachers could cooperate, reflect and
plan with each other in small groups.
In a recent and more comprehensive analysis of the teacher networks, Zhao and Rop (2001) report that the
teacher networks were claimed to have a number of positive effects on their participants such as reduced teacher
isolation, enabled curriculum development, facilitated dissemination of information, and provided easy access to
curricular materials. To enable teachers to use computer mediated communication (CMC), this study showed
again that not only general purpose tools like e-mail and discussion lists, but also content specific, tailorable and
flexible courseware may be designed and used by teachers. Tools similar to Workers’ World would develop
teacher reflective discourse communities, because the real value of CMC may lie in information sharing around a
set of materials and sharing student data which could be the focus of discussion and reflections. When similar
student-facilities specific to learning tasks increase in number and quality, teachers may then enrich their
reflective discourse in different learning units of varying content areas.
The study developed, implemented and evaluated a set of software environments with Student Activity
Environment, Curriculum Authoring Center, Teacher Collaboration Tools and Global Activity Center. These
components were built according to the recommendations of previous studies for collaboratively set learning
environments. The studies with students and teachers validated the approach. However, the student activity
environment may be enriched with new task models. The models may include script flexibility for expert
teachers or programmers. The system may support external objects with predefined standards (e.g., SCORM
compliant), and include a learning object or other multimedia objects as video segments, pictures or animations.
Possible external objects may include updates or support for other learning units. They may also be linked to a
new science unit. In this sense, the student activity environment may be used for only its core functions such as
student record repository, learning object adaptivity, portability and feedback mechanism. The feedback in the
student activity environment may be changed in several ways: If there is more than one type, it should be under
teacher control. Further studies should investigate the different usage of the facilities and extensions. Prime
interest in those research is to collect data at the operational level (e.g., to what extend are students and teachers
able to use the system and do the facilities motivate them?) and at a reflection level (e.g., is the learning
environment encouraging tactical thinking?) so that meta-cognitive and problem solving skills can become part
of the learning objectives.
Teacher collaboration tools should further be studied to increase experience-sharing among teachers in terms of
similar activity center applications. Teacher actions in the center may be observed for further expansions of the
system. The collaboration tools may be strengthened with the new online messaging tools. Task regimes and
worksheets may be adapted for collaborative learning strategies to increase efficiency of the learning gains.
Acknowledgements
We would like to thank three anonymous reviewers and Dr. Gunizi Kartal for comments on earlier drafts of this
paper.
References
Akpinar, Y., & Hartley, J. R. (1996). Designing interactive learning environments. Journal of Computer Assisted
Learning, 12 (1), 33-46.
Amasci, R. (2004). Errors and science myths, retrieved January 9, 2006 from http://www.amasci.com/miscon.
Bell, B. (1999). Supporting educational software design with knowledge-rich tools. International Journal of
Artificial Intelligence in Education, 10 (1), 46-74.
93
Briscoe, C., & Peters, J. (1997). Teacher collaboration across and within schools: Supporting individual change
in elementary science teaching. Science Education, 81 (1), 51-65.
Cadmus, R. R. Jr. (1990). A video technique to facilitate the visualization of physical phenomena. American
Journal of Physics, 58 (4), 397-399.
Chandler, P. (2004). The crucial role of cognitive processes in the design of dynamic visualizations. Learning
and Instruction, 14 (3), 353-357.
Collins, A., & Brown, J. S. (1988). Computer as a tool for learning through reflection. In Mandl, H. and Lesgold,
A. (Eds.), Learning Issues for ITSs, Berlin, Germany: Springer Verlag, 1-18.
Draper, S. W., Driver, R. J., Hartley, R., Hennessy, S., Mallen, C., Mohamed, R., O’Malley, C., O’Shea, T.,
Scanlon, E., & Twigger, D. (1991). Design considerations in a project on conceptual change in science.
Computers and Education, 17 (1), 37-41.
Dunlap, D. R., Neale, D. C., & Carroll, J. M. (2000). Teacher collaboration in a networked community.
Educational Technology and Society, 3 (3), 1-17.
Escalada, L. T., Grabhorn, R. P., & Zollman, D. A. (1996). Applications of interactive digital video in a physics
classroom. Journal of Educational Multimedia and Hypermedia, 5 (1), 73-97.
Hasebrook, J. P. & Gremm, M. (1999). Multimedia for vocational guidance: Effects of individualised testing
videos and photography on acceptance and recall. Journal of Educational Multimedia and Hypermedia, 8 (4),
377-400.
Hwang, F. K. & Esquembre, F. (2003). Easy java simulations: An interactive science learning tool. Interactive
Multimedia Electronic Journal, 5 (2), retrieved May 1, 2006 from http://0-imej.wfu.edu.csulib.ctstateu.edu/
articles/2003/2/01/index.asp.
Jauhiainen, J., Lavonen, J., Koponen, I., & Kurki-Suonio, K. (2002). Experiences from long-term in-service
training for physics teachers in Finland. Physics Education, 37 (2), 28-35.
Jonassen, D. H. (1996). Computers in the classroom: Mindtools for critical thinking, Englewood Cliffs, New
Jersey, USA: Prentice-Hall.
Kong, S. C., & Kwok, L. F. (2005). A cognitive tool for teaching the addition/subtraction of common fraction: A
model of affordances. Computers and Education, 45 (2), 245-265.
Murray, T. (2003). An overview of intelligent tutoring system authoring tools: Updated analysis of the state of
the art. In Murray, T., Blessing, S. & Ainsworth S. (Eds.), Authoring tools for advanced technology learning
environments, Dordrecht, The Netherlands: Kluwer, 491-510.
Putnam, R. T., & Borko, H. (1997). Teacher learning: Implications of new views of cognition. In Biddle, B. J.,
Good, T. L. & Goodson, I. F. (Eds.), The international handbook of teachers and teaching, Dordrecht, The
Netherlands: Kluwer, 1223-1296.
Schneiderman, B. (1998). Designing the user interface: Strategies for effective human-computer interaction,
Boston, MA, USA: Addison Wesley.
Smeets, E. (2005). Does ICT contribute to powerful learning environments in primary education? Computers
and Education, 44 (3), 343-355.
Tait, K. (1994). Discourse: The design and production of simulation based learning environments. In Jong, T. de
& Sarti, L. (Eds.), Design and production of multimedia and simulation-based learning material, Dordrecht, The
Netherlands: Kluwer, 113-131.
Thompson, P. W. (1992). Notations, conventions and constraints. Journal for Research in Mathematics
Education, 23 (2), 123-147.
94
Tsai, C. C., & Chou, C. (2002) Diagnosing students' alternative conceptions in science. Journal of Computer
Assisted Learning, 18 (2), 157-165.
Welch, M., & Brownell, K. (2000). The development and evaluation of a multimedia course on educational
collaboration. Journal of Educational Multimedia and Hypermedia, 9 (3), 169-194.
Welmar, H. G. (1996). Assessing the impact of computer-based learning in science. Journal of Research on
Computing in Education, 28 (4), 461-85.
Zhao, Y., & Rop, S. (2001). A critical review of the literature on electronic networks as reflective discourse
communities for inservice teachers. Education and Information Technologies, 6 (2), 81-94.
95
Gülseçen, S. & Kubat, A. (2006). Teaching ICT to Teacher Candidates Using PBL: A Qualitative and Quantitative
Evaluation. Educational Technology & Society, 9 (2), 96-106.
Teaching ICT to Teacher Candidates Using PBL: A Qualitative and
Quantitative Evaluation
Sevinç Gülseçen
Faculty of Sciences and Informatics Department, Istanbul University, 34452 University, Istanbul, Turkey
[email protected]
Arif Kubat
Faculty of Economics and Administrative Sciences, Inönü University
Malatya, Turkey
ABSTRACT
The idea underlying this study is that the prospective teachers develop their Information and Communication
Technology (ICT) skills throughout the learning process supported by the Problem Based Learning (PBL), a
method that produces independent learners who can, then, continue to learn on their own in their chosen
careers. The major goal of the study, conducted in two phases during two academic years, is to investigate the
differential effects of PBL and conventional teacher-centred instruction on cognitive and a group of affective
variables which have been coined “technophobia”. Phase I of the study, entirely qualitative in character,
covered a case study in which a total of 111 students from three departments of the Faculty of Education took
part. Phase II of the study was quasi-experimental in nature with 79 students taking part. Most of the
outcomes from Phase II were based on statistical measurements. At the end of the instruction process, the
findings indicated that, while there was no significant difference in the anxiety levels of the two groups, the
number of successful students had registered a considerable increase. Furthermore, the students, actively
involved in the learning process, solving real problems, viewed the PBL as an effective learning tool rich in
motivation. Within this framework, the motivating role of ICT as a method of learning is automatically
recognised.
Keywords
Problem Based Learning, Information and Communication Technology, Cognitive Tools, Improving Clasroom
Teaching.
Introduction
In general, when the related literature is reviewed, it appears that several international and national studies point
in the direction of the central role teachers play in the implementation of educational change. In particular,
however, the main concern is the integration of Information and Communication Technology (ICT) into the
teaching process. According to Fullan (1991), “Educational change depends on what teachers do and think – it is
as simple and as complex as that.” On the other hand, while only a few teachers have had the opportunity to
develop more than the basic skills in the use of ICT, even fewer have developed confidence and competence in
the integration of ICT into their classrooms (Orhun, 2002a; Albion, 2003). A 1992 OECD report entitled,
“Education and New Information Technologies: Teacher Training and Research” emphasizes in its conclusion
that, “...the potential of the new ICT for improving learning and teaching will not be realised unless teachers are
well trained and retrained in the pedagogical use of technology in the classroom (Orhun, 2002b: Yıldırım &
Kiraz, 1996).
While the ICT receives wider acceptance in the field of education, computer anxiety remains to be a challenging
concern. Generally speaking, some teachers still exhibit a certain degree of anxiety toward computers as a tool to
be employed in the fields of education and learning. Yıldırım and Kiraz (1996) state the fact that even the
utilisation of e-mail is adversely effected by the negative approach of some teachers towards it. It is obvious that
the teachers themselves must have necessary training and support to properly integrate the related technology
into their practices even before employing methods and techniques to improve the process of teaching and
learning. In particular, the teachers are to be healed of the anxiety of computers utilised in such processes.
Moreover, the virtues of self-confidence and positive approach toward computers, as well as belief in the ability
to employ them in an effective and efficient manner should be instilled in the teachers.
Computer-based cognitive tools have a higher potential to foster meaningful learning, which requires a
constructivist view of learning (Jonassen, 2000; Gibson & Silverberg, 2000). According to Jonassen, Howland,
Moore and Marra (2003), constructivism, like nearly all contemporary theories of learning (situated learning,
social cognition, activity theory, distributed cognition, ecological psychology and case-based reasoning) share
96
convergent beliefs about how people naturally come to know. One of the best exemplars of a constructivist
learning environment is Problem Based Learning (PBL) as described by Barrows who was involved in the early
stages of the development of PBL at McMaster University in Canada (Savery & Duffy, 1995; de Graff &
Kolmos, 2003). He defines the concept in terms of its specific attributes such as being student-centred, taking
place in small groups with the teacher acting as a facilitator, and being organised around problems. Liu (2004)
suggests that the aspect of anchoring learning in real world contexts in PBL has been found to provide
opportunities for transferring knowledge and skills from the classroom to workplace more easily.
According to Albion (2003) teachers’ use of ICT may be influenced by their self-efficacy beliefs, that these, in
turn, may be enhanced by access to examples of successful practice, and PBL may provide an especially
effective vehicle for this purpose. We also believe that the attitudes of the teachers can be influenced and
changed positively during the early periods at the teacher training institutions. By creating a learning
environment which promotes professional development, teacher candidates should be made aware in terms of the
ways in which they could benefit from existing ICT tools.
The main goal of the study presented in this paper was to investigate the differential effects of PBL and
conventional teacher-centred instruction on performance of teacher candidates of the Faculty of Education,
Istanbul University (hereafter Faculty of Education) and a group of affective variables such as “technophobia” as
coined by Rosen and Well (1992), i.e., Computer Anxiety Rating Scale (CARS), Computer Thoughts Survey
(CTS) and General Attitudes Towards Computers Scale (GATCS). The performance and attitudes of students in
the PBL curriculum has been compared with that of the students in the standard curriculum longitudinally over
two years.
The rest of the paper is organised in four sections. Based on the literature review in “General Rationale and
Hypothesis” section, the basic definitions for constructivism and PBL are provided in that order, followed by the
main hypothesis of the study. Subsequently, Phase I and Phase II detail a model of PBL program used in a
second-year course on “Introduction to ICT” at the Faculty of Education taught by same instructor (the first
author of this paper) during the academic years 2001-2002 and 2002-2003 respectively. Both phases introduce
an “experiment” in developing ICT skills of the second-year students by having them design a multimedia
presentation for their future students, working in groups undertaking PBL-based project work. While Phase I is
entirely qualitative, Phase II is more quantitative and much of what is described in these two sections stems from
students evaluation and feedback obtained over two academic years. A general discussion and conclusion follow
in the last section.
General Rationale and Hypothesis
The Context
Providing schools with ICTs promises a high return on investment. The presence of computers and Internet
access raises ICT literacy and skills, better preparing the future generations to participate in the information
society. To this end, the schools represent ideal access points because they cover a large part of the population,
especially in developing countries. Connecting schools through technology also bring online that part of the
population that can learn quickly how to use ICT (World Telecommunication Development Report, International
Telecommunication Union, 2003).
The education policies and systems of Turkey, a country with young population, are being revised on an ongoing
basis so as to train competent and fully equipped teachers challenged to meet the needs of eight years of
elementary compulsory schooling recently put in place. Orhun (2002b) stated that main part of Computer-Aided
Education (CAE) project was initiated in 1990-1991 as part of the World Bank National Education Development
Project. It included a program to introduce computer literacy and computer aided instruction in grade 10 at
selected secondary schools. A report by the Research, Planning and Coordination Board of the Ministry of
National Education (http://www.meb.gov.tr) titled, “The National Education at the Beginning of 2002”, it is
stated that “... using the credit received from the World Bank, 3188 IT classrooms have been built in 2802
schools, tenders have been opened to build more of them, and to obtain software in 3000 more primary schools.”
The Faculty of Education is one among several others to meet the ever increasing demand of teachers and
educational specialists in Turkey. The initial teacher preparation involves the development of discipline-based
knowledge, curriculum, and pedagogy skills as well as professional ethos. A course named “Introduction to ICT”
has also been included as a compulsory course in the curriculum. The history of the course reaches back to 1999
97
when the Faculty began receiving students. At the end of the course, students are expected to gain specific ICT
skills and knowledge which they will then transfer on to the workplace through opportunities provided.
Most of the students come from the middle or lower classes with no or very little access to technology. Although
the students seem to be motivated and keen to learn when introduced to the topic itself, a certain level of anxiety
stemming from their fear of what is technological is also noted. It cannot be said that the overall attitudes of the
students concerning computers, their application, or impact on the society is negative. Nevertheless, the fact is
that the dominant feeling in most students’ approach to technology is one of anxiety based on a technophobic
tendency pointed out earlier in this paper. As indicated by Weil and Rosen (1995), the problem of technophobia
is fairly common. Erktin and Gülseçen (2001) define it as an important psychological factor efficacious in the
use of computers in education. According to Rosen and Weil (1992), the technophobic is an individual
evidencing at least one of the following, ranging from severe reactions to mild discomfort: anxiety about present
or future interactions with computers or computer related technology, specific negative cognition or self critical
internal dialogues during actual computer interaction or when contemplating future computer interaction.
During the academic year 1999-2000, the “Introduction to ICT” course has been taught using traditional weekly
lectures and laboratory sessions throughout the semester four hours a week. By the end of the semester, it was
obvious that there was not sufficient time available to cover the entire content of the course; individual students
were at varying levels of knowledge and unhappy with this fact; many students suffered the same level of
technophobia as they did in the beginning; the instructor was dissatisfied with her performance – as there were
too many students in one class, it was impossible to interact equally with every student on one-to-one basis
causing an adverse affect on the outcome. Moreover, both the instructor and the students themselves were
dissatisfied by the performance of the students who exhibited signs of lack of motivation. In order to alleviate the
problem of unmotivated students, a plan could be brought forth to overcome the lack of time, as well as to
facilitate the learning process. Designing an interactive and highly effective learning environment conducive to
cooperation, self-assessment, and provision of prompt feedback would allow for better opportunities for students
taking into account their personal learning preferences. According to Frank et al. (2003), self-direction, a passion
for learning, and strong individual responsibility are important influences on achievement. Therefore, in order to
allow the students to learn different ways in pace with their individual nature while encouraging them to evaluate
their own way of learning, a more flexible approach to teaching was adopted. Such an approach requires the
instructor to change as well. To wit, the teacher is to cease being the center of attention and the source of all
knowledge, becoming instead a coach, a facilitator of the knowledge to be acquired. The next section explains,
PBL has several distinct characteristics that may be identified and utilised in designing such environments.
Constructivism and PBL
Constructivism as a philosophical view on how we come to understand or know, holds that any so-called reality
is the mental construction of those who believe they have discovered and investigated it. Its view is characterized
in terms of three primary propositions. Knowledge is in our interactions with environment. Cognitive conflict is
the stimulus for learning. Understanding is influenced through the social negotiation of meaning (Saunders,
1992; Savery and Duffy, 1995). From these three propositions, a set of instructional principles that can guide the
practice of teaching and the design of a learning environment emerged. These principles are, anchoring all
learning activities to a larger task or problem, supporting the learner in developing ownership for the overall
problem or task, designing an authentic task and the learning environment to reflect the complexity of the
environment, giving the learner ownership of the process used to develop a solution, designing the learning
environment to support and challenge the learners’s thinking, encouraging testing ideas against alternative views
and alternative context providing opportunity for and support reflection on both what is learned and on the
learning activity (Savery and Duffy, 1995).
As stated in the literature since its first prominence in the late 1970s, PBL has lent its increasingly important
voice to the ongoing debate on how to organize teaching and learning at the universities. It restructures
traditional instructor/student interaction to emphasize active, self-directed learning by the student, rather than
didactic, teacher-directed instruction (Harland, 2002; Maxwell, Bellisimo, & Mergendoller, 2001). According to
Driessen and Vlueten (2000), PBL is characterized by problem-orientation, interdisciplinary work and selfdirected learning and focuses on interpersonal and professional skills. Having taken its place among the
curricular innovations hotly debated in the circles of the higher education over the last 30 years, learners exposed
the PBL are gradually allowed to acquire more and more responsibility, who, then, become increasingly
independent of the teacher in terms of their own education. Providing educational materials and guidance that
facilitate learning are the responsibilities of a PBL instructor. Several studies have shown that PBL is a
98
successful method compared with the more traditional curricula with regard to internal motivation and long-term
retention of the learned knowledge. It brings about procedural changes in a direction that can support a shift in
perspectives from teaching to learning (Dahlgren, 2003). The PBL covers any learning environment in which the
problem drives learning. That is to say, before students acquire knowledge, they are given a problem so posed
that the students discover that they need to learn something new before they can solve the problem at hand.
Tse and Chan (2003) assert that in the PBL environment, students act as professionals and confront problems as
they actually occur – with fuzzy edges, insufficient information, and a need to determine the best solution
possible by a given date. The primary distinction is the focus on introducing concepts to students by challenging
them to solve a real world problem. Duch (1997) states that in contrast to the more traditional approach of
assigning an application problem at the end of a conceptual unit, the PBL uses problems to motivate, focus and
initiate student learning. A critical factor in the success of PBL is the problem itself. PBL problems should strive
to induce students to learn at the higher Bloom levels (see Table 1), where they analyze, synthesize and evaluate
rather than simply define and explain. A Level 3 problem is a good PBL problem, at Bloom’s Analysis,
Synthesis or Evaluation levels. It is related to the real world, drawing the students into the problem.
Table 1. Bloom’s cognitive levels and student activities (Bloom, 1956)
Bloom’s Cognitive Level
Student Activity
Evaluation
Making a judgment based on a pre-established set of criteria
Synthesis
Producing something new or original from component parts
Analysis
Breaking material down into its component parts to see interrelationships/hierarchy
of ideas
Application
Using a concept or principle to solve a problem
Comprehension
Explaining/interpreting the meaning of material
Knowledge
Remembering facts, terms, concepts, definitions, principles
The characteristics that may be identified and utilized in designing a PBL curriculum are as follows (Savery &
Duffy, 1995; Tse and Chan, 2003):
1. Reliance on problems to drive the curriculum – the problems do not test skills; they assist in development of
the skills themselves.
2. The problems are authentically ill-structured – there is not meant to be one solution and, as new information
is gathered in a reiterative process, perception of the problem, and thus the solution, changes.
3. Students solve the problem – lecturers are coaches and facilitators.
4. Students are only given guidelines for how to approach problems – there is no one formula for student
approaches to the problem.
5. Authentic, performance-based assessment – this is a seamless part and end of the instruction.
Hypothesis
During the first meeting with the students at the beginning of academic year 2001-2002, a female student
(Cigdem) felt extremely insecure with the idea of using a computer and said, “I am afraid to break it?” Also the
students encountered difficulties stemming from lack of basic technical skills such as using the keyboard. One of
the male students (Hasan Ali) expressed this difficulty as, “I will never be able to get familiar with all keys on
this keyboard!” The fear and anxiety reached to such a degree that some students were compelled to ask
instructor’s guidance during what is their first meeting with the computer. The reactions of Cigdem and Hasan
Ali, also observed in most of the students during their first experience with the computers, was the moment the
instructor made the decision to leave conventional teaching and shift to the PBL model of instruction to provide
the students with a more effective learning environment.
Phase I
Method
Participants
The sample of the study consists of a total of 111 sophomore students (28 female and 83 male at average age of
21) from the Department of Primary Education (Division of Classroom Teaching and Division of Science
99
Teaching) and Department of Foreign Languages Teaching (Division of English Language Teaching)
respectively.
At the beginning of the course, it was observed that the students, exhibiting various levels of anxiety concerning
the usage of computers, were also at different levels of basic computer knowledge. In order to measure the said
knowledge, a simple test comprised of ten multiple choice questions has been applied. In accordance with the
results of the test, students have been classified as “Computer Illiterate”, “Moderately Computer Literate” and
“Fairly Computer Literate.” The percentage of the “Fairly Computer Literate” was only 3%. As stated before,
high levels of anxiety has been observed in vast majority of students who were, then, classified on such basis as
“Technophobics” and “Non-technophobics”.
Procedure
During the first two weeks of the course, the introductory concepts of ICT (basics of computer hardware and
software, familiarisation with Windows, and a word processor) were taught at what could be described as
conventional lectures. One of the objectives of the said lectures was to eradicate the vast differences of basic
technical knowledge determined to have existed among the students followed by the application of the PBL. In
order to encourage the students to assume the responsibility and take ownership for the problem, it was presented
as a Level 3 problem, classified as “good problem” at Bloom’s higher levels presented in the previous section.
The students were, then, confronted with a real world scenario (Tse and Chan, 2003) through a seemingly
authentic correspondence by utilizing a Power Point presentation for their future students on a topic relevant to
the academic field they have chosen, e.g. language or science teaching, by working in groups of 3 to 5. In the last
10 years, Power Point presentations have become the most prevalent form of multimedia in education as the
students prefer it to presentations from transparencies (Bartsch & Cobern, 2003; Jonassen et al., 2003). Not all
the information needed was divulged concerning the problem at hand. Students were provided only with enough
information about the goals and the process they would go through during the program. They were given a four
week free study time to complete the work. De Graf and Colmos (2003) suggest that the PBL education is based
on students’ background, expectations, and interests. Thus the topic selection is left to the students themselves
without imposing any limit at all with the supposition that the groups would be more enthusiastic about studying
a topic of their own choosing. The students were also informed that they needed to research, discover new
material, arrive at judgements and decisions based on the information accessed through print (books, papers and
magazines), human (other instructors from the Faculty of Education, classmates, and other experts), and
electronic information resources (CD-ROMs and Internet web sites). They have especially been encouraged to
browse Internet resources to obtain contemporary electronic material (pictures, images, sound effects, visual
effects and video clips) and to integrate them into their presentations. Knowledge concerning Power Point
presentation was not imparted to the students. They, were, however, encouraged to access computers whenever
they can from computer laboratories within the university campus, Internet cafés, student dormitories and homes.
During this period, tasks such as collecting materials, learning the related software, preparing the initial design of
the presentations and getting feedback from the instructor were successfully performed.
At the end of the four week period the presentations were examined and the deadline was extended for two
weeks in order to let students to make the necessary modifications and to finalise the presentations. During that
time the role of the instructor was that of being a coach and facilitator.
Measurements
Measurements were based on observations, interviews and group portfolios. Also a test was performed in order
to assess the academic achievement of students. At the end of the semester, a performance-based assessment of
final presentations was made.
Findings and Discussion
At the end of the course, all groups were able to complete their presentations from which the following were
chosen as the most successful ones: “Substances”, “Holiday”, “Communication Devices”, “Population of our
Country” (Classroom Teaching); “Ants”, “Genetics”, “Our Mountains” and “Harmful Insects” (Science
Teaching); “Solar System”, “Indians”, “The Ancient Egypt”, “Earthquake” and “Seasons” (English Language
100
Teaching). Assessment of presentations and data that was collected by the instructor using measurements
mentioned above revealed a number of themes that have to be discussed.
Once students began working with the problem they are confronted with, they gradually identified what they
needed to learn. Most of them were engaged in a self-directed study to gather the information necessary to solve
the problem. Some visited Internet cafés, others met at the homes of their group-mates with computers. A couple
of students visited primary schools in order to obtain information from the school teachers. One group (the “Our
Mountains” group) interviewed a science teacher in preparation of their presentation and discovered that she got
into trouble with the section describing the mountains because all third graders tried to memorize the text in the
book. Another group (the “Communication Devices” group) visited the main post office of the city where there
is a museum and obtained a large amount of material about early predecessors of contemporary communication
devices in order to insert them into their presentation.
In a PBL model of instruction, the focus is on learners as constructors of their own knowledge within a context
similar to the context in which they would apply that knowledge. As expected, during the self-study period,
students taught both critically and creatively and monitored their own understanding. Most of the groups
prepared several additional slides including an explanation of initial preparations and characteristics of the
presentation imparted (see Table 2). It was observed that the students worked interactively rather than dividing
the workload on an individual basis. The intimacy of learning in small-group setting was determined to be part
of the pleasure that intrinsically exists in an integrated educational process.
During the meetings in the class, it was determined that the anxiety observed at the beginning of the course had
decreased rapidly and the desire and motivation to learn had pulled ahead. The reality that “Ants” and
“Substances” presentations were made by the students with the highest level of anxiety (Cigdem and Hasan Ali)
was an important sign of the success of the PBL. Smiling faces with bright eyes conveyed enthusiasm and
motivation of the students. At the end of the course, when the performance-based assessment was conducted,
most of the students tried to behave as teachers. Self-confidence was another important characteristic observed
during the assessment process. Most of the students expressed their feelings with such sentences as, “I can
perform better” and “I feel proud of myself”.
Table 2. Explanations about the presentation “Seasons”
Characteristics
Explanation
We worked on vocabulary teaching. We try to teach the new words by direct association. That
Purpose
means we try to make a relation between words and pictures. This method is being used since
the direct method. We tried to adapt it to our presentation. The pictures are very colourful. We
wanted to take the students’ attention to the presentation.
In our work, colours are very important. Each season is depicted in different colours. The
Colours
seasons and their colours are relevant. For example, white for winter, green for spring etc. In
each season, there are three parts. In the first part, we examine the nature; in the second,
people; and in the third, the fruits and vegetable in each season.
The verbs we used are in the present tense generally. The pictures are suitable for the verbs.
Pictures
We try to animate the verbs with the pictures. The words in the sentences (for example; dark,
thick clothes) can be related with the pictures: ski on the mountains, wear dark and thick
clothes, have a picnic, pick flowers, plants start growing, go swimming.
Graphics are used to make the students recognise the comparisons.
Graphics
The language is relatively simple; it is for lower grades. We want the students to understand
Language
the sentences.
Games could be designed in this unit to make use of interaction while learning.
Additional
Phase II
Rationale and Hypothesis
The findings from the Phase I have encouraged us to apply the PBL program for the students taking the same
course during the academic year 2002-2003. This time our aim was to investigate the differential effects of the
PBL and teacher-centred instruction on cognitive and a group of affective variables which have been coined
technophobia. We had to teach at least one class in conventional teacher-centred way. Our hypothesis was that
the class taught by utilizing the PBL would be more successful and experience a decrease in their anxiety level
more rapidly than with the class receiving teacher-centred instruction.
101
Method
Design
A quasi-experimental study was designed to evaluate the cognitive and affective outcomes of the PBL program.
Sample consisted of university students from the Division of Classroom Teaching (experimental group) and
from the Division of English Language Teaching (control group). Pre-test and post-test results for the
experimental and the control groups on cognitive and affective variables were compared. The differences in the
scores of the experimental and the control groups were also reported.
Participants
The experimental and control groups taking the same course consisted of 49 and 30 sophomore students
respectively. The SPSS-X for Windows programme was used to explore the demographics and the frequency of
responses. The experimental group consisted of 15 (31%) male and 34 (69%) female students. The mean age
was 20.5 with a standard deviation of 1.3. About 25% of the students possessed a PC. Twenty six percent of the
students reported they had taken a computer course before attending the university, and 74% had not.
According to the demographics, 68% of mothers had elementary school degree, while 32% have middle/high
school or university degree. Also 68% of mothers were housewives and 32% working somewhere. Forty percent
of fathers had elementary school degree, 60% graduated from middle/high school or university. Forty six percent
were workers and 54% were civil servants.
The control group consisted of 8 (27%) male and 22 (73%) female students. Mean age was 20 with a standard
deviation of 0.4. In this group, about 40% of the students possessed a PC. Thirty three percent of the students
reported that they had taken a computer course before attending the university and 67% had not. From the
control group demographics it was determined that 50% of mothers had elementary school degree and 50%
middle/high school or university degree. While 54% of them were housewives, 46% have worked somewhere.
Of fathers, 27% had elementary school degree and 73% graduated from middle/high school or university. The
amount of workers among fathers was 43% while the amount of civil servants was 57%. The experimental group
demonstrated low competency level on using computers.
Procedure
The same instructor taught in both classes. A PBL program was to be used in the experimental group. The
control group was given the usual teacher-centred instruction. The PBL program was employed by assigning a
group project to the class based on creativity designing a Power Point presentation for their future students as in
Phase I, to be completed in a month. As it was mentioned in previous sections the critical factor in the success
of PBL is the problem itself. It was presented as a Bloom’s Level 3 problem and it was similar to those students
would face in their future profession. This made the students even more enthusiastic than instructor and they
immediately realise that they had to make analysis, synthesis and evaluation.
Instruments and Measurements
After the instruction the students were given the same tests and statistical analyses were run to compare the
corresponding performances of the two groups on all the tests. The pre and post tests were also compared for the
two groups separately. Both classes were administered the achievement test, CARS, CTS and the GATCS prior
to the instruction. Of the two classes, the one with the lower mean score on the achievement pre-test (39.8) was
chosen to be the experimental group.
The achievement test consisted of 16 items based on the objectives of the course. For the reliability of the scale
Cronbach alpha was calculated to be 0.7775. Technophobia was measured by three scales, namely CARS, CTS
and GATCS (Rosen and Weil, 1992). The scales were translated into Turkish. CARS consists of 20 items.
Higher scores represent more situations for which the respondent feels anxious. The results of the item analysis
for the Turkish form were deemed to be satisfactory. For the internal consistency of the Turkish form, Cronbach
alpha was calculated to be 0.8975. The second scale was based on the translations of the items of the CTS.
Higher scores represent more positive cognition concerning computers. Cronbach alpha was 0.7487 for the
102
Turkish form. For attitudes towards computers, the GATCS, the Cronbach alpha correlation coefficient was
determined to be lower (0.6551) than the other two scales. But, this result paralleled other reliability studies
conducted by Rosen and Weil on GATCS (1992).
In order to gather adequate information concerning the change of students’ attitudes and their level of
knowledge, other data sources were used. Among these were, the observations made throughout the course, the
group interviews during the performance-based assessment of the presentations, and the final test at the end of
the semester examining all the learning objectives covered over the whole semester.
Results
Considering the sample sizes, non-parametric tests were used for comparing the experimental and control groups
as well as the pre-test and the post-test results. The two groups were compared in pre and post measures of
achievement (variable ACH) and affective variables (variables CARS, CTS and GATCS) by Independent
Samples T-Test. Results of the comparisons of the two groups are presented in Table 3.
Paired Samples T-Test was run for the experimental and control groups separately as a non-parametric
counterpart of the t-test for dependent groups. Result of the comparisons of the pre and the post test measures are
presented in Table 4.
When the achievement pre-test results were compared, they indicated a difference at .001 level between the two
groups. The control group mean score was significantly higher than the experimental group mean score prior to
the instruction. When the test was run for the achievement post test, no significant difference was determined
between the scores of the two groups.
The group given the PBL instruction was disadvantaged as to the cognitive entry behaviours in the beginning of
instruction. At the end of the instruction, the difference disappeared. When the pre and the post test measures of
achievement were compared for the two groups separately, the difference in the experimental group scores was
about 0.38 points and significant at 0.0001 level, whereas, the difference in control group was .08 points,
significant at 0.05 level. We concluded from these findings that PBL instruction applied in our experiment can
be claimed to be just as good as the teacher centred instruction for achievement, if not better.
Then tests were run for the affective variables. The results of the test for anxiety CARS pre-test given in Table 3
showed no significant differences in the anxiety levels of the two groups although the mean of the control group
was a little higher. Similarly, for the post test of anxiety no significant differences existed between the scores of
the two groups.
When the pre and post measures of computer anxiety were compared, no significant differences were observed,
although for both groups, the level of anxiety was lower after instruction. There were no significant differences
in the computer cognition measured by CTS in the two groups. But, when pre and post measures were analyzed
separately, the cognition of the group which was given the teacher-centred instruction was significantly more
positive after the instruction. The change in cognition was not significant after PBL instruction. The results of
the pre-test on GATCS showed no significant difference between the attitudes of the two groups as shown in
Table 3. The results of the post-test however showed a near significant difference in the attitudes of the
experimental and the control groups where the attitude of the control group was more positive.
The pre and post test differences in the attitudes of the control group was significant as observed in Table 4. The
attitude of the control group was more positive at the end of the instruction, while there was no change in the
attitudes of the PBL group.
Exp./
Contr.
Diff.
ACH
CARS
CTS
GATCS
Mean
Exp./Contr.
2.1645/2.4792
2.0959/1.9750
2.9684/2.9067
3.3418/3.4417
Table 3. Independent Samples t-test Results
PRETEST
St.Dev.
t
Sign.
Mean
Exp./Contr.
Exp./Contr.
0.6382/0.4803 -1.821
.04
2.5421/2.5563
0.6415/0.5663 .849
.398
2.0286/1.9667
0.4323/0.3352 .668
.506
2.9071/2.9367
0.5381/0.2736 -.943
.349
3.3051/3.4650
POSTTEST
St.Dev.
t
Exp./Contr.
0.3883/0.3059 -.170
0.5762/0.6469 .442
0.3798/0.3598 -.342
0.6679/0.2761 -1.245
Sign.
.886
.660
.733
.217
103
Pre/
Post
Diff.
ACH
CARS
CTS
GATCS
Table 4. Paired Samples t-test Results
EXPERIMENTAL
Mean
St.Dev.
t
Sign.
Mean
Pre/Post
Pre/Post
Pre/Post
2.1645/2.5421 0.7629/0.3883 -3.783 .0001 2.4792/2.5563
2.0959/2.0286 0.6415/0.5762 -.755
.454
1.9750/1.9667
2.9684/2.9071 0.4323/0.3798 1.051
.299
2.9067/2.9367
3.3418/3.3051 0.5381/0.6679 .481
.633
3.4417/3.4650
CONTROL
St.Dev.
t
Pre/Post
0.7157/0.3059 -.746
0.5663/0.6469 .114
0.3352/0.3598 -.555
0.2736/0.2761 -.502
Sign.
.461
.910
.583
.619
General Discussion and Conclusion
The motivational role of ICT in learning is widely recognised. How well prospective teachers learn to use ICT
will profoundly influence their effectiveness as future educators. This paper demonstrates how we employed the
PBL method in the course titled “Introduction to ICT”, providing the students at the Faculty of Education with a
lively and effective learning environment. As researchers, we are interested in determining whether or not such
an environment helps the students improve their academic performance. Another point of interest, as significant
as the preceding one, is to note the difference in the level of anxiety about present or future interaction with the
computers, if indeed there is any, in comparison with traditional teaching and learning environment.
In both phases of the study, it was rewarding to see the students so interested in and deriving pleasure from their
learning experience with the PBL. It seemed obvious that they were more motivated, working much harder with
this model compared with the students to which the traditional methods were applied.
As Milliken and Barnes (2002) states, there is a recognition that traditional, university teaching methods have
served the interests of lecturers and educational institutions more than they have served the interests of students.
It is possible to state that with the application of the PBL, the interests and learning needs of students are more
fully and sufficiently acknowledged and satisfied.
The attitude towards the computers were consistently determined to be an important variable in educational
computing research playing a pivotal role in student success in computer-related tasks. Hong, Ridzuan, and Kuek
(2003) stated that the students with positive attitudes towards the computers are also favorably disposed towards
them as an instrument of learning. On the other hand research in medical education indicate that the students
develop improved attitudes towards learning when received PBL training in advance (Lim, 2002; Liu, 2004). In
our study we observed that, in addition to pedagogical benefits obtained as a byproduct, the learning process was
driven by the problem initially posed: the learning process became interactive involving the students and the
teachers in unison.
By working in groups, the students developed collaborative work skills (Grabinger et al., 1997) and learned how
to work within complex environments. Although groups spent a great deal of time on PBL work usually
breaking the time limit, they took longer than expected to complete the presentations. Two factors seemed to
account for this situation from which the second one might be considered as a limitation for this study. Firstly,
they found that conducting Internet searches for specific information was quite different and generally needed
more time. Secondly, they had limited computer access out of class hours and this made the process of working
together slower.
As (Tynjälä, 1999) and Willis (2002) state, student learning is strongly steered by its assessment tool. Traditional
methods of assessment have often used only “objective” measures such as examinations and tests to assess
learning. Student’ perception of assessment requirements direct their approaches to learning and affect their
learning outcomes At the beginning of the semester students from the experimental group had negative feelings
about performance based assessment which is another key point in PBL, they found it encouraging by the end of
the course.
The level of knowledge the students reached were deemed to be satisfactory. During the time they spent
together, they also tried to teach each other the more complex features of the Power Point, most of them
expressing their hope of better performance in the near future. The evidence gathered from the whole process
indicate that the students were inclined to self-direction. Another promising development was observed in the
area of strong individual responsibility, further motivating some students to prepare presentations for other
courses, e.g. “The History of Art” and “Instruction Technologies”. Thus, the generic course could be said to have
achieved its aims.
104
Although our expectation was an evidence of a significant difference between the changes in attitudes of the two
groups in Phase II, we were encouraged to see, the students overcame, at least to some degree, their initial
feelings of technophobia. In our opinion, this particular result concerning Phase II could be explained by taking
into consideration the human factor: the instructor as the source of information. The instructor is still the
dominant side in the relationship between the elements that partake in the learning process where technology is
both a tool and the object. A different explanation can be derived from the fact stated by Liu (2004) that the
original discomfort with an initial degree of freedom as exhibited in most of the students during PBL, is an
additional source of anxiety.
As it was stated in Grabinger et al. (1997), creating autonomous, life-long learners, the PBL specifically teach
students to manage their own learning: identify learning needs, set learning objectives, select and employ
learning strategies, determine and use appropriate resources and assess the overall process. Such an approach
requires that we, the instructors, change also, from being the center of attention and the source of all knowledge
to the coaches and facilitators of the acquisition of knowledge sought for by the students.
Although the PBL is perceived as costly, for it requires a great investment of faculty time to provide tutors, we
believe that teaching through this method produces independent learners who can continue to learn on their own
in their chosen careers.
Acknowledgements
We wish to acknowledge the assistance of Çiğdem Selçukcan, Fidan Bayraktutan and Ceyda Cimilli in
conducting interviews with the students and making classroom observations for this study. Appreciation is
extended to Hulusi Gülseçen for his support during all stages of the study. We also thank the reviewers of
“Educational Technology and Society” who gave constructive and helpful comments.
References
Albion, P. R. (2003). PBL + IMM = PBL2: Problem-based learning and interactive multimedia development.
Journal of Technology and Teacher Education, 11 (2), 243-257.
Bartsch, R. A., & Cobern, K. M. (2003). Effectiveness of Power Point presentations in lectures. Computers &
Education, 41, 77-86.
Bloom, B. (1995). Taxonomy of Educational Objectives, New York, USA: McKay.
Dahlgren, M. A. (2003). PBL through the looking-glass: comparing applications in computer engineering,
psychology and physiotherapy. The International Journal of Engineering Education, 19 (5), 672-681.
De Graaf, E., & Kolmos, A. (2003). Characteristics of Problem-Based Learning. The International Journal of
Engineering Education, 19 (5), 657-662.
Driessen, E., & Vlueten, C. V. D. (2000). Matching student assessment to problem-based learning: lessons from
experience in a law faculty. Studies in Continuing Education, 22, 235-248.
Duch, B. (1997). Problems: A key
www.udel.edu/pbl/cte/spr96-phys.html.
factor
in
PBL,
retrieved
November
25,
2005
from
Erktin, E., & Gülseçen, S. (2001). Psychological factors affecting the use of IT in education. Education and
Science, 26 (121), 7-11.
Frank, M., Reich, N., & Humphreys, K. (2003). Respecting the human needs of students in the development of
e-learning. Computers & Education, 40, 57-70.
Fullan, M. (1991). The new meaning of educational change, London, England: Cassell Educational Ltd.
Gibson, K. E., & Silverberg, M. (2000). A two-year experience teaching computer literacy to first-year medical
students using skill-based cohorts. Bulletin of Medical Library Association, 88 (2), 157-164.
105
Grabinger, S., Dunlop, J., & Duffield, J. A. (1997). Rich environments for active learning in action: problem
based learning. ALT-J, 5 (2), 3-17.
Harland, T. (2002). Zoology Students’ Experiences of Collaborative Enquiry in Problem-based Learning.
Teaching in Higher Education, 7 (1), 6-17.
Hong, K.-S., Ridzun, A. A., & Kuek, M.-K. (2003). Students’ attitudes toward the use of the Internet for
learning: A study at a university in Malaysia. Educational Technology & Society, 6 (2), 45-49.
International Telecommunication Union (2003). Access Indicators for the Information Society, World
Telecommunication Development Report, Geneva (ISBN 92-61-10541-6).
Jonassen, D. (2000). Computers as Mindtools for Schools: Engaging Critical Thinking, Prentice Hall.
Jonassen, D. H., Howland, J., Moore, J. L., & Marra, R. M. (2003). Learning to solve problems with technology:
a constructivist perspective, New Jersey, USA: Pearson Education Inc.
Lim, K. (2002). Impacts of personal characteristics on computer attitude and academic users information system
satisfaction. Journal of Educational Computing Research, 26 (4), 395-406.
Liu, M. (2004). Examining the performance and attitudes of sixth graders during their use of a problem-based
hypermedia learning environment. Computers in Human Behaviour, 20, 357-379.
Maxwell, N. L., Bellisimo, Y., & Mergendoller, J. (2001). Problem Based Learning: modifying the medical
school model for teaching high school economics. The Social Studies, March/April, 73-78.
Milliken, J. & Barnes, L. P. (2002). Teaching and technology in higher education: student perceptions and
personal reflections. Computers & Education, 39, 223-235.
Orhun, E. (2002a). Computer-Based Cognitive Tools. In E. Orhun, & P. A. M. Kommers (Eds.), Information
and communication technologies in education: a focus on cognitive tools, İzmir, Türkiye: Ege University Press,
17-26.
Orhun, E. (2002b). Computer Innovation in Education in Turkey: A Study of the Initiation Conditions in
Secondary Schools in İzmir. In E. Orhun, & P.A.M. Kommers (Eds.), Information and communication
technologies in education: a focus on cognitive tools, İzmir, Türkiye: Ege University Press, 343-375.
Rosen, L. D., & Weil, M. M. (1992). Measuring Technophobia, A Manual for the Administration and Scoring of
the Computer Anxiety Rating Scale, Computer Thoughts Survey and the General Attitudes towards Computers
Scale, California State University Dominguez Hills, Computerphobia Reduction Program.
Saunders, W. L. (1992). The Constructivist Perspective: Implications and Teaching Strategies for Science.
School Science and Mathematics, 92 (3), 136-141.
Savery, J. R., & Duffy, T. M. (1995). Problem Based Learning: An instructional model and its constructivist
framework. Educational Technology, 35, 31-38.
Tynjälä, P. (1999). Toward Expert Knowledge? A Comparison Between a Constructivist and a Traditional
Learning Environment in the University. International Journal of Educational Research, 31, 357-442.
Tse, W. L., & Chan, W. L. (2003). Application of problem based learning in an engineering course. The
International Journal of Engineering Education, 19 (5), 747-753.
Weil, M. M., & Rosen, L. D. (1995). The psychological impact of technology from a global perspective: a study
of technological sophistication and technophobia in university students from twenty three countries. Computers
in Human Behaviour, 11 (1), 95-133.
Willis, S. C., Jones, A., Bundy, C., Burdett, C. R., Whitehouse, C. R., & O’Neill, P. A. (2002). Small-group
work and assessment in a PBL curriculum: a qualitative and quantitative evaluation of student perception of the
process of working in small groups and its assessment. Medical Teacher, 24 (5), 495-501.
Yıldırım, I. S. & Kiraz, E. (1996). Distance does not really matter: a case study. Proceedings of the Türkiye First
International Distance Education Symposium, Ankara: UZEV, 673-682.
106
Zheng, R. & Zhou, B. (2006). Recency Effect on Problem Solving in Interactive Multimedia Learning. Educational
Recency Effect on Problem Solving in Interactive Multimedia Learning
Robert Zheng
Assistant Professor, Department of Psychological Studies in Education (PSE)
College of Education, Temple University, Pennsylvania, USA
[email protected]
Bei Zhou
Department of Curriculum, Instruction, and Technology in Education (CITE), College of Education
Temple University, Ritter Hall 472, 1301 Cecil B Moore Ave, Philadelphia, PA 19122, USA
Tel: +1 215 204 6372
[email protected]
ABSTRACT
This study investigated the impact of recency effect on multiple rule-based problem solving in an
interactive multimedia environment. Forty-five college students were recruited and assigned to two groups:
synchronized and unsynchronized interactive multimedia groups based on their spatial ability score. Results
show that students in the synchronized interactive multimedia group outperformed their counterparts in the
unsynchronized interactive multimedia group in terms of response time and test scores. Results also
indicated that low spatial ability learners in the synchronized interactive multimedia showed an
improvement in problem solving.
Keywords
Multimedia, Problem solving, Recency effect, Rule-based reasoning, Working memory
Introduction
Problem-solving skill is highly valued. In the last five decades, many theorists and educational institutions have
placed a heavy emphasis on this ability. For example, the movement of “discovery learning” (e.g., Bruner, 1961)
was spawned, at least in part, by the perceived importance of fostering problem-solving skills. This emphasis on
problem solving was not associated, however, with the knowledge of cognitive resources involved in the
problem solving process. That is, it focused on procedures of problem solving rather than investigating the
relationship between procedures of problem solving and cognitive resources that affect such procedures (Sweller
& Low, 1992; Hanley, 1987). In the last twenty years, this state of affairs has begun to change with our
knowledge of relevant mechanism (e.g., working memory, cognitive load, etc.) increasing markedly. This study
investigated the recency effect – a phenomenon in working memory that affects learners’ holding of information
during problem solving - and how such phenomenon may affect learners’ multiple rule-based reasoning in
particular, and problem solving skills in general.
Literature Review
Working Memory and Cognitive Load
Cognitive scientists believe that the input information such as auditory, visual, and kinesthetic, etc. is processed
through a temporary storage before it gets coded into the long term memory (Baddeley & Logie, 1992; Logie,
1995). This temporary storage, also called the working memory, comprises two major components. One is the
executive controlling mechanism that is believed to be related to cognitive activities such as reasoning and
problem-solving. The other is the visuo-spatial sketchpad (VSS) that is believed to process and manipulate
visuo-spatial images (Logie, 1995; Pearson & Logie, 2000). For example, the ability to mentally manipulate 3D
images by rotating them in mind is largely determined by the VSS function. The two components are closely
related and interacted with each other in the process of problem solving (Bollaert, 2000; Mayer & Moreno,
2003).
Studies show that the working memory is very limited in both duration and capacity. Van Merrienboer and
Sweller (2005) observe that the working memory stores about seven elements but normally operates on only two
or three elements. They also find that the working memory can deal with information “for no more than a few
seconds with almost all information lost after about 20 seconds unless it is refreshed by rehearsal” (p. 148).
When the working memory becomes overloaded with information, learning can be adversely affected (Paas,
Tuovinen, Tabbers, & Gerven, 2003; Sweller & Chandler, 1994; Marcus, Cooper, & Sweller, 1996). Sweller and
107
Chandler (1991) studied the relationship between the cognitive load and the learning effectiveness and found that
learning became improved when the extraneous cognitive load was reduced. According to Cognitive Load
Theory (CLT), three types of cognitive load exist: intrinsic load, extraneous or ineffective load, and germane or
effective load. The intrinsic cognitive load refers to cognitive load that is induced by the structure and
complexity of the instructional material. Usually, teachers or instructional designers can do little to influence the
intrinsic cognitive load. The extraneous cognitive load is referred to the cognitive load caused by the format and
manner in which information is presented. For example, teachers may unwittingly increase learner’s extraneous
cognitive load by presenting materials that “require students to mentally integrate mutually referring, disparate
sources of information” (Sweller et al., 1991, p.353). Finally, the germane cognitive load refers to cognitive load
that is induced by learners’ efforts to process and comprehend the material. The goal of CLT is to increase this
type of cognitive load so that the learner can have more cognitive resources available to solve problems
(Brunken, Plass, & Leutner, 2003; Marcus, et al., 1996).
Cognitive Load Theory has been recently introduced into multimedia studies where researchers try to find out
whether the use of multimedia can improve learning by reducing extraneous cognitive load. Tabbers, Martens
and Van Merrienboer (2004) studied the effects of modality and cueing on learning in terms of cognitive load
theory and found that adding visual cues to the pictures enhanced learners’ retention scores. Mayer and his
colleagues (Mayer, 1997; Mayer et al., 2003) conducted a series of studies on the effect of text and image on
problem solving. They noticed that when learners were presented with synchronized texts and images, they were
able to better recall and answer transfer questions. However when learners were presented with unsynchronized
instructional material, i.e., the text was presented first followed by an image (a filled delay phenomenon),
learners’ ability to recall and answer transfer questions decreased. Mayer (1997) concluded that the synchronized
multimedia could alleviate the extraneous cognitive load and provide extra cognitive resources for problem
solving whereas the unsynchronized multimedia could cause cognitive overload, thus decreased performance in
learning. Although the CLT theory provides insights on problem solving in terms of cognitive load, cautions
must be taken in interpreting learners’ ability to solve problems. Studies show that factors such as recency effect,
problem types, etc. are related to learners’ problem-solving skills (Delisle, 1997; Logie, 1995).
Recency Effect and Cognitive Load
While the process of problem solving draws on resources from both long-term and short-term memories, it is
believed to rely heavily on the working memory for the working information during the problem-solving process
(Baddeley et al., 1992; Logie, 1995). When this temporary holding of information in the working memory is
interrupted, the ability to solve problems can be affected. Capitani, Della Sala, Logie, and Spinner (as cited in
Logie, 1995) conducted a study on the working memory. They found a high recall by subjects immediately after
the items have been presented. However if there was a filled delay before the recall was required, only the first
few items on the list could be recalled. Logie (1995) described the former phenomenon as the recency effect and
the latter as the primary effect. He believed that “recency reflected the operation of a short-term or primary
memory system” (p. 5) which was critical to problem solving process. It is essential to distinguish between
cognitive load and recency effect since both involve cognitive resources in working memory pertaining to
learning. The notion of cognitive load refers to the load imposed on the learner’s working memory while
performing a particular task whereas the recency effect refers to the information, specifically the most recent
information that can be recalled within working memory. The cognitive load study is focused on working
memory architecture and its limitations relating to the design of instruction (Van Merrienboer et al., 2005; Paas
et al., 2003; Tabbers et al., 2004). The recency effect is, however, focused on the state of recalling and
maintaining much needed information during problem solving process.
Of particular interest to researchers is the issue of effective time period in recency effect. Studies on effective
time period in recency effect have so far produced mixed results. For example, studies by Posner, Boies,
Eichelman and Taylor (1969, cited from Baddeley, 1997) suggest that the information resulting from a visual
trace has a 2-second decay rate. However, Glanzer and Cunitz (1966, cited from Haberlandt, 1999) observed that
the recency effect was wiped out when recall was delayed by 30 seconds. Contrary to the findings of Posner et
al. (1969) and Glanzer et al. (1966), Bjork and Whitten (1974, cited from Haberlandt, 1999) showed that the
recency effect could survive a delay. They had subjects learn a word list in a free-recall experiment. After a 30second interval of backward counting that should have resulted in a loss of the most recent items from the shortterm memory, they found the subjects were still able to recall the most recent items in the list. Obviously, the
inconsistency in the above findings raises the question of whether the effective time period should be used to
measure the impact of recency effect on problem solving. Richardson (1996) states that “working memory is not
a general system with unitary capacity. Rather, the capacity of working memory will vary as a function of how
efficient the individual is at the specific processes demanded by the task to which working memory is being
108
applied” (p. 12) which means the measurement of the impact of recency effect on problem solving should focus
on the effectiveness that learners make use of the limited working memory capacity in information processing
(Perfetti & Goldman, 1976). In other words, the recency effect should be measured in terms of learner efficiency
in problem solving, that is, the ability to achieve optimal results in problem solving with minimum amount of
time possible.
Problem Types
Delisle (1997) states that problem types are related to thinking procedures in problem-solving process. For
example, causal relationship problems require a linear thinking procedure that has a strong linear direction
emphasizing the cause and effect whereas multiple rule-based problems involve simultaneously weighing several
conditions/rules in mind in order to make a decision (Frye, Zelazo, & Palfai, 1995; Price & Yates, 1995). Zheng,
Miller, Snelbecker, & Cohen (2005) assert that different types of problem may require different levels of
working information in problem solving process. For instance, the multiple rule-based problem solving may
require more working information than does causal relationship problem solving or single rule-based problem
solving. The single rule-based problem solving, according to Frye et al. (1995), requires a straight-forward
deductive thinking such as applying the rule of card sorting to the action of sorting a deck of cards whereas the
multiple rule-based problem solving involves a more complex, nonlinear thinking where the learner reaches a
solution by engaging in a series of cognitive thinking activities such as analyzing, synthesizing, evaluating, and
so forth while holding several conditions and rules in mind within a short time framework provided by the
working memory (Johnson, Boyd, & Magnani, 1994; Price et al., 1995). Obviously, multiple rule-based problem
solving is likely to increase intrinsic cognitive load more than other two types of problem solving.
Studies showed that the interactive multimedia can enhance students’ recalling and maintaining of working
information in multiple rule-based problem solving (Zheng et al., 2005). According to the dual coding theory
(Paivio, 1986), learning is more effective with two sensory channels (i.e., visual and verbal) than one channel
alone. When information is processed through multiple sensory channels, learners’ ability to hold such
information is improved (Zheng et al., 2005). However, this ability is also determined by the types of problem
and thinking procedures during the problem solving process.
Learner Characteristics
One of the issues that frequently surfaced in multimedia problem-solving research is whether demographic
factors such as age, education, ethnicity, etc. would affect learners’ problem solving (Forcier & Descy, 2005).
Hall and Cooper (1991) report a gender difference in computer use. Passig and Levin (2000) concur that gender
preferences exist in multimedia related problem solving. Another issue of interest is individual differences in
multimedia problem solving. Fink and Neubaurer (2005) observed that individuals differ in speed of information
processing and working memory. Mayer and Sims (1994) conducted a study on spatial ability and problem
solving with multimedia and concluded that the spatial ability was critical to problem solving and “appears to
enhance coordinated visual and verbal instruction” (p. 399).
Hypotheses
Based on the above discussion, the following hypotheses were proposed that formed the basis for this study:
Hypothesis 1
Participants in the synchronized interactive multimedia group will outperform their counterparts in the
unsynchronized interactive multimedia group in multiple rule-based problem-solving.
Hypothesis 2
Different interactive multimedia such as synchronized and unsynchronized multimedia can significantly affect
learners’ spatial ability and their performance relating to such ability.
109
Hypothesis 3
Learners’ ability to solve problems can be affected by demographic factors such as education, ethnicity, gender,
and hobbies.
Methodology
Participants & Design
Participants consisted of 45 students recruited from a large comprehensive research university on the east coast
of the United States in the fall 2004. Of 45 participants, 89% (n = 40) were undergraduate teacher education
majors who received course credit for participating in the study and 11% (n = 5) were graduate students who
volunteered to participate in this study. Approximately 65 % (n = 29) of the participants were Caucasian, 22% (n
= 10) were African-American, and 13% (n = 6) were Asian or Asian American. The average age of the
participants was 21 with a range from 19 to 31.
Two levels of interactive multimedia (synchronized vs. unsynchronized) were crossed with two levels of spatial
ability (high vs. low) to form a 2 x 2 between subjects factorial design. Participants were blocked by spatial
ability and then randomly assigned to one of the multimedia groups. The independent variables included
multimedia learning (synchronized vs. unsynchronized) and spatial ability (high vs. low) with two dependent
variables: response time and test scores. In this study, demographic information such as age, education, ethnicity,
and hobbies was collected and served as covariates in MANCOVA analysis. All statistical tests were performed
with alpha at .05.
Measures
Two instruments were used to assess students’ spatial ability and problem-solving skills.
Guilford-Zimmerman Aptitude Survey Test 5: Spatial Orientation (Guilford-Zimmerman, 1956). The GuilfordZimmerman spatial orientation test requires participants to observe the position of the bow of a boat relative to
the horizon and then to correctly identify the change in the boat’s orientation by the change in horizon relative to
the bow. The purpose of the Guilford-Zimmerman test is to assess participant’s ability to mentally maneuver
objects in terms of the special relationship which is consistent with the reasoning tasks set up in this study, that
is, the participant came up with a solution by maneuvering the figures and determining the spatial relationship
between the figures based on the conditions and rules set in the tasks. The Guildford-Zimmerman spatial
orientation test has reported a high reliability (alpha = .88) (Price & Eliot, 1975).
Interactive Multimedia Problem-Solving Test. The interactive multimedia problem-solving test was developed
by the first author using Flash MX and Microsoft Active Server Page. The test comprised five sub-tests that
included air traffic control, Tower of Hanoi, sailing boat, taking pictures, and office inspection. Sub-test 1 Air
Traffic Control had a set of conditions that restricted the order and parking positions of airplanes. The subject
had to consider all the conditions and decided which flight would park at which gate without violating the
conditions (Figure 1). Sub-test 2 Tower of Hanoi had a set of rules restricting the movement of disks. The
subject was required to move three disks from one stack to another without violating the rules. Sub-test 3 Sailing
Boat described six boats anchoring at four different directions. The subject was to find out the spatial
relationship between the boats based on the conditions given. Sub-test 4 Taking Pictures had a set of rules that
determined who could stand next to each other in the line. The subject was required to come up with a list of
persons who would take the pictures in an order that did not violate the rules set by the task. Finally, Sub-test 5
Office Inspection was about office inspections by five people who conflicted with each other in terms of the
schedule and the order of inspection. The subject had to consider these conflicting conditions and decided who
got assigned to inspect which office without violating the conditions/rules. For each sub-test, there was a
problem (text) and an interactive multimedia with which the subject could move the figures around to help solve
the problem. Two versions of tests were created: synchronized and unsynchronized. The synchronized
interactive multimedia test displayed both text and interactive multimedia at the same time on the same page
whereas the unsynchronized interactive multimedia test separated the text from the interactive multimedia by
presenting the text first followed by an interactive multimedia.
110
The problem-solving test consisted of five sub-tests, each contained three parts: the problem with the text format,
interactive multimedia, and two problem-solving questions that measured participants’ multiple rule-based
reasoning skills. Each sub-test had a timer recording the start and the end of the response time (Figure 1).
Figure 1. Sample of Multiple Rule-Based Problem-Solving Tasks
To ensure the content validity, the instrument was reviewed by a panel of instructors and a selected group of
graduate students whose feedback was carefully reviewed and changes were made based on the feedback. Next,
a pilot test was conducted to a group of undergraduates (n = 10). Further changes were made based on the results
of the pilot test and comments from instructors. A reliability analysis was done on test items. The Pearson
correlation analysis showed that nine items had a significant correlation between sub-items (Table 1). The item
reliability analysis using Cronbach alpha showed a moderate coefficient of .71.
Table 1. Pearson Correlation Matrix
111
Procedure
Using a blocked random sampling procedure, participants were first given the Guilford-Zimmerman Aptitude
Survey Test 5: Spatial Orientation (Guilford & Zimmerman, 1956). The middle value of the distribution (median
= 7.25) was determined and chosen as the cut score for defining subjects’ high and low spatial ability. Subjects
were then blocked into high and low spatial ability groups based on their spatial ability test score. Subjects were
randomly drawn from each of the blocked groups (i.e., high and low spatial ability) to form two separate groups:
the synchronized interactive multimedia group and the unsynchronized interactive multimedia group. A paired
samples t-test was performed on the synchronized and unsynchronized multimedia groups. The result indicated
no significant difference between two groups, t (1, 21) = 1.00 (2-tailed), p = .329, ns.
Participants were given a test number and assigned to one of the interactive multimedia groups based on the
result of randomization. They were provided a URL to take the problem solving test online in a computer
classroom proctored by the investigator. The participants were required to fill out a demographic information
sheet online which included age, education, ethnicity, hobbies, etc. After completing the demographic
information, participants began to take the problem solving test. Participants were first presented a problem that
included a scenario with some conditions and rules. They were then told to use the interactive multimedia to help
solve the problem by moving images or figures around to simulate various solutions until the correct solution
was reached. The total score equaled 10 points. If an error was made by the participant, a point would be
deducted from the total score. All participants were given a consent form to sign for their participation in the
study.
Results
Descriptive Statistics
The means and standard deviations are reported in Table 2.
Table 2. Descriptive Statistics for Response Time and Test Scores
With regard to the response time the synchronized group (Mean sync = 16.87) spent less time than the
unsynchronized group (Mean unsync = 18.89) in problem solving test. Further, high spatial ability subjects in the
synchronized group (Mean sync = 15.89) spent less response time than their counterparts in the unsynchronized
group (Mean unsync = 18.64). However, low spatial ability subjects in the synchronized group (Mean sync = 17.95)
112
seemed to spend more response time than their counterparts in the unsynchronized group (Mean unsync = 17.25).
With regard to test scores the synchronized group (Mean sync = 5.43) scored higher on problem-solving test than
the unsynchronized group (Mean unsync = 4.41). Both high and low spatial ability subjects in the synchronized
group outperformed their counterparts in problem-solving test.
MANCOVA Tests
A multivariate analysis of covariance (MANOVA) was conducted with spatial ability and interactive multimedia
group as independent variables, the response time and the test score as dependent variables, and education,
gender, ethnicity, and hobby as covariates. The Wilks’ Lambda estimate was used to determine the main effects.
The results indicated that there was a main effect for multimedia group (Wilks’ Lambda = 5.642; p = .007) and
for spatial ability (Wilks’ Lambda = 5.397; p = .008). There was also an overall interaction between the
multimedia group and the spatial ability (Wilks’ Lambda = 6.480; p = .004). The covariance analysis indicated
that none of the covariates were significant: education (Wilk’s Lambda = 1.824, p = .176), gender (Wilk’s
Lambda = 2.167, p = .129), Ethnicity (Wilk’s Lambda = .736, p = .486), and hobby (Wilk’s Lambda = .944, p =
.398) (Table 3).
Table 3. MANOVA Tests
The between-subjects analysis showed that there was a significant difference between the synchronized and the
unsynchronized groups in terms of the response time (F (1, 44) = 4.345; p = .043) and test scores (F (1, 44) = 6.187; p
= .017). There was a significant difference between the high and the low spatial ability people for test scores (F
(1, 44) = 10.091; p = .003) but no significance was found for the response time (F (1, 44) = .482; p = .491). With
regard to the multimedia group and spatial ability interaction, there was an overall interaction for the response
time (F (1, 44) = 11.979; p = .001). But no interaction was found for test scores (F (1, 44) = .679; p = .415) (Table 4).
Efficiency Scores
The efficiency score is the total test score divided by the total response time in seconds. The total test score was
calculated by adding up the correct scores from each task and the total response time was calculated by summing
up the response time of all tasks. An efficient problem solver is defined as a person who achieved higher test
113
scores with less response time. Descriptive data (Table 5) and the results of univariate analysis of variance
(Table 6) are reported as follows.
Table 4. Tests of Between-Subjects Effects
Table 5. Descriptive Statistics for Efficiency Scores
Table 6. Univariate Analysis of Variance
114
Discussions
The discussion of the results will focus on three hypotheses proposed earlier with an emphasis on (a) the impact
of recency effect on problem-solving, (b) multimedia and spatial ability, and (c) demographic factors as
covariates.
Hypothesis 1
Hypothesis 1 tried to investigate whether the use of different interactive multimedia (synchronized vs.
unsynchronized) would affect learners’ performance in problem solving. The results of the study showed a main
effect for multimedia groups as well as an overall interaction between multimedia groups and the spatial ability,
which means learners’ performance in problem solving was affected by the type of media employed. This was
supported by the mean difference in test scores between the synchronized group (Mean sync = 5.43) and the
unsynchronized group (Mean unsync = 4.41) (Table 1). The between-subjects tests indicated a statistical
significance between two groups (F (1, 44) = 6.187; p < .05) (Table 4). However, when learners’ performance was
measured by the response time instead of test scores, low spatial ability subjects in the synchronized group
performed less well than high spatial ability subjects in the unsynchronized group (Table 2). There was a
marginal significance between two groups (synchronized vs. unsynchronized) in terms of response time (F (1, 44)
= 4.345; p = .043). Nonetheless, such difference was compromised by a small effect size (η2 = .096) and weak
statistical power (.530) (Table 4), which means the difference may have little practical significance in reality.
Results also show that when provided with the synchronized interactive multimedia, higher spatial ability
learners had a higher efficiency score than their counterparts in the unsynchronized interactive multimedia
(Table 5). There was a significant difference between multimedia groups (F (1, 44) = 11.99, p < .05) (Table 6).
Since the efficiency score indicates learners’ ability to solve problems in terms of the ratio between total test
scores and total response time (in seconds), it would be reasonable to assume that learners who solved more
problems with less response time had more cognitive resources during problem solving process. It would also
imply that such learners were more effective in recalling and maintaining critical working information while
solving problems since recalling and maintaining working information is critical to problem solving (Loggie,
1995).
The results of MANCOVA and univariate analysis of variance have consistently shown that the synchronized
interactive multimedia facilitated learners’ ability to solve problems due to a prompt recall and retrieval of
working information whereas the unsynchronized interactive multimedia decreased the learners’ ability to solve
problems due to a filled delay phenomenon caused by scrolling back and forth between pages. Evidently, the
synchronized interactive multimedia facilitates the recall of on-demand working information – a recency effect and enables learners to solve problems more efficiently in a short time framework provided by the working
memory. The findings of this study concurred with previous studies (e.g., Mayer et al., 1994; 2003) that the
synchronized multimedia enhanced students’ problem solving whereas the unsynchronized multimedia could
affect students’ ability to recall and maintain working information in the working memory due to a filled delay
phenomenon.
Hypothesis 2
Hypothesis 2 tried to find out whether different interactive multimedia such as synchronized and unsynchronized
multimedia would affect learners’ spatial ability and their performance. The MANCOVA analysis revealed an
interaction between multimedia group and spatial ability (Wilks’ Lambda = 6.480; p = .004). Overall, high and
low spatial ability subjects in the synchronized group outperformed their counterparts in the unsynchronized
group (F (1, 44) = 10.091, p < .05). It should be noted that with synchronized interactive multimedia low spatial
ability subjects outperformed their counterparts and did almost as well as the high spatial ability subjects in the
unsynchronized group. This is perhaps for low spatial learners who are known for linear and abstract thinking
(McGrew & Flanagan, 1998), multiple sensory inputs from the synchronized interactive multimedia can provide
extra cognitive resources that enable them to recall and maintain on-demand working information while
engaging in multiple rule-based reasoning. This suggests that media attributes like synchronized interactive
multimedia can compensate learner’s deficit in spatial ability such as visualization (Reiser, 1994; Salomon,
1979).
115
The efficiency analysis showed that the synchronized interactive multimedia facilitated learning for both high
and low spatial ability learners. High spatial ability learners in the synchronized group (Mean = .387)
outperformed all three groups including low spatial ability learners in the synchronized group, high and low
spatial ability learners in the unsynchronized group. The mean score of low spatial ability learners in the
synchronized group (Mean = .260) was higher than its counterparts (Mean = .226) and almost the same as the
high spatial ability learners in the unsynchronized group (Mean = .262) (Table 5). There was a significant
difference between spatial groups in terms of efficiency (F (1, 44) = 12.60, p < .05). This suggests that with the
synchronized interactive multimedia both high and low spatial ability learners were able to perform well in
problem solving.
Hypothesis 3
Hypothesis 3 tried to determine if demographic factors such as education, ethnicity, gender, and hobbies would
affect learners’ ability to solve problems. Although studies (Forcier et al., 2005; Hall et al., 1991; Passig et al.,
2000) showed that demographic factors could influence learners’ attitude, motivation, and their way of learning,
the results of this study showed that none of the demographic factors that is, education, ethnicity, gender, and
hobbies, were significant (Table 3), which suggested that in this particular study main effects for multimedia
group and spatial ability were accounted for by the differences between multimedia groups, not affected by any
of the covariates mentioned above.
Conclusion
As it was mentioned elsewhere in this paper, the ability to hold information in one’s working memory is critical
to multiple rule-based reasoning problem solving. This study indicated that the recency effect, which is related to
learners’ ability to recall and maintain working information in working memory, is related to the media
employed in problem solving. The analysis of efficiency scores revealed that the synchronized interactive
multimedia could facilitate recalling and maintaining on-demand information in working memory - a recency
effect, thus enabled learners to solve problems more efficiently whereas the unsynchronized interactive
multimedia hinders immediate information retrieval due to a filled delay phenomenon. This was supported by
the observation we made in which subjects in the synchronized group were more focused on problem solving
whereas subjects in the unsynchronized group frequently scrolled back and forth between the pages trying to
update the information in working memory while working on the problems.
This study concurred with previous studies that spatial ability is related to multimedia problem solving (Mayer et
al., 1994). However, it went further to conclude that the synchronized interactive multimedia could compensate
low spatial ability learners for lacking visualization in problem solving. Contrary to the findings of previous
studies (Forcier et al., 2005; Hall et al., 1991; Passig et al., 2000; Fink et al., 2005), this study did not find any
significant relationship between demographic factors and learners’ ability to problem-solving.
Overall, we can derive some useful theoretical and practical implications from this study. On the theoretical side,
the results have extended the exiting studies on multimedia such as cognitive load theory and called attention to
an important cognitive phenomenon – recency effect - in the interactive multimedia problem solving. The study
provides a better understanding of the relationship between spatial ability and media as well as the relationship
between recency effect and multimedia presentation mode, which promotes further research on the constructs of
working memory, complex reasoning such as multiple rule-based reasoning, and interactive multimedia. On the
practical side, our work reveals that differences in multimedia design can result in different cognitive
consequences. Therefore, teachers and other professional educators need to be aware of such issues as media
attributes, problem types, and learner characteristics in the process of designing multimedia problem solving,
especially in designing multiple rule-based problems.
The importance of teaching problem solving to students has been widely recognized by teachers, administrators,
and other educational stakeholders. The advent of computer technology, particularly multimedia learning, has
changed the landscape of problem-solving instruction. Computer-based problem simulations, for example, begin
to replace the traditional paper and pencil approach with more vivid, interactive approach that provides both
auditory and visual information to problem solving. The psychological and cognitive benefits of using
multimedia to teach problem solving are palpable: the multimedia motivates students to learn, promotes deep
understanding, and engages them in problem solving (Fulford, 2001; Mayer et al., 2003; Rieber & Hannafin,
1988). This study has provided initial results on the impact of recency effect on multimedia problem solving.
116
Further research is needed to investigate the relationship between cognitive load and recency effect and perhaps
include cognitive load as a possible covariate to determine the impact of recency effect on problem solving.
Research in the future should include a larger, more diverse population in terms of ethnicity, age, education,
social and economic status. It is suggested that a broader research agenda is needed to address the whole range of
issues in the study of recency effect that includes the relationship between recency effect and its related cognitive
constructs such as learner aptitudes, learning styles, mental synthesis, etc. in the context of working memory and
multimedia learning.
References
Baddeley, A. D., & Logie, R. H. (1992). Auditory imagery and working memory. In Reisberg, D. (Ed.), Auditory
imagery, Hillsdale, NJ, USA: Lawrence Erlbaum Associates, 179-197.
Baddeley, A. D. (1997). Human memory: Theory and practice, Hove, UK: Psychology Press.
Bollaert, M. (2000). A connectionist model of the processes involved in generating and exploring visual mental
images In Nuallian, S. O. (Ed.), Spatial cognition: Foundations and applications, Amsterdam/Philadelphia: Joint
Benjamins Publishing, 329-346.
Bruner, J. (1961). The act of discovery. Harvard Educational Review, 31 (1), 21-32.
Brunken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning.
Educational Psychologist, 38 (1), 53-61.
Delisle, R. (1997). How to use problem-based learning in the classroom, Association for Supervision and
Curriculum Development: Alexandria, VA, USA.
Fink, A., & Neubauer, A. C. (2005). Individual differences in time estimation related to cognitive ability, speed
of information processing and working memory. Intelligence, 33 (1), 5-26.
Forcier, R., & Descy, D. (2005). The computer as an educational tool: Productivity and problem solving (4th
Ed.), Upper Saddle River, NJ, USA: Pearson-Merrill/Prentice Hall.
Frye, D., Zelazo, P. D., & Palfai, T. (1995). Theory of mind and rule-based reasoning. Cognitive Development,
10 (4), 483-527.
Fulford, C. P. (2001). A model of cognitive speed. International Journal of Instructional Media, 28 (1), 31-42.
Guilford, J. P., & Zimmerman, W. S. (1956). Guilford-Zimmerman aptitude survey, Beverly Hills, CA, USA:
Sheridan.
Haberlandt, K. (1999). Human memory: Exploration and application, Boston, MA, USA: Allyn and Bacon.
Hall, J., & Cooper, J. (1991). Gender, experience, and attributions to the computer. Journal of Educational
Computing Researcher, 7 (1), 51-60.
Hanley, G. L. (1987). The origin of information and its effect on problem solving. Paper presented at the Annual
Meeting of American Psychological Association, New York, USA, ERIC Document Reproduction Service No.
ED291791.
Johnson, J. T., Boyd, K. R., & Magnani, P. S. (1994). Causal reasoning in the attribution of rare and common
events. Journal of Personality and Social Psychology, 66 (2), 229-242.
Logie, R. H. (1995). Visuo-spatial working memory, Hove, UK: Lawrence Erlbaum Associates.
Marcus, N., Cooper, M., & Sweller, J. (1996). Understanding instructions. Journal of Educational Psychology,
88 (1), 49-63.
Mayer, R. E. (1997). Multimedia learning: Are we asking the right questions? Educational Psychologist, 32 (1),
1-19.
117
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational
Psychologist, 38 (1), 43-52.
Mayer, R. E., & Sims, V. K. (1994). For whom is a picture worth a thousand words? Extensions of a dual-coding
theory of multimedia learning. Journal of Educational Psychology, 86 (3), 389-401.
McGrew, K. S., & Flanagan, D. P. (1998). The intelligence test desk reference (ITDR): Gf-Gc cross-battery
assessment, Needham Heights, MA, USA: Allyn and Bacon.
Paas, F., Tuovinen, J. E., Tabbers, H., & Gerven, P. W. M. (2003). Cognitive load measurement as a means to
advance cognitive load theory. Educational Psychologist, 38 (1), 63-71.
Paivio, A. (1986). Mental representations: A dual coding approach, Oxford, England: Oxford University Press.
Passig, D., & Levin, H. (2000). Gender preferences for multimedia interfaces. Journal of Computer Assisted
Learning, 16 (1), 64-71.
Pearson, D. G., & Logie, R. H. (2000). Working memory and mental synthesis: A dual task approach. In
Nuallian, S. O. (Ed.), Spatial cognition: Foundations and applications, Amsterdam/Philadelphia: Joint
Benjamins Publishing, 347-359.
Perfetti, C. A., & Goldman, S. R. (1976). Discourse memory and reading comprehension skill. Journal of Verbal
Learning and Verbal Behavior, 15 (1), 33-42.
Price, L., & Eliot, J. (1975). Convergent and discriminant validities of two sets of measures of spatial orientation
and visualization. Educational and Psychological Measurement, 35 (4), 975-977.
Price, P. C., & Yates, J. F. (1995). Associative and rule-based accounts of cue interaction in contingency
judgment. Journal of Experimental Psychology, 21 (6), 1639-1655.
Reiser, R. (1994). Clark’s invitation to the dance: An instructional designer’s response. Educational Technology
Research and Development, 42 (2), 45-48.
Richardson, J. T. E. (1996). Evolving concepts of working memory. In Richardson, J. T. E., Engle, R. W.,
Hasher, L., Loggie, R. H., Stoltzfus, E. R. & Zacks, R. T. (Eds.), Working memory and human cognition,
Oxford, England: Oxford University Press, 3-30.
Rieber, L. P., & Hannafin, M. J. (1988). Effects of textual and animated orienting activities and practice on
learning from computer-based instruction. Computers in the Schools, 5 (1-2), 77-89.
Salomon, G. (1979). Interaction of media, cognition, and learning, San Francisco, CA, USA: Jossey-Bass.
Sweller, J., & Chandler, P. (1991). Evidence for cognitive load theory. Cognition and Instruction, 8 (4), 351362.
Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and Instruction, 12 (3),
185-233.
Sweller, J., & Low, R. (1992). Some cognitive factors relevant to mathematics instruction. Mathematics
Education Research Journal, 4 (1), 83-94.
Tabbers, H. K., Martens, R. L., & van Merrienboer, J. J. G. (2004). Multimedia Instructions and Cognitive Load
Theory: Effects of Modality and Cueing. British Journal of Educational Psychology, 74 (1), 71-81.
Van Merrienboer, J. G., & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments
and future directions. Educational Psychology Review, 17 (2), 147-177.
Zheng, R., Miller, S., Snelbecker, G., & Cohen, I. (2005). Use of multimedia for problem-solving tasks. Paper
presented at the annual conference of American Educational Research Association, Montreal, Canada.
118
Pragnell, M. V., Roselli, T. & Rossano, V. (2006). Can a Hypermedia Cooperative e-Learning Environment Stimulate
Constructive Collaboration? Educational Technology & Society, 9 (2), 119-132.
Can a Hypermedia Cooperative e-Learning Environment Stimulate
Constructive Collaboration?
Mary Victoria Pragnell
Faculty of Medicine and Surgery, University of Bari - P.za Giulio Cesare, 1 – 70124 Bari – Italy
[email protected]
Teresa Roselli and Veronica Rossano
Department of Computer Science, University of Bari - Via Orabona,4 - 70126 Bari - Italy
Tel: +39 080 5443276
Fax: +39 080 5442424
[email protected]
[email protected]
ABSTRACT
The growing use of the Internet in learning environments has led to new models being created addressing
specific learning domains, as well as more general educational goals. In particular, in recent years
considerable attention has been paid to collaborative learning supported by technology, because this mode
can enhance peer interaction and group work.
Among the different active learning strategies, Cooperative Learning has found in the Internet and the
World Wide Web the ideal technological support. In this scenario, we have implemented a web-based
environment, endeavouring to reproduce within it the traditional teaching methodologies typical of
cooperative learning. Herein we describe two experiments aiming to assess the quantity and quality of the
interaction promoted by the system and how such factors as gender, background knowledge and role affect
communication. We compared the communication and the learning gain achieved in the experimental group
that worked with the system, named Geometriamo, with those achieved in the control group working in
class with the teacher. The results demonstrate that the use of an environment mediating peer-to-peer
collaboration can be highly beneficial, even in primary school age. A high number of messages were
exchanged among all pupils and, notably, the highest learning gain was recorded in the less able students in
the experimental group.
Keywords: Cooperative learning, Collaborative learning, CSCL, E-learning environment, Educational technology
Introduction
In advanced nations, computers and the WWW in particular are having a strong impact on education. These tools
must clearly be regarded as versatile aids rather than as a replacement for face-to-face teaching methods.
Nevertheless, the growing use of the Internet in learning environments has enabled new learning models to be
created addressing specific learning domains, as well as more general educational goals.
In particular, considerable attention has been paid in recent years to collaborative learning supported by
technology (Hiltz 1994), because this learning mode can enhance peer interaction and group work. Although in
the past the effectiveness of collaborative learning was not widely accepted because academic excellence was
defined by individual achievement, the area of research known as Computer-Supported Collaborative Learning
(CSCL) has recently shown rapid growth (Johnson et al. 1998, Choen & Scardamalia, 1998, Hakkarainen et al.
1999, Hoadley & Linn 2000, Enyedy et al. 1997).
One of the aims of CSCL is to increase the quality of the learning/teaching process by engaging students and
teachers in coordinated efforts to build new knowledge and solve problems together (Dillenbourg 1996). This
approach offers a good medium for classroom discussions that can facilitate participation and social interaction
among students, and between the teacher and students (Shellens & Valcke 2005). In the face-to-face discussions
which normally arise in the traditional classroom during collaborative type learning, the teacher’s role is
fundamental and shifts from that of unique information source to that of communication facilitator (William &
Peters 1997).
119
To obtain successful online learning, in both a collaborative and an individual learning situation, three types of
interaction must be present, namely learner-content, learner-learner and learner-instructor (Moore 1993). The
first type, learner-content, is strictly dependent on the construction and organisation of the material, a task which
is traditionally carried out by the teacher. The other two forms of interaction have to do with communication
processes in the strictest sense, and depend on the quality of the tools made available in the given CSCL
environment.
Among the different active learning strategies, much attention has in recent years been paid to Cooperative
Learning, that has found in the Internet and the World Wide Web the ideal technological support for setting up
collaborative activities with no space-time restrictions. There is still some terminological confusion between the
two concepts, although John Myers endeavoured to clarify this, by declaring: “The cooperative learning tradition
tends to use quantitative methods which look at achievement: i.e., the product of learning. The collaborative
tradition takes a more qualitative approach, analyzing student talk in response to a piece of literature or a primary
source in history.” (Myers 1991). Our approach addresses both aspects, analyzing both the quantitative learning
gain and the qualitative communication engendered.
Naturally, synchronous and asynchronous tools are not enough to ensure effective communication between
learners and teacher and the interaction needs to be encouraged by appropriate stimuli leading to positive
exchanges of opinion and outlook on the problem to be solved. A web-based cooperative learning environment,
therefore, must help to build up a communicative network and provide an environment where student groups of
varying sizes can work together to achieve common learning goals (Johnson et al. 1999).
Vygotskij (Vygotskij 1978) appraised the psychological implications of cooperative learning and asserted that
human beings are products of their cultures, thus people learn from interacting with others around them.
According to an analogous approach to the one adopted for intelligent tutoring systems, featuring the
introduction of Artificial Intelligence techniques and methods (Schank et al., 1993, Schank et al. 1995, Roselli
1995), in cooperative learning environments the role of communication facilitator can conveniently be assumed
by a virtual tutor, that simulates the teacher’s role in a traditional setting. Cooperation has been found to act as a
catalyst for the development of particular problem-solving abilities, because learners are able to continue to use
the reasoning techniques and strategies assimilated while working with their companions or the teacher, when
they later have to solve similar problems alone (van der Meijen & Veenman 2005).
This is the theoretical framework within which our research work is inserted. We have implemented a web
environment, endeavouring to reproduce within it the traditional teaching methodologies typical of cooperative
learning. Our WWW Hypermedial Cooperative Environment, called Geometriamo, is addressed to pupils
attending fifth-grade at the elementary school. The experiments we describe were conducted last year and one of
the aims, whose results are illustrated herein, was to assess the quantity and quality of the interaction promoted
by the system in pupils aged 10-11 years, and how factors such as gender, background knowledge and role can
affect communication.
Theoretical Background
Considerable research has been devoted to the benefits of CSCL. In particular, CSCL environments are
considered as satisfactory tools that can: facilitate task-oriented and reflective activity (Choen & Scardamalia,
1998; Hakkarainen et al. 1999, Wegerif, 2004, Gillies, 2004), encourage complex reasoning and deeper levels of
argumentation (Hoadley & Linn 2000), support mathematical problem solving (Enyedy et al. 1997, Nason &
Woodruff 2003), improve the use of conceptual models (Bell, 1997, Wegerif 2004) and increase students’
cognitive and metacognitive understanding (Brown, et al. 1998, Cohen & Scardamalia 1998, Shellens & Valcke,
2005). A large number of studies comparing CSCL and non-CSCL environments (Lamon et al. 1996) in high
school or university settings have demonstrated that CSCL can achieve better results in terms of student learning
gain (Berge 2003, Charnistski et al. 2003, Veerman et al. 2001). Only few experiences have yet been reported in
the primary school setting (Lipponen 2003, Anastasiades 2003, Tapola 2001, van der Meijen & Veenman 2005),
but in these, too, the value of the combination of new educational technologies, computers and the WWW,
integrated with a traditional learning pedagogical model such as cooperative learning, has been stressed.
However, although many studies of CSCL environments have demonstrated positive results as regards both
individual and group learning, others have pointed out some limitations (Guzdial 1997, Guzdial & Turns 2000).
Poor student participation has been recorded, measured in terms of the number of messages exchanged and the
number of lines per message, while another side effect (Feldman et al. 1999) is that students have been found to
120
tend to indulge in discussion more as a form of social interaction than for true learning purposes. It is now
recognised that in distance learning situations, characterised by geographic and physical isolation, new
communication strategies need to be implemented, which were not required in face-to-face communication.
Thus, the role of online tutor, be it virtual or human, must concentrate on fostering constructive, focused
communication achieving a balance between learning of the domain and social exchange.
In any case, considerations of a purely quantitative type may not be a sufficient metric for the assessment of the
contribution of CSCL to the quality of the interaction, let alone of the efficacy of the resulting learning. As has
been shown (Lipponen 2003), other conclusions may be drawn and results obtained if in addition to quantitative
assessment some qualitative factors are taken into account, focusing on gender, motivation, communication
direction, etc.
In this context, the effectiveness of the computer as a support for group work, and its effect on social interaction,
take on great importance. In CSCL environments, unlike in face-to-face discussion, communication occurs by
means of text messages. Several researchers have recognized the strength of writing as a communication tool.
Writing messages gives students the opportunity to reflect, to share their ideas and expertise (Collins et al. 1991),
to become more aware of their knowledge and to ask and answer questions without any problem of time limits.
All these factors encourage a deeper level of thinking in virtual communities.
From the research standpoint, the possibility of recording discussions conducted in cooperative work sessions is
of fundamental importance, as a basis for conducting in-depth analysis of the quantity and quality of such
communication and its effects on the overall learning process.
Cooperative Learning and the method STAD for setting up the CL
Already by the beginning of the 1990s, in the USA considerable debate had arisen as to the best way to reform
teaching methods in the fields of math and science. Teachers propend for the choice of integrating the traditional
face-to-face formula, in which students are provided with information to be memorized, with active learning
activities, in which the students are given the opportunity to build on the knowledge gained.
Cooperative Learning is achieved by dividing the class up into small groups that work together to achieve the
best group results by means of mutual assistance among the group members. All the members of the group must
work on the task assigned by the teacher and each is aware that the success or failure of each individual will
affect the results of the whole group (“your success benefits me and my success benefits you, in short we all sink
or swim together here” (Johnson et al. 1999)). This fosters both a team spirit and a more questioning approach to
learning. The learner-interaction variable is obviously the key to success or failure, on which all the other
variables depend (motivation, cognitive processes, class organization, assessment, etc.).
There are many collaborative and cooperative learning methods, which also can be considered as group learning
methods and used in both classroom-based and web-based environments. One of the methods adopted for
achieving Cooperative Learning is Student Team Learning (STL) defined by Slavin (Slavin 1980, Slavin 1990).
It is fundamentally centred around interaction in small groups but, above all, on individual responsibility and the
provision of incentives and rewards to stimulate the group’s individual and collective commitment. STL is one
approach featuring various cooperative techniques, including the Student Teams Achievement Divisions (STAD)
by Slavin, that we have chosen to implement in Geometriamo because it is suited to teaching disciplines such as
mathematics. The focal point of this teaching strategy is interaction among the members of small groups, during
which the notion of individual responsibility is reinforced by the fact that it has a strong effect on the final group
assessment (Zhao, 2002). This individual accountability motivates students to do a good job of peer tutoring and
explaining concepts, as the only way for a team to succeed is if all the team members have mastered the
information or skills being taught. The effects achieved with this method are not confined to the cognitive sphere
but include fostering interpersonal relationship skills. The teacher, in this context, is seen more as a source,
rather in the light of a guide and a sounding board, than as an authoritative figure, and studies conducted by
psychologists have demonstrated that this has remarkable positive character-forming effects.
The STAD technique subdivides the learning/teaching process into different phases: explanation by the teacher
of the topic to be studied; subdivision of the class into small heterogeneous groups; group work; individual
assessment; correction of the tests and final grading.
121
Groups should consist of 4/5 students and must be heterogeneous, so that in each group the different levels
(good, fair, sufficient, poor) are represented, as well as both sexes and different socio-cultural backgrounds. In
the group work the students’ task is to assimilate the concepts learnt during the lesson and help their companions
to do so. This work is carried out in two phases: solution of a series of simple exercises supplied by the teacher
to learn the concepts studied during the theoretical lesson, and then solution of a complex exercise: the latter is
the main goal of the group work.
During the subsequent individual assessment tests the students may not help one another because each student
must be aware that he or she is responsible for his/her own level of understanding. The individual assessment is
transformed by the teacher into a group assessment by summing the individual progress marks. This system of
progress assessment allows each student to contribute to the group if, and only if, he or she does his/her best and
demonstrates a substantial improvement as the work develops.
In the design of our cooperative learning environment the various components needed to reproduce the model in
a virtual classroom were defined.
Geometriamo, a www hypermedial cooperative environment
The idea of building Geometriamo grew out of our awareness that nowadays the Net is very much used in
schools, where an increasing number of cooperative experiences have been set up involving several different
nations (Olimpo & Trentin 1993). In fact, in advanced nations the schools are equipped with online laboratories
that allow large-scale exchanges of experiences at all levels. To examine the needs of schools at different
academic levels, we decided to build a hypermedial environment for cooperative learning, Geometriamo, that
aims to bring in virtual contact pupils from different classes and schools as a means of fostering constructive
exchanges of opinion.
In the design and development phases the pedagogical model determined the building of the other components,
in order to achieve the ideal environment for cooperative learning according to the STAD. In addition, when
implementing the domain, the teachers contributed actively by providing theoretical material and exercises, to
ensure the typical content for the scholastic level addressed by Geometriamo, as well as the appropriate level of
terminology.
The result was found to be a versatile tool combining the features of a tutorial system and a CSCL environment.
In fact, in Geometriamo the domain is plane geometry, and the tutorial system is organized in hypermedial form,
while the cooperative activity involved is that of assigning an exercise to each group, to be solved together
reaching a unanimous conclusion.
In the tutorial part, the first knowledge transfer is in the form of “explanation by the teacher of the topic to be
studied”. Each topic is organized in pages on Theory and Examples (Fig. 1a and 1b). The former contain the
explanations of the various geometric figures and the relative direct and inverse formulas; the latter pose model
exercises (already solved) that teach the user the method for solving a geometric exercise.
The “group work” phase is set up by the cooperative environment. Each group can resort to a series of Simple
Exercises (Fig. 1c) serving to apply what was explained in the Theory, and to the model exercises, as well as
carrying out a series of Complex Exercises (Fig. 1d) that represent the task assigned. Each group must solve
these by communicating among themselves and agreeing on the problem solving strategy to be used. At the end
of the task the solution is communicated to the virtual tutor, which will provide positive feedback and reward, or
suggest areas of further study.
The chance to work in a group and to interact, discussing a study topic or the solution of a task, was provided by
a dedicated component allowing one-to-one and one-to-many communication. To exchange opinions, ask
questions and make suggestions, the members of each group can use a messaging notice board that works like email. Figure 2a shows the notice-board of the leader of Group 5. As can be noticed the group members have sent
two requests for help. For each message the notice-board displays the sender, the receiver, the date and the
number of responses received. Figure 2b shows the form for writing the message, the student has to fill the
sender field (implemented using a drop-down menu), the subject and the text field. Moreover, in the left corner
of both figures (2a and 2b) of the leader’s notice-board there are three buttons that allow the leader to ask the
tutor for help for a particular student, for the group that is having difficulty in cooperating or with a difficult
exercise. In the right corner there is the virtual tutor’s notice-board where all the answers sent by the tutor are
122
listed. In the example (Fig 2a) the virtual tutor invites the leader before asking for help for one of the group
members, to try to discuss the problem better with him/her and to involve him/her more in the group discussion.
1a. Theory:
Pentagons and
Hexagons as
regular polygons
1c. Simple
Exercise:
Calculate the
perimeter of the
Hexagon
1b. Example:
calculation of
the perimeter of
a pentagon
1d. Complex
Exercise:
calculate the
perimeter of
GIGI
Figure1: Some examples of pages (a)Theory, b) Example, c)Simple and d)Complex exercises)
As pointed out above, one of the advantages of text-based communication is that of keeping trace of the
contributions by each member. All the messages sent and received are stored and can then be used as tools for
assessment of the learning process, providing useful and detailed information on the dynamics of the interaction.
For this reason, each member has to register. When a sufficient number of users have registered for the system
to create heterogeneous cooperative groups, the collaboration activities can be started.
In addition, a leader must be individuated in each group, whose task is that of guiding and moderating the group
work. In Geometriamo the leader interacts with the Tutorial Component, and asks for suggestions as to how to
“promote” peer interaction. This tutorial component, implemented using Artificial Intelligence techniques,
intervenes to stimulate collaborative activities, peer interaction and to suggest revision contents, as the Teacher
does in the traditional collaborative classroom. This is possible because Geometriamo includes not only the
typical components of an Intelligent Tutoring System (Knowledge Base, Student Module, Tutor Module and
Student Interface) (Barr et al. 1982) but also a Group Modeling component (Group Module), that keeps track of
the group history, providing the data for suitable intervention by the tutorial component.
Moreover, Geometriamo provides an assessment area that includes the “individual assessment” and “correction
of the tests and final grading”. In this phase the pupil can take a final test that assesses not only individual
improvement but also and above all, the group improvement, calculated as defined in the STAD. This places the
emphasis on the group rather than the individual, in accordance with the cooperative philosophy.
123
Figure 2a: Example of leader’s notice-board
Figure 2b: Writing messages in notice-board
Finally, there is an area devoted to the Teacher, who acts as supervisor, monitoring the work done in each group,
in terms of exercises done, pages visited and assessment tests completed.
The Effectiveness of Geometriamo as a tool promoting communication
To assess the effectiveness of communication promoted by Geometriamo in a computer supported learning
situation, two controlled experiments were conducted: a pilot study and a follow-up study.
Aim of the evaluations
The experiments were carried out on the one hand to assess the learning efficacy achieved in a distance
cooperative learning environment and on the other, to see how successfully the electronic tool could mediate
communication, and make interaction among the group members more active and productive.
Learning Effectiveness Questions
Both experiments aimed to answer the following questions:
1. Can children really improve their knowledge of geometric concepts by using the hypermedia?
2. Can the use of the cooperative learning system be as effective as cooperative learning in the classroom
mediated by a teacher?
3. Can the use of the cooperative learning be as effective for less gifted students as for more gifted students?
Promoting Collaboration Questions
The follow-up study also aimed to probe the following aspects:
1. How many notes did each student send to his/her companions?
2. What is the relationship between communication and learners’ ability?
3. How much does the factor of not knowing the other members of the group affect the interaction in a
cooperative learning environment?
4. What relationship is there between gender and ability to stimulate communication among group members?
These questions were answered according to the method described below.
Samples
Two experiments were conducted:
¾ A first pilot study was carried out to see whether the environment or the content needed improvements.
For this study only 24 pupils were selected.
124
¾
A follow-up study involved a larger sample: 152 pupils.
The pupils in both experiments were attending the fifth class at the primary school “XX Circolo Didattico E. De
Amicis” in Bari. Learning using hypermedia was a new experience for all of them, but as we expected, some
pupils were familiar with the use of PCs, primarily for playing games and for navigating on Internet to find
information. In any case, the fact that some of them had no experience of computers was not a handicap because
appropriate training in the use of the PC and the specific environment was given to all the participants.
The pupils firstly underwent an individual pre-test, developed in collaboration with the teachers, to assess their
prior knowledge of geometry. For each test the teachers assigned an overall mark corresponding to the four
levels (excellent, good, fair and poor).
The experimental designs
Both the experiments were designed using a mixed approach, with pre-test post-test as the within-subjects factor
and the between-subjects factor. The children, 24 in the pilot study and 152 in the further investigation, were
divided into two groups: the members of the Experimental Group (EG) used the Geometriamo hypermedia and
the members of the Control Group (CG) carried out cooperative learning in class with the teacher.
The two main groups (EG and CG) were subdivided into cooperative subgroups, 6 in the pilot study and 19 in
the follow-up study, each numbering 4 pupils. This number was defined on the basis of the STAD methodology,
that prescribes small groups of 4/5 students, as well as convenience, as the system is currently able to manage
four unit groups.
When forming the groups, care was taken to ensure homogeneity within and among groups both in the
experimental and control groups, using both the marks obtained in the pre-test and the teacher’s overall
assessment of each pupil for the previous year. Particular care was taken to match for sex, (2 males and 2
females per group), as this factor has a strong effect on communication among 10-11 year-old pupils.
In each group it was also necessary to appoint a leader to coordinate the group activities. The EG leaders could
ask the virtual tutor for help in motivating the group cooperation, while the CG leaders could ask the teacher.
The Experimental Group worked in the two laboratories available in the two school premises connected online to
the server made available by our research lab, where the software was installed. The Control Group worked in
the classroom, putting the desks of the group members together so that they could work and share materials,
knowledge and skills.
In the follow-up study, because of the larger sample size, we were able to create more heterogeneous groups. As
Figure 3 shows, in each group half of the subjects worked in groups of four including only classmates
(denominated as EGClass and CGClass) and the other half in groups of four pupils from other classes
participating in the experiment (denominated as EGMix and CGMix). This enabled closer simulation of on-line
cooperative learning, in which participants do not know one another. This further subdivision allowed us to
assess how much this factor of not knowing the other members of the group can affect the interaction in a
cooperative learning environment.
Figure 3: Distribution of the pupils in the EG and CG
125
Procedures
As stated above, the pupils firstly underwent an individual pre-test to assess their prior knowledge of geometry.
Both experimentations consisted of three sessions, each one lasted an hour and a half, with a 2-day interval
during the sessions. In the Theory session the teaching unit on the perimeter of a regular polygon was studied by
the Experimental Group with the aid of Geometriamo and by the Control Group with the aid of the teacher. In
the second session they worked on simple exercises, and in the third, they had to solve a complex exercise
consisting of calculation of the perimeter of a compound figure (in Figure 4 the last two sessions are
denominated Work Group sessions).
One week after the experiment, all the pupils were given a post-test. To try and control confounding variables
related to the participants’ history, no other geometric lessons were taught during the period of the experiment
and the pupils were not given any homework on the topic.
Finally, comparison of the results obtained by each participant, in the pre-test and in the post-test, allowed us to
evaluate not only the learning gain but also how much the peer to peer interaction had contributed to improve
their knowledge.
Figure 4: A graphic depiction of the experimental procedures
Results
Data analysis yielded the following answers to the research questions listed above.
Question 1. Can children really improve their knowledge of geometric concepts by using the hypermedia?
From an effectiveness point of view a first important result is shown in Figure 5 and Figure 6.
8
6.4
7.5
6.2
7
6
6.5
Pre-t est
6
Post-test
5.5
Pre-test
5.8
Post-test
5.6
5
5.4
4.5
4
EG
CG
Figure 5: Average pre-test and post-test score in pilot
study
5.2
EG
CG
Figure 6: Average pre-test and post-test score in follow-up
study
126
Moreover, Geometriamo provides an assessment area that includes the “individual assessment” and “correction
of the tests and final grading”. In this phase the pupil can take a final test that assesses not only individual
improvement but also and above all, the group improvement, calculated as defined in the STAD. This places the
emphasis on the group rather than the individual, in accordance with the cooperative philosophy.
Question 2. Can the use of the cooperative learning system be as effective as cooperative learning in the
classroom mediated by a teacher?
Table 1 and Table 2 report the t-test analysis, which was applied to the results of the post-test to test the
difference between the performances of the experimental and control groups. The predetermined alpha level
adopted for hypothesis testing was 0.05. The results confirmed the experimental hypothesis claiming an equal
improvement over time in the two experimental conditions. This implies that all the children learned during the
experiment and that this learning was almost identical in both groups. Therefore we can assert that no significant
difference due to the mode of instruction was found between the two groups in either experiment.
Table 1 t-test analysis in pilot study
EG
CG
N
Mean learning
gain
Standard
Deviation
12
12
1.33
1.17
2.67
1.59
Total Standard Deviation =
T(22) =
t=
2.198
0.186
1.717
Table 2. t-test analysis in follow-up study
N
EG
CG
76
76
Mean learning
gain
0.66
0.21
Total Standard Deviation =
T(150) =
t=
Standard
Deviation
2.30
3.08
2.722
1.007
1.67
Question 3. Can the use of the cooperative learning be as effective for less gifted students as for more
gifted students?
Scattering of data evinced an interesting finding reported in Figure 7, which shows that in the EG group pupils
with a lower score in the pre-test had a higher learning gain. These results seem to confirm what previous
research had pointed out: the use of the computer may promote better and freer communication among members
of a group, reinforcing the inherent characteristics of the cooperative learning model. Very likely, the less
directly personal confrontation occurring online encourages students with greater learning difficulties to voice
their opinions and doubts, and this freer sounding-board effect benefits their overall understanding of the
concepts under discussion.
Furthermore, the use of text-based communication gives students more opportunities for reflection and
exploration of their own knowledge and this encourages deeper levels of thinking, a point which is particularly
useful in less gifted students, who can thus realize what it is that they don’t know. The final result seems to be
that of reducing the gap between less and more able students.
Question 4. How many notes did each pupil send to his/her companions?
The above results were confirmed by the number of messages exchanged during interaction with Geometriamo.
The data collected in the Follow-up study demonstrate that the messaging component was amply used. In fact,
during the three working sessions the 20 groups exchanged an average of 100 messages. Consistent use of the
messaging function is confirmed by analysis of the individual messages sent (Figure 8), which showed that only
7 of 76 pupils sent less than 10 messages per session, while all the others had taken full advantage of the
communication tool.
127
30
5%
25
4%
Less able
3%
More able
2%
Number of students
6%
20
15
10
5
1%
0
0%
0-10
Pilot Study
Follow-up Study
11-20 21-30 31-40 41-50 51-60 61-70
Number of messages sent by each student
Figure 7. Learning gain of less gifted and more gifted
pupils in Pilot study vs. Follow-up study
Figure 8: Number of messages sent per individual
Question 5. What is the relationship between communication and learners’ ability?
Moreover, analysis of the number of messages exchanged according to the classification of the learner’s ability
(excellent, good, fair and poor) showed that as expected, the group leaders exchanged the highest number of
messages because they had been made responsible for the promotion of communication, and more gifted pupils
also communicated very extensively, as shown in Figure 9. On the face of it, and comparing these with the
results discussed in Question 3, this would seem to imply that a greater level of interaction does not therefore
result in a true learning gain. In fact, however, even if more gifted pupils sent more messages than less gifted
ones, the learning gain was actually higher in the latter pupils. It would appear that even if they tended to
experience the discussion rather more passively, less gifted students in any case felt themselves to be involved in
the developing debate and were able to absorb the growing knowledge. In fact, the rare messages they sent
tended to be requests for further explanation. This tendency was less marked in the EGClass than in the EGMix,
showing that familiarity does, to a certain extent, smooth out such communication difficulties. In any case, even
if messages were not directly addressed to them, the most gifted students took part in subsequent explanations,
thanks to the stimulation provided by the Tutorial Component, that had the express task of promoting peer
interaction and reviving it in case of difficulties.
In short, closer analysis shows that the high level of communication by the leaders is a demonstration of the
achievement of another objective of the cooperative activity, that of encouraging more gifted students to help
their less gifted peers.
Furthermore, it is interesting to observe that the tendency towards a higher number of messages exchanged
among students in the EGClass than in the EGMix is reversed in the group of students classified as poor,
reinforcing the concept that they are less overawed in an online and therefore more impersonal setting.
40
12%
35
10%
30
8%
25
20
EG Class
15
EG Mix
6%
Class Group
4%
Mix Group
2%
10
5
0%
0
-2%
Excellent
Good
Fair
Poor
CG
EG
Figure 9: Number of messages exchanged per class of ability Figure 10 Learning gain in the class groups and mixed groups
in CG and EG
128
Question 6. How much does the factor of not knowing the other members of the group affect the
interaction in a cooperative learning environment?
Further analyses in this direction yielded particularly interesting results as shown in Figure 10, illustrating the
comparison of the results obtained in the follow-up study, in the two groups (EG e CG) in their respective
subdivisions (EGClass and CGClass versus EGMix and CGMix). As expected, even if all the groups showed an
overall improvement in the EG, this was higher in the class groups than in the mixed classes groups, because
children find it easier to work with their classmates. Instead, in the CG, although the class groups improved, the
mixed class groups had negative results. One of the factors which could explain the results obtained is the
communication medium used. In fact, while face-to-face communication inhibited pupils in the control group
mixed classes, especially the shy ones, in the experimental group the mediation of the electronic system tended
to overcome the psychological barriers created in mixed cooperative situations (a fear of asking for help,
jealousy, pride in one’s own abilities, etc,…).
Question 7. What relationship is there between gender and ability to stimulate communication among
group members?
During group work sessions, the figure of the leader took on a fundamental importance in the task of stimulating
communication by encouraging cooperation among his/her companions. Analysis of the messages sent by each
leader revealed that female leaders appeared to be more able to involve the group and encourage cooperation
among members than their male counterparts (see Figure 11).
However, this first impression was revised when the type of messages sent was considered in greater depth.
Figure 12 compares the number of messages sent (quantity) with their type (quality). In this analysis we
assessed how many of the messages sent by each group leader were directly related to the domain topic. The
results were found to be approximately equal for male and female group leaders. In short, the difference in the
results for overall communication seems to be due to a greater number of messages having been sent for
relational purposes by female group leaders. Again, this result is not unexpected in the age group involved in
our experimentation. In any case, overall, the results show that regardless of gender, the group leader is
absolutely essential as the fulcrum around which cooperation can be built up and a miniature learning
community created.
Are girls or boys m ore successful as
group leader?
30
25
20
Girls
15
Girls
Boys
Boys
10
5
0
Figure 11 Correlation between gender and ability
to stimulate communication in the group
Relationship messages
Topic messages
Figure12. Ratio of relational versus topic-related
messages sent by leader
Conclusions and future works
A multiplicity of different learning methods using the web as the communication tool, many of which are
labelled as “non traditional”, are now available at all levels of education, and addressing all types of learners.
They are now so widespread, and the quality of the learning promoted is so strongly supported by experimental
evidence, that it is no longer justifiable to consider distance learning facilities purely as an alternative method for
stimulating motivation in learners or for solving space-time problems. The ability to develop metacognitive
skills fostered by CSCL environments is a very important aspect. The experiments conducted with our
cooperative environment also aimed to address the synergic aspects of cooperative and collaborative learning;
they demonstrate that a number of benefits can be gained with the use of an environment that mediates peer-topeer communication even in primary school age. Collaboration skills are, after all, indispensable in our modern
world, where a multidisciplinary approach is becoming the norm. After the first sense of diffidence toward the
different communication tool had worn off, the experimental subjects appreciated the chances offered to talk to
129
pupils in other buildings and to share not only their knowledge and skills in the field of geometry but also the
pleasure of building relationships with new, unknown peers.
Our experimental results confirm the value of our environment as a tool for stimulating collaboration in a
cooperative learning situation. A high number of messages was sent in all EG groups. Comparison between the
communication that took place in the EG and in the CG showed that although the efficacy results (Roselli 2003)
demonstrated a learning gain in both groups, Computer-Mediated Communication was more successful than
classroom communication for the purposes of fostering collaborative learning. Indeed, we found that the less
directly personal confrontation occurring online seems to encourage students with greater learning difficulties to
voice their opinions and doubts, and this freer sounding-board effect benefits their overall understanding of the
concepts under discussion. Even if students classified in the lower aptitude groups tended to send less messages,
and hence to experience the discussion rather more passively, in any case the less gifted students appeared to feel
themselves involved in the developing debate and were able to absorb the growing knowledge. Another
interesting aspect, namely analysis of the communication from the gender standpoint, was that while boys tended
to send less messages than girls, their messages were more topic-related, while girls also sent many messages
having a more relational purpose. Overall, the topic-related communication was comparable for the two
genders.
All these results have encouraged us to extend research in this area. We are at present carrying out a
reorganisation of the teaching content, modifying the environment to make it suitable for use with high school
students. Further investigations will also be made of Geometriamo in the current version but in different
contexts, for example by using it with children of different nationalities, to see whether the results obtained in
the present experience are confirmed in other cultural milieus.
References
Anastasiades, P. S. (2003). Distance learning in elementary schools in Cyprus: the evaluation methodology and
results. Computers & Education, 40 (1), 17-40.
Barr, A. & Feigenbaum, E. (1982). The handbook of Artificial Intelligence (Vol. 1 & 2), Los Altos, CA, USA,
Morgan Kaufmann.
Bell, P. (1997). Using argument representations to make thinking visible for individuals and groups. In
Koschmann, T., Hall, R., & Miyake, N. (Eds.), CSCL II: Carrying Forward the Conversation, NJ, USA:
Lawrence Erlbaum Associates, 10-19.
Berge, Z. L. (2003). Interaction in Post-Secondary Web-Based Learning, retrieved November 29, 2005 from
http://www.saskschools.ca/~parkland/interaction.htm.
Brown, A. L., Ellery, S., & Campione, J. C., (1998). Creating zones of proximal development electronically. In
Greeno, J. G. & Goldman, S. V. (Eds.), Thinking practices in mathematics and science learning, NJ, USA:
Charnistski, C. W., Molinaro, J. A., Corabi, J., & Nolan, K. (2003). Comparing Student Achievement in a
Graduate Level Research Methods Course Using Face-to-Face and Web-based Instruction: Result of a Pilot
Study. In Rosset, A. (Eds.) World Conference on E-Learning in Corp., Government., Health., & Higher
Education (ELEARN), VA, USA: Association for the Advancement of Computing in Education (AACE), 174177.
Choen, A., & Scardamalia, M. (1998). Discourse about ideas: monitoring and regulating in face-to-face and
computer-mediated environments. Interactive Learning Environments, 6 (1-2), 93-113.
Collins, A., Brown, J. S., & Holum, A. (1991). Cognitive apprenticeship: making thinking visible. American
Educator, 15 (6-11), 38-46.
Dillenbourg, P., Baker, M., Blaye, A., & O’Malley, C (1996). The evolution of research on collaborative
learning. In Reimann, P. & Spada, H. (Eds.), Learning in Humans and Machines: Towards an interdisciplinary
learning science, UK: Elsevier, 189-211.
130
Enyedy, N., Vahey, P., & Gifford, B. R. (1997). Active and supportive computer-mediated resources for student
to student conversations. In Koschmann, T., Hall, R. & Miyake, N. (Eds.), CSCL II: Carrying Forward the
Conversation, NJ, USA: Lawrence Erlbaum Associates, 27-36.
Feldman, A., Konold, C., Coulter, B., Conroy, B., Hutchison, C., & London, N. (1999). Network science, a
decade later. The internet and classroom learning, NJ, USA: Lawrence Erlbaum Associates.
Gillies, R. M. (2004). The effects of cooperative learning on junior high school students during small group
learning. Journal of Learning and Instruction, 14 (2), 197-213.
Guzdial, M. (1997). Information ecology of collaboration in educational settings: influence of tool. In
Koschmann, T., Hall, R. & Miyake, N. (Eds.), CSCL II: Carrying Forward the Conversation, NJ, USA:
Guzdial, M., & Turns, J., 2000. Effective discussion through a computer-mediated anchored forum. Journal of
the Learning Sciences, 9 (4), 437-469.
Hakkarainen, K., Lipponen, L., Jarvela, S., & Niemivirta, M. (1999). The interaction of motivational orientation
and knowledge-seeking inquiry in computer-supported collaborative learning. Journal of Educational
Computing Research, 21 (3), 261-279.
Hiltz, S. R. (1994). The virtual classroom: Learning without limits via computer networks, NJ, USA: Ablex
Publishing Corporation.
Hoadley, C. M., & Linn, M. C. (2000). Teaching science through online peer discussions: speakeasy in the
knowledge integration environment. International Journal of Science Education, 22 (8), 839-857.
Johnson, D. W., Johnson, R. T., & Smith, K. A. (1998). Cooperative learning returns to college. Change, 30 (4),
26-35.
Johnson, D. W., & Johnson, R. T. (1999). Learning together and alone: Cooperative, competitive and
individualistic learning (5th Ed.), Boston, USA: Allyn &Bacon.
Lamon, M., Secules, T., Petrosino, A., Hackett, R., Bransford, J., & Goldman, S. (1996). Schools for thought:
overview of the project and lessons learned from one of the sites. In Schauble, L. & Glaser, R. (Eds.),
Innovations in learning: new environments for education, NJ, USA: Lawrence Erlbaum Associates, 243-288.
Lipponen, L. Rahikainen, M., Lallimo, J., & Hakkarainen, K. (2003). Patterns of participation and discourse in
elementary students’ computer-supported collaborative learning. Journal of Learning and Instruction, 13 (5),
487-509.
Moore, M. (1993). Three types of Interaction. In Harry, K. Hohn, M & Keegan, D (Eds.), Distance Education:
New Perspective, London, UK: Routledge, 19-24.
Myers, J. (1991). Cooperative learning in heterogeneous classes. Cooperative Learning, 11 (4).
Nason, R., & Woodruff, E. (2003). Fostering Authentic, Sustained, and Progressive Mathematical KnowledgeBuilding Activity in Computer Supported Collaborative Learning (CSCL) Communities. Journal of Computers
in Mathematics and Science Teaching, 22 (4), 345-363.
Olimpo, G., & Trentin, G. (1993) La telematica nella didattica: come e quando. Tecnologie Didattiche, 2, 4-17.
Roselli, T. (1995). Artificial Intelligence Can Improve Hypermedia Instructional Technologies for Learning.
ACM Computing Surveys, 27 (4), 624-626.
Roselli, T., Faggiano, E., & Rossano, V. (2003) A Further Investigation On The Effectiveness Of A WWW
Hypermedial System for Cooperative Learning. In Tin Lee, K. & Mitchell, K. (Eds.), International Conference
on Computer in Education, VA, USA: Association for the Advancement of Computing in Education (AACE),
440-442.
131
Schank, R. C., Davenport, G., Ghislandi, P., Spohrer, J. C., & Wilson, K. (1993). A.I. Multimedia and
Education. In Bajcsy, R. (Ed.), Thirteenth International Joint Conference on Artificial Intelligence. CA, USA:
Morgan Kaufmann Publishers, 1667-1672.
Schank, R. C., Korcuska, M., & Jona, M. (1995). Multimedia Application for Education and Training:
Revolution or Red Herring? ACM Computing Surveys, 27 (4), 633-635.
Shellens, T., & Valcke, M. (2005). Collaborative learning in asynchronous discussion groups: What about the
impact on cognitive process? Computers in Human Behavior, 21 (6), 957-975.
Slavin, R. E (1980). Using Student Team Learning, Baltimore, MD: The John Hopkins University, Center for the
Social Organization.
Slavin, R. E. (1990). Cooperative Learning: theory, research, and practice, Englewood Cliffs, NJ, USA:
Prentice-Hall.
Tapola, A. (2001). Participation and motivation in computer-supported collaborative learning. In proceedings of
World Conference on Educational Multimedia, Hypermedia, and Telecommunications, VA, USA: Association
for the Advancement of Computing in Education (AACE), 1873-1878.
van der Meijen, H., & Veenman, S. (2005). Face-to-Face versus computer-mediated communication in a primary
school setting. Computers in Human Behavior, 21 (5), 831-859.
Veerman, A., & Veldhuis-Diermanse E. (2001). Collaborative learning through computer-mediated
communication in academic education, retrieved March 20, 2005 from http://www.ll.unimaas.nl/eurocscl/papers/166.doc.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes, Cambridge, MA,
USA. Harvard University Press.
Williams, V., & Peters, K. (1997). Faculty Incentives for the preparations of Web-based Instruction. In Khan, B.
H. (Ed.), Web-based Instruction, NJ, USA: Educational Technology Publications.
Wegerif, R. (2004). The role of educational software as a support for teaching and learning conversations,
Computers & Education, 43 (1-2), 179-191.
Zhao, J., McConnel, D., Li, K., & Akahori, K. (2002). STAD Modeling Method for Computer Supported Group
Learning. In Kinshuk, Lewis, Akahori, Kemp, Okamoto, Henderson & Lee (Eds.), International Conference on
Computer in Education, Los Alamitos, CA, USA: IEEE Computer Society Press, 344-348.
132
Chan, H., Tsai, P., & Huang, T.-Y. (2006). Web-based Learning in a Geometry Course. Educational Technology & Society, 9
(2), 133-140.
Web-based Learning in a Geometry Course
Hsungrow Chan
Associate Professor, Department of Mathematics Education, National Pingtung University of Education
No.4-18, Minsheng Rd., Pingtung City, Pingtung County 900, Taiwan (R.O.C.)
[email protected]
Tel: +886-8-7226141#6200 or 6210
Fax: +886-8-7239860
Pengheng Tsai
Doctoral Student, Department of Industrial Technology Education, National Kaohsiung Normal University
No.58, Lane 59, Longsing St., Cianjhen District, Kaohsiung City 806, Taiwan (R.O.C.)
[email protected]
Tel: +886-7-7510048
Fax: +886-7-7510445
Tien-Yu Huang
Professor, Department of Computer Science, National Pingtung University of Education
No.4-18, Minsheng Rd., Pingtung City, Pingtung County 900, Taiwan (R.O.C.)
[email protected]
Tel: +886-8-7226141#3107 or 3110
Fax: +886-8-7215034
ABSTRACT
This study concerns applying Web-based learning with learner controlled instructional materials in a
geometry course. The experimental group learned in a Web-based learning environment, and the control
group learned in a classroom. We observed that the learning method accounted for a total variation in
learning effect of 19.1% in the 3rd grade and 36.5% in the 6th grade. The main factor in the difference was
attributed to the students’ ability to apply information technology. We further found that students need
more communication with peers, teachers, and lecture materials when learning via the Internet. We are
encouraged that progress in the technological advances of the Internet will make it more convenient and
more effective for use in Web-based learning.
Keywords:
Classroom learning, Geometry curriculum, Learner control, Van Hiele level of geometric thought, Web-based
learning
Introduction
Elementary students in Taiwan usually learn geometric concepts by recitation and through practice. Most of
them fail to learn basic geometric concepts and geometric problem-solving, so they obviously don’t view
themselves as being successful in geometry courses. Some elementary geometry students have a lot of difficulty
with ordering properties logically and general definitions. This situation gets worse later in Junior and High
School. Taiwan is not alone in this. Much learning of geometric concepts by U.S. students has been by rote; they
frequently do not recognize components, properties, and relationships between properties (Clements & Battista,
1992). Many mathematics educators have studied children’s geometric conceptions. The three dominant lines of
inquiry concerning children’s geometric conceptions are based on the theories of Piaget, the van Hieles, and
cognitive psychologists (ibid). While the Piagetian and cognitive psychology studies were not grounded in
educational concerns, the van Hielian research, by contrast, was so-grounded (Clements, Swaminathan,
Hannibal, & Sarama, 1999). Thus, this study is based on their work. According to the van Hiele theory, students
progress through levels of thought in geometry when aided by instruction (van Hiele, 1986). Thinking on
geometric concepts develops from an initial, Gestalt-like visualization through increasingly sophisticated levels:
descriptive and analytic, abstract and relational, formal deductive, and mathematically rigorous.
The five van Hiele levels are thought of in the following ways.
1. The visual level: Children identify shapes according to appearance, recognizing them as visual gestalts using
visual prototypes, saying, for instance, that a given figure is a circle because “it looks like a moon.”
2. The descriptive and analytic level: Students reason about geometric concepts by means of an informal
analysis of component parts and attributes.
133
3.
4.
5.
The abstract and relational level: Students order properties logically and begin to appreciate the role of
general definitions. They can form abstract definitions and distinguish between the necessity and sufficiency
of a set of properties in determining a concept.
The formal deductive level: The role of axioms, undefined terms, and theorems is fully understood, and
original proofs can be constructed.
The mathematically rigorous level: Students can compare various axiomatic systems based on various
axioms, and study various geometries in the absence of concrete models.
Helping children move through these levels may be taken as a critical educational goal (van Hiele, 1986). The
main aim of geometric education is to change students’ geometric conceptions through teaching activities, and
move students from one level of van Hiele thinking to the next. Using an elementary geometry curriculum, the
first purpose of this study was to find a way to help elementary students use the learning effect to advance their
van Hiele levels of geometric thought.
Previous lines of inquiry into children’s geometric conceptions have provided useful foundations, but have also
left gaps that impede curriculum development and teaching improvements. And as briefly explained above,
much learning of geometric concepts by Taiwan students has been by recitation and through practice. We should
allow students to explore geometric concepts and shapes, and must provide experiences in informal geometry for
students throughout their school years. However, because of limited classroom space, teaching time, and
inability to interact, classroom teaching and learning contexts in traditional schools haven’t been successful in
geometry courses, and students always treat the geometric models as toys. Moreover, instructional control and
program control are administered by teachers in classrooms, and interactive activities are few or even
nonexistent. So even though teachers perform quite well, no instructional records are preserved and it is
impossible for students to review the instructional interactive process.
The concept of learner control, defined as an individual ability to sequence the content while engaged in an
instructional activity, is now extended to include allowing students to control the pace of instruction (Mager &
Clark, 1963; Gay, 1986; Reiser, 1984). Several studies suggest that learner control of instruction can result in
more positive student attitudes, higher levels of student engagement, and less anxiety (Fowler, 1983; Hannafin,
1984). Accordingly, this study involved using the concept of learner control in conducting a Web-based
geometry course. However, implementation did entail some risk. Learner-controlled instructional materials may
improve students’ performance and attitudes toward the subject matter, and increase efficiency, but they also
produce large variations in students’ performance that may result from instructional tasks, instructional designs
or assessment procedures (Hannafin, 1984).
Lepper (1985) suggests that providing learner control increases the relevance of instruction especially if the
activity involves the external environment. Through careful combinations of course exposition text, images,
expository videos, and 3D virtual geometric objects, the prototypic module strives to simulate instructional
activities and geometric mental manipulation action. Our creation of Web-based materials had the ultimate goal
of generating complete support for vertical and horizontal integration in the geometry course. Our experimental
design embedded the instructional material scaffolding in the Internet, and allowed students to arrange their
programs, manipulate 3D virtual geometric objects, and discuss online with teachers and peers. That is,
instructional control and program control were returned to students. Here, teachers served as counselors and
guides. Hence, the second purpose of this study was to promote the learning effect and level of geometric
thought of elementary students with Web-based learning. This study designed and applied Web-based cube and
cuboid materials for 3rd grade students, and prism, pyramid, cylinder, and cone materials for 6th grade students.
Upon completion, we used our own test to assess the learning effect and the van Hiele level of geometric
thought. We also interviewed the sampled students to determine how thoroughly they grasped the geometric
concepts, how well they accepted this different learning method, and their perceptions on using Web-based
learning.
Method
Participants
The experimental group, which received Web-based instruction, included thirty-five 3rd grade students and
thirty-nine 6th grade students. The control group, which received classroom instruction, included thirty-four 3rd
grade students and forty 6th grade students. All subjects were from an elementary school in Kaohsiung City,
Taiwan.
134
Web-based learning environment
This study applied the Web-based learning environment whose structure is shown in Figure 1. The environment
is divided into a teachers’ management interface, students’ learning interface, and system support interface.
Teachers’ management interface: Enables teachers to manage all parts of the courses, design Web-based
materials, record lecture videos, administer teaching files, and test students.
Students’ learning interface: Students logging on to this environment can choose studying the geometry course,
controlling the instruction program and content, asynchronous and real-time communication, and taking tests. In
particular, they can manipulate 3D virtual geometric objects using a mouse.
System support interface: This provides course management, including page setting, personal e-mail, and
discussion and assignment areas so that instruction activities proceed smoothly.
These interfaces are integrated by the database management function which, includes the user, curriculum, Webbased materials, and instruction activities record databases.
Figure 1 Web-based learning environment
Achievement tests
The aim of this study was to find a way to help promote learning and van Hiele levels of geometric thought
among elementary students. We developed our pre- and post-tests according to the criterion-referenced van Hiele
theory of content validity. Each test consisted of three parts:
1. identifying and denominating geometric shapes on level 1;
2. informal analysis of geometric component parts and geometric shape attributes on level 2;
3. forming abstract definitions and distinguishing geometric shapes on level 3.
The Pearson correlation coefficients for the pre- and post-tests were 0.8054 for the 3rd grade and 0.736 for the 6th
grade as alternate-form reliability, and the Cronbach coefficient for the pre- and post-tests were 0.8289 for the 3rd
grade and 0.8482 for the 6th grade, with a 0.05 confidence level.
Results
Learning effect
Table 1 shows descriptive statistics for both groups’ pre- and post-tests. We found that learning effects in both
groups were promoted in the geometry course.
135
Table 1. Summary of means; SD: the learning effect determined by achievement test
6th Grade
3rd Grade
Pre-test
Post-test
Pre-test
Post-test
Group
n
Group
n
Means
SD
Means
SD
Means
SD
Means
SD
Ex
35
66.86
10.42
78.03
13.36
Ex
39
46.77
18.99
72.21
23.23
Con
34
71.94
9.96
80
16.89
Con
40
46.98
17.88
61.93
22.54
Ex = experimental group, Con = control group
Table 2. Summary of ANCOVA: the learning effect of different learning methods, *p<.05
Sources of variation
df
Mean Square
F
Sig.
Type Ⅲ Sum of Squares
Between groups
37.061
1
37.061
0.2
0.656
3rd Grade
(learning method)
Error
12214.466
66
185.068
Eta Squared = 0.003, Observed Powerα =0.073, r2 = 0.191
Between groups
1756.824
1
1756.824
5.136
0.026
6th Grade
(learning method)
Error
25994.814
76
342.037
Eta Squared = 0.063, Observed Powerα =0.609, r2 = 0.365
Grade
After removing the influence of the covariance, F = 0.2 for the 3rd grade in Table 2 is below the 0.05 significance
level, which means there was no significant difference in learning effect between the Web-based and classroom
learning environments. Taking a closer look at the strength of association (r2 = 0.191), we see that learning
method accounted for 19.1% of the total variation in learning effect in the 3rd grade class, but in the 6th grade
class, F = 5.136 reached the 0.05 significance level, which means the learning effect in the Web-based learning
environment differed significantly from the learning effect in the classroom learning environment. Checking the
strength of association (r2 = 0.365), we see that the learning method accounted for 36.5% of the total variation in
learning effect in the 6th grade class.
Tabl e 3 . van Hiele Level Distribution on Pre-test and Post-test
3rd Grade
6th Grade
Test Level
Ex (n＝35),％
Con (n＝34),％
Ex (n＝39),％
Con (n＝40),％
Prism,
Pyramid,
Prism,
Pyramid,
Cube
Cuboid
Cube
Cuboid
Cylinder
Cone
Cylinder
Cone
30.8
20.5
42.5
30
*
0
0
0
0
48.6
45.7
26.5
44.1
30.8
48.8
30
50
1
Pre51.4
54.3
70.6
53
35.9
28.2
22.5
15
2
test
0
0
2.9
2.9
2.5
2.5
5
5
3
7.7
5
12.5
12.5
*
0
0
0
0
8.6
11.5
2.9
5.9
17.8
28.2
15
30
1
Post88.6
85.7
79.4
76.4
41
33.4
62.5
47.5
2
test
2.8
2.8
14.7
14.7
33.4
33.4
10
10
3
* = unreached Level 1
Table 3 shows that the van Hiele level of geometric thought in both groups was promoted from level 1 to level 2
or level 2 to level 3. The promotion percentage in the 3rd grade control group was greater than that of the
experimental group, however, in the 6th grade, the promotion percentage was greater in the experimental group
than in the control group.
Group interview
Students were questioned in a group interview about the various learning settings, with the interviewer recording
the students’ answers (I = interviewer). After the interview, the students were asked to make two perspective
drawings. Six students from each group were sampled and answers were coded as follows: E = experimental
group, C = control group, 3 = 3rd grade, 6 = 6th grade; score intervals are: 1 = high score, 2 = middle score and 3
= low score, and serial numbers are 1 and 2. Thus, code E312 represents the 2nd sample in the high score interval
of 3rd grade students in the experimental group. Students’ verbal statements are given below:
136
Experimental group
I: Just tell me about the activities of the geometry
course via the Internet?
E312: I watch the videos and the Web, then I discuss
with teacher and classmates in the chat room.
E321: I do the same things, and I think the 3D things
in the screen are interesting.
E331: But, sometimes I don’t know what to do.
E611: I can receive the reply from teacher by e-mail or
bulletin board.
E622: When I click the hyperlink, I can explore the
information of the geometry course.
E631: Yes, if I can’t understand the geometry course, I
can watch the videos again.
Control group
I: Just tell me about the activities of the geometry
course in the classroom?
C311: Teacher asks us to open the textbook, we go
page by page.
C322: And we make some paper models and play with
them.
C332: I don’t know what teacher says, I forget.
C612: We read and listen to what teacher teaches.
After school, I do homework and complete exercises.
C622: At times, we make too much noise, and teacher
silences us. But that is not all my fault.
C631: I don’t know who I can ask about the geometry
problems. Our teacher just teaches all the time.
I: Can you tell me how you play with the 3D
animations?
E311: I can rotate and move them with the mouse.
E332: I can find their surfaces, sides and corners.
E612: I can operate them easily with the mouse, their
solid shapes are clear.
E632: I have seen 3D things in video games, so I know
how to use them.
I: Can you tell me how you play with the paper
models?
C312: I can make the paper models and observe them.
C331: The paper models are like building blocks, I can
use them to build a house.
C611: Teacher tells us how to make them, I can
understand more about geometry.
C632: I can’t make the paper models.
I: How do you feel the Internet discussions went?
E311: I can discuss with teacher and classmates in the
chat room, and send my questions. I can answer others’
questions, too.
E322: I don’t worry about making too much noise on
the Web, or teacher asking me to be quiet.
E331: I can see what I don’t understand while they
talk.
E612: It’s perfect. I can communicate with teacher and
classmates by typing.
E621: And we share our thinking on the bulletin board
or by e-mail.
E632: Although I type slowly, everyone can see what I
type in the list.
I: How do you feel the classroom discussions went?
C312: I can answer every question teacher asks.
C321: Teacher doesn’t notice when I raise my hand
most times.
C331: Nobody asks me, and I don’t want to reply to
any questions.
C612: In my group, they don’t listen to me. How can I
complete the task?
C621: I don’t need to do too much. Jay and Lin usually
will do the whole job.
C632: I don’t know how to join them, they are too
smart.
I: What is the most difficult for you on the Internet?
E312: Nothing at all! I feel it is interesting. Will we
have another course?
E321: I don’t see a real teacher.
E332: Typing is the most difficult thing for me. I can’t
keep up with others.
E611: Too many others want me to help them and
reply to them. I don’t like it.
E622: People saying something dirty in the chat rooms
is unlovely.
E631: I can’t type fast, because it is hard for me to
spell well.
I: What is the most difficult for you in the classroom?
C311: Nothing is difficult for me in the classroom, I do
whatever teacher says.
C322: I can’t solve some homework exercises of the,
and I can’t find any help.
C332: The geometry course is the most difficult for me
anyway.
C611: I think I don’t have enough ability to be the
leader of the team.
C622: I can’t join discussions. Some people are
uncooperative.
C631: Everything is difficult for me. I wish I could
understand geometry better.
Interpretation of the students’ responses to the questions above helped to explain the quantitative results obtained
with the ANCOVA. The group interview results can be classified into three main viewpoints on the Web-based
and classroom-based learning settings. The first viewpoint is that students in high score intervals could complete
most of the learning objectives and solve problems in both 3rd grade and 6th grade settings. The second viewpoint
is that students in the middle score interval of control groups of both grades hardly gained any help when they
faced problems with geometry questions. Students in the middle score interval of the 6th grade experimental
137
group could acquire help from the chat room, bulletin board, or e-mail, but not the 3rd grade. The third viewpoint
is that the main difficulty students in the low score intervals of both grades’ control groups had was in the
content of the geometry course. Another difficulty students in the low score intervals of both grades’
experimental groups had was inability to use computers and the Internet. In particular, they all complained of
slow keying-in and spelling.
The Perspective drawings
Code
E311
Cube
Table 4. Perspective drawings by 3rd grade and 6th grade students
Cuboid
Code
Cube
C311
E312
C312
E321
C321
E322
C322
E331
C331
E332
C332
Code
E611
Prism or cylinder
Pyramid or cone
Code
C611
E612
C612
E621
C621
E622
C622
E631
C631
E632
C632
Prism or cylinder
Cuboid
Pyramid or cone
138
Students need both spatial and drafting ability. From Table 4, we can see that the students with higher van Hiele
levels made better drawings. However, while subject C632 had better sketching ability, he couldn’t reach level 1
of the van Hiele theory.
Discussion
The results above show that learning method accounted for a total variation in learning effect of 19.1% in the 3rd
grade and 36.5% in the 6th grade. And the promotion percentage in van Hiele level of geometric thought in the
3rd grade control group was greater than the promotion percentage in the experimental group, but the promoted
percentage in van Hiele level of geometric thought in the 6th grade experimental group was greater than the
promotion percentage in the control group. The group interview results explain why learner-control instruction
indeed improved students’ achievements, but also produced large variations in students’ performance. Those
variations may come from the instructional task, instructional design, or assessment procedure results. In other
words, students in the experimental groups thought that learning in the Web-based learning environment was
interesting, helpful, and convenient, but some had difficulty typing and manipulating computers. A few also felt
that the lecture video wasn’t real. The influence of these problems was more serious than the shortcomings of
classroom learning in the 3rd grade, however, the reverse was true in the 6th grade. That is, the ability to learn
with computers and the Internet were the important factors in learner-controlled geometry instruction in the
Web-based learning environment. For reasons mentioned above, the 6th grade were possessed of better computer
skills and experiences than 3rd grade, so they could benefit more learning effect than the 3rd grade in a geometry
course via the Web-based learning environment of learner control.
Conclusion
The Web-based geometry course in this study was implemented with learner controlled instructional materials.
We observed that the 6th grade students’ ability to learn with computers and the Internet was sufficient to handle
problems and promote their van Hiele levels of geometric thought. But this instructional technology seemed to
threaten 3rd grade students’ learning. Our research goals were confined by the 3rd grade students’ ability to apply
information technology. Research into the application of the Internet has been an amazing journey and led to
some interesting observations. The evidence from the group interview pointed out that students need more
communication with peers, teachers, and lecture materials, but this need wasn’t satisfied for students in the
control group. However, students in the experimental group could be included in a navigational scheme that
reflected a balance between instructional control and program control based on learner control. This balance
could let students in the experimental group meet the requirements of curricular discussion. Thus, it will help
them to construct their geometric concepts and get greater achievement. We realize that this is but one small step
in a long and challenging journey. The ever-growing presence of the Web should facilitate necessary levels of
communication to continue our research and development. The technological advances in Internet progress will
make it more convenient and more effective for use in Web-based learning.
References
Clements, D. H., & Battista, M. T. (1992). Geometry and spatial reasoning. In Grouws, D. A. (Ed.), Handbook of
reasoning on mathematics teaching and learning, New York, USA: Macmillan, 420-464.
Clements, D. H., Swaminathan, S., Hannibal, M. A. Z., & Sarama, J. (1999). Young children’s concepts of
shape. Journal for Resaerch in Mathematics Education, 20 (2), 192~212.
Fowler, J. F. (1983). Use of computer-assisted instruction in introductory management science. Journal of
Experimental Education, 52, 22-26.
Gay, G. (1986). Interaction of learner control and prior understanding in computer-assisted video instruction.
Journal of Educational Psychology, 78, 225-227.
Hannafin, M. (1984). Guidelines for using locus of instructional control in the design of computer-assisted
instruction. Journal of Instructional Development, 7, 6-10.
139
Lepper, M. R. (1985). Microcomputers in education: Motivational and social issues. American Psychologist, 40,
1-18.
Mager, R. F., & Clark, C. (1963). Explorations in student-controlled instruction. Psychological Reports, 13, 7176.
Resier, R. A. (1984). Reducing student procrastination in a personalized system of instruction course.
Instructional-design Theories and Models: An Overview of Their Current Status, Hillsdale NJ, USA: Lawrence
Erlbaum Associates.
Van Hiele, P. M. (1986). Structure and insight. A theory of mathematics education, Orlando, FL, USA:
Academic Press.
140
Askar, P., Usluel, Y. K. & Mumcu, F. K. (2006). Logistic Regression Modeling for Predicting Task-Related ICT Use in
Teaching. Educational Technology & Society, 9 (2), 141-151.
Logistic Regression Modeling for Predicting Task-Related ICT Use in
Teaching
Petek Askar
Department of Computer Education and Instructional Technology
Faculty of Education, Hacettepe University, 06800, Beytepe, Ankara, Turkey
Tel: + 90 312 2977180
Fax: + 90 312 2977176
[email protected]
Yasemin Kocak Usluel
Tel: +90 312 2978569
Fax: + 90 312 2977176
[email protected]
Filiz Kuskaya Mumcu
Tel: +90 312 4207924
Fax: +90 312 4207274
[email protected]
ABSTRACT
The main goal of this study is to estimate the extent to which perceived innovation characteristics are
associated with the probability of task related ICT use among secondary school teachers. The tasks were
categorized as teaching preparation, teaching delivery, and management. Four hundred and sixteen teachers
from secondary schools in Turkey, completed a questionnaire, which was designed to determine the taskrelated usage and the perceptions of the teachers in regard to ICT. Logistic stepwise regression analysis
showed that complexity or ease of use was found to be a common perceived innovation characteristic for
teaching delivery, preparation and managerial tasks in schools. Another result of this survey lead one to
conclude that observability is a perceived attribute in teaching delivery in some specific tasks performed
during the class period whereas relative advantage and compatibility are for teaching preparation tasks.
Keywords
Innovation characteristics, Logistic regression, ICT, Secondary schools
Introduction
The way of the influence on education system exerted by innovations in information and communication
technologies (ICT) had previously been confined to training individuals for technology–literacy. This function,
has gradually changed, and developed into a new dimension, affecting the learning–teaching processes in a direct
way. Indeed, when looked at the National Educational Technology Standards (NETS) of the International
Society for Technology Education (ISTE), it is seen that the skills required for teachers were no longer limited to
knowing the basic processes and concepts related to technology, but developed into a wider spectrum,
comprising the integration of technology into education, and knowing and implementing to ethical principles
related to the use of these technologies (ISTE 2004). Apart from these skills that teachers are expected to have, it
has also been put forward through researches that teachers, in connection with the use of ICT in classes, have
developed their own principles, ideas and judgements, and have influenced their implementation (Galanouli,
Murphy, & Gardner, 2004; Cope & Ward, 2002; Mumtaz, 2000). For this reason, it is seen that teachers have
important role in ICT use in schools. Furthermore, the fact that educational changes rely largely on the adoption
of this change by teachers (Van den Berg, Vandenberghe, & Sleegers, 1999; Fullan, 1991; Hall & Hord 1987), is
also indicated in literature study.
Diffusion of ICT in Education
Use of ICT in schools for the purpose of teaching and learning is a kind of diffusion process in which ICT is an
innovation which is defined by Rogers (2003, p.12) as “any idea, practice or object that is perceived as new by
141
an individual or other unit of adoption”. In fact, ICT as a relatively new building block in the educational system,
causes innovations which ranges from way of communications, and interactions to teaching methods, and
materials (Askar & Usluel, 2005). Rogers’ (2003, p. 11) research indicates, “Technological innovations are not
always diffused and adopted rapidly, even when the innovation has obvious, proven advantages”. Along with a
process for adoption, Rogers provides a theory of how the innovation itself can affect this process. He identified
five innovation characteristics that influence the decision to adopt an innovation.
The literature on diffusion of innovation highlights the factors that may influence a teacher’s likelihood of
utilizing computers for instructional purposes. Dooley and Murphrey (2000) stated that how teachers perceive
and react to these technologies is far more important than the technical obstacles in influencing implementation
and use. Four main elements in the diffusion of innovation are stated by Rogers (2003) as innovation,
communication channels, time, and social system.
The characteristics of an innovation, as perceived by the members of a social system, determine its rate of
adoption. Five attributes of innovations (Rogers, 2003) are; relative advantage, compatibility, complexity,
trialability, and observability. In other words, when an innovation is perceived by users as having greater relative
advantage, compatibility, trialability, observability, and less complexity, the innovation will be adopted more
rapidly.
The first characteristic, relative advantage of an innovation, as perceived by the members of a social system, is
positively related to its rate of adoption. Relative advantage is the degree to which an innovation is perceived as
better than the idea it supersedes. If the technology provides some type of increased effectiveness or efficiency,
then individuals are more likely to adopt the technology (Rogers, 2003). The nature of the innovation largely
determines what specific type of relative advantage is important to adopters expressed as economic profitability,
social prestige, decrease in discomfort, low initial cost, time savings, savings in effort, and immediacy of
rewards, or other benefits. The degree of relative advantage is often expressed in economic profitability and in
status giving. This suggests the need to focus on the specific pedagogical advantages of the instructional
technology over more conventional teaching tools in the in-service training program (Bennett & Bennett, 2003).
So, relative advantage is the degree to which a teacher perceives a new technology as superior to existing
substitutes.
The second characteristic, compatibility of an innovation, as perceived by the members of a social system, is
positively related to its rate of adoption. Compatibility is the degree to which an innovation is perceived as being
consistent with the existing values, past experiences, and needs of potential adopters (Rogers, 2003. p. 15).
Socio-cultural values and beliefs, previously adopted ideas and the needs determine compatibility or
incompatibility. So, compatibility is the degree to which ICT in education is perceived as being consistent with
the teachers’ values, past experiences and needs.
The third characteristic, complexity of an innovation, as perceived by the members of a social system, is
negatively related to its rate of adoption. Complexity is the degree to which an innovation is perceived as
difficult to understand and use (Rogers, 2003). Complexity refers to the extent to which an innovation is difficult
to understand; the greater the difficulty, the more reluctant potential adopters will be to embrace the change. To
ensure that the fear of technical complexity does not present itself as an obstacle, it is important that the content
and outcomes of the development and training program be consistent with the knowledge, skills, and abilities of
the teacher involved. Complexity often found to be inversely related to the diffusion of an innovation, while
simplicity, or ease of use, makes for wider and more rapid acceptance (Serow & Zorowski, 1999)
The fourth characteristic, trialability of an innovation, as perceived by members of a social system, is positively
related to its rate of adoption. Trialability is the degree to which an innovation may be experienced with on a
limited basis. New ideas which go through the experimental process can be adopted faster (Rogers, 2003).
The last characteristic, observability in the results of an innovation is positively related to the rate of adoption.
Observability is the degree to which the results of innovation are visible to others (Rogers, 2003). If the
technology has a high degree of observability, it will be relatively easy for the teacher to learn about it and judge
its potential benefits. This, in turn, can increase the likelihood of adoption.
Bennett and Bennett (2003) studied the perceived characteristics of instructional technology that may influence a
faculty members’ willingness to integrate it in his/her teaching. They have expressed that the most important
barrier that teachers face using technology is not lack of technology or funds but teachers’ lack of willingness
and their belief that technology is not useful. Butler and Sellbom (2002) examined the factors affecting teachers
142
in adopting new teaching technologies and barriers emerging during adoption. Surveys have been mailed to 410
teachers, about 30% have responded. As a result of the research, trust in technology has been identified as the
most important factor in teachers’ decision whether or not to adopt. Know-how, difficulty in learning and time
required to learn appear as the second most important factor in adoption. Believing that technology enriches and
improves education, difficulty using technology and management support appears as other factors affecting
adoption. Another study investigating the rate of adoption with respect to computers in three primary schools
was conducted by Askar and Usluel (2003). During the research, the factors such as perceived attributes of
computers and some characteristics of schools which have an effect on the rate of adoption were taken into
consideration. For this purpose, in the year 2000, 27 teachers and in 2002, 31 teachers from the same schools
were interviewed. Changes in the ratio of computer use were identified over two years. Four variables which
might be effective in the differences were: relative advantage, observability, factors which are encouraging and
hindering the use of computers in the schools. Mumcu (2004) highlighted in her research on the diffusion of ICT
in vocational and technical schools that the most critical obstacles were found as insufficient budget, hardware
and in-service training. In addition, a positive relationship between relative advantage, compatibility and
visibility with the use of ICT were reported.
Dooley, Metcalf, and Martinez (1999) explained that public schools are installing computer technology in
classrooms at an alarming rate. However, the training for this infusion of technology does not always transfer to
integration of the technology into the curriculum. With the introduction of computers in schools, there are
significant changes in the school organization and the roles of the teachers, administrators, parents and students.
For institutional change, it is imperative that school personnel understand the diffusion process and its
implications for success or failure of innovations.
In addition to ICT usage, Internet adoption is attracting the researchers. Jebeile and Reeve (2003) reported the
findings of a study of teacher adoption of Web technology in a secondary college. The results showed that the
innovation adoption variables of relative advantage, compatibility, visibility, ease of use, results demonstrability,
and trialability should be considered by school administrators seeking to increase the use of e-learning within
their organizations. Braak (2001a) examined the factors influencing the use of computer-mediated
communication (CMC) by teachers in secondary schools in Brussels. The survey compared a group of CMC
users with non-CMC users. It was demonstrated that language teaching was the best predictor for the use of
CMC. The main reason for this is that education policy within the area under investigation has developed a
specific CMC project that is primarily oriented towards a target group of language teachers. A second predictor
of CMC use was the degree of technological innovativeness. This instrument is a measure of the willingness of
the teacher to adopt technological innovation in his own teaching practice. A third predictor was perceived CMC
attributes. This instrument indicates the degree to which users observe any congruence between the
characteristics of CMC as a medium and their own teaching practice. Martins, Steil, and Todesco (2004)
conducted a survey on 92 language schools in Brazil. Results revealed that the Internet is adopted in 55% of the
schools analyzed. Both the model of linear multiple regression and the model of logistic regression predicted
77% of the cases of adoption and, therefore, represented satisfactorily the data from the questionnaire used. The
variables observability and trialability were found to be the two most significant predictors of adoption.
In one of the studies (Braak, 2001b), logistic regression analysis results showed that technological
innovativeness, teaching a technology-related subject, and computer experience were more important in
explaining the computer use in the classroom than the computer attitude, general innovativeness, age, and
gender.
In summary, as Surry and Gustafson (1994) concluded, compatibility, complexity and relative advantage can be
important considerations when introducing an innovation into instructional settings. In parallel Bussey,
Dormody, and VanLeeuwen (2000) stated that the strongest predictor of the level of adoption of technology
education was the perception of the teacher of the attributes of technology education. The researchers also
concluded that Rogers’ theory of perceived attributes can be a valuable tool for instructional developers working
to increase the utilization of their products.
ICT in schools in Turkey
ICT was first introduced to schools in Turkey in 1984. Since then, Ministry of National Education has allocated
considerable amount of budget for the diffusion of computers in the teaching and learning process. For the
diffusion of ICT into schools, so many efforts have been undertaken such as in-service training of teachers and
administrators, courseware and educational material development and training of computer coordinators.
143
The body of authority is the key policy makers in the Ministry of National Education for all levels of schools in
Turkey. However the implementation of any innovation, including ICT is under the responsibility of the schools.
In a longitudinal and qualitative study (Askar & Usluel, 2003) held in three primary schools, the change in the
situation of teachers related to their use of computers were observed throughout the two years (2000 and 2002)
with respect to each school; and it was seen that there were differences, with respect to each school, in the use of
computers by teachers. Therefore the adoption rate varies due to schools and teachers. To install a computer lab
or to get a trained teacher (computer coordinator) don’t imply that adoption is successful even it is an authority
innovation-decision. Indeed, as a result of the analysis of the data obtained through questionnaire from 638
teachers in 27 primary schools with the aim of identifying the purposes of Turkish teachers in using ICT, it was
noticed that computers in schools have become widely used in administrative work – such as, preparation of
lecture plans and unit plans, organizing scores and reports of students, writing official letters. However, this wide
use of computers has not yet been observed in the instructional activities – either using a presentation tool during
class or using computers for experiments (Usluel & Askar, 2002). In addition it was observed that instructional
usage could be grouped as teaching preparation and delivery (Askar & Usluel, 2003).
It is supported by literature that ICT have entered the lives of teachers and are used in administrative tasks at
schools, but not in instructional tasks. Studies pointed out that teachers’ use of ICT for instructional purposes is
insufficient (Martins, Steil, & Todesco, 2004; Askar & Usluel, 2003; Szabo & Suen, 1998; Proulx & Campbell,
1997). Hence task-related analyses in the framework of innovation characteristics (Rogers, 2003) would be
important for developing strategies in the diffusion of ICT in schools.
Research Questions
This research investigates the adoption of task-related ICT by teachers. The task-related ICT usage was modeled
by the four perceived attributes: relative advantage, compatibility, complexity, and observability. The trialability
was not considered to be a differentiating attribute to ponder on, since almost all teachers respond to trialability
questions affirmatively.
Do the four ICT innovation characteristics discriminate the users and non-users on the following tasks?
1. Preparation for a lesson such as developing worksheets or internet search for course content (Teaching
preparation)
2. Preparation of lesson plans, unit plans and yearly plans (Teaching preparation)
3. Use of educational multimedia while giving lectures, doing practices reviewing lessons (Teaching delivery)
4. Using a presentation tool during class. (Teaching delivery)
5. Using ICT for experimental study in the laboratories (Teaching delivery)
6. Preparation of examination or organizing student scores (Management)
7. Writing and saving official letters (Management)
Methodology
This research was based on causal-comparative research design. In causal-comparative research, investigators
attempt to determine the cause or consequences of differences that already exist between or among groups of
individuals (Fraenkel & Wallen, 2003). The concern of this study was to identify of membership of users and
non users given in the research questions according to the perceived attributes. Since the outcome variable was
dichotomous the binary logistic regression model was used.
Participants
The participants of the study were 416 teachers from 8 secondary schools in Ankara, capital of Turkey. There are
8 counties in the provincial capital of Ankara. One county has been chosen among these, and efforts have been
exerted to reach all secondary school teachers. There were 710 teachers in the secondary schools of that county;
548 of them have been reached through the survey, 425 of them have replied and 416 questionnaire forms have
been evaluated. The demographical information is shown on Table 1.
144
Years in this
school
Years in
teaching
Age
Gender
Table 1. Demographical information of the teachers
TOTAL
Demographical Information
f
Female
286
%
68.8
Male
130
31.2
20-29 age
30-39 age
40-49 age
50-59 age
1-5 years
6-10 years
11-15 years
16-20 years
21-25 years
26 years and more
1-5 years
6-10 years
11-15 years
16-20 years
21-25 years
44
170
174
28
42
74
119
82
69
30
191
94
80
48
3
416
10.6
40.9
41.8
6.7
10.1
17.8
28.6
19.7
16.6
7.2
45.9
22.6
19.2
11.5
0.7
100
TOTAL
Data Collection and Analysis
The questionnaire were developed by the researchers and administered to the teachers one by one. Since the
items in the questionnaire were used individually in the data analysis, the total score was meaningless. The
questionnaire was divided into three sections. Section 1 consisted of demographical information dealing with
gender, age, educational qualifications, years in teaching, and years in this school concerning the respondents.
Section 2 included tasks eliciting the views of the teachers toward the use of ICT in teaching preparation,
teaching delivery, and management. Section 3 consisted of the views of the teachers regarding the innovation
characteristics toward the use of ICT on the tasks.
An example was given for section 2 and for section 3;
Section 2
“Do you use educational multimedia while giving lectures, doing practices or reviewing
lessons?”
The responses were taken as ‘yes’ or ‘no’.
Section 3
“Rate your opinion whether the following task is observable or not.”
Use of educational multimedia while giving lectures, doing practices reviewing lessons.
The responses were taken as ‘not observable’, ‘undecided’, and ‘observable’.
“Rate your opinion whether the following task is complex or not.”
Preparation of lesson plans, unit plans and yearly plans
The responses were taken as ‘not complex’, ‘undecided’, and ‘complex’
“Rate your opinion whether the following task is compatible with your job.”
Using a presentation tool during class.
The responses were taken as ‘not compatible’, ‘undecided’, and ‘compatible’
Since the outcome variable is dichotomous (binary) the binary logistic regression model was used. It was defined
as Y=0 (non-use), or Y=1 (use). X denotes the vector of independent variables or predictors. The independent
variables are perceived attributes of innovation; relative advantage, compatibility, complexity and observability.
The binary logistic regression is stated in terms of the probability that Y=1 given X:
145
P (Y = 1 | X ) =
1
1 + Exp(− βX )
Ζ = βX is called the linear predictor and stands for β0 + β1X1 +……….+ βpXp . By solving this equation Y, the
form for the binary logistic regression model is obtained:
ln
P (Y = 1 | X )
= logit (Y ) = β X (Košmely & Vadnal, 2003).
P (Y = 0 | X )
The recommended sample size for this kind of research is calculated by taking into consideration the minimum
ratio (sample size/ predictor variables) of 10 to 1, with a minimum sample size of 100 or 50, plus a variable
number that is a function of the number of predictors (Peng, Lee & Ingersoll, 2002). It was satisfied in this
study.
Findings
Task 1: Preparation for a lesson such as developing worksheets or internet search for course content
(Teaching preparation)
The binary logistic stepwise regression was conducted of four innovation characteristics: relative advantage,
compatibility, complexity and observability on the dependent variable task 1 (use or not use). Out of 407, 175
teachers coded as users and 232 non-users. Variables in the equation are complexity and relative advantage (for
complexity β=-1.014, SE=0.141, Wald’s χ 2=51.401, p=0.000 and for relative advantage β=0.789, SE=0.397,
Wald’s χ 2= 3.942, p=0.047).
The equation for task 1 is Z = (-0.492) + (-1.014) * Complexity + 0.789 * Relative Advantage. The Nagelkerke
R2 was reported for each predictor (R2 =0.208, R2(complexity)=0.195). Figure 1 shows the probability of a
teacher using ICT while preparing a lesson as a function of complexity and relative advantage.
0.8
Estimated
Probability
0.7
0.6
0.5
0.4
Relative
Advantage
0.3
Yes
0.2
0.1
Undecided
Not
0
Not Complex
Undecided
Complex
Complexity
Figure 1. The likelihood of a teacher using ICT for task 1 by perceived complexity and relative advantage
Complexity and relative advantage are predictors that distinguish between the teachers who do and who do not
benefit from ICT in preparing instructional activities. The highest probability of usage is the category of teachers
who found computers for this task advantageous and who found it easy to use (P=0.7028). The second highest
probability is the category of teachers who couldn’t decide whether it is advantageous or not, but found it easy to
use (P=0.5180). The lowest probability is the group who thought that it had no advantage to him (or her) and it
was complex to use.
146
Task 2: Preparation of lesson plans, unit plans and yearly plans (Teaching preparation)
The binary logistic stepwise regression analysis for this task showed that two variables were significant for
discriminating the users and non-users. Out of 406, 147 were non-users, 259 were users. Variables in the
equation were complexity and compatibility (for complexity β=-1.376, SE=0.177, Wald’s χ 2=60.674, p=0.000
and for compatibility β=1.330, SE=0.364, Wald’s χ 2=13.368, p=0.000).
Estimated
Probability
The equation for task 2 is Z = (-1.305) + (-1.376) * Complexity + 1.330 * Compatibility (R2=0.354, R2
(complexity) =0.307). Figure 2 shows the probability of a teacher using ICT while preparing of lesson plans, unit
plans and yearly plans experiments in the laboratories as a function of complexity and compatibility.
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Not Compatible
Not Complex
Undecided
Undecided
Compatibility
Compatible
Complex
Complexity
Figure 2. The likelihood of a teacher using ICT for task 2 by perceived complexity and compatibility
Preparing lesson plans, which is one of the teaching preparation tasks, indicates the highest usage probability. In
this task, as well, compatibility and ease of use come to the foreground. The highest probability of usage is the
categories of teachers who found computers for this task compatible and who found it easy to use (P=0.7873).
The second highest is the teachers who had no idea for compatibility, but found it easy to use (P=0.4947).
Estimated
Probability
Task 3: Use of educational multimedia during lecturing, practices and reviewing lessons (Teaching
delivery)
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
Observability
Observable
Undecided
Not Observable
Not Complex
Undecided
Complex
Complexity
Figure 3. The likelihood of a teacher using ICT for task 3 by perceived complexity and observability
The binary logistic stepwise regression analysis for this task showed that two variables were significant for
discriminating the users and non-users. Out of 406, 296 were non-users, 110 were users. Variables in the
147
equation were complexity and observability (for complexity β=-0.871, SE=0.186, Wald’s χ 2=21.851, p=0.000
and for observability β=0.449, SE=0.155, Wald’s χ 2= 8.412, p=0.004).
The equation for task 3 is Z = (-0.680) + (-0.871) * Complexity + 0.449 * Observability (R2 =0.173, R2
(complexity) =0.145). Figure 3 shows the probability of a teacher using ICT for educational multimedia during
lecturing, practices and reviewing lessons as a function of complexity and observability.
The probability of using multimedia during lessons has been found lower than the probability of using computers
for preparation of instructional activities. This result indicates that diffusion of using ICT for preparation of
lesson is more rapid than using ICT during lesson. The highest probability of usage is the category of teachers
who found computers non complex for this task and who observed its usage (P=0.4491). However still the
probability is lower than 0.50. While ICT usage during lesson is related to the teachers’ skills in one respect, it
may also be related to the infrastructure, technology, and physical features of classrooms.
Task 4: Using a presentation tool during class (Teaching delivery)
For this task out of 407, 323 were non-users, 84 were users. The variables in the equation were complexity and
compatibility (for complexity β=-1.447, SE=0.314, Wald’s χ 2=21.300, p=0.000 and for compatibility β=4.565,
SE=0.570, Wald’s χ 2=7.533, p=0.006).
The equation for task 4 is Z = (-3.777) + (-1.447) * Complexity + 1.565 * Compatibility (R2=0.291, R2
(complexity) =0.248). Figure 4 shows the probability of a teacher using a presentation tool during class as a
function of complexity and compatibility.
0.4
0.35
Compatible
0.3
Estimated
Probability
0.25
0.2
0.15
0.1
0.05
Compatibility
Undecided
Not Compatible
0
Not Complex
Undecided
Complex
Complexity
Figure 4. The likelihood of a teacher using ICT for task 4 by perceived complexity and compatibility
The two innovation characteristics which distinguish between the teachers who use presentation program during
lesson and who do not are ease of use and compatibility. The group with highest probability is the one who find
it easy to use and compatible (P=0.3707). What draws attention here is presence of compatibility characteristic
together with the ease of use. As is known, one indication of compatibility is the habits and how compatible they
are to the way of life. The reason why the probability of use of presentation program by teachers in classes is
high is that they likely perceive the presentation program as the extension of the white-board and overhead
projector.
Task 5: Using computers for experiments in the laboratories (Teaching delivery)
The binary logistic stepwise regression analysis showed that complexity and observability were significant for
discriminating the users and non-users (for complexity β=-1.614, SE=0.284, Wald’s χ 2=32.360, p=0.000 and for
observability β=0.602, SE=0.170, Wald’s χ 2=12.562, p=0.000). Out of 407, 313 were non-users, 94 were users.
148
The equation for task 5 is Z = (-0.257) + (-1.614) * Complexity + 0.602 * Observability (R2=0.308, R2
(complexity) =0.266). Figure 5 shows the probability of a teacher using ICT for experiments in the laboratories
as a function of complexity and observability.
0.6
Observable
Estimated
Probability
0.5
0.4
0.3
0.2
0.1
Observability
Undecided
Not Observable
0
Not Complex
Undecided
Complex
Complexity
Figure 5. The likelihood of a teacher using ICT for task 5 by perceived complexity and observability
The highest probability of usage is the categories of teachers who found computers non complex and who
observed its usage (P=0.4837). In this task, observability arises as a discriminating variable with ease of use. The
highest probability for this task is category of teachers who find it observable and ease of use. This finding
indicates that teachers should be given chances to observe the ICT integration done by their associates in the
classroom.
Task 6- Preparation of examination or organizing student scores (Management) and Task 7- Writing and
saving official letters (Management)
The binary logistic stepwise regression analysis for task 6 and 7 showed that complexity is the only
discriminating variable. For task 6, out of 407, 128 were non-users, 279 were users (for complexity β=-1.416,
SE=0.183, Wald’s χ 2=59.658, p=0.000). For task 7, out of 407, 164 were non-users, 243 were users (for
complexity β=-1.416, SE=0.183, Wald’s χ 2=59.825, p=0.000). While the probability of ICT usage by teachers
who find it easy are 0.7835 (task 6), and 0.7164 (task 7), the probability of ICT usage by them who don’t find it
easy are 0.1756 (task 6) and 0.1107 (task 7). The equation for task 6 is Z = 2.702 + (-1.416) * Complexity (R2
=0.245). The equation for task 7 is Z = 2.432 + (-1.505) * Complexity (R2 =0.264).
Conclusion & Discussion
The role that perceptions play in the adoption of an innovation has been examined by different researchers
(Martins, Steil, & Todesco, 2004; Bennett & Bennett, 2003; Jebeile & Reeve, 2003; Braak, 2001; Bussey,
Dormody, & VanLeeuwen, 2000; Serow & Zorowski, 1999; Surry & Gustafson, 1994). These researchers also
confirm the importance of the study of perceived attributes and rate of adoption.
As is seen, complexity comes out as the predictor variable in all tasks. While complexity and relative advantage
comes forth in preparation for lesson activities, being predictor of compatibility together with complexity in
presentation during lesson and preparation of lesson plans, and also coming forth of observability together with
complexity in the use of multimedia during lesson and experiments in the lab, prove that perceived attributes
could vary according to tasks.
Complexity or ease of use was found to be a common perceived innovation characteristic for teaching delivery,
preparation and managerial tasks in schools. As a predictor variable, complexity (ease of use) explains the
user/non user between 15% and 31% according to the tasks. Complexity as defined before is the degree to which
149
the technology is difficult to understand or use. The strategies for decreasing the perceived complexity are highly
important for diffusion of ICT in schools. The design of staff development and the ongoing and immediate
technical and educational support could be key considerations. The barriers teachers stated in several studies are
likely to be related to the perceived complexity.
Another result of this survey lead one to conclude that “observability” is a perceived attribute in teaching
delivery in some specific tasks performed during the class period. Besides professional development of teachers,
the classroom activities should be open for all teachers in the school. Teachers should be given chances to
observe the ICT integration done by their associates in the classroom. Therefore, it is easy to learn about it and
judge its potential benefits. The importance of observability indicates how critical it is that the instructional
technology be demonstrated in the in-service training program. Ideas that can easily be observed and
communicated to others will be adopted more quickly than ideas that are more difficult to see and communicate.
It is recommended that the best practices in ICT integration in schools should be shared by the teachers with
active participation are more useful than the traditional in-service teacher training programs. A program that
focuses on the interactive instructional properties of the technology would be of greater interest to this teacher
than one that failed to discuss how the technology is consistent with his/her teaching philosophy. Unfortunately,
in many circumstances, the introduction of instructional technology will require the rejection of one set of values
and ideas about education and the adoption of a new set with regards to what constitutes effective pedagogy
(Bennett & Bennett, 2003).
It can be argued that installing ICT for schools solely as product doesn’t carry much meaning. For this reason
besides of the nationwide solutions, school based solutions will be more realistic in the diffusion of computers
for instructional purposes. One of the results of this research is that, complexity comes out as the predictor
variable in all tasks, and that other variables vary according to the tasks. School administration should focus their
efforts on decreasing complexity. Technical support, on going teacher professional development, early
familiarity with the ICT, sharing best practices as well as barriers and difficulties in real teaching-learning
environments could be some strategies in the diffusion process. On the other hand, the importance of school’s
implementations according to a phased plan by way of determining its priorities with respect to ICT, and shaping
it according to the task, cannot be ignored. Further detailed studies on the ICT diffusion in schools will
illuminate this subject.
References
Askar, P., & Usluel, Y. K. (2003). A longitudinal study related to the rate of adoption of computers: Comparison
of three schools. Hacettepe University Journal of Education, 24, 15-25.
Askar, P., & Usluel, Y. K. (2005). The diffusion of computers in the schools: A comparison of three cases on the
basis of complex systems. In Berg, G. A. (Eds.), Encyclopedia of International Computer-based Learning: IDEA
accepted for publication.
Bennett, J., & Bennett, L. (2003). A review of factors that influence the diffusion of innovation when structuring
a faculty training program. Internet and Higher Education, 6, 53-63.
Braak, J. Van (2001a). Factors influencing the use of computer mediated communication by teachers in
secondary schools. Computers & Education, 36, 41-57.
Braak, J. Van (2001b). Individual characteristics influencing teachers class use of computers. Journal of
Educational Computing Research, 25 (2), 141-157.
Bussey, J. M., Dormody, T. J. & VanLeeuwen, D. (2000). Some factors predicting the adoption of technology
education in New Mexico public schools. Journal of Technology Education, 12 (1), retrieved May 1, 2006 from
http://scholar.lib.vt.edu/ejournals/JTE/v12n1/bussey.html.
Butler, D. L., & Sellbom, M. (2002). Barriers to adopting technology. Educause Quarterly, 2, 22-28.
Cope, C., & Ward, P. (2002). Integrating learning technology into classrooms: The importance of teachers’
perceptions. Educational Technology & Society, 5 (1), 67-70.
150
Dooley, L. M., Metcalf, T., & Martinez, A. (1999). A study of the adoption of computer technology by teachers.
Dooley, K. E. & Murphrey, T. P. (2000). How the perspectives of administrators, faculty, and support units
impact the rate of distance education adoption. Online Journal of Distance Learning Administration, 3 (4),
retrieved May 1, 2006 from http://www.westga.edu/~distance/ojdla/winter34/dooley34.html.
Fraenkel, J. R., & Wallen, N. E. ( 2003). How to design and Evaluate Research in Education (5th Ed.), New
York: McGraw-Hill.
Fullan, M. G. (1991). The new meaning of educational change (2nd Ed.), New York, USA: Teachers College
Press.
Galanouli, D., Murphy, C., & Gardner, J. (2004). Teachers’ perceptions of the effectiveness of ICT-competence
training. Computers & Education, 43, 63-79.
Hall, G. E., & Hord, S. M. (1987). Change in Schools: Facilitating the Process, USA: Suny Press.
ISTE (2004). The National Educational Technology Standards, retrieved November 30, 2005 from
http://www.iste.org/standards/.
Jebeile, S., & Reeve, R. (2003). The diffusion of e-learning innovations in an Australian secondary college:
Strategies and tactics for educational leaders. The Innovation Journal, 8 (4), retrieved May 1, 2006 from
http://www.innovation.cc/peer-reviewed/jebeile-reeve-elearning.pdf.
Košmely, K., & Vadnal, K. (2003). Comparison of two generalized logistic regression models: A case study.
Paper presented at the 25th International Conference on Information Technology Interfaces (ITI) 2003, June 1619, 2003, Cavtat, Croatia.
Martins, C. B. M. J., Steil, A. V., & Todesco, J. L. (2004). Factors influencing the adoption of the internet as a
teaching tool at foreign language schools. Computers & Education, 42, 353-374.
Mumcu, F. K. (2004). Diffusion of information and communication technologies in vocational and technical
schools, MS Thesis, Hacettepe University, Turkey.
Mumtaz, S. (2000). Factors affecting teachers’ use of Information and Communications Technology: A review
of the literature. Journal of Information Technology for Teacher Education, 9 (3), 319-333.
Peng, C. J., Lee, K. L., & Ingersoll, G. M. (2002). An introduction to logistic regression analysis and reporting.
The Journal of Educational Research, 96 (1).
Proulx, M., & Campbell, B. (1997). The professional practices of faculty and the diffusion of computer
Technologies in university teaching. Electronic Journal of Sociology, retreived November 30, 2005 from
http://www.sociology.org/content/vol002.003/proulx.html.
Rogers, M. E. (2003). Diffusion of innovations (5th Ed.), New York, USA: The Free Press.
Serow, R. C., & Zorowski, C. F. (1999). Diffusion of instructional innovations in engineering education. Paper
presented at ICEE’99, Czech Republic, Paper No. 258.
Surry, D. W., & Gustafson, K. L. (1994). The role of perceptions in the adoption of computer-based learning,
Education Resources Information Service (ERIC), ED 374 788.
Szabo, M., & Suen, C. (1998). A study of the impact of a school district computer technology program on
adoption
of
educational
technolog,
retreived
November
30,
2005
from
http://www.quasar.ualberta.ca/edmedia/Suenszabo.html.
Usluel, Y. K., & Askar, P. (2002). Teachers and use of computers. The Proceedings of BTIE -5, Informatics
Association Of Turkey Ankara-Turkey, 169-180.
Van Den Berg, R., Vandenberghe, R., & Sleegers, P. (1999). Management of innovations from a cultural
individual perspective. School Effectiveness and School Improvement, 10 (3), 321-351.
151
Kiraz, E. & Ozdemir, D. (2006). The Relationship between Educational Ideologies and Technology Acceptance in Preservice Teachers. Educational Technology & Society, 9 (2), 152-165.
The Relationship between Educational Ideologies and Technology
Acceptance in Pre-service Teachers
Ercan Kiraz
Middle East Technical University, Faculty of Education, Department of Educational Sciences, Ankara, Turkey
Tel: +90-312-210 40 37
Fax: +90 312 210 12 54
[email protected]
Devrim Ozdemir
Virginia Polytechnic Institute and State University, 220 War Memorial Hall 24061 Blacksburg, VA, USA
Tel: +1-540-239-37 61
[email protected]
ABSTRACT
After the evaluation of numerous technology integration programs in school districts and universities, it is
recognized that the existence of technology does not guarantee its utilization in the classroom environment.
Although many models and theories have tried to explain the contributing factors in technology acceptance,
most of the models and theories have focused on technology-related factors. This study focused on
educational ideology, a factor not related to technology that also affects decisions in terms of educational
applications. Based on the literature review, we hypothesized a new model of technology acceptance which
includes educational ideology as an external factor. We attempted to create a model that was compatible
with our hypothesized model by collecting data from surveys completed by 320 pre-service teachers.
Structural Equation Modeling was employed to create the path analytic model. The variables used in the
path analytic model were the components of the original Technology Acceptance Model and six different
educational ideologies. The results showed that the new model was consistent with the hypothesized model.
Therefore, the results illustrate that different educational ideologies may have different effects on teachers’
technology acceptance.
Keywords
Educational Ideologies; Technology Acceptance; Technology Adoption, Teacher Education
Introduction
With the swift advent of technology in previous decades, Information and Communication Technologies (ICT)
have pervaded the workplace and fostered modern corporations along with providing governments with a
proficient infrastructure. Besides these dramatic changes in many aspects of society, education remained by and
large a traditional craft (Perkins 1992, p.3).. According to Strommen (1992), “technological changes that
affected society left educational systems largely unchanged” (as cited in Semple, 2000, p. 21). Since the users
are imperative and have played a major role in the utilization of technology, factors affecting their technology
use became an important concern for researchers. This is because merely the existence of technology in the
classroom does not guarantee the utilization of that technology. Teachers are less likely to integrate technology
into their instruction unless they accept the notion of the requirement of educational technology use in their
classroom environment. (Stetheimer and Cleveland, 1998; Weiss, 1994). The central questions with regard to
technology acceptance are how individuals perceive technology and which factors contribute to the lack of
utilization (Rogers, 1995; Surry, 2000).
Technology Acceptance
It is commonly accepted that today’s teachers may benefit from educational technology to a great extent.
Therefore, failure in the usage or acceptance of technology is an important issue. At this point, it is necessary to
define the term ‘technology acceptance’ to determine the factors affecting the actual use of educational
technology in the classroom environment. Davis, Bagozzi and Warshaw (1989) defined significant factors
affecting technology acceptance in their Technology Acceptance Model (TAM). The TAM stems from
behavioral theory, and is a well-known model that has undergone significant developments since its conception.
A substantial number of factors have been added to TAM as possible significant determinants of technology use.
However, only some of the factors remained as major determinants of technology use. In order to understand the
rationale in the TAM model, it is necessary to mention the supporting theories.
152
One of these behavioral theories is the Social Cognitive Theory (SCT) (Bandura, 1986). The SCT outlines three
reciprocal factors, the person, behavior, and the environment and explains the relationship among these factors.
In this reciprocity, personal and environmental factors affect behavior and vice versa. SCT mentions two
different channels, outcome expectation and self-efficacy, between the two factors, the person and behavior.
According to this theory, people ask two questions before behaving, ‘Can I do it?’ and ‘Should I do it?’ If they
believe that they are able to perform the behavior and also if they will attain good outcomes afterwards, they are
in close proximity to perform. Although the SCT model showed how the two main factors, outcome expectation
and self-efficacy, affected behavior, attitudinal factors were left unmentioned. In 1975, Fishbein and Ajzen
produced the Theory of Reasoned Action (TRA) by focusing on attitudinal factors on behavior. TRA posited
behavioral intention as a measure of one’s intention to perform a specific behavior and represented the primary
predictor of actual behavior (as cited in Brosnan, 1999). Therefore, according to TRA, self-efficacy and
outcome expectation does not directly affect the actual behavior; however, it affects behavioral intention. In
1988, the TRA model was modified and the theory was changed to the Theory of Planned Behavior (TPB).
According to the TPB, behavior is influenced solely by behavioral intention and behavioral intention in turn is
influenced by attitudes toward behavior, by subjective norms and by perceived behavioral control (Bajaj and
Nidumolu, 1998).
Finally, in 1989, Davis, Bagozzi and Warshaw defined a model, named the “Technology Acceptance Model”
(TAM) which covers some significant factors affecting the use of technology (Davis et al., 1989). Similar to the
TRA, the TAM explained how one primary factor, the behavioral intention to use, affected actual system use.
Although the behavioral intention to use is the only factor related to actual system use, recent studies confirmed
that this is only a mediating factor - a hypothetical factor and not used in most of the studies. Corresponding to
the TPB, the TAM explains attitudinal factors affecting the behavioral intention to use focusing on the factor,
attitudes towards the use of technology. In addition, similar to the SCT, the TAM demonstrates the effects of
self-efficacy and outcome expectation on attitudes toward use of technology. The TAM defined self-efficacy and
outcome expectation factors as the perceived ease of use of technology and perceived usefulness of technology
which are related to self-efficacy and outcome expectation. In TAM, all other factors related to technology
acceptance are called external variables.
In summary, technology acceptance is related to four main factors: the perceived ease of use of technology; the
perceived usefulness of technology; the attitudes toward the use of technology; and the frequency of use of
technology. For instance, if people perceive that technology is easy to use and useful in their field, this builds
positive attitudes toward the use of that technology. Positive attitudes toward the use of technology have a
positive relationship with the behavioral intention to use the technology. In this sense, four main factors of the
TAM, shown in Figure 1, have been used as major determinants of technology acceptance in this study.
Percieved
Usefullness
External
Variables
Attitude
Towards
Behavioural
Intention to
use
Actual
System Use
Percieved
Ease of Use
Figure 1. Original Technology Acceptance Model (Source: Legris et. al., 2003)
Thus far, many external variables have been tested in the model and few studies have explained more than 40%
of the variance in technology use (Legris, Ingham & Collerette 2003). In addition, Legris, Ingham and Collerette
stated that “TAM is a useful model, but has to be integrated into a broader model which would include variables
related to both human and social change processes, and to the adoption of the innovation model” (p.191). In
accordance with this claim, the main question of this study was “which factors influence the decision directly or
indirectly”. This study suggests that educational ideology may affect one’s decisions directly or indirectly, since
every individual has a belief system.
153
Educational Ideologies
Educational ideologies are rooted in political ideologies and affect decisions related to education. In 1981,
William F. O’Neill grouped educational ideologies into six different groups under two main categories. O’Neill
categorized these ideologies as conservative and liberal educational ideologies with three subgroups for each
category. The conservative educational ideologies are educational fundamentalism, educational intellectualism,
and educational conservatism. The liberal educational ideologies are educational liberalism, educational
liberationism, and educational anarchism. O’Neill (1990) mentioned that educational ideologies have an impact
on individuals’ beliefs with regard to the overall goals of education, the objectives of the school, the child as a
learner, administration and control, the nature of the curriculum, instructional methods and evaluation and
classroom control. However, O’Neill did not mention the effects of educational ideologies on the acceptance of
educational technology. At this point, it would be beneficial to delineate what educational ideologies are. The
information below is summarized from “Educational Ideologies: Contemporary Expressions of Educational
Philosophy” (O’Neill, 1990).
In educational fundamentalism, knowledge is a tool for reconstructing society in pursuit of a predetermined
pattern of moral excellence where man is a moral agent. The approach is tacit anti-intellectualism and is opposed
to the critical examination of preferred patterns of belief and behavior. Education is considered as moral
regeneration and the ideology focuses on the original purposes of the existing social traditions and institutions,
placing emphasis on a return to the past as a corrective reorientation.
In educational intellectualism, knowledge is viewed as an end to itself and truth has an intrinsic value, where
man is man. That is, man’s universal nature transcends specific circumstances. The approach is traditional
intellectualism (stressing reason and speculative wisdom). Education is an orientation to life in general. It
focuses on the intellectual history of a man, generally identified with the dominant Western intellectual tradition
of classicism.
In educational conservatism, knowledge is for social utility and a means of realizing existing social values. Man
is a citizen, who finds his highest fulfillment as an effective member of the established social order. This
approach is based on reasoned conformity and reliance on the best answers of the past as the most trustworthy
guide to present action. Education is considered as socialization to the established system. The ideology focuses
on existing social traditions and institutions and places emphasizes on the present situation, viewed in a
relatively shallow historical perspective, that is, conventionalism.
In educational liberalism, knowledge is a necessary tool used in practical problem solving. The individual is a
unique personality, who finds his greatest satisfaction in self-expression in response to changing conditions. The
approach is effective thinking (practical intelligence), and the ability to solve personal problems effectively.
Education is the development of personal effectiveness.
In educational liberationism, knowledge is a necessary tool for required social reforms. Man is a product of
culture, who finds his highest fulfillment along the lines defined and controlled by the existing social system.
The approach is based on the objective (rational- scientific) analysis and evaluation of existing social policies
and practices. Education is the fullest realization of each person’s unique potentialities as a distinctive human
being. This ideology focuses on social conditions that block the fullest realization of individual potentialities,
and emphasizes on the future (that is, on changes in the present system required to bring about a more
humanistic and humanizing society). The purpose of individual is to bring about immediate large scale changes
within the existing society. The ideology stresses the significant changes that affect the basic nature and conduct
of the established social system
In educational anarchism, knowledge is a natural by-product of daily living. Individual personality is a value
that transcends the requirements of any particular society. The approach is based on free choice and self
determination in a sane and humanistic social setting. Education is considered as a natural function of everyday
living in a rational and productive social environment and the ideology focuses on the development of an
“educational society” that either eliminates or radically minimizes the necessity for formal schools and other
such institutional constraints on personal behavior, and emphasizes on a post-historical future in which people
function as self-regulating moral beings. The purpose of the ideology is the continuous change and self-renewal
within a constantly emerging society; stresses the need for minimizing and / or eliminating institutional restraints
on a personal behavior (deinstitutionalization).
154
It is clear that each ideology focuses on different definitions of knowledge and of the individual, and each
ideology has different approaches to reach different goals related to education. In this sense, comparing the
differences in the general characteristics of educational ideologies, perceptions and attitudes regarding
educational technologies may change depending on ideology. While some teachers may think that educational
technology is crucial to reach the goal of education, some others may think that educational technologies are not
as useful to reach the goals, as the goal of technology may oppose the ideology of the individual.
The aforementioned shed light on the effect of factors such as the teachers’ perceptions, attitudes, and beliefs on
the use of educational technology. Therefore, using educational ideologies and attitudes, beliefs, and perceptions
as meaningful categories, this study investigated whether educational ideologies of pre-service teachers have an
effect on technology acceptance.
Method
Based on the literature, this study hypothesized a model including educational ideologies as an external variable
of the Technology Acceptance Model. The methodology section of this study outlines the sample, the
instrument, the data analysis stages, and presents the stages necessary to obtain a model compatible with the
hypothesized model
Sample
Since this study is related with technology acceptance and educational ideologies, whether participants have a
well formed educational philosophy is an important issue. Participants of the study were graduates of Teachers’
High Schools and are currently attending the Faculty of Education in the Middle East Technical University
(METU), Ankara, Turkey. They were selected through nation wide examinations and are training to be teachers.
During both their high school and university educations, they have taken courses relating to teaching and
learning, such as “Introduction to Educational Sciences”, “Introduction to the Teaching Profession”,
“Instructional Planning and Evaluation”, “Development and Learning”, “Introduction to Computer Education”,
“Instructional Materials Development”, “Classroom Management”, and so forth. They have had exposure to
subjects such as philosophy, educational philosophy, terrains of philosophy, schools of philosophy, and the
relationship between philosophy and educational practices. They have also had work experience during their
undergraduate practicum period, a requirement for all teacher candidates set by the Faculty of Education.
Therefore, it can be assumed that these students have satisfactory theoretical and practical backgrounds
regarding the formation of a philosophical context.
The sample consisted of 320 students. In terms of their departments, the majority of these students were
attending the Department of Elementary Education (50.3%) followed by the Department of Foreign Language
Education (38.4%). Furthermore, most of the students (43.4%) who responded to the survey were freshmen,
14.1% of the students were sophomore, 22.2% of the students were junior, and 17.2% of the students were at the
senior level. The majority of the sample was female (66.9%). Finally, 50.3% of the students were between the
ages of 15 and 20, while 41.3% of the students were between the ages of 21 and 25.
Instruments
The data collection instrument consisted of five sections − demographics, educational ideologies, perceived ease
of use, perceived usefulness, attitudes toward computer use and the frequency of use. The second section was
adapted from O’Neill (1990) in order to identify the educational ideologies. Perceived ease of use and perceived
usefulness sections were adapted from Legris et al (2003). The Attitudes toward computer use section was
adapted from a questionnaire developed by Dusick & Yildirim (2000). A committee approach was used in the
adaptation process of the instrument. The items were translated by eight different instructors from the
Department of Foreign Language Education at METU. An expert familiar with both American and Turkish
cultures was responsible for the appropriate translation while subject matter experts adapted the translations. A
pilot study was conducted to correct some misunderstandings. The researchers also used a pilot study to check
the validity of the items in the instrument.
The educational ideology part consisted of 104 questions that measured the two general and six specific
educational ideologies. 14 questions were used to identify each of the six specific educational ideologies and 10
155
questions for each general educational ideology. Given that this study investigated the effect of specific
educational ideologies, a total of 84 questions related to specific educational ideologies appeared in the
instrument. The perceived ease of use and perceived usefulness sections included 10 questions on each topic.
The attitudes toward educational technologies section included 23 questions. Finally, the actual use section had
only one question. With the exception of the question related to actual use, all of the questions utilized a fivepoint Likert-type scale with responses ranging from Strongly Agree to Strongly Disagree. The question in the
actual use section utilized a six-point Likert-type scale ranging from “more than once a day” to “less than once a
month”.
Data Analysis
The data analysis of the study was conducted in three sections: item analysis, factor analysis, and path analysis.
Item Analysis: For the item analysis, the statistical program SPSS 10.0 (Statistical Package for Social Sciences)
was used. Considering the dispersion of the data in each item and the corrected total item-scale correlations, the
items were analyzed and the valid and reliable items were selected for the factor analysis stage of the study.
Factor Analysis: Factor analysis was a necessary step to confirm the assumption of the structural equation
modeling, that all of the variables in the hypothesized model are independent from each other. Therefore, the
results of the factor analysis will prove that the items selected for the path analysis are discriminatory for the
factors and that all these variables are independent factors in the model. This part of the analysis includes two
subsections: exploratory and confirmatory factor analysis. According to the principal component analysis results,
21 out of 84 questions in the educational ideologies section, 10 out of 10 questions in the perceived usefulness of
technology section, 7 out of 10 questions in the perceived ease of use section, 9 out of 23 questions in the
attitudes toward use section, and the 1 question in the actual use section were determined to be useable for this
study. The details of the exploratory and confirmatory factor analyses will be presented in the next section.
The study first analyzed the dimensionality of the 84 items selected from the questionnaire. The purpose in this
stage was to find out which questions statistically represent the six different educational ideologies. Since each
participant in the sample may have different ideas stemming from the different ideologies as mentioned above,
the educational ideology questions do not divide the sample into groups clearly. Moreover, there are some
commonalities among the educational ideology questions. For exploratory factor analysis, principal component
analysis was used to determine the discriminatory educational ideology questions. The items determined to be
theoretically and statistically separate from the other factors were evaluated using confirmatory factor analysis.
After the confirmatory factor analysis, 21 educational ideology questions were chosen for further analysis. Six
factors representing educational ideologies were retained for further analysis. The validity values of the factors
(eigenvalues) were 3.66, 2.45, 1.44, 1.38, 1.14, and 1.11.
Table 1 presents items grouped as a result of principal component analysis, with their respective factor loadings.
Six factors explained 53% of the total variance in this particular analysis. The first factor represents educational
fundamentalism, the second educational conservatism, the third educational liberationism, the fourth educational
liberalism, the fifth educational intellectualism, and the sixth educational anarchism. The numbers in bold text
indicate significant factor loading of each question.
Educational Ideologies Section
Considering the factor structure as indicated in Table 1, we formed latent variables and educational ideologies,
for the path analytic model. In this process, two important criteria were used. Firstly, we attempted to keep the
number of observed variables to a minimum of three (Schumacher & Lomax, 1996); secondly, we gave greater
preference to the items with greater factor loadings.
The first factor in the table above represents the educational fundamentalism ideology and includes questions 90,
101, 97, 87, and 62. The second factor, educational conservatism includes questions 4, 72, 46, 12, and 83. The
third factor, educational liberationism includes questions 84, 99, and 69. The fourth factor, educational liberalism
includes questions 77, and 53. The fifth factor, educational intellectualism includes questions 2, 23, and 14. The
sixth factor, educational anarchism includes Questions 6, 45, and 39. These factors were used as latent factors for
path analysis.
156
Table 1: Factor Loadings of Educational Ideologies Items for Principal Component Factor Analysis
Factor Loading
Items
1
2
3
4
5
90. The school should encourage a return to the simple and
straightforward virtues of an earlier day, to the older and
better ways.
101. The schools should emphasize the virtues of the historical
past as a way of correcting the existing overemphasis on the
present and the future.
97. The individual finds his greatest fulfillment in a voluntary
subordination to the ends of the State.
87. A central purpose of education should be to revive and
reaffirm an almost religious commitment to certain profound
national goals.
62. The history of this nation is preeminently a spiritual history
guided by Providence.
4. In the final analysis, human happiness derives from adapting
oneself to prevailing standards of belief and behavior.
72. Students should be trained to be good citizens in terms of
prevailing cultural views about the nature of good
citizenship and proper conduct.
46. The basic value of knowledge is its contemporary social
utility; knowledge is primarily a means of adapting
successfully within the existing social order.
12. The school should encourage an appreciation for time – tested
cultural institutions, traditions, and processes.
83. Schools should be run in a manner consistent with the
conventional wisdom (the common sense beliefs) of society
at large.
84. The schools should stress the critical analysis and evaluation
of prevailing social beliefs and behavior.
99. The schools should encourage students to recognize and
respond to the need for particular kinds of liberalizing social
reforms.
69. The teacher should be a model of intellectual commitment
and social involvement.
77. The teacher should be basically an organizer and expediter of
learning activities and experiences.
53. Knowledge is ultimately a tool, a means to be used in solving
the problems of everyday living.
14. Secondary education should provide the student with an
orientation to life in general, emphasizing his role as a
human being rather than training him for any particular
social role or position.
23. The schools should place their basic emphasis on man as
man; that is, on the sort of abiding human nature which all
individuals share.
2. The most valuable type of knowledge is that which involves
symbolism and abstract thinking.
6. Individual differences (physical, psychological, and social)
are so significant that they dictate against the wisdom of
prescribing the same or similar educational experiences for
all people.
45. Problems associated with student control and discipline are
frequently caused by a society which blocks the
development of personal responsibility by over controlling
everyone, including students.
39. Conventional teaching ordinarily subverts the child’s capacity
for self – learning.
-.192
.781
.758
.662
.274
.568
.236
.132
.492
.135
-.361
.245
.189
-.147
-.401
.186
-.138
.256
.734
-.125
.128
.719
.160
.134
.655
.246
.176
.276
.540
.125
.164
.215
.447
-.318
.235
.734
.139
-.124
.486
.102
.139
.158
.175
.156
.180
.625
.273
.119
6
.284
.118
.787
.176
-.132
.714
.206
.610
.253
-.140
.594
.132
.244
-.121
.148
.180
.130
-.165
.211
.592
-.286
-.130
.699
.332
.598
.535
Note: Loadings below 0.10 were suppressed in the table.
157
Technology Acceptance Section
This section of the questionnaire consisted of 4 different subsections: the perceived usefulness of technology,
perceived ease of use, attitudes toward use, and actual use sections. Using exploratory factor analysis, it was
ensured that each subsection included only one factor. Therefore, three different exploratory factor analyses were
conducted for the perceived ease of use, perceived usefulness, and attitudes toward use sections at this stage.
Since the actual use section included one question, exploratory and confirmatory factor analysis was not
conducted for this section. Ten questions in the perceived usefulness of technology section, seven questions in
the perceived ease of use section, nine questions in the attitudes toward use section, and one question in the
actual use section were determined to be used for this study.
Table 2: Principal Component Analysis Results for Technology Acceptance
Factor Loading
Items
Perceived Usefulness
8. Computers enhance my educational effectiveness.
6. Computers increase my performance in education.
5. Computers increase my productivity in education.
10. Overall, I found computers to be useful in education.
3. Computers enable me to accomplish tasks more quickly in education.
9. Computers make my work easier in education.
2. Computers give me greater control over my work in education.
1. Computers improve the quality of the work I do in education.
4. Computers support the critical aspects of my work in education.
7. Computers allow me to accomplish more work than otherwise be possible in education.
Perceived Ease of Use
8. My interaction with the computers are clear and understandable.
3 Interacting with the computers is often frustrating.
2. Learning to operate computer applications is easy for me.
4. I find it easy to get the computers to do what I want to do.
10. Overall, I find computers easy to use.
6. It is easy for me to remember how to perform tasks using the computers.
5. The computers are rigid and inflexible to interact with.
Attitudes toward use
9. I feel anxiety when I am using computers.
4. I have self-confidence in using computers.
7. I feel terrible while my friends are talking about computers.
5. It is hard to learn new computer applications.
3. I am not the one who can work with computers.
2. It is easy to learn how to use computer software.
10. I fell confused while I am working with computers.
20. The use of computers in classrooms are useful and it is worth of endeavor.
18. I am also using computers out of school.
1
.845
.824
.810
.801
.777
.774
.767
.721
.672
.648
.712
.685
.673
.661
.649
.633
.488
.792
.741
.725
.717
.688
.663
.593
.555
.507
Note: Loadings below 0.10 were suppressed in the table.
Table 2 presents items grouped as a result of the principal component analysis, with their respective factor
loadings. Each variable has one factor in this analysis. The perceived usefulness section explained 59%, the
perceived ease of use section explained 42% and the attitudes toward use section explained 45% of the total
variance in this particular analysis. The questions in the Table 2 represent the three factors of technology
acceptance.
Table 3 indicates the Lambda-x estimates and standard errors as obtained for each of the observed variables from
the confirmatory factor analysis, with their abbreviations, the names of the latent variables, response modes, and
respective item means. Lambda-x values, which are the loadings of each observed variable on the respective
latent variable, indicate reasonable sizes to support the idea of using these latent variables in the proposed path
analytic model to explain significant factors in educational technology acceptance.
After the confirmatory factor analysis of each questionnaire, 21 questions in the educational ideologies section,
five questions in the perceived usefulness section, four questions in the perceived ease of use and attitudes
toward use section and one question for the actual use section were selected for the path analysis. Considering
the factor structure as indicated in Tables 1 and 2, we have formed latent variables for the path analytic model. In
158
this process, the two important criteria were used again. First, the number of observed variables was kept to a
minimum of three (Schumacher & Lomax, 1996); second, preference was accorded to the items with greater
factor loadings.
Table 3: LISREL Estimates, Standard Errors for Confirmatory Factor Analysis and Item Means with Response
Modes
Latent and Observed Variables
Educational Fundamentalism
Question 62
Question 87
Question 90
Question 97
Question 101
Educational Intellectualism
Question 2
Question 14
Question 23
Educational Conservatism
Question 4
Question 12
Question 46
Question 72
Question 83
Educational Liberalism
Question 53
Question 77
Educational Liberationism
Question 69
Question 84
Question 99
Educational Anarchism
Question 6
Question 39
Question 45
Perceived Usefulness of Technology
Question 2
Question 4
Question 7
Question 8
Question 9
Question 2
Question 6
Question 8
Question 10
Attitudes toward Use
Question 2
Question 4
Question 5
Question 16
Lambda-x
SE
Mean
0.48
0.44
0.82
0.58
0.68
0.06
0.06
0.05
0.06
0.05
2.97
3.18
2.54
2.63
2.36
0.44
0.47
0.54
0.07
0.07
0.07
2.96
4.02
3.86
0.60
0.57
0.59
0.76
0.49
0.06
0.06
0.06
0.05
0.06
3.33
3.57
4.05
3.76
3.02
0.69
0.65
0.07
0.07
3.87
3.87
0.53
0.68
0.63
0.06
0.06
0.06
4.05
3.87
3.94
0.26
0.43
0.66
0.07
0.07
0.09
3.46
3.81
3.93
0.73
0.65
0.74
0.94
0.84
0.05
0.05
0.05
0.04
0.05
4.03
3.87
3.89
4.11
4.23
0.61
0.62
0.74
0.63
0.05
0.05
0.05
0.05
3.25
3.41
3.56
3.44
0.69
0.81
0.70
0.62
0.05
0.05
0.05
0.05
2.87
3.41
3.43*
4.05
Response Mode
1 (Strongly Disagree) to
5 (Strongly Agree)
* This item was reversed for the analysis.
Path Analysis: In the study, LISREL 8.30 for Windows (Joreskog & Sorbom, 1999) with SIMPLIS command
language was used to analyze data for path analysis with latent variables. The maximum likelihood estimation
method was used in all of the LISREL analyses. For the model data fit assessment, Standardized Root Mean
Squared Residual (SRMR), and Root-Mean-Square Error of Approximation (RMSEA) were used in the study. In
this study, we determined a 0.073 Standardized Root Mean Square (SRMR) and a 0.080 Root Mean Square
(RMSEA) index. These indices were deemed adequate to treat the respective item groups as distinct latent
variables in the path analytic model. The alpha reliability coefficients for the latent variables were 70 for
fundamentalism, 40 for intellectualism, 69 for conservatism, 52 for liberalism, 56 for liberationism, 37 for
anarchism, 83 for perceived usefulness of technology, 69 for perceived ease of use of technology and 76 for
159
attitudes toward use. These reliability coefficients are significant to indicate that the questions under each
category are reliable.
Based on the evidence from the literature, it would be possible to claim that educational ideologies might affect
educators’ perceptions on the usefulness of technology and their attitudes toward technology directly. Therefore,
they might affect technology use indirectly. This study has focused on educational ideologies as possible
external factors of the TAM. In this study, educational ideologies and perceived ease of use are exogenous
variables, the use of educational technology is endogenous variables and finally, perceived usefulness and
attitudes toward use are both exogenous and endogenous variables.
Edu. Fund.
Edu. Intel.
Perc. Useful.
Edu. Cons.
Edu. Lib.
Edu. Libt.
Attitudes
Use
Edu. Anar.
Perc. E. U.
Figure 2: Hypothesized Model
Results
In order to predict the most compatible model, t-test results for both exogenous and endogenous variables and
model data fit indexes (such as SRMR and RMSEA) were taken into consideration. The paths that indicated nonsignificant t-values were deleted from the model. In accordance with the LISREL analysis, some minor
modifications were made to the model to obtain the most compatible model. Finally, the model in Figure 3 was
obtained with 0.073 SRMR and 0.080 RMSEA fit index values. These values were deemed adequate to interpret
the significant relationship among the latent variables. Table 4 shows Lambda-x estimates, t-values and standard
errors for the educational technology acceptance model. The Lambda-x estimates and t-values reflect the
reliability of the items in the final model. None of the t-values are below 0.30 which means that all these
questions could be kept for the best-fit model.
Table 4: LISREL Estimates, t-values, and Standard Errors for LISREL Model
Latent Variables
EDUCATIONAL FUNDAMENTALISM
EDUCATIONAL INTELLECTUALISM
EDUCATIONAL CONSERVATISM
Observed Variables
Question 62
Question 87
Question 90
Question 97
Question 101
Question 2
Question 14
Question 23
Question 4
Question 12
Question 46
Question 72
L
Lambda-x
0.49
0.45
0.82
0.58
0.68
0.36
0.42
0.58
0.59
0.57
0.55
0.78
t
8.40
7.88
15.51
10.23
12.34
5.10
6.09
8.07
10.55
10.14
10.21
14.95
SE
0.06
0.06
0.05
0.06
0.05
0.07
0.07
0.07
0.06
0.06
0.05
0.05
160
EDUCATIONAL LIBERALISM
EDUCATIONAL LIBERATIONISM
EDUCATIONAL ANARCHISM
PERCEIVED USEFULNESS OF TECHNOLOGY
PERCEIVED EASE OF USE
ATTITUDES TOWARD USE
Question 83
Question 53
Question 77
Question 69
Question 84
Question 99
Question 6
Question 39
Question 45
Question 2
Question 4
Question 7
Question 8
Question 9
Question 2
Question 6
Question 8
Question 10
Question 2
Question 4
Question 5
Question 16
0.49
0.63
0.59
0.22
0.72
0.66
0.26
0.40
0.66
0.75
0.66
0.77
0.93
0.84
0.59
0.63
0.75
0.60
0.71
0.84
0.69
0.33
8.39
10.21
9.56
8.56
11.89
11.07
3.71
5.56
7.77
11.54
10.42
11.81
13.58
12.64
10.66
11.56
14.53
10.94
5.06
5.11
5.03
4.18
0.06
0.06
0.06
0.07
0.06
0.06
0.07
0.07
0.08
0.06
0.06
0.06
0.07
0.07
0.05
0.05
0.05
0.05
0.14
0.16
0.14
0.08
Table 5 presents the Beta estimates, which are the coefficients among attitudes toward computers, perceived
usefulness, and frequency of use. The table also presents the Gamma estimates, which are the coefficients among
the endogenous and exogenous variables and t-values.
Table 5: LISREL Estimates and t-values for LISREL Model
Latent Variables
Attitudes toward use & Perceived Usefulness of Technology
Perceived Usefulness & The Frequency of Use
Educational Fundamentalism
& Attitudes toward Technology Use
Educational Conservatism
Educational Liberalism
Educational Anarchism
& Perceived Usefulness of Technology
Educational Liberationism
Educational Intellectualism
Beta
0.43
0.25
-
Gamma
0.96
-0.15
0.26
-0.32
-0.33
0.49
0.38
t
4.01
4.24
4.28
-1.95
1.97
-2.22
-2.49
3.73
4.33
Table 5 and Figure 2 both indicate the structural model of technology acceptance developed. In this model, the
standardized path coefficients varied between –0.33 and 0.96 in the fitted model. Cohen (as cited in Kline, 1998)
made some suggestions about the interpretations of the absolute magnitudes of path coefficients. Cohen (1988)
explained that standardized path coefficients that have absolute values less than 0.10 might indicate a “small”
effect; whereas values around 0.30 indicate a “medium,” and values above 0.50 indicate a “large” effect,
respectively (Kline, 1998). In accordance with these suggestions, the path coefficient from attitudes towards use
to the perceived ease of use could be considered as a high effect in the model. All the other path coefficients
indicated medium effect sizes in the model.
The model developed in this study has some similarities as well as differences in regard to the TAM. Similar to
the hypothesized model, the fitted model in figure 3 shows that there is a direct effect of perceived usefulness on
the actual use, and there is also a direct affect of perceived ease of use on attitudes. However, unlike the TAM,
the fitted model shows that attitudes towards use affect perceived usefulness, and there is no significant effect of
attitude towards actual use. Also, the fitted model shows that there is no significant relationship between
perceived ease of use and perceived usefulness.
PEU: Perceived Usefulness of Technology, ATT: Attitudes toward Technology Use, PU: Perceived Usefulness
U: The Frequency of Use F: Educational Fundamentalism, I: Educational Intellectualism, C: Educational
Conservatism, L: Educational Liberalism, LB: Educational Liberationism, A: Educational Anarchism.
161
Figure 3 shows that there is a strong positive relationship between pre-service teachers’ perceived ease of use of
educational technology and their attitudes toward use. Therefore, pre-service teachers who are competent with
technology in a classroom environment have highly positive attitudes towards the use of educational
technologies. In addition, fundamentalist and liberalist pre-service teachers have low values for attitudes
towards the use of educational technologies. While educational fundamentalists and educational liberalists
demonstrate negative linear relationships in relation to attitudes towards use, pre-service teachers who embrace
conservative educational ideologies demonstrate high attitude values towards the use of educational
technologies. Moreover, pre-service teachers who believe in educational anarchism suppose that educational
technologies are not useful in the classroom environment. However, educational liberationists and
intellectualists believe that educational technologies are very useful in the classroom environment.
Figure 3: Structural Model of Educational Technology Acceptance Model Integrated with Educational
Ideologies
Arguments over the integration of technology in schools have met with many obstacles. Most decision makers
or administrators want to benefit from technology in educational practices. For this reason, teacher education
institutions as well as training programs have placed a strong emphasis on technology education courses in both
pre-service and in-service education. However, the literature shows that self-efficacy of the teachers, in other
words the perceived ease of use, is not sufficient to allow teachers to use technology in education. This study
shows that the perceived ease of use may not directly influence the frequency of use of computer technologies in
education. This result might give some clues about the ineffectiveness of computer literacy courses and
technology education courses in pre-service and in-service education and training.
Based on the results of this study, we assert that there is a medium effect of perceived usefulness of technology
on the frequency of use of technology in education. This result is consistent with the suggestions of Moore et al
(1999) who suggested that teachers’ technology competencies consist of four major categories: prerequisite
technical skills, technical skills, instructional uses, and professional roles (Moore et al, 1999). They explained
that teachers react negatively to courses that emphasize technical skills without practice. For this reason,
instructional technology use must be related with teaching practices in actual professional implementation. In
this sense, we can say that prospective teachers consider educational technologies to be more useful in real
educational settings, which also promotes the use of technology in education effectively.
Besides the effect of the perceived ease of technology use on the frequency of use, perceived ease of use
positively affected the attitudes toward use. According to our model, attitudes towards use are the only
162
determinant of perceived usefulness. The pre-service teachers who perceived the use of technology in education
to be crucial probably have high positive attitudes towards use.
Although the perceived ease of use is the primary determinant of attitude, three educational ideologies,
fundamentalism, conservatism, and liberalism, also have significant effects on attitudes towards use. For
example, educational fundamentalist ideology has a negative affect on attitudes towards the use of technology.
This result might be stemming from the characteristics of educational fundamentalism, since it tends to
implement uncritical acceptance and urges individuals to conform to the existing social order. However,
computer technologies, especially the Internet with access to various different sources, provide opportunities for
increased critical thinking. From a broader vision, students might get a chance to question common sense by
accessing different sources, and consequently, this might lessen the authority of the teacher in the classroom
environment. In this aspect, fundamentalist pre-service teachers might see technology as possible threat to their
existing routines.
Conservatism has a positive affect on attitudes towards use. According to the model, the educational
conservative view appears more moderate to change when compared with the fundamentalist view. As Skolnik
(1998) stated, “It has been pointed out that since computers first appeared, predictions have been made about
how they would revolutionize education, but that has not happened” (p. 644). Based on the literature, it could be
appropriate to say that technology is a tool for changing society. Nevertheless, belief, understanding and
acceptance of technology illustrate a complex pattern in relation to the implementation of educational
technology. Therefore, there may be a consistency between attitudes towards the use of technology and the
conservative educational view.
As opposed to conservatism, the educational liberalist ideology has a negative affect on the attitude towards use.
In fact, liberalist pre-service teachers do use technology, especially computer technology, for a significant
amount of time and become accustomed to technology as a part of their college life. Interestingly, even though
they use technology, they may not be conscious concerning the need of technology in their prospective job
settings. In 2001, Steel and Hudson conducted a study regarding academics’ perceptions of educational
technologies and found that some of the academics thought the use of educational technology was a potential
threat to meaningful face-to-face interaction. Whether liberalists assume technology as potential threat or not
should be studied in-depth.
All of the ideologies mentioned above have effects on attitudes toward use. Moreover, educational anarchism,
educational liberationism, and educational intellectualism have an effect on the perceived usefulness of
technology.
Educational Anarchism has a negative effect on the perceived usefulness of technology. Some applications of
computer technology intend to reach a wide population to provide them more a systemized education via
synchronous or asynchronous technologies. This is contradictory with what anarchists deem. Because, preservice teachers who might think of educational technology as a mediator of a centralized and institutionalized
society might perceive educational technology as not useful since some of the technology integration programs
create centralized and institutionalized society, such as the use of learning management systems and learning
content management systems.
There is a positive relationship between educational liberationism and the perceived usefulness of technology. It
is an expected result since the literature supports the consistency between the goal of education from an
educational liberationist perspective and the general perceived role of technology in education. The educational
liberationist belief emphasizes the fullest development of each person’s unique potentialities as a human being.
Steel and Hudson (2001) also support this notion by underlining the introduction of technology into the learning
process as a potential quality-enhancing act.
Educational Intellectualism has positively affected the perceived usefulness of technology. Since educational
intellectualism seeks to change existing educational practices in order to make them conform more perfectly to
some established and essentially unvarying intellectual or spiritual ideal, technology in education may be seen as
a tool to change existing educational practices to be more consistent with those purposes.
Discussion
This study has indicated that attitudes towards use and the perceived usefulness of technology differentiate
depending on the educational ideologies of pre-service teachers. These attitudes and perceptions about the
163
usefulness of technology directly or indirectly affect the frequency of use of technology. Technology education
or computer literacy courses in pre-service and in-service teacher education is not enough to increase the
frequency of use of educational technologies if they focus only on technical skills, rather than including the
instructional use and emphasizing the professional roles of teachers in the environment integrated with
technology (Inan, F., Yildirim, S. & Kiraz, E. 2004; Yildirim, S. & Kiraz, E. 1999).
According to a report published by UNESCO (2002), there is continuum of approaches through which schools
and education systems proceed in their adoption of ICT. In the first stage of technology adoption, namely the
applying approach, teachers and school administrators start using ICT for daily tasks and they adopt curriculum
to create more space for ICT in teaching. In the infusing approach, ICT is integrated throughout the school
curriculum. School personnel start using more advanced and customized software to increase their productivity
as well as their professional competencies. Finally, in schools that demonstrate the transforming approach, ICT
becomes an internal part of the curriculum and it is used as a catalyst for school reform. Teachers take more
initiative to move from teacher-centered teaching approach to student-centered learning activities, and schools
serve as learning centers for the surrounding communities (p. 15-16).
It should be noticed that technology integration in schools largely depends on individuals, especially teachers,
since they are obviously gatekeepers for all kind of innovation introduced to the education system. Thus, it is
crucial to scrutinize teacher’s ICT utilization and reveal factors that contribute to their skeptical practices of
teaching with technology. Unquestionably, this study revealed the fact that educational ideologies cultivate the
type and extent of teacher’s technology use and thus it has a paramount effect on a successful integration of
technology throughout the system.
There could be two different approaches according to the findings of this study. Firstly, educational ideology is a
system of belief that has established through a long period of time and it is hard to change. If the truth is that
educational technology is useful for every educational practice, its benefits should be emphasized for different
individuals rather than imposing specific educational technologies. Therefore, stakeholders and teacher
educators should try to familiarize educators by considering their various educational beliefs. Secondly, this
study shows that the use of educational technology and the benefits to education is an arguable matter since there
is a contradiction between the overall goals of some educational ideologies and the factors of technology
acceptance. Two questions arise from the findings of this study: is the acceptance of educational technology
necessary for every single person? Or can we achieve success in education without using educational technology
in every situation? The answer is complex, as none of the ideologies is superior to the others. Each ideology is
effective in its own scope and has a possible explanation for accepting or rejecting technology. Therefore, some
pre-service teachers may not consider the use of educational technology necessary in some situations since their
educational goals might be attainable without using the means of technology.
References
Bajaj, A, & Nidumolu, S. R. (1998). Techniques A feedback model to understand information system usage.
Information & Management, 33 (4), 213-224.
Bandura, A. (1986) Social Foundations of Thought and Action: A Social Cognitive Theory, Englewood Cliffs:
NJ: Prentice Hall.
Brosnan, M. J. (1999). Modeling Technophobia: a case for word processing. Computers in Human Behavior, 15,
105-121.
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: a comparison
of two theoretical models. Management Science, 35 (8), 982-1003.
Dusick, D., & Yıldırım, S. (2000). Faculty Computer Use and Training: Identifying Distinct Needs for Different
Populations. Community College Review, 27 (4), 33-47.
Inan, F., Yildirim, S., & Kiraz, E. (2004) The Design and Development of an Online Learning Support System
for Preservice Teachers: A Discussion of Attitudes and Utilization. Journal of Interactive Instruction
Development, 17 (1), 3388-3393.
Kline, R. B. (1998). Principles and practice of structural equation modeling, New York: Guilford.
164
Joreskog, K. G., & Sorbom, D. (1996). LISREL 8 user's reference guide, Chicago, Illinois, USA: Scientific
Software International.
Legris, P., Ingham, J., & Collerette, P. (2003). Why do people use information technology? A critical review of
the technology acceptance model. Information & Management, 40, 191-204.
Moore, J., Knuth, R., Borse, J., & Mitchell, M. (1999). Concepts and Procedures. Paper presented at the 10th
International Conference of Society for Information Technology and Teacher Education, February 28 - March 4,
1999, San Antonio, Texas, USA.
O'Neill, W. F. (1990). Educational Ideologies Contemporary Expressions of Educational Philosophy, Dubuque,
Iowa: Kendall / Hunt Publishing Company (Original work published 1981).
Perkins, D. (1992). Smart schools: Better thinking and learning for every child, New York: The Free Press.
Rogers, E. M. (1995). Diffusion of Innovations (4th Ed.), New York: Free Press.
Schumacher, R. E., &. Lomax, R. G. (1996). A beginner's guide to SEM, Manwah, New Jersey, USA: Lawrence
Erlbaum Associates.
Semple, A. (2000). Learning theories and their influence on the development and use of educational
technologies. Australian Science Teachers Journal, 46 (3), 21-28.
Skolnik, M. L. (1998). Higher Education in the 21st Century. Futures, 30 (7), 635-650.
Steel, J., & Hudson, A. (2001). Educational Technology in Learning and Teaching: The Perceptions and
Experiences of Teaching Staff. Innovations in Education and Teaching International, 38, 103-111.
Stettheimer, T. D., & Cleveland, A. D. (1998, November). Modeling Utilization of Planned Information
Technology. Paper presented at The 1998 American Medical Informatics Association Annual Symposium,
November 7-11, 1998, Walt Disney World Resort, USA.
Surry, D. W., & Land, S. M. (2000). Starategies for motivating higher education faculty to use technology.
Innovations in Education and Training International, 37 (2), 145-153.
UNESCO (2002). Information and Communication Technology in Education: A curriculum for schools and
programme of teacher development, Paris: UNESCO.
Weiss, J. (1994). Keeping up with the research. Technology & Learning, 14 (5), 30.
Yildirim S., & Kiraz, E. (1999). Obstacles in Integrating Online Communication Tools into Preservice Teacher
Education: A Case Study. Journal of Computing in Teacher Education, 15 (3), 23-28.
165
Serce, F. C. & Yildirim. S. (2006). A Web-Based Synchronous Collaborative Review Tool: A Case Study of an On-line
Graduate Course Educational Technology & Society, 9 (2), 166-177.
A Web-Based Synchronous Collaborative Review Tool: A Case Study of an
On-line Graduate Course
Fatma Cemile Serce
Information Systems, Informatics Institute, Middle East Technical University, 06531 Ankara, Turkey
Tel: +90 312 210 3747
Fax: +90 312 210 3745
[email protected]
Soner Yildirim
Middle East Technical University, Faculty of Education, 06531 Ankara, Turkey
Tel: +90 312 210 4057
Fax: +90 312 210 1112
[email protected]
Abstract
On-line collaboration is an instructional method that facilitates collaboration in an on-line learning setting.
To promote effective collaboration, it is vital to reveal both the student’s and the instructor’s point of view
pertaining to effective on-line collaboration. In this study, the effectiveness of a learning management
system in on-line collaboration was first investigated in a graduate course offered through the means of
distance learning. In this first phase of this study, the nature of collaboration, the students perceptions of the
effectiveness of the tool in on-line collaboration, the factors contributing to effective peer interaction among
students and the role of the instructor as perceived by the students in the on-line course were explored. This
phase of the study also involved the reviews of other learning management systems, course management
systems and groupware systems regarding the tools used to encourage collaboration. Also revealed was a
lack of diversity in collaboration tools. Based on these preliminary findings, an on-line document reviewing
tool was developed and pilot tested in the second phase of the study. In this phase, a web-based
synchronous collaborative review tool called WebSCoRe is proposed to promote online collaboration.
WebSCoRe is proposed as an attempt to develop and implement a new platform for on-line document
reviewing, to promote effective on-line collaboration among students and the instructor.
Keywords
Collaborative learning, Distance learning, Learning technologies, On-line review tools
Introduction
The advances in technology and changes in the organizational infrastructure put an increased emphasis on
teamwork within the workforce. Employees need to be able to think creatively, solve problems, and make
decisions as a team. Therefore, the development and enhancement of collaborative skills and producing
graduates who are flexible and have market-related skills and abilities has been among the primary goals of
higher education. On-line education provides collaboration opportunities both to the educational institution and
to learners.
Several studies have shown that the implementation of collaborative learning strategies result in higher student
involvement in the course and more engagement in the learning process and collaborative learning methods are
more effective than traditional methods in promoting student learning and achievement (Hiltz, 1998). Gokhale
(1995) stated that if the purpose of instruction is to enhance critical- thinking, problem-solving and collaborative
skills, then the utilization of collaborative learning techniques will be more beneficial.
The learning management systems and groupware systems are the main platforms for distance education (Horton
& Horton, 2003). It is essential to understand the factors affecting on-line collaboration in these platforms. In
this study, several widely used learning management and groupware systems were first reviewed as tools
provided to promote collaboration. Moreover, in order to obtain the users’ point of view on on-line collaboration,
an in-depth assessment study was conducted on a learning management system, called NET-Class. NET-Class
has been developed at the Informatics Institute at Middle East Technical University. It has been used for
delivering on-line graduate programs and some general education courses given at METU. It is also been used as
a tool to support face-to-face instruction. As a tool used for promoting computer-mediated communication, the
first part of the study examined the effectiveness of NET-Class on on-line collaboration for project based
166
learning in a graduate course offered through the means of distance learning. The student’s perceptions of their
experiences in on-line collaboration in the course studied were obtained. Moreover, the nature of collaboration in
the course was identified from the postings in the forum with respect to the issues of interaction, participation,
feedback and the utterances for on-line collaboration.
Findings indicated that NET-Class failed in providing diverse collaboration tools and platforms. Thus, a webbased synchronous collaborative review tool named WebSCoRe was developed and implemented. WebScoRe is
a real-time collaborative meeting-based document review system that allows all comers to participate regardless
of location. An object-oriented approach was used in the design and development of the tool and Java was
chosen as the development language to ensure compatibility with the NET-Class Learning Management System.
Collaborative Learning
Learning is never a passive act. The key to the learning process is the interactions among students themselves,
the interactions between the faculty and the students, and the collaboration that results from these interactions.
Learning by collaborating is a social process and leads to learning being not only active, but also interactive. The
requirements for education in the twenty-first century result in more emphasis on active participation rather than
passive approaches presenting learning (Hiltz, 1998)
Collaboration is a synchronous activity of a gathering of parties with diverse skills and backgrounds,
contributing those skills and resources in an atmosphere of trust, respect and flexibility, in order to achieve
shared goals and objectives. Collaboration is something that human beings have been experiencing from early
times and applying throughout their lives (Muronaga & Harada, 1999).
Collaborative learning is an instructional method that encourages students to work together in groups toward a
common goal. (Bruner, 1991) The students learn or attempt to learn something together and they are responsible
for one another’s learning as well as their own. Thus, the success of one student helps other students to be
successful.
Gokhale(1995) examined the effectiveness of individual learning versus collaborative learning in enhancing
drill-and-practice skills and critical-thinking skills. Regarding gaining factual knowledge, both individual
learning and collaborative learning were found to be equally effective. The study reveals that collaborative
learning fosters the development of critical thinking through discussion, clarification of ideas and evaluation of
others’ ideas. Therefore, the study concludes that if the purpose of instruction is to enhance critical-thinking and
problem-solving skills, then collaborative learning will be more beneficial.
The need for collaborative skills is commonly emphasized in information systems. Layzell, Brereton, and French
(2000) argue that software engineering is no longer the preserve of individuals but is essentially a team-based
activity involving a wide variety of stakeholders and thus making the need for communication and co-operation
an inherent characteristic. Because of the changes in support technology, economic factors and globalization of
the software process, the personnel are separated. This results in the need for effective communication and
collaboration skills of globally distributed personnel. However, many of the barriers to the successful
implementation of team-based collaborative software engineering practices trace directly to the poor
understanding of and inadequate training for the interaction skills software professionals need to successfully
enact such collaborative activities, such as requirements elicitation, project management and peer review
(Schoeder and Brunner, 1996). The course assessed in this study was an information systems project course,
which requires software engineering practices.
Promoting Collaborative Learning
Dillenbourg and Schneider (1995) stated some conditions for effective collaborative learning, such as group
composition, task features and communication media. Group composition includes the age and levels of the
participants. The size of the groups and the difference between group members are the variables affecting the
group composition. Task features stress the relation between the nature of the task and the effectiveness of
collaboration. There are some tasks that may not be shared, requiring individual work; some tasks are
straightforward without the potential to lead to any disagreement or misunderstanding. The communication
media is another essential condition for effective collaborative learning. According to the task in hand and the
group members that have been selected, the collaborative process may not work, due to the inadequacy of the
medium used for communication.
167
Dillenbourg (1999) argues that collaborative learning is not one single mechanism. If one talks about “learning
from collaboration”, one should also talk about “learning from being alone”. Individual cognitive systems do not
learn because they are individual, but because they perform some activities (reading, building, predicting), which
trigger learning. Similarly, peers do not learn because they are many, but because they perform some activities,
which trigger specific learning mechanisms. Additionally, the author argues that the interaction among students
generates extra activities (explanation, disagreement, and mutual regulation), triggering knowledge elicitation,
and internalization, and reduces cognitive load. The field of collaborative learning is precisely about these
activities and mechanism. In this study, collaborative reviewing was chosen as the activity to promote
collaboration.
Collaborative reviewing involves negotiation of the meaning of facts and demands for consensus for an
appropriate solution. Working together on a common document allows collaborators to share and discuss their
ideas about further revisions (Kim, 2002). Collaborative reviewing is one of the instructional strategies used to
promote collaboration in both face-to-face and on-line settings. Collaborative reviewing provides participants to
be engaged in rich interactions, which is the necessity of an effective collaborative learning (Dillenbourg, 1999).
Pratt and Pallof (2001) argue that the elements of on-line groups are the individuals, the group, the facilitator
(instructor) and the technology. In collaborative learning environments, the role of the instructor and the role of
the students are important.
The Role of the Instructor
Teaching at a distance involves the use of a different set of skills for instructors, compared to those used in a
traditional classroom. There is a consensus in the literature that getting people to participate and making learning
active at a distance is much more important than presenting the information.
According to Moore and Kearsley (1996), the most critical skills that the distance educator must develop are;
making the students active participants, inducing inter-learner interaction among the participants, keeping the
discussion on track and providing immediate feedback
Pallof and Pratt (1999) summarize the role of the on-line instructor in four general areas: Pedagogical, Social,
Managerial and Technical. The pedagogical function involves providing guidance and a framework as a
“container” for the course and allowing students to explore the course material, as well as related materials. The
role of instructor becomes that of an educational facilitator. The social function of the instructor is to facilitate
and make room for the personal and social aspects of on-line community. The managerial function consists of
objective setting, rule making, decision making, posting a syllabus for the course, including assignments and
some other initial guidelines for the group discussions. The technical function is the provision of technical
facilitation by the on-line instructor, to the students. This function depends on the instructor being confident
concerning the use of technology and making the students comfortable by providing guidance.
The Role of the Learner
Pallof and Pratt (1999) describe the responsibilities of the learner in distance learning environments. The authors
categorized the role of learner into three areas: Knowledge Generation, Collaboration and Process Management.
Knowledge Generation involves roles such as active solution seeking, viewing problem from number of
perspectives and questioning the assumptions presented by the instructor and others. Collaboration consists of
working together to generate deeper levels of understanding and critical evaluation of material under study,
sharing resources, giving each other feedback etc. Learners have roles in process management such as being
active, participating within minimal guidelines, interacting and engaging with one another and taking the
responsibility for formation of on-line learning communities.
Collaborative Learning and Technology
Technology is accepted as a facilitator in applications of collaboration. The Internet is a natural enabler of
collaboration in the context of on-line learning. On-line collaboration can be broadly defined as the cooperation
of individuals engaged in a common task, using electronic technologies.
168
In the current literature, many benefits including motivating learners, increasing participation in online courses
and empowering learners are directly associated with the use of effective online collaboration strategies.
(Kubala, 1998; Allen, 2002). On the other hand, many researchers acknowledge that online collaboration is not
an easy task but rather it remains a challenging task for all who teach online. For example Horton (2000)
stressed that good design is a must for successful online collaboration, otherwise some learners may not be able
to benefit fully. Especially students with high anxiety regarding communication with others may hesitate to
participate in online discussions. Another challenge pertaining to online collaboration is that compared with
face-to-face interaction; students may demand more interaction with the instructor. As a result, instructor should
dedicate more time for discussion sessions. Finally, online communication lacks some of the features of face-toface communication including gestures, tone of voice and body language. On the other hand, most of these
drawbacks result from poorly designed or implemented applications, thus they can be overcome by proper
design and use of appropriate technologies.
Most collaboration tools can be divided into three categories, being asynchronous collaboration, synchronous
collaboration and integrated collaboration tools. Many businesses employ some type of asynchronous
collaboration technology. These basic collaboration tools include e-mail, bulletin boards and intranets. They
allow people to interact in a more friendly way, but disconnections remain. Information sharing cannot occur in
real-time. Synchronous solutions represent the next level of collaboration, enabling real time communication.
Virtual meetings using text chat tools, audio/video conferencing, and shared whiteboards can provide immediate
connections between users. However, security can be a concern. In integrated collaboration, virtual co-location
products employ leading edge collaboration technology to create permanent virtual workplaces. These
environments allow people to work together in one secure place, wherever they are located, through rich
communications tools like text chat, audio/video conferencing presence awareness and document sharing.
Edutech (2004) has developed an interactive platform to compare web-based learning environments according to
108 criteria. The results of Edutech (2004) study shows that WebCT, BlackBoard and LearningSpace are among
the most powerful widely-used tools. In this study, these three platforms were analyzed in detail by focusing on
the collaboration tools. It is found that chat, whiteboard and forum tools are commonly used for providing
interaction and collaboration among the on-line participants. Apart from the learning and course management
systems, the study also involves the reviews of the groupware systems. In the study, 96 groupware systems were
found, the descriptions of each system were collected by means of their web sites and each was reviewed
according to the tools that they provide for collaboration and communication. Table 1 summarizes the results of
the analysis of groupware systems.
Apart from these, various web-based review tools were examined. wbART, a web-based asynchronous review
tool, was designed to supporting technical asynchronous review (Yanbaş & Demirörs, 2003). The other review
tool examined is iMarkup. This tool also facilitates asynchronous web-based collaboration on web content.
Table 1: The results of analysis of groupware systems
Tools for Collaboration
Forum
Textual Chat
File Sharing
Audio Communication
Screen Sharing
Integrated E-Mail
Instant Messages
Polling
Group Calendar
Video Communication
%
49
47
30
29
24
23
22
22
21
21
Tools for Collaboration
Whiteboard
Workspace Awareness
Application Sharing
Floor Sharing
Version Control
Collaborative Browsing
Virtual Hand Raising
Voice Chat
Collaborative Viewing
Synchronization of Content
%
16
16
15
12
7
6
6
5
3
2
Moreover, MS Word also provides some tools for collaboration and review of the documents. Although
asynchronous tools have powerful features for document review, synchronous communication is more
appropriate for collaborative activities, because collaboration is synchronous by nature. This study proposes a
review tool which promotes collaboration by making use of synchronous communication.
169
An Assessment of On-line Collaboration: A case study of an on-line graduate course
To investigate the effectiveness of the learning management system in on-line collaboration, an exploratory case
study was conducted based upon descriptive data. Berg (2004) describes case studies as a method which “. . .
involves systematically gathering enough information about a particular person, social setting, event, or group to
permit the researcher to effectively understand how the subject operates or functions.” (p.251). Further,
exploratory case studies can be considered as a preliminary phase of a large study. Thus, for the first phase of
this study, the exploratory case study method was employed. The participants of the study included 16 students
enrolled in the course offered in a graduate program at the Informatics Institute at Middle East Technical
University during spring 2002, as well as the instructor and the teaching assistant. The course was offered
through a learning management system, called NET-Class as a case that grounds the whole research process.
All the activities involving collaborative tasks in the course were analyzed. Accordingly, the collaborative
groups and peers were determined. The collaborative groups were whole class, the discussion groups formed in
the forum environment and peers working together to develop the project required in the course.
A combination of quantitative and qualitative research methods were employed in the study: a questionnaire
survey, observation, interviews and document analysis. The qualitative analysis method was applied to the
transcripts of the interviews, the outcomes of the observation reports and the outcomes of the document analysis
of messages posted on the forum. The coding schema developed by Curtis and Lawson (2001), was used to
describe the utterances in on-line collaboration in discussion forums. The coding scheme consists of 15
behaviors and 5 behavior categories as shown in Table 2. The messages posted in the discussion groups were
also analyzed according to 3 important issues, participation, interaction types and feedback.
Table 2: Coding Scheme
Behavior categories
Planning
Contributing
Seeking Input
Reflection / Monitoring
Social Interaction
Behaviors
Group skills
Organizing work
Initiating activities
Providing help
Giving feedback.
Exchanging resources and information
Sharing knowledge
Challenging others
Explaining or elaborating
Seeking help
Seeking feedback
Advocating effort
Monitoring group effort
Reflecting on medium
Social interaction
Findings
Although the study is based on a one-semester observation rather than longitudinal observation, it contributes to
understanding of the effectiveness of NET-Class on-line collaboration, the factors of effective on-line
collaboration and the role of the instructor in an on-line collaborative learning environment.
The findings provide information concerning collaboration in class, the factors affecting on-line collaboration,
the instructor’s role in on-line collaborative learning environments and the suggested collaborative learning
environment.
Collaboration in Class
The findings indicate that except from the initial phase in which learners start to get to know each other, some
peers worked individually throughout most of the phases. They preferred to divide each task into sub-tasks and
distribute the sub tasks among themselves. Their work is more cooperative than collaborative work. These peers
170
stressed that the time, work overload and knowledge levels drove them to work individually in most of the
phases.
The findings regarding both discussion groups and whole class were similar. The on-line interactions in both
collaborative groups show that the interactions provide contributions to the participants. According to evidence
for collaboration in on-line interactions to the findings, it is found that the Contributing behavior category has
the highest proportion among the other behaviors for the whole class. Sharing Knowledge, and Exchanging
resources and information are prominent behaviors of this category respectively. On the other hand, the Social
Interaction behavior category was not present in the coded behaviors in either of the collaborative groups. This
means that the discussions were task focused.
According to the quantitative results, it is found that the average feedback provision time was about 2 days and 3
hours. Regarding the type of interactions, 34 % of the messages involved student-student interactions and
participation of the students was about 40 % for the whole class. Moreover, disorganized messages and threads
in the forum, lack of student-student interaction, the language of the discussions, lack of participation and
activities in discussions in the forum were the major problems expressed by the participants of the study.
The Factors Affecting On-line Collaboration
According to the findings of the study, the factors affecting collaboration in on-line courses are given in Figure
1. Apart from these factors, the participants proclaimed that some strategies used in on-line discussions such as
discussion circles were effective, as student-student interaction had the highest value. According to student
feedback, it was also found that the provision of a platform in which all students or the group members could
meet at a particular time and discuss the projects each week may be an effective strategy to promote effective
peer interaction. The discussion is synchronous in nature; however, the forum tool will be used instead of chat
tool as in the traditional synchronous communication. This is something similar to 8:30 classes in the traditional
learning settings.
Figure 1: The factors affecting on-line collaboration
Role of the Instructor
It was found that the objectives of the course, the content of the course and the nature and purpose of the task
were the factors affecting the role of the on-line instructor. Moreover, almost all of the students emphasized
giving immediate feedback, keeping discussions active, monitoring group work and motivating learners as the
major roles of the on-line instructor in on-line collaborative learning environments.
Suggested Collaborative Learning Environment
The participants were asked to describe the desired tool with which they thought on-line collaboration could be
conducted successfully. Responses indicated that participants consider the forum, instant messaging, common
library and review tools as essential to successful online collaboration. Participants also valued inter-group
interactions as much as intra-group interactions. Figure 2 depicts the structure and components of an online
collaboration session proposed by the students.
171
Figure 2: The structure of on-line class with collaborative groups
WebSCoRe: Web-based Synchronous Collaborative Review Tool
According to the results, it was found that in the majority of groupware systems, collaboration is provided with
some particular tools such as chat, whiteboard, file sharing, audio communication, discussion board, e-mail etc.
These tools are important and essential parts of an on-line learning environment. However, it is essential to
enrich the learning environment with variety of tools, which support different instructional strategies such as
reviews.
Because of the lack of diversity of tools, the contribution of collaborative reviewing strategy to gain
collaborative skills and the suggestions of the participants of the study, a synchronous web-based tool called
WebSCoRe, used for promoting on-line collaborative reviewing, is proposed in this study. WebSCoRe supports
distributed and synchronous collaborative document reviewing in on-line learning settings. It is a real-time
system that lets everyone participate regardless of location.
The users of WebSCoRe are categorized into two groups: Students and Facilitators. A Student can be either the
author, who is the creator of the document or a reviewer who gives comments about the document. The
facilitator is the manager of the review session, and s/he initiates the review. The facilitator also has the same
roles given to the reviewers.
The document constitutes the core of the review session. We assume that authors have created the document and
sent it to the instructor; therefore reviewers have already read the document and have prepared for commenting.
WebSCoRe provides a meeting-based communication platform between reviewers and authors in order to share
ideas and comments after the examination of the document created by the authors with an eye to criticism and
correction. Discussion is the backbone of the WebSCoRe which promotes collaboration through rich interactions
among on-line participants.
The facilitator is responsible for facilitating the review session. Figure 3 displays a typical view from the review
room screen. This is the main meeting space for on-line review. The facilitator, reviewers and authors examine
the document and discuss about the comments in turn through this screen. The review room screen involves
three parts. In the first part, reviewers and authors are listed and the content of the document is displayed. The
content area is the place where the reviewers and the facilitator select the text from the content and give
comments to the selected area. The second part is called the comment area. The comments given by either the
facilitator or a reviewer are displayed in this area. Lastly, the third part is the discussion area where all
participants of the review session may send messages to each other. The structure of the review room screen is
almost the same for each user. However, according to the roles of the users, some of the functions may be
disabled in the review room screen. For example, authors are not allowed to give comments about their own
document. They may discuss comments given by the reviewers and the facilitator. Therefore, “Add Comment”
and “Accept” and “Reject” functions are not displayed in their review room screens.
172
Figure 3: Review room screen
Color is one of the most important features of the WebSCoRe. Color is used very simply to differentiate the
users from each other. Each user chooses his/her color that will represent the user. It is assumed that this
prevents confusion when the number of participants is high.
Figure 4 displays an exemplary review session with four users (screenshots 1, 2, 3, and 4): the facilitator, two
reviewers (Reviewer A and Reviewer B) and the author. The color of each user is reflected in the review room
screens. The selected text area, the name of the users and the messages sent by the users are colored with a
differently for each user (screenshot 1). In the example given below, Reviewer A selected the text from the
content and gave the comment about the selected content. The color of the selected area is the same color with
the color of Reviewer A, who sent the comment (screenshot 2).
The other important feature of the WebSCoRe is that each comment is examined and discussed in turn.
Therefore, when any of the reviewers or the facilitator provides comments, the functions for providing comments
become disabled in the review room screens of the other users who are not providing comments. For example, in
the example below, “Add Comment”, “Accept” and “Reject” buttons in the review room screens of other users
become disabled when the Reviewer A sends his/her comment (screenshot 3). After discussing the comment, a
common decision should be taken to accept or reject the comment. Then, Reviewer A finalizes the examination
of his/her comment by either accepting or rejecting the comment.
The review session is finalized by the facilitator (screenshot 4). It is assumed that the participants will come to a
decision to approve or disapprove the document. The facilitator is responsible for approving or disapproving the
document according to the decision taken by the review session members. It becomes clear that WebSCoRe
focuses mainly the discussions among the participants.
Utilization of WebSCoRe
WebSCoRe was applied to review user manuals of a learning management system by the developers of the user
manuals. Although the application is an initial experiment of the tool rather than a case study to validate the
effectiveness, it contributes to understanding of effectiveness of WebSCoRe in on-line collaboration, and the
strengths and weaknesses of the tool.
173
Screenshot 1
Screenshot 2
Screenshot 3
Screenshot 4
Figure 4: View from a review session with four users
The participants of the application were the research assistants and graduate students. They were responsible for
creating and reviewing the user manuals developed for a learning management system. All participants have the
knowledge and experience regarding document reviewing. We constructed each review with at least three users:
a facilitator, a reviewer and the author. At the end of the application of WebSCoRe, we had six review sessions.
We then interviewed the participants and asked about the effectiveness, strengths and weaknesses of
WebSCoRe. The semi-structured interview approach was used. The average time for the interviews was 40
minutes. We analyzed the interviews using qualitative analysis procedures.
According to the findings of the analysis of the responses, we found that all of the participants believed that the
WebSCoRe could promote both effective on-line collaborative learning and collaborative working. The
participants generally compared document reviews in on-line settings with document reviews in face-to-face
settings. Few of them believe that document reviews in on-line environment could not be effective as in the faceto-face settings. They argued that in on-line settings, it is necessary to keep sentences as simple and small as
possible. However, in face-to-face environments, they argued, even mimics can explain lots of things, which
cannot be expressed in another way. On the other hand, all of the participants argued that the distance education
and on-line learning settings have become necessary in this era, and therefore, there is a need for variety of tool
that might increase the quality of the instruction given through the Internet. They believed that the WebSCoRe
could be an example of this kind of tool. In fact, one of the participants said, “… the WebSCoRe was developed
for conducting documents reviews in on-line settings and it has the closest design to the document reviews in the
face-to-face settings…”
174
The findings also revealed the strengths and weaknesses of WebSCoRe (see Table 3).
•
•
•
•
•
•
•
Table 3: The Strengths and Weaknesses of WebSCoRe
Strengths
Weaknesses
Immediate feedback
• Tracking is difficult when the group size is high
Discussion-oriented
• Communication is slow
Color coding
• There is a need for writing short sentences
Providing ease of tracking comments and discussions
• The details cannot be considered at all
Providing high participation
• The effectiveness is highly dependent on of the facilitator
Increasing motivation
Better than e-mail or forum tools
The findings indicate that determining the review strategy, providing guidelines for review process, engaging the
group in discussion of guidelines, formulating shared understanding, getting people to participate to the
discussions, and providing support as needed are found as the major roles of the on-line facilitator in
synchronous document review environment.
The findings also provide some recommendations for the further improvement of the WebSCoRe such as giving
more control to the participants, increasing the use of keyboard and extending the features of the tool by adding
sound facilities. The findings state that like color, sound might also help to track the review session.
Conclusion
“Collaborative learning”, “team work”, “student-centered learning” and “students taking responsibility for their
own learning” have become essentials in education in the information age. There is a high demand from staff and
employers for students to graduate with good interpersonal skills, knowledge of group dynamics, the flexibility
to work in teams, the ability to lean, to solve problems and to communicate effectively.
This study indicates that collaboration in on-line learning environments depends on both instructional and
technical issues. Although technical aspects are important, the spirit of distance education lies in the instructional
strategies applied in the learning settings. Therefore, it is suggested to design and develop new instructional
strategies and methods and then to design and develop necessary tools accordingly. Conducting reviews on a
document is one of those effective instructional strategies and WebSCoRe, the collaborative review tool
proposed, was found to be effective in facilitating on-line document reviews. According to Dillenbourg (2000),
interaction can be rich when the participants explain themselves in terms of conceptions and not simply answer,
when they argue about the meaning of terms and representations and shift roles. It is found that WebSCoRe
encourages participants to build such rich interactions through providing a platform for working together on a
common document, which provides for the collaborators to share and discuss their ideas about further revisions.
Coomey and Stephenson (2001) argue that getting immediate feedback is one of the essentials of an effective on-line
learning environment. Moore and Kearsley (1996) also add two more issues as the essentials such as increasing
motivation and participation. The study also confirms that the giving immediate feedback, increasing participation,
encouraging student-student interaction and social interaction are the three essential tasks of an effective on-line
collaborative learning environment. According to the findings, it can be concluded that WebSCoRe satisfies these
essentials of on-line learning environment.
The study showed that pedagogical and the managerial functions of the facilitator to be more important and
highly emphasized in on-line document reviews. The group size, the quality of facilitation, the speed, and the
lack of facial facilities such as mimics, the limited size of discussions were stressed as the important issues
affecting the effectiveness of WebSCoRe and given as the weaknesses of the tool. In fact, these issues are the
weaknesses of the distance education environment rather than the weaknesses of WebSCoRe.
To summarize, collaborative review is one of the instructional methods that promotes rich interactions. It is
effective to apply collaborative reviewing strategy to develop collaborative skills and to promote collaborative
learning. According to the findings of the study, it can be concluded that the WebSCoRe is a promising tool that
might be a positive influence in developing collaborative skills and supporting effective collaborative learning.
175
References
Allen, M. (2002). Guide to e-learning: building interactive, fun, and effective learning programs for any
company, New Jersey, USA: John Wiley & Sons.
Berg, B. L. (2004). Qualitative research methods for the social sciences, Boston, MA, USA: Pearson Education.
Bruner, C. (1991). Thinking collaboratively: Ten questions and answers to help policy makers improve
children's services, Washington, DC, USA: Education and Human Services Consortium.
Coomey M., & Stephenson, J. (2001). On-line Learning: it is all about dialogue, involvement, support and
control - according to the research. In Stephenson, J. (Ed.), Teaching and Learning On-line: Pedagogies for New
Technologies, Sterling, VA, USA: Kogan Page, 37-52.
Curtis, D. D., & Lawson, M. J. (2001). Exploring collaborative on-line learning. Journal of Asynchronous
Learning Networks, 5 (1), 21-34.
Dillenbourg, P., & Schneider, D. (1995). Collaborative learning and the Internet. Proceedings of International
Conference on Computer Assisted Instruction (ICCAI 95), retrieved January, 05 2006 from
http://tecfa.unige.ch/tecfa/research/CMC/colla/iccai95_1.html
Dillenbourg, P. (1999). Introduction: What do you mean by "collaborative learning"? In Dillenbourg, P. (Ed.),
Collaborative learning: Cognitive and computational approaches, Amsterdam: Pergamon, 1-19.
Dillenbourg, P. (2000). Virtual learning environments. Paper presented at the Workshop on Virtual Learning
Environments of the EUN Conference: Learning in the New Millennium: Building New Education Strategies for
Schools, retrieved January, 05 2006, from http://tecfa.unige.ch/tecfa/publicat/dil-papers-2/Dil.7.5.18.pdf.
Edutech (2004). Evaluation of Learning M<anagement Systems - 2003, retrieved January 05, 2006 from
http://www.edutech.ch/lms/ev2.php.
Gokhale, A. (1995). Collaborative learning enhances critical thinking. Journal of Technology Education, 7 (1),
22-30.
Hiltz, S. R. (1998). Collaborative Learning in Asynchronous Learning Networks: Building Learning
Communities. Invited address at WEB98 Conference, Orlando Florida, November 1998, retrieved January, 05
2006 from http://eies.njit.edu/~hiltz/collaborative_learning_in_asynch.htm.
Horton, W., & Horton, K. (2003). E-Learning Tools and Technologies, Indianapolis, IN. USA: Wiley.
Horton, W. (2000). Designing Web-Based Training: How to Teach Anyone Anything Anywhere Anytime, New
York, USA: John Wiley and Sons.
Kim, H. C. (2002). From comments to dialogues: a study of asynchronous dialogue processes as part of
collaborative reviewing on the Web. Proceedings of 35th Annual Hawaii International Conference on System
Sciences (HICSS-35) (Vol. 4), retrieved January, 05 2006 from http://csdl2.computer.org/comp/proceedings/
hicss/2002/1435/04/14350116.pdf.
Kubala, T. (1998). Addressing student needs: Teaching on the internet. Technological Horizons in Education
Journal, 25 (8), 71-74.
Layzell, P., Brereton, O. P., & French, A. (2000). Supporting collaboration in distributed software engineering
teams. Paper presented at the Seventh Asia-Pacific Software Engineering Conference (APSEC'00), December 58, 2000, Singapore.
Moore, M. G., & Kearsley, G. (1996). Distance education: A systems view, Belmont, CA, USA: Wadsworth.
Muronaga, K., & Harada, V. (1999). The art of collaboration. Teacher Librarian, 27 (1), 9-14.
176
Palloff, R. M., & Pratt, K. (1999). Building Learning Communities in Cyberspace, San Francisco, USA: JosseyBass.
Palloff, R. M., & Pratt P (2001). Lessons from the Cyberspace Classroom: The Realities of On-line Teaching,
San Francisco: Jossey Bass.
Schoeder, U., & Brunner, M. (1996). Opinion - Distributed Discussion Support Based on the Metaplan Method.
Technical Report No. PI-R 3/96, Software Engineering Group, Department of Computer Science, Technical
University of Darmstadt, retrieved January, 05 2006 from http://www.pi.informatik.tu-darmstadt.de/
publikationen/technische%20Berichte/1996/TR_1996.3/opinion.html.
Yanbaş, E., & Demirörs, O. (2003). A Web-based Review System and its Usability in a Small Software
Organization. Paper presented at the Applied Informatics 2003 International Conference (IASTED), February
10-13, 2003, Innsbruck, Austria.
177
Nokelainen, P. (2006). An empirical assessment of pedagogical usability criteria for digital learning material with elementary
school students. Educational Technology & Society, 9 (2), 178-197.
An empirical assessment of pedagogical usability criteria for digital learning
material with elementary school students
Petri Nokelainen
Research Centre for Vocational Education, University of Tampere, FIN-13101 Hämeenlinna, Finland
http://www.uta.fi/u/petri.nokelainen
Tel: +358 40 557 4994
Fax: +358 3 6145 6111
[email protected]
ABSTRACT
This paper presents the pedagogical usability criteria for evaluating the digital learning material.
Pedagogical aspects of designing or using digital learning material are much less frequently studied than
technical ones. Further, there are relatively few inventories measuring subjective end-user satisfaction with
the pedagogical aspects of digital learning material and not a single inventory has undergone a rigorous
process of empirical psychometric testing. They include the following components: 1. Learner control, 2.
Learner activity, 3. Cooperative/Collaborative learning, 4. Goal orientation, 5. Applicability, 6. Added
value, 7. Motivation, 8. Valuation of previous knowledge, 9. Flexibility and 10. Feedback. The pedagogical
usability criteria have been operationalized into an on-line Likert -scale self-rating Pedagogically
Meaningful Learning Questionnaire (PMLQ) that has 56 items. In the PMLQ, separate items have been
developed to measure the usability of the learning management system (LMS) and the learning material
(LM). When evaluating the usability of a LMS, it is possible in the pedagogical context to evaluate the kind
of learning material it enables the users to produce. Evaluation of the usability of a LM is based on a fact
that the learning content is based on a certain learning goal or goals. The criteria of pedagogical usability
presented here have undergone two-step psychometric testing process using empirical samples of 5th and
6th grade elementary school students (n = 66 and n = 74). Students evaluated one LMS and four LM’s with
PMLQ. Results supported the existence of theoretical dimensions of the criteria. The PMLQ was able to
capture differences in the pedagogical usability profiles of the learning modules. Generalizability of the
pedagogical usability criteria to other domains is limited by the small sample size, the small age range of
respondents’ and the small number of learning material evaluated. However, empirical studies that aim at
evaluating a more representative set of learning material in different domains targeted for both adolescent
and adult learners are currently conducted.
Keywords
Pedagogical issues, Usability, Evaluation methodologies, Evaluation of CAL systems, Authoring tools and
methods
Introduction
New software has eliminated some previously prerequisites student skills, while simultaneously creating new
requirements for the computer systems they are using. The usability analysis of learning material is intended to
influence the process by which the correct applications and target groups find one another so that a student can
focus his/her energy on the content of the learning material rather than technical issues caused by the software
design or user interface.
Several sets of recommendations for the evaluation of technical usability have been developed over the last
twenty years (e.g., Shneiderman, 1998; Chin, Diehl & Norman, 1988; Nielsen, 1993; 1994; Lin, Choong &
Salvendy, 1997; Preece, Rogers & Sharp, 2002; Chalmers, 2003; Tognazzini, 2003). However, pedagogical
aspects of designing or using digital learning material are much less frequently studied than technical ones. This
paper presents criteria for evaluating the pedagogical usability of digital learning material. The purpose of the
criteria is not to brand any learning material as “good” or “bad,” but to help users choose the most suitable
alternative for any particular learning situation.
When evaluating technical usability, the basic assumption is that it should be easy to learn to use the central
functions of the system and the functions are efficient and convenient in use. Another assumption is that error
responses to incorrect operation of the software should help teach the user to use the system as intended so that
the error will not be repeated. When evaluating pedagogical usability, the assumption is that the designers of the
learning platform or learning unit were guided by either a conscious or subconscious idea of how the functions
178
of the system facilitate the learning of the material it is delivering. Examples of the range of learning theories
that influence design paradigms are objectivism (instructivism, behaviourism), and constructivism (focus on
learner, learner’s active role in learning and learning from experience).
The criteria presented here have been operationalized into an on-line application (Nokelainen, 2004a) that
applies a Likert –scale self-rating questionnaire with 56 items (see Appendix). In the Pedagogically Meaningful
Learning Questionnaire (PMLQ), separate items have been developed to measure the usability of the learning
platform and the learning unit it is delivering. When evaluating the usability of a specific learning platform
(learning management system), we can evaluate how easy it is to use the platform itself (technical usability), or
what kind of learning material it enables the users to produce (pedagogical usability). When evaluating the
usability of a learning unit or a learning object, we assume that each learning unit has its own interface relating
to the content, and learning content based on a certain learning goal. When evaluating the pedagogical usability
of a learning unit, we must try to control the effect of the pedagogical solutions of a learning platform.
The compilation of criteria for usability of digital learning material presented here is based on the following
preparatory analyses: 1. Charting of systems and taxonomies developed for the evaluation of digital learning
materials, 2001-2002, 2. Analysis of learning theories suitable as the basis of evaluation, 2001-2002, 3.
Compilation of the first draft of criteria for pedagogical usability, 2001-2002, and 4. Testing of the first draft of
criteria for pedagogical usability with empirical samples, 2001-2002.
Central Concepts
Digital learning material means in this research all material that is designed for educational purposes, published
in a digital form and intended to be accessed by computer.
A Learning object (LO) is, as the name implies, the smallest reasonable unit of learning material. Examples of
this are a pronunciation sample from an English language-teaching program or an animation clip that describes
how a dangerous procedure is completed safely.
Learning material (LM) can include such things as 1) a WWW page introducing the effects of image
compression on the quality of an image, 2) a JAVA applet to practice touch-typing, 3)an individual learning
module in a learning platform, the topic of which might be, for example, positioning decimal numbers on a
continuum. Regardless of the type, each individual learning material has its own user interface, the usability of
which can be evaluated, as well as a definable learning goal.
A Unit of learning material (ULM) consists of several individual learning materials connected by a common
main goal. A ULM is a concept that combines several learning objects (and possible meta, history and relation
data) related to the same subject as a single entity.
A Virtual learning environment (VLE) is an application that uses learning materials, or units of learning
material. There are numerous commercial and non-commercial virtual learning environments available. They
can be divided into two groups according to whether they include prepared learning material (e.g., eWSOY’s
OPIT) or offer a presentation service for learning material provided by the user (e.g., BlackBoard, Belvedere,
Dyn3W, Future Learning Environment, Knowledge Forum, WebCT)
Usability can, according to Keinonen (1998, 62) be defined as a characteristic related to 1)the product’s design
process, 2) the product itself, 3) use of the product, 4) user experiences of the product or 5) user expectations. In
the present research, usability is understood primarily as the product’s usability attributes, which are measured
through subjective user experiences with a self-evaluation questionnaire.
The operationalisation of the usability attributes in this study is done according to the model presented by
Nielsen (1990, 148). Figure 1 shows that the top-level concept is system acceptability. Acceptability is further
divided into two parts: 1) Social acceptability and 2) Practical acceptability. The implementation of a system that
serves as a presentation platform reveals the views of its developers about both social and practical acceptability.
The example given by Nielsen to illustrate social acceptability (ibid. 147-148) is a curriculum that lets only the
teacher make changes that increase viewpoints on the studied subject, which could in some societies be seen as
inappropriately emphasising the teacher’s authority and restricting the students’ individual knowledge formation.
The tone of Nielsen’s later example of social acceptability (1993, 24) puts more emphasis on the ethical and
179
moral choices of the program developers: Is it right that people applying for unemployment benefits would have
to use an application form that attempts to use their input (questions) to reveal those who are applying for
benefits for invalid reasons? According to Nielsen (1990, 148), practical acceptability means, among other
things, cost, compatibility, reliability and usefulness of the system. Nielsen states (1990, 148) that usefulness can
be further divided into utility and usability. Utility refers to the ability of the system to generally provide
functions that correspond with the needs of the users and usability refers to how well the users are able to use the
functions offered by the system (ibid. 148-149). According to Nielsen’s examples (1990, 151), an entertainment
application (e.g., a game) is utilisable for users if they enjoy themselves with it, and learning material is
utilisable if students learn with it. Nielsen (ibid. 151) further divides usability into learnability, efficiency,
memorability, errors, and satisfaction, calling them usability attributes. Learnability depends on how long
beginners’ use a system before they learn the essential skills necessary to perform their tasks. Efficiency refers to
how well experienced users can operate an application after they have mastered it. Memorability means the
ability of an occasional user who has previously used the system to remember its operational principles. Errors
are divided into two groups: less serious errors that disturb the work of the user and serious errors that, for
example, endanger the preservability of the users’ outputs. Satisfaction is a subjective judgment by the user.
Figure 1. Conceptual mapping of the technical and pedagogical usability to the Nielsen’s usability model
(Adapted from Nielsen 1990, 148)
Pedagogical usability, which is defined in this research according to Nielsen’s classification (1990, 148), is a
sub-concept of utility, and technical usability is a sub-concept of usability (Figure 1). Thus, in addition to the
dialogue between a user and a system, the pedagogical usability of a system and/or learning material is also
dependent on the goals set for a learning situation by the student and teacher.
Evaluation in this research is a subjective judgment made by the user of digital learning material. The evaluation
criteria are applied using a self-evaluation questionnaire that employs a Likert scale 1 (strongly disagree) – 5
(strongly agree), 6 (not applicable), 7 (don’t know). The teacher and learners/students each have their own
versions of the evaluative statements.
Pedagogical Usability
Learning is a largely unobservable and uncontrollable process that happens all the time. Attributes attached to
learning such as ‘effectiveness’ or ‘informality’ are both difficult to understand and measure. The term
education, on the other hand, is easier to analyse because it is more often bound to observable artefacts such as
books or computer programs that are related to a curriculum. Some researchers term education that is tied to a set
180
curriculum as ‘formal,’ and education that is more conversation based in nature as ‘informal’ (Livingstone,
2000). Although we will go no further with this topic, one may ask what portion of measurement error of the
pedagogical aspects of usability is related to the difference between the formal and informal aspects of
education.
Education can be provided in groups or according to individual needs as teacher directed, cooperative, or
individual practice. When computers and digital learning material are used in a learning situation, it is expected
that this is done to introduce identifiable added value to the learning in comparison to, for example, printed
material, and material produced by the teacher or the students themselves. The expectation of “added value,” not
merely “reaching the same level,” is demonstrated by the fact that each group of students is under the charge of
an educational expert who has primary responsibility for the provision of education that agrees with the general
goals, and further supported by computers and software that require not only capital investment but also
continuous maintenance. In the case of distance education, the equation is different and so it may be that
reaching the same level as in contact teaching is considered acceptable. In this context, the level of educational
quality desired is at least that of the average performance in a good quality contact teaching context – it is
unlikely that one would want to match digital material with poor learning materials and teaching practices! In the
following the problematic concepts of the quality of education and learning are discussed in more detail.
Many of the “innovations” that have been executed in computer environments have their equivalents in the
world of traditional education. For example, instead of a “mind map” type of computer application that collects
arguments and counter-arguments, cooperative knowledge construction can be achieved with large pieces of
paper on the walls of a classroom, on which various points and opinions that have been raised during a week can
be recorded. The added value of computers in this case is technical, because it gives the students a chance to
work simultaneously on several different things (for example in different subjects) and save each phase of their
work on their own workspace. In a classroom, the activity is more limited and inflexible because the space on the
walls of the classroom is limited. Then again, if the unfinished work and problems were visible at all times, the
students could work on them in the background for the whole time that they spend in the classroom. A counterargument to this is the Sartrean thought about comprehensive thinking that follows the learner from school to
home. In this case, a computer-based application allows the learners to continue their work at home, in a library
or in a coffee shop with the help of a computer network.
Developing the New Criteria for Pedagogical Usability
We analysed the criteria that have been developed for the factors in learning materials that promote learning
(Reeves, 1994; Squires & Preece, 1996; Quinn, 1996; Albion, 1999; Squires & Preece, 1999; Horila,
Nokelainen, Syvänen & Överlund, 2002). The summary of different criteria is presented in Table 1. The small
number of existing models shows that pedagogical aspects of designing or using digital learning material are
much less frequently studied than technical ones. The table shows that one criteria is a theoretical model (Squires
& Preece, 1996), four are theoretical models with heuristic checklists (Reeves, 1994; Quinn, 1996; Albion, 1999;
Squires & Preece, 1999) and only one is inventory measuring subjective end-user satisfaction with the
pedagogical aspects of digital learning material (Horila et al., 2002). Further, not a single heuristic checklist or
inventory had undergone a process of empirical psychometric testing.
The most common pedagogical usability features of the criteria presented in Table 1 are “learner control”,
“possibility for cooperative or collaborative learning activities”, “explicit learning goals”, “authenticity of
learning material” and “learner support (scaffolding)”.
In this study, we created new criteria for the assessment of the pedagogical usability of digital learning materials,
as the earlier research work had not addressed all the relevant issues of the topic neither on a theoretical nor
practical level. The existing criteria neglect partially the role of learner’s activity, added value of digital learning
material, learning motivation and feedback related to user input. We found that none of the existing criteria
included such concepts as valuation of previous knowledge and role of pre-testing and diagnostics. Our model
includes ten dimensions as follows: 1. Learner control, 2. Learner activity, 3. Cooperative/Collaborative
learning, 4. Goal orientation, 5. Applicability, 6. Added value, 7. Motivation, 8. Valuation of previous
knowledge, 9. Flexibility and 10. Feedback.
181
Next we proceed to describe the theoretical structure of each dimension. The model presented here contains 51
sub dimensions. To get the picture of the overall structure of the criteria, the reader is encouraged to see the
pedagogical usability sub dimensions in the Appendix 2.
1 Learner Control
When learning a new topic, the learner’s memory should be burdened to an optimal level (Miller, 1956;
Shneiderman, 1998, 355). It is difficult to define a universally optimal level (generally people can have 5 to 9
items in their short term work memory), but it is certainly helpful to break down the material to be learned into
meaningful units (Wilson & Myers, 2000). In so-called structured learning materials, the learned material has
already been broken down into meaningful units from the point of view of the students (albeit usually by the
teacher). Such a “one-size-fits-all” approach has been criticized for the fact that the students are required to
adjust their learning to fit the teacher’s conception of the best way of learning certain material (Jonassen, Myers
& McKillop, 1996). The factor structure of this dimension is presented in Table 2.
2 Learner Activity
A teacher’s “didactic role” in a learning situation may strongly scaffold the learners’ own activity, and,
correspondingly, the learners’ independent activity may be increased when the teacher stays in the background,
as a “facilitator” (Reeves, 1994). Learners’ activity is determined in large measure but the characteristics of the
learners themselves, but the learning material can affect it through assignments that support student activity by
being interesting and based on real life. An alternative approach to structured material is problem-based learning
(e.g., Savery & Duffy, 1995). In this case, the teacher gives the students a certain amount of source material from
which the students (individually or in groups) construct their own conception of the topic to be learned.
However, even in this approach all available information (i.e. “source material”) has usually already been
structured by someone else! Problem-based learning or accessory software also increases student activity
(Jonassen, Peck & Wilson, 1999). Items 8, 9 and 10 reflect more collaborative nature of learning (Forman &
Cazden, 1985; Palloff & Pratt, 2005; Tudge & Rogoff, 1989). Study of collaborative activity in computer-based
settings, often cited as computer supported collaborative learning (CSCL), can enhance our understanding of the
processes of productive interaction (e.g., Littleton & Häkkinen, 1999). The factor structure of this dimension is
presented in Table 3.
3 Cooperative/Collaborative Learning
Cooperative and collaborative learning means studying with other learners to reach a common learning goal.
According to Barab and Duffy (2000, 27-28), learners are moving away from acquisition metaphor (i.e.,
acquiring knowledge that is constituted of symbolic mental representations) to participation metaphor (i.e.,
knowledge is considered fundamentally situated in practice). Instead of acquiring personal knowledge, learners
construct knowledge as members of communities in practice (Lave & Wenger, 1991). To be more specific,
cooperative learning is more structured than collaborative learning, as the teacher has the control. Learning takes
place in groups in which the members gather and structure information, in which case the system or learning
material should offer the learner tools that can be used to communicate and negotiate different approaches to a
learning problem (Jonassen, 1995). Through the use of computer-supported learning material it is possible to
practice cooperative learning so that all students are connected to each other over a distance, for example
through discussion groups and chat forums (Quinn, 1996; Reeves 1994). In a system or learning material
supporting cooperative knowledge construction (KB, knowledge building or KC, knowledge construction) may
include, for example, a visual tool that the learners can use to fashion simultaneous mind maps of the topic.
Often such a system also includes tools for social navigation (Mayes & Fowler, 1999; Kurhila, Miettinen,
Nokelainen & Tirri, 2002) with which a learner gains information about what the other learners have done
(asynchronic social navigation) or are presently doing (synchronic social navigation).
4 Goal Orientation
As learning is a goal-oriented activity, goals and objectives should be clear to the learner (Quinn, 1996). The best
results are attained when the goals of the learning material, teacher and student are closely aligned. The goals
182
may vary from concrete (for example the basic techniques of first aid) to abstract (learning material aimed to
develop the appreciation of modern arts) (Reeves, 1994). If the learners themselves do not set the goals, their
meaningfulness should be justified from the point of view of motivation. The students should have a chance to
pursue their own interests in relation to the learning goals. According to instructivist learning theory, learners
should be introduced to only with a few, clearly specified goals at any one time. According to constructivist
learning theory, the goals should be clearly defined, but they have to originate, as much as possible, with the
learners themselves. (Wilson & Myers, 2000.) If the goals do not originate with the students, their meaning
should be explained to them. The students should have a chance to make choices with respect to the course of
their studies in relation to the learning goals. For example, in some learning materials, the assignments are
graded and granted points and thus the student can access the next assignment after garnering enough points
from the previous assignments. The downside of the point system is the fact that the students are less able to
complete the material in the order of their own preference, or bypass areas that they already know.
Table 1. Summary of the pedagogical usability criteria research
Reeves (1994)
“Pedagogical
dimensions” b
1. Learner control
2. Pedagogical
philosophy
3. Underlying
psychology
4. Goal orientation
Quinn (1996)
Squires & Preece
“Educational
(1996) “JIGSAW
design heuristics” b
model” a
1. Clear goals and
objectives
2. Context
meaningful to
domain and learner
3. Content clearly and
multiply represented
and multiply
navigable
4. Activities
scaffolded
Albion (1999)
“Content
heuristics” b
1. Establishment of
context
2. Relevance to
professional practice
3. Application
operation tasks
3. Representation of
3. Match between
professional responses designer and learner
to issues
models
3. Technical
requirements
4. General system
operation tasks
4. Relevance of
reference materials
4. Intuitive
efficiency
5. Elicit learner
understandings
5. Presentation of
video resources
6. Teacher role
6. Formative
evaluation
6. Assistance is
supportive rather than
prescriptive
7. Program
flexibility
7. Performance
should be ‘criteriareferenced’
7. Materials are
engaging
8. Value of errors
8. Support for
transference and
acquiring 'selflearning' skills
9. Support for
collaborative learning
8. Presentation of
resources
10. Motivation
11. Epistemology
9. Overall
effectiveness of
materials
1. Appropriate levels
of learner control
2. Navigational
fidelity
Horila,
Nokelainen,
Syvänen &
Överlund (2002)
“Pedagogical
usability of digital
learning
environments” c
1. Specific learning
tasks
2. General learning
tasks
5. Experiential value
(Authenticity)
9. Cooperative
learning
Squires & Preece
(1999) “Learning
with software
heuristics” b
4. Prevention of
peripheral cognitive
errors
5. Understandable and
meaningful symbolic
representation
1. Learnability
2. Graphics and
layout
5. Suitability for
different learners
and different
situations
6. Support personally 6. Ease of use:
significant approaches Technical and
to learning
pedagogical
approach
7. Strategies for the
7. Interactivity
cognitive error
recognition, diagnosis
and recovery
8. Match with
8. Objectiveness
curriculum
9. Sociality
10. Motivation
11. Added value for
teaching
12. User activity
13. Accommodation
of individual
differences
(Scaffolding)
14. Cultural
sensitivity
a
= Theoretical model. b = Theoretical model and heuristic checklist. c = Theoretical model and subjective enduser inventory.
183
Table 2. Factor structure of the first pedagogical usability dimension (1. Learner control)
Target group
Child
Factors
1.1 Minimum memory load
Items
LMS
1.2 Meaningful encoding
1.3 Take responsibility for one’s own learning
1.4 User control
2
Teacher
Items
LM
47
Items
LMS
Adolescent/adult
Items
LM
47
Items
LMS
Items
LM
47
48, 49
48, 49
48, 49
1
1
1
2
2
2
2
2
8.2 Valuation of Previous Knowledge
40
40
40
-> Elaboration
Note. LMS = Learning management system. LM = Learning module/material. See Appendix for the item labels.
Italicised items show relationship to the other usability dimensions.
5 Applicability
The approach taken in learning material should correspond to the skills that the learner will later need in
everyday and working life (Jonassen, Peck & Wilson, 1999; Quinn, 1996). The skills or learned knowledge
should be transferable to other contexts (Quinn, 1996; Reeves, 1994). Learning something new is most
effectively accomplished through so-called learning by doing methods that involve practical tasks. Learning by
doing has been found to be an effective learning method for both practical (changing some part of a paper pulp
machinery) and in abstract (integration in mathematics) issues. Learning material should always be at an
appropriate level from the point of view of a learner’s learning process (Wilson & Myers, 2000). For example,
students in the early grades of comprehensive school have limited ability to adopt abstract concepts, but it
becomes far easier for fifth and sixth graders. It is possible to adapt learning material to meet the needs of a
student for, example, by periodically asking for feedback about the experienced difficulty of the material. In a
dynamic system, it is possible to follow a student’s activity, such as the time it takes to finish assignments and
the number of mistakes, and adapt future tasks according to this information. Learning material should be
planned and executed in cooperation with both groups of end-users (students and teachers); systems produced by
experts rarely meet their everyday needs without modifications. In the planning of learning material special
attention can be given to those issues that are most likely to cause problems for the learners (for example the
adoption of a new concept) and build support structures that the students can use if needed. The system can also
observe the learner’s activity and proactively offer hints (prompting and fading, Hannafin & Peck, 1988, 47).
When students are in a so-called ZPD (zone of proximal development, Vygotsky, 1978, 84-91), they are in the
process of rising into a new level of skills and knowledge and progress depends on the understanding of some
detail. When they acquire a hint to solve this problem (scaffold), they experience a break through and can
progress to the next step in their studies (Chalmers, 2003).
6 Added Value
A formal learning situation, planned by a teacher or a mentor, can be carried out in many ways, for example
through cooperative or individual learning approaches, directed by a teacher or as group work or individual
practice. When computers and digital learning material are used in a learning situation, it is expected to introduce
definite added value to the learning in comparison to, for example, printed material, and material produced by
teacher or the students themselves. The added value is usually in the form of creative use of the possibilities that
the computer offers, for example voice, image and video files: the learners can choose a media that best fits their
preferences. Jansen, van den Hooven, Jägers and Steenbakkers (2002) point out that especially young students
are familiar with computers and multimedia programs (for example so-called video games) and so similar
components in learning material suit their life styles and future work. Jansen et al. (2002) have devised a list of
aspects of computer-assisted learning that offer added value: (1) adaptability to individual needs, (2) number of
flexible options, (3) learning is controlled by the learner, initiated by the learner and is in the form that the
learner desires, (4) interesting contents, (5) development of communication, and (6) active participation of the
students. In practice, the realization of all the items on this list requires that the developers of the learning
184
material have multidisciplinary skills and knowledge, experience or teaching and time to develop the learning
material. The system or learning material should offer the students with tools that are suited to control the
contents of the learning material and that make the use of the material more effective and economic. The student
should have a feeling that the topic is best learned through the use of a computer.
Table 3. Factor structure of the second pedagogical usability dimension (2. Learner activity)
Target group
Child
Factors
2.1 Reflective thinking
2.2 Problem-based learning (PBL)
Items
LMS
Teacher
Items
LM
3, 4r
Items
LMS
Adolescent/adult
Items
LM
Items
LMS
Items
LM
3
5, 9
5, 9
5, 9
2.3 Use of primary data sources
6
6
6
2.4 Immersion
7
7
7
2.5 Ownership
8, 10
8, 10
8, 10
Note. LMS = Learning management system. LM = Learning module/material. r = A reversed item, i.e. values in
1 to 5 Likert scale change as follows: 1 -> 5, 2 -> 4, etc. See Appendix for the item labels.
7 Motivation
Motivation (Latin: Movere, to move) affects all learning and makes people behave the way they do.
Behaviourists explain the motivation to do things by reference to instincts, desires and reinforcement; cognitive
theorists rely on models of cognitive processes and analysis (Wilson & Myers, 2000, 65). Motivations, which
can be either consciously or subconsciously goal-oriented, support the direction of an individual’s general
behaviour (Ruohotie, 1996). Key concepts of motivation include incentives, self-regulation, expectations,
attributions of failure and success, performance or learning goals, as well as intrinsic or extrinsic goal orientation
(Reeves, 1994; Ruohotie & Nokelainen, 2003). Someone with intrinsic goal orientation strives to reach learning
goals for his or her own purposes, because the material is interesting in itself. Someone with an extrinsic goal
orientation strives to achieve better results than others (highest grades in the class), to achieve an extrinsic
reward (pay raise, grant) or to avoid punishment (for example repeating a course). Contextual motivation,
relating, for example, to the interest of the studied topic, varies dynamically. General level motivation is more
static, because it can change according to a stage of life. Attitude and motivation can be distinguished, because
attitude affects the quality of one’s work, while motivation affects one’s alertness and vigour (Ruohotie, 1996).
8 Valuation of Previous Knowledge
Learning material that presumes previous knowledge from the learner may expect the learner to already possess
some skills or knowledge that have been presented, for example, in some earlier learning materials. Learning
material that respects the learner’s previous knowledge takes into account individual differences in skills and
knowledge and encourages them to take advantage of it during studies. This approach favours learner’s
elaboration, contemplation or new issues and the analysis of their relationship with learner’s earlier knowledge
constructs (Wilson & Myers, 2000). Computer-assisted learning material can include various predefined “paths”
that demonstrate the use of the learning material depending on the previous knowledge of the learner. The
learning material may review the central concepts from earlier studies that are important for understanding the
present material. In this case, the importance of the learner’s previous knowledge is diminished, but the
importance of the previous material and the cumulative nature of knowledge become clear to the learner.
9 Flexibility
Flexible learning material takes into account learners’ individual differences. For example, a test given at the
beginning of the studies can provide information about previous knowledge, interest towards learning the topic,
and expectations of what the learner seeks to gain from the studies. Information gained in pre-testing can be used
185
to provide the learner with optional or alternate routes in the studies (Hannafin & Peck, 1988, 48; Wilson &
Myers, 2000). The learner should be given a chance to navigate freely through the learning material. Flexibility
in the contents of the learning material means that the material contains diverse assignments (Quinn, 1996). The
more adaptable and broadly defined the assignments are (content description with meta coding), the easier it is to
combine them to fit the student’s individual needs (Leflore, 2000). Collis and Moonen (2002, 10) address the
need for social organization of learning (face-to-face, group, individual), language to be used during the learning
situations, modality and origin of the learning resources (teacher, student, library, Internet) and instructional
organization of learning (assignments, monitoring). They further suggest that students should share the
responsibility of identifying appropriate additional learning resources and even contribute to the learning
resources (ibid, 13).
10 Feedback
The system or learning material should provide the student with encouraging (Albion, 1999; Quinn, 1996) and
immediate feedback. Encouraging feedback increases learning motivation; immediate feedback helps the student
to understand the problematic parts in his or her learning. Immediate feedback is particularly important in
behaviourist (stimulus – reaction) learning materials (Wilson & Myers, 2000). In interaction between a student
and a computer, the benefit is that feedback can be given immediately after the student’s action. On the other
hand, computer provided feedback is rarely so valuable and well timed that it can support learning by itself. If
the other side of the dialogue is a human (for example, a peer learner on a discussion forum for the learning
material), the feedback is more likely to support reflection, depending, of course, on the quality of interaction
inside the peer group (Kolodner & Guzdial, 2000).
In so-called faultless learning, the learning is based on the fact that mistakes are corrected immediately and one
cannot progress to the next part in the material before this is done. On the other hand, in a constructivist learning
situation, in which the problem is solved in cooperation, the learners may sometimes come up with a faulty
solution to an ill-structured problem (Jonassen, Peck & Wilson, 1999, 196). In this case, the teacher or mentor
can react to the mistake after a delay and give the students hints at the correct solution or links to additional
material.
Empirical Evaluation of the Criteria
A PMLQ (Pedagogically Meaningful Learning Questionnaire) research instrument was developed on the basis of
the aforementioned technical and pedagogical usability criteria. The first version of the instrument contained 92
multiple-choice items. The five-point scale ranged from 1 (totally disagree) to 5 (totally agree). The sixth
response option was “Not applicable”.
The PMLQ has three parts. The first part concerns technical and pedagogical usability of the learning platform
(or system) containing 43 items; the second is about technical usability of the learning material (24 items), and
the third part measures pedagogical usability of the learning material (25 items). The propositions are clearly
marked when measuring issues about system or contents.
Empirical measurements were carried out in October 2003 with 5th and 6th grade elementary school students (n
= 66) and their teachers (n = 4). Twenty-four of the children were boys and 42 were girls. Three teachers were
male and one was female. Participants evaluated the OPIT learning system and four learning modules embedded
into the system. Two of the modules were about mathematical topics (decimal numbers and fractions) and the
remaining two modules were about the third foreign language of Finnish school children, English (singular vs.
plural and knowledge test about British Isles).
The design of this empirical test contained three stages: First, each participant was profiled using a previously
developed ACALQ (Abilities for Computer Assisted Learning Questionnaire) instrument that characterizes
respondents by their self-rated motivational level, metacognitive preparedness and social abilities (Nokelainen &
Ruohotie, 2004). This information was intended for future purposes in order to control individual differences in
the evaluation of learning materials. Second, participants filled out a PMLQ questionnaire for the OPIT platform.
Shortly after using each of the four modules they filled out a similar usability questionnaire for each module.
Third, for each module, individual score and turnaround time was recorded.
186
Results of this first version of the PMLQ instrument showed that full scale of 1 to 5 was in use. Dependencies
between the variables were investigated with Bayesian dependency modeling (Myllymäki, Silander, Tirri &
Uronen, 2002; Nokelainen & Tirri, 2004) due to discrete measurement scale and small sample size. The results
supported the chosen dimensionality (Nokelainen, 2004b).
The next step in the development process was to revise the old items because interviews with the children and
teachers revealed some deficiencies in wording. The second version of the PMLQ’s pedagogical usability section
contained 56 items after five new items were included (see Appendix). Empirical measurements were carried out
in February 2004 with 5th and 6th grade elementary school students (n = 74) with age median of 12 years.
Participants evaluated the OPIT learning system and two learning modules embedded into the system. First
module was about decimal numbers, and the second one was about English language singulars and plurals. Both
modules were chosen by the teacher and research team. The structure of math module was non-linear, allowing
the learner to choose from several equal paths to complete the sequence of numbers tasks. The English module
represented typical fill-in test that proceeds in a linear fashion with minimal user control.
Analysis showed that although the distribution of most items was skewed, the full five-point Likert scale was in
use.
Table 4 contains the Cronbach’s alpha reliability scores for the first part of the study measuring usability of the
OPIT learning management system. Fifty-two out of 74 students evaluated the system with the PMLQ. Only a
few items measure the pedagogical usability of a learning management system, as our main focus of interest is in
the contents (i.e., learning material). As the alpha coefficient tends to grow as the number of items increases, it is
not surprising that the third dimension, Cooperative/Collaborative learning, has higher alpha value (.86) than the
fourth dimension, Goal orientation (.72).
Table 5 contains the reliability coefficients for the dimensions measuring the pedagogical usability of the two
learning modules. Thirty-four students used and evaluated the “Decimal numbers in a continuum” math learning
material. Reliability coefficients range from .75 (7. Motivation) to .87 (5. Applicability). Although high alpha
value indicates unidimensionality, the inner structures of the dimensions described in this paper are in most cases
quite complex, as presented earlier in tables 2 and 3. The second part of the Table 5 shows reliability coefficients
for the English “singular or plural” learning material. The alphas are satisfactory ranging from .80 (9. Flexibility)
to .92 (5. Applicability).
When we compare the pedagogical usability profiles of the two modules (presented in Table 5), we notice that
learner control, activity and flexibility dimensions have higher mean scores for the math module (M = 3.7, SD=
.94; M = 3.6, SD= .79; M = 3.4, SD= .66) than for the English module (M = 3.4, SD= .87; M = 3.3, SD= .94; M =
3.2, SD= .68). Although the mean differences are quite small, the measurement instrument is able to capture
differences in the pedagogical usability profiles. This leads us to cautiously conclude that the finding supports
our earlier hypotheses regarding the different roles of the two learning modules: The math module is designed to
allow a learner possibility to complete exercises of a different kind and to decide where and when to proceed, the
English module presents more inflexible and linear way of practising singulars and plurals with only one type of
exercise.
Description of the Utilization of the Criteria
A typical user of the criteria for pedagogical usability of digital learning material is a teacher, application
programmer, or student. The following describes the use of the criteria on a general level:
1. The user logs into the eValuator system. S/he submits background and profile information by completing
self-assessment questionnaire scales on, for example, motivation, learning strategies and social skills.
2. The user chooses to either search earlier evaluations or to assess new learning material.
3. The user provides criteria for the search or the evaluation (e.g., target group, topic, purpose of the material),
defines the target (e.g., LM, ULM or LMS) and, if needed, the usage context of the material (e.g.,
autonomous studying at school, autonomous studying at home, teacher-directed learning) and the type of
material (e.g., supporting/additional assignments, learning a new topic).
4. On the basis of the above criteria, the application generates an evaluation form, which contains statements
drawn from the criteria. Each target group has a different set of statements (e.g., children in the 5th or 6th
grade who evaluate learning material from the point of view of their own learning, teachers who read
evaluations from other teachers or evaluate material that they themselves use, adolescents in polytechnics or
187
5.
6.
7.
universities who choose material for their own use or evaluate material that they use, and adults in working
life).
The user provides his or her subjective evaluation of the usability of the material by responding to
statements.
At the end of search or evaluation, the user is presented with an overview of how the material corresponds
with the different parts of the technical and pedagogical usability criteria, as evaluated by the user.
The system serves also as a search tool. When the evaluation database contains more evaluations, user is
able to conduct personalised searches in order to find the most appropriate learning material. The personal
profile (collected with ACALQ and various background questions) and pedagogical usability scores
(collected with PMLQ) act as filters. For example, user is able to ask for 5th grade elementary school math
LM’s that have something to do with decimal numbers and are suitable for collaborative working. A second
example might be to ask for such university level statistical LM’s that are rated high by registered users
whose personal profile is close to the users’ profile.
Table 4. Reliability coefficients of the dimensions measuring usability of a learning management system
Sample 1a
Pedagogical Usability Dimensions
Items
M
SD α
1. Learner control
—
2. Learner activity
—
3. Cooperative/Collaborative learning
11, 12, 13, 14, 15
3.4 1.22 .86
4. Goal orientation
23, 24
3.7 1.11 .72
5. Applicability
—
6. Added value
35
3.6 1.36 —
7. Motivation
—
8. Valuation of previous knowledge
—
9. Flexibility
—
10. Feedback
52
3.7 1.30 —
a
Elementary school students, n = 52, age range 10 – 13 years, age median 12 years.
Note. See Appendix for the item labels.
Conclusion
A rapidly growing quantity of digital learning material is available to fill various learning needs via online (e.g.,
Internet, mobile networks) and offline (e.g., DVD, CD-ROM) media. A broad array of learning material is
offered to learners all over the world. However, whether the application selected achieves the desired ends
depend on various factors; for example, whether the learner needs to learn new things or just refresh old
knowledge, is working alone or collaboratively with other learners, or is willing to use commercial or noncommercial material.
This paper presents criteria for evaluating the pedagogical usability of digital learning material. In practice the
role of the criteria is to give the learner a chance to choose the most suitable learning material possible for any
learning situation. Previously constructed criteria for evaluating pedagogical usability have been introduced, and
differences between them have been considered. The various phases of creating criteria in this study have been
described and the empirical research setting has been evaluated. The 56 –item multiple-choice inventory
(PMLQ) that operationalises the pedagogical usability criteria was presented. Results of a two-stage empirical
evaluation of the criteria with 5th and 6th grade elementary school children (2003 n = 66, 2004 n = 74) was
reported. Results supported all theoretical dimensions of the pedagogical usability criteria. The PMLQ was able
to capture differences in the pedagogical usability profiles of the learning modules.
Table 5. Reliability coefficients of the dimensions measuring usability of two learning modules (math and
English)
Math
English
Sample 2a
Sample 3b
Pedagogical Usability Dimensions
Items
M
SD
M
SD
α
α
1. Learner control
1, 2, 47, 48, 49, 40
3.7 .94 .86
3.4 .87 .90
2. Learner activity
3, 4, 5, 6, 7, 8, 9, 10
3.6 .79 .86
3.3 .94 .91
3. Cooperative/Collaborative learning 11, 12, 13, 14, 15
3.4 .90 .76
3.4 1.05 .88
188
4. Goal orientation
5. Applicability
20, 21, 22, 23, 24
3.5 .91 .78
3.3 .97 .85
25, 26, 27, 28r, 29, 30, 31, 54,
3.5 .75 .87
3.4 .85 .92
55, 56, 40, 48, 49
6. Added value
32, 33, 34, 35, 45
3.1 1.05 .82
3.2 1.10 .88
7. Motivation
13, 36, 37, 38
3.6 .90 .75
3.4 1.03 .83
39, 40, 41
3.4 .96 .79
3.3 .97 .85
9. Flexibility
42, 44, 46
3.4 .66 .77
3.2 .68 .80
10. Feedback
50, 51, 52, 53
3.7 .76 .79
3.6 .97 .89
a
b
Note. r = A reversed item, i.e. values in 1 to 5 Likert scale change as follows: 1 -> 5, 2 -> 4, etc. See Appendix
for the item labels.
Discussion
The first drawback of the pedagogical usability criteria presented here is their generalisability to other domains.
In this study, generalisability is limited by various factors, for example, the small sample size, the narrow age
range of respondents’ and the limited number of learning material evaluated. To address these issues, we are
currently conducting empirical studies that aim at evaluating more representative set of learning material in
different domains targeted for both adolescent and adult learners. The criteria of pedagogical usability will in the
future be complemented by a segment evaluating “mobile usability” (Syvänen, Nokelainen, Ahonen & Turunen,
2003).
The second drawback, although in a more debatable sense, is the methodology we have applied to measure
usability, that is, a self-report questionnaire aiming to measure subjective end-user satisfaction. According to
researchers specialising in the measurement of usability (Rubin, 1994; Nielsen, 1993; Kirakowski, 2003), the use
of questionnaires is normally justified in usability research as part of a test arrangement where a user answers a
pre-test questionnaire, uses the application that is to be evaluated and then answers a post-test questionnaire
evaluating the usability. However, Kirakowski (2003) sees self-evaluation questionnaires as particularly
appropriate for measuring subjective matters (e.g., the enjoyability of use) in usability research. We all agree,
that various other test arrangements (Rubin, 1994; Nielsen, 1993, 192-200) are better for gathering factual data
(e.g., time spent in doing a particular task). On the other hand, the criteria developed here can also be used as the
basis for heuristic evaluation (Nielsen & Molich, 1990).
The third drawback is that the criteria do not directly address the issue of cultural sensitivity (Reeves, 1994).
Authority (e.g., who is ‘permitted’ to ask questions or ‘obligated’ to lead group work on the basis of his/her
social status) and collaboration (e.g., are questions regarding the peer-learners presentation or evaluation of other
learners’ work allowed) are examples of cultural issues that are strongly connected to pedagogics. The problem
of describing cultural sensitivity via pedagogical criteria is two-fold. First, it is to some extent connected with
the metadata (the definition of creator, ideal target group, religion, language and country) and technical usability
(graphical user interface colours and symbols) issues. Second, cultural sensitivity is very strongly latent variable
by nature, and hence it is not trivially operationalised into questions that end-user is able to answer.
Finally, if we try to verbalise what the pedagogical usability criteria aim to address, we may ask: “Does the
system, and/or learning material it contains, make it possible for the student and the teacher to achieve their
goals?” The important thing to acknowledge is that the dimensions of technical and pedagogical usability
(operationalised dimensions) might correlate and thus describe (i.e., in factor analysis language “load on”) both
usability and utility (latent dimensions). From the perspective of the user/learner, the number of things to be
memorised and the feedback required are examples of these kinds of common components between the user and
the system, and between the user and the contents of the learning material. From the viewpoint of usability, the
system works well in user terms when the user does not have to resort to any help and is able to control the
program, and not the other way around. The number of things to be memorised is, in terms of utility, related to
the functioning of the learning material because this will be minimized if the studied subject matter is presented
in manageable sections that make sense to the users. There may be a certain degree of tension between some of
the criteria. For example, usability is enhanced if the number of things to be remembered and errors are kept to a
minimum, but for the utility of the learning material it is desirable for the user to remember the core set of
functions so that they can be used readily without prompting. Another example is authenticity and usability.
Using real representations from the outside world in the user interface may not be as efficient as a metaphoric
189
design. In the design of learning materials, the aim is often to correspond closely with the real world so that it
will be easier for users to link new knowledge to earlier knowledge structures. On the other hand, learning
materials intended to inculcate a certain skill by repetition can be weak in their real-world correspondence but
still produce good learning results, particularly if the skill being learned is not derived directly from the real
world, either.
The criteria described here are not the first attempt to provide an analysis tool for the evaluation of the usability
of digital learning material; both its structure and content are to great extend based on previous research.
However, the criteria and research that led to its creation, have the following original features that bring added
value for research on the pedagogical usability of digital learning material: 1) a multidisciplinary approach
involving the fields of both computer technology and education, 2) a review of previous research, 3) the
development of the criteria on the basis of systematic empirical research, performed with the help of real users;
and 4) a multiple choice inventory that was developed on the basis of the criteria and can be used independently
or as part of the eValuator computer application. A preliminary version of the eValuator online software is
available for educational and research purposes at http://evaluator.hamk.fi.
Acknowledgements
This study is part of Digital Learning research project (http://dll.hamk.fi/dl2/en/) financed by National
Technology Agency in Finland (http://www.tekes.fi/eng/), Research Centre for Vocational Education, University
of Tampere, Finland (http://rcve.uta.fi/), and Hame Polytechnic, Finland (http://www.hamk.fi/in-english/).
Author wishes to thank Mikko Horila, Miikka Miettinen, Dr. Jorma Saarinen, Antti Syvänen, Tuomo Tammi and
Jan Överlund for their contribution to this project. Dr. Bruce Beairsto’s comments helped to improve the quality
of this manuscript.
References
Albion, P. (1999). Heuristic evaluation of educational multimedia: From theory to practice. Paper presented at
the 16th Annual conference of the Australasian Society for Computers in Learning in Tertiary Education,
retrieved November 24, 2005 from http://www.usq.edu.au/users/albion/papers/ascilite99.html.
Barab, S. A., & Duffy, T. M. (2000). From Practice Fields to Communities of Practice. In Jonassen, D. H. &
Land, S. M. (Eds.), Theoretical Foundations of Learning Environments, Mahwah, NJ, USA: Lawrence Erlbaum
Associates, 25-56.
Chalmers, P. (2003). The role of cognitive theory in human -computer interface. Computers in Human Behavior,
19, 593-607.
Chin, J., Diehl, V., & Norman, K. (1988). Development of an Instrument Measuring User Satisfaction of the
Human-Computer Interface. Proceedings of ACM CHI'88 Conference on Human Factors in Computing Systems,
New York: ACM Press, 213-218.
Collis, B., & Moonen, J. (2002). Flexible Learning in a Digital World. Experiences and expectations, London:
Kogan Page.
Forman, E. A., & Cazden, C. B. (1985). Exploring Vygotskian perspectives in education: The cognitive value of
peer interaction. In Wertsch, J. (Ed.), Culture, communication, and cognition: Vygotskian perspectives,
Cambridge: Cambridge University Press, 323-347.
Hannafin, M., & Peck, K. (1988). The Design, Development and Evaluation of Instructional Software, New
York: Macmillan.
Horila, M., Nokelainen, P., Syvänen, A., & Överlund, J. (2002). Criteria for the pedagogical usability, version
1.0, Hämeenlinna, Finland: Häme Polytechnic and University of Tampere.
Jansen, W., van den Hooven, H., Jägers, H., & Steenbakkers, G. (2002). The Added Value of E-learning. Paper
presented at the Informing Science & IT Education, June 19-21, 2002, Cork, Ireland.
190
Jonassen, D. H. (1995). Supporting Communities of Learners with Technology: a Vision for Integrating
Technology with Learning in Schools. Educational Technology, 35 (4), 60-63.
Jonassen, D. H., Myers, J., & McKillop, A. (1996). From Constructivism to Constructionism: Learning with
Hypermedia/Multimedia Rather Than from It. In Wilson, B. (Ed.), Constructivist Learning Environments,
Englewood Cliffs, NJ, USA: Educational Technology Publishers, 93-106.
Jonassen, D. H., Peck, K., & Wilson, B. (1999). Learning with Technology. A Constructivist Perspective, Upper
Saddle River, NJ, USA: Merrill.
Keinonen, T. (1998). One-dimensional usability – Influence of usability on consumers’ product preference,
Saarijärvi, Finland: Gummerus.
Kirakowski, J. (2003). Questionnaires in Usability Engineering, retrieved November 24, 2005 from:
http://www.ucc.ie/hfrg/resources/qfaq1.html.
Kolodner, J., & Guzdial, M. (2000). Theory and Practice of Case-Based Learning Aids. In Jonassen, D. H. &
Land, S. (Eds.), Theoretical Foundations of Learning Environments, Mahwah, NJ, USA: Lawrence Erlbaum,
215–242.
Kurhila, J., Miettinen, M., Nokelainen, P., & Tirri, H. (2002). Use of Social Navigation Features in Collaborative
E-Learning. Paper presented at the E-Learn 2002 Conference, retrieved May 1, 2006 from http://cosco.hiit.fi/
edutech/publications/elearn2002.pdf.
Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation, New York, USA:
Cambridge University Press.
Leflore, D. (2000). Theory Supporting Design Guidelines for Web-Based Instruction. In Abbey, B. (Ed.),
Instructional and Cognitive Impacts of Web-Based Education, Hershey: Idea Group Publishing, 102-117.
Lin, H., Choong, Y., & Salvendy, G. (1997). Proposed Index of Usability: A Method for Comparing the Relative
Usability of Different Software Systems. Behaviour and Information Technology, 16 (4/5), 267-278.
Littleton, K., & Häkkinen, P. (1999). Learning Together: Understanding the Processes of Computer-Based
Collaborative Learning. In Dillenbourg, P. (Ed.), Collaborative Learning. Cognitive and Computational
Approaches, Amsterdam: Pergamon, 20-30.
Livingstone, D. W. (2000). Exploring the Icebergs of Adult Learning: Findings of the First Canadian Survey of
Informal Learning Practices, NALL Working Paper #10-2000, Ontario, Canada: Ontario Institute for Studies in
Education, University of Toronto.
Mayes, J., & Fowler, C. (1999). Learning technology and usability: a framework for understanding courseware.
Interacting with Computers, 11, 485-497.
Miller, G. (1956). The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for
Processing Information. The Psychological Review, 63, 81-97, retrieved November 24, 2005 from:
http://www.well.com/user/smalin/miller.html.
Myllymäki, P., Silander, T., Tirri, H., & Uronen, P. (2002). B-Course: A Web-Based Tool for Bayesian and
Causal Data Analysis. International Journal on Artificial Intelligence Tools, 11 (3), 369-387.
Nielsen, J. (1990). Evaluating Hypertext Usability. In Jonassen, D. H. & Mandl, H. (Eds.), Designing
Hypermedia for Learning, Berlin: Springer-Verlag, 147-168.
Nielsen, J. (1993). Usability Engineering, Boston: Academic press.
Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. Proceedings of the ACM CHI'90
Conference, New York, USA: ACM Press, 249-256.
191
Nokelainen, P. (2004a). Designing an Evaluation Tool for Digital Learning Materials. Paper presented at the
Society for Information Technology and Teacher Education International Conference, March 1-6, 2004, Atlanta,
USA.
Nokelainen, P. (2004b). Conceptual Definition of the Technical and Pedagogical Usability Criteria for Digital
Learning Material. Paper presented at the EdMedia 2004 Conference, June 21-26, 2004, Lugano, Switzerland.
Nokelainen, P., & Ruohotie, P. (2004). Empirical Validation of Abilities for Computer Assisted Learning
Questionnaire. In Marsh, H. W., Baumert, J., Richards, G. E. & Trautwein, U. (Eds.), Proceedings of the 3rd
International Self-Concept Research Conference, Sydney: University of Western Sydney, Self Research Centre,
676-689.
Nokelainen, P., & Tirri, H. (2004). Bayesian Methods that Optimize Cross-cultural Data Analysis. In Campbell,
J. R., Tirri, K., Ruohotie, P. & Walberg, H. (Eds.) Cross-cultural Research: Basic Issues, Dilemmas, and
Strategies, Hämeenlinna, Finland: RCVE, 141-158.
Palloff, R. M., & Pratt, K. (2005). Collaborating Online. Learning Together in Community, San Francisco, CA,
USA: Jossey-Bass.
Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction Design: Beyond Human-Computer Interaction, New
York, NY, USA: John Wiley & Sons.
Quinn, C. (1996). Pragmatic Evaluation: Lessons from Usability, retrieved November 24, 2005 from:
http://www.ascilite.org.au/conferences/adelaide96/papers/18.html
Reeves, T. C. (1994). Evaluating what really matters in computer-based education. In Wild, M. & Kirkpatrick,
D. (Eds.) Computer education: New Perspectives, Perth, Australia: MASTEC, 219-246.
Rubin, J. (1994). Handbook of Usability Testing: How to plan, design, and conduct effective tests, New York,
USA: John Wiley & Sons.
Ruohotie, P. (1996). Professional Growth and Development. In Leithwood, K. (Ed.), International Handbook of
Educational Leadership and Administration, Dordrecht: Kluwer Academic Publishers, 419-445.
Ruohotie, P., & Nokelainen, P. (2003). Practical Considerations of Motivation and Computer-supported
Collaborative Learning. In Varis, T., Utsumi, T. & Klemm, W. R. (Eds.), Global Peace Through The Global
University System, Hämeenlinna, Finland: RCVE, 226-236.
Savery, J. R., & Duffy, T. M. (1995). Problem based learning: An instructional model and its constructivist
framework. Educational Technology, 35 (5), 31-7.
Shneiderman, B. (1998). Designing the User Interface: Strategies for Effective Human-Computer Interaction (3rd
Ed.), Menlo Park, CA, USA. Addison Wesley.
Squires D., & Preece, J. (1996). Usability and Learning: Evaluating the Potential of Educational Software.
Computers & Education, 27 (1), 15-22.
Squires, D., & Preece, J. (1999). Predicting quality in educational software: Evaluating for learning, usability
and the synergy between them. Interacting with Computers, 11, 467-483.
Syvänen, A., Nokelainen, P., Ahonen, M., & Turunen, H. (2003). Approaches to Assessing Mobile Learning
Components. Paper presented at 10th Biennial Conference of the European Association for Research on
Learning and Instruction, August 26-30, 2003, Padova, Italy.
Tognazzini,
B.
(2003).
First
principles,
http://www.asktog.com/basics/firstPrinciples.html.
retrieved
November
24,
2005
from:
192
Tudge, J., & Rogoff, B. (1989). Peer influences on cognitive development: Piagetian and Vygotskian
perspectives. In Bornstein, M. H. & Bruner, J. H. (Eds.), Interaction in human development, Hillsdale, NJ, USA:
Lawrence Erlbaum, 17-40.
Vygotsky, L. S. (1978). Mind in Society. The Development of Higher Psychological Processes, London,
England: Harvard University Press.
Wilson, B., & Myers, K. (2000). Situated Cognition in Theoretical and Practical Context. In Jonassen, D. H. &
Land, S. (Eds.), Theoretical Foundations of Learning Environments, Mahwah, NJ, USA: Lawrence Erlbaum
Associates, 57-88.
193
Appendix 1
The pedagogical usability items for 4 to 6th grade elementary school students
1. When I worked on this assignment, I felt that I, not the program, held the responsibility for my own
learning. (Definition: I’m not repeating same kind of steps all the time in my studying, but the assignments
make me think and make a different solution for each one.)
2. When I used this learning material, I felt that I controlled what it did and not vice versa. (Definition: The
program does not lead me from one step to another, but I can choose by myself in which order I want to
finish my tasks.)
3. I have to think and make my own solutions to learn this learning material. (Definition: I have to concentrate
on this material, I cannot complete the tasks simply with role learning.)
4. This learning material has been divided into sections, my task is to learn them in pre-defined order (and
possibly respond to assignments).
5. This learning material provides learning problems without a pre-defined model for the solution.
6. This material does not have material in itself, but links to various other sources, which I have to use to learn.
(Definition: If the topic is ”a dumb yard”, there is no ready-made presentation of the topic. The learning
starts, for example, with a short tip by the teacher, which tells what kinds of things are needed in order to
build a dumb yard. Some of the information that you need may be in the system, but you have to find most
of it from newspapers, books or the Internet in order to make your own presentation.)
7. I get so deep into this learning material that I forget all about what is happening around me and how much
time I spend on it.
8. When I work with this learning material, I feel that I know more about some topics than others, I’m ”an
expert”. (Definition: The learning material may involve an individual information gathering task, for
example, an interview of neighbours or measuring the depth of packed snow in one’s home garden over the
period of one month.)
9. When I work in this learning material, I (or us, if a group work) have to find out own solutions without the
teacher’s or the program’s model solutions.
10. I am proud of my own solution, or one that I made with others, to the problem presented in the learning
material. (Definition: I feel that I, or we together, have made something that is significant.)
11. This learning material lets me talk with my classmates. (Definition: For example, messages in chat or notice
board.)
12. I can do group work with my classmates in this learning material. (Definition: If I wanted, I could do
assignments together with my classmate so that we both used our own computers.)
13. It is pleasant to use the learning material with another student on the same computer.
14. This learning material lets me know what other users have been doing in the system. (Definition: For
example, which learning materials have been read the most or assignment that have been the most popular.)
15. This learning material lets me know what other users are doing when I’m using the system. (Definition: For
example, the most read material at the moment or the assignment with which most people are working on.)
16. This learning material offers me simple utility programs (for example, a calculator).
17. This learning material offers me versatile utility programs (for example, Excel sheets, a HTML editor, text
processor, etc.).
18. In this learning material the utility programs have a central role. (Definition: I have to, for example, edit an
Excel sheet to solve a problem.)
19. I can save my work on this learning material and use or evaluate others’ work. (Definition: I can, for
example, explore or evaluate other groups’ group works and use them in my own studies.)
20. This learning material tells me clearly what I’m expected to know (or learn) after I’ve used it. (Definition:
The learning goals are clearly set, for example, ”After this assignment, you will know how to divide with
decimal fraction” or ”After this assignment you can form interrogative clauses in English”.)
21. This learning material tells me clearly why it is useful for me to learn this material. (Definition: The learning
goals are justified, for example, ”This assignment will help you to make interrogative phrases in English.”)
22. The learning material assesses my achievements with scores. (Definition: For example, the system gives a
score at the and of an assignment and shows the maximum score.)
23. This learning material tells me how much progress I have made in my studies. (Definition: I know what I
have practiced or learned thus far.)
24. This learning material is strictly limited. (Definition: For example, the topic of a math learning material is
“Calculating the mean”.)
25. This learning material teaches me skills that I will need. (Definition: I will be able to, for example, convert
euros into crowns or marks, or help my parents to choose between different-sized packages according to
their prize difference.)
194
26. I feel that I will be able to use the skills and knowledge this learning material has taught me in the future.
27. This learning material is based on the idea that ”one learns the best by doing stuff by oneself”. (Definition:
The material offers more assignments than, for example, PowerPoint presentations.)
28. I feel that this learning material will help me to do better in the test. (Definition: I think the assignments in
the material are similar to the assignment that we usually have in tests.)
29. This learning material is suitably challenging to me. (Definition: The assignments are not too easy or too
hard.)
30. I feel that this learning material has been designed for me. (Definition: The material suits your own needs,
and it does not feel that you are considered too smart or too dumb.)
31. This learning material adjusts the difficulty to suit my skills. (Definition: I can practice something that is
hard for me until I have learned it and before I move on to the next topic.)
32. The images in this learning material help me to learn.
33. The sounds in this learning material help me to learn.
34. The animations in this learning material help me to learn.
35. It is more useful to me to learn topics with this learning material than with conventional methods in a
classroom. (Definition: Think if you would be more willing to do this assignment with a computer or with a
normal study book or exercise book.)
36. I try to achieve as high a score as I can in this learning material.
37. I want to learn the topics of this learning material as deeply as I can.
38. I’m interested in the topic of this learning material.
39. This learning material required me to know something that had been taught in some other learning material.
(Definition: This material made a reference to some other learning material.)
40. I can use my earlier knowledge when I study with this material.
41. This learning material goes over earlier material before starting to teach a new topic. (Definition: For
example, in mathematics, the material first goes over simpler calculations that are needed to learn a more
difficult topic.)
42. This learning material offers optional routes for my progress. (Definition: I can choose different assignments
each time I use the system.)
43. This learning material does not let me proceed to the next point before I have answered correctly to every
question. (Definition: For example, in an English language assignment one has to answer correctly to every
question, even with the help of the program, before it lets you proceed to the next topic.)
44. This learning material has many similar, consecutive assignments. (Definition: For example, an English fillin assignment that has many consecutive assignments for am, are, and is sentences.)
45. This learning material makes it quick and easy for me to learn a new topic or recap an earlier topic.
46. If I cannot remember a particular word or concept while using this learning material, I can go back and
check its meaning in previous material.
47. When I used this learning material, I felt that I had to remember too many things at the same time.
(Definition: I felt at some points that I should have used a paper to write some things down.)
48. This learning material presents information in a format that makes it easy to learn. (Definition: Information
is presented in meaningful, interconnected entities, and not in separate pieces that are hard to understand.)
49. This learning material presents new material (or recaps old) in ”portions” suitable for me. (Definition: There
are not too many new things presented at once, I have time to learn them before moving onto the next topic.)
50. I can make a certain number of mistakes with this material (for example, wrong answers to calculus tasks),
after which the program shows me the correct answer.
51. When I make a wrong solution in an assignment, the program gives me a friendly note.
52. This learning material gives me motivating feedback. (Definition: I am willing to try out the less used
functions in the learning material, because I know that the system will give me all the advice that I need.)
53. This learning material provides me with immediate feedback of my activities. (Definition: When I write my
response to a calculus task, the system shows me immediately whether the answer is correct or not.)
54. This learning material gives first an example of the correct solution. (Definition: Multiplying with decimal
fractions is started with a model performance, after which I will calculate on my own.)
55. In this learning material, I get to carry the responsibility for the solution of an assignment in small portions.
(Definition: For example, in a math task, I will be first shown the task and then the result. Next, I see the
task but not the result, which I have to solve on my own.)
56. I think I learn more quickly with this material than normally. (Definition: This learning material provides
me with the right kind of support when I need it.)
195
Appendix 2
The pedagogical usability subdimensions
Note. Related items are presented in parenthesis (see Appendix 1). r = rewersed, i.e., negatively worded, item.
1. Learner control
1.1 Minimum memory load (47)
1.2 Meaningful encoding (48,49)
1.3 Take responsibility for one’s own learning (1)
1.4 User control (2)
8.2 Elaboration (40)
2. Learner activity
2.1 Reflective thinking (3, 4r)
2.2 Problem-based learning (5, 9)
2.3 Use of primary data sources (6)
2.4 Immersion (7)
2.5 Ownership (8, 10)
2.6 Primary data source (for PBL) (Items for teacher only)
2.7 Facilitative teacher (Items for teacher only)
2.8 Didactic teacher (Items for teacher only)
2.9 Individual/distance learning (Items for teacher only)
3. Cooperative/Collaborative learning
3.1 Support for conversation and dialog (11)
3.2 Group work (12,13)
3.3 Asynchronous social navigation (14)
3.4 Synchronous social navigation (15)
3.5 Asynchronous social navigation monitoring (Items for teacher only)
3.6 Synchronous social navigation monitoring (Items for teacher only)
3.7 Tertiary courseware (19)
4. Goal orientation
4.1 Explicit goals (20)
4.2 Usefulness of goals (21)
4.3 Focus on results (22)
4.4 Focused goals (34)
4.5 Monitor one’s own studies (pedagogic feedback) (23)
4.6 Set one’s own goals (Items for teacher only)
5. Applicability
5.1 Authentic material (25,28r)
5.2 Perceived usefulness (26)
5.3 Learning by doing (27)
5.4 Adequate material for the learners needs (human development) (29)
5.5 Pretesting and diagnostics (30,31)
5.6 Prompting (54)
5.7 Fading (55)
5.8 Scaffolding (56)
1.2 Meaningful encoding (40,48,49)
6. Added value
6.1 Overall added value for learning (35)
6.2 Effectiveness for learning (45)
6.3 Added value of pictures (32)
6.4 Added value of sounds (33)
6.5 Added value of animations (34)
196
7. Motivation
7.1 Intrinsic goal orientation (37)
7.2 Extrinsic goal orientation (36)
7.3 Meaningfulness of studies (38)
2.4 Immersion (7)
8.1 Prerequisites (39)
8.2 Elaboration (40)
8.3 Examples (41)
9. Flexibility
9.1 Pretesting and diagnostics (42)
9.2 Task decomposition (43r,46)
9.3 Repetitive tasks (44)
10. Feedback
10.1 Encouraging feedback (52,51)
10.2 Accurate feedback (53)
10.3 Errorless learning (50)
197
Zualkernan, I. A. (2006). A framework and a methodology for developing authentic constructivist e-Learning
environments. Educational Technology & Society, 9 (2), 198-212.
A framework and a methodology for developing authentic constructivist eLearning environments
Imran A. Zualkernan
School of Engineering, American University of Sharjah, P.O. Box 26666, Sharjah, UAE
Tel: +971-06-515-2953
[email protected]
ABSTRACT
Semantically rich domains require operative knowledge to solve complex problems in real-world settings.
These domains provide an ideal environment for developing authentic constructivist e-learning
environments. In this paper we present a framework and a methodology for developing authentic learning
environments for such domains. The framework is based on an ecological view and characterizes
dimensions of a typical constructivist environment in terms of pedagogical design, architecture, the
environmental context and what is actually learned. A case-study illustrating the use of the framework to
develop a just-in-time game-based learning environment is also presented.
Keywords
Computer assisted instruction, Expert problem solving, Operative knowledge, Development methodology
Introduction
This paper focuses on learning that occurs in professional problem solving domains that require a very high level
of skill. In some sense, these domains are characterized by knowledge that is “operative” as the professionals in
these domains are required to do work in real settings. This paper presents a framework that serves as the
foundation for conceptualizing the development of authentic constructivist environments in such domains.
The constructivist view of learning has its foundations in Piaget (1975) who believed that learning is not
transmitted passively, but attained through well-defined stages by active participation of a learner. Vygotsky
(1980) presented similar ideas but focused on the importance of socio-cultural activity in learning in addition to
introducing flexible stages of development. More recently, the importance of context and “authenticity” in
learning has been emphasized by Brown, Collins and Duguid (1989). According to them, “authentic activity is
the ordinary practices of cultures.” (p. 36). Lave and Wenger (1991) further extend this view in their influential
work on situated learning to point out that, “…Learning occurs through centripetal participation in the learning
curriculum of the ambient community” (p. 100). Where the learning curriculum consists of “…situated
opportunities (thus, including exemplars of various sorts often thought of as “goals”)” (p. 97).
For the purpose of this paper, constructivist authentic learning environments are defined as those learning
environments whose design is consistent with the principles of the more recent constructivist tradition on how
people learn as exemplified by the works of Lave and Wenger (1991) and Brown et al. (1989). As Herrington
and Oliver (2000) point out, such learning environments typically provide authentic contexts and activities,
access to expert performances, and support multiple roles and perspectives. In addition, such environments also
support collaborative construction of knowledge and promote reflection and articulation. Finally, such
environments may include coaching and scaffolding by the teacher and provide for authentic assessment of
learning within tasks.
Constructivist learning environments are ideally suited to teaching professional problem solving. However, the
complexity and ill-structure of such problem solving activity raises particular issues when it comes to actually
building a constructivist e-learning environment. For example, what one means by an “authentic context” or an
“authentic activity” in a particular problem solving context is often unclear. Prescriptive guidelines (Herrington
& Oliver, 2000) like using “activities which have real-world relevance” or “ill-defined activities” or “a single
complex” task to define an “authentic activity” attack such difficult questions at a syntactic level and as such
provide only a good starting point. Constructing authentic learning environments in professional problem
solving domains requires an analysis at the semantic level based on a deeper understanding of what constitutes
learning in these environments.
This paper presents a framework and a methodology specifically developed for such complex domains.
198
Framework
The key construct of this framework is the concept of a “learning curriculum.” A learning curriculum consists of
situated opportunities (Lave & Wenger, 1991). The principles emerging from this framework will be later used
to prescribe the general parameters that govern the construction of authentic e-learning environments. In
addition, a methodology based on these principles will also be presented.
The primary components of the framework used to describe a learning curriculum are derived from (Johnson,
Kochevar & Zualkernan, 1992a) as shown in Figure 1. Briefly, the “Physical Environment” is a description of
the objectively observable characteristics (e.g., a disease or defect). The available information part of the
physical environment may also consist of artifacts such as books, manuals, databases that exist as well as
interaction with peers, experts and teachers. Specific characteristics of the environment require specific actions
by the learner (e.g., diagnosis or repair). A task can only be performed by a learner because the information in
the task is lawfully related to some physical occurrence (Turvey, Carello & Kim, 1981).
The “Task Environment” is the subset of the physical environment that is relevant to a class of agents (e.g.,
surgeons). The adaptation is the primary construct in this framework and represents what is “learned” under the
constraints of the task environment and the constraints from the learner. The constraints on the learner may
contain cognitive constraints (e.g., short-term memory, processing capabilities (Anderson, 1990)) and learning
styles (e.g., holistic, analytical, field independent vs. field dependent (Rumetshofer & Woss, 2003)) or based on
theory of multiple intelligences (see Gardiner (1993), for example) on one side and goals and motivation on the
other.
Figure 1. A framework for developing authentic constructivist learning environments
The first key construct in the framework is the “adaptation” itself. Adaptation is a construct that develops under
the constraints of the task environment and the learner (this is similar to Simon’s notion of adaptation as the
interface between the outer and inner environment (Simon, 1996)).
Adaptation, however, does not exist inside the “head” or “mind” of the learner. It is a construct that represents
what evolves as a set of routines (including asking for, and retrieving information, for example) or dynamics that
allow the learner to be “fit” for the particular task environment.
The second key construct of the framework is the concept of “fit” (this is not the same as the evolutionary
biology’s notion of a fit, but it describes a psychological fit). Loosely, the fit describes how well a learner is
adapted to the task environment (e.g., how good is the surgeon?). Fit can be classified into two dimensions;
semantic and structural.
The semantic dimension is a measure of how well the learner’s actions are acceptable in the particular
environment (e.g., how well are surgeon’s operations received in the physical world? Or, how many patients
actually die under her care?). Hence the semantic fit is primarily related to how well the goals and the intentions
of the learner are realized in the actions he/she takes in the physical environment.
199
The structural dimension of fit describes how closely the cognitive constraints and learning styles of the learner
“match” to the information present in the environment (e.g., does the surgeon accept a particular type of surgery
as suited to her/his skills?). Manifestation of failure of structural fit occurs when, for example, an individual
refuses to accept the information provided in the environment as “valid” for their task environment.
Authentic Learning Environments
Authentic learning environments in the constructivist tradition are situations that allow a learner to create their
own personal knowledge in a particular task environment. In simplified manifestation, an authentic learning
environment is a surrogate to the actual problem-solving environment (e.g., surgery room as opposed to the
surgery simulator).
An authentic learning environment can more generally be described as a manifestation of a “learning
curriculum” as described by Lave and Wenger (1991). In terms of the framework, the learning curriculum, then,
is simply a set of situated opportunities that allow the adaptation to eventually attain a high degree of fit between
the task environment and the learner.
The design of a good authentic learning environment, therefore, consists of creation of an appropriate set of
situated opportunities. Each situated opportunity is described by 4-tuple (I, A, C, G) where
I: Information in the environment
A: Successful actions in the environment
C: Cognitive constraints and learning styles of the learner
G: Goals and intentions of the learner
A successful authentic environment has to create enough (and the right) situated opportunities to ensure that
adaptation that arises for a specific learner has both a high structural as well as a semantic fit.
A fundamental problem that arises with using authentic environments is their validity. How does, for example,
one ensure that adaptation thus evolved within the authentic environment will in fact transfer to the real
environment?
The key problem that needs to be solved in developing an authentic constructivist environment, therefore, is to
ensure that adaptation that emerges in response to the “fit” requirements (both semantic and structural) of the
constructed authentic environment, also establishes a high degree of fit in the real environment. This is exactly
what the framework and the resulting methodology helps achieve in a specific situation.
Methodology
This section presents the various dimensions of a typical authentic learning environment and shows how the
framework presented earlier provides conceptual footing for a development methodology based on a)
pedagogical design, b) architectures, c) the environmental context and d) what actually gets learned in such
environments. These four dimensions culminate in a four step methodology where each step is tied to exploring
one dimension.
Step 1: Complete pedagogical design
The primary objective of this step is to determine what constitutes a “situated opportunity” (i.e., I, A, C and G
according to the framework) within a particular pedagogical design paradigm. A pedagogical design imposes
broad constraints on what a situated opportunity can be. In doing so, it outlines the possible space of situated
opportunities for the learning environment. In addition, based on the properties of the semantic and structural fit
in the real environment, pedagogical design also determines constraints on the four components of a typical
situated opportunity.
Some commonly used pedagogical design paradigms for “authentic” learning environments in the constructivist
tradition are given below (Ip & Naidu, 2002; Oliver, 2001)
200
¾
¾
¾
¾
¾
¾
¾
Problem-based Learning (PBL) - In PBL, a convincing scenario problem (scenario) is created where
learners are supported by stories as told by various actors. The primary premise behind these environments
is to allow the learners to fail in a “safe” context and to receive feedback as a third-person.
Distributed Problem-Based Learning (DPBS) brings the additional element of a group of individuals
using the network as a medium to work on and solve a common problem.
Inquiry-based learning (IBL) is one variant of PBL that poses ill-structured tasks to the students.
Role-Play Simulation and Game-based Learning (RSL) creates situations where learners take on the
role-profiles of various characters in contrived educational games.
Case Studies-based Learning (CSBL) uses actual events to force students to “practice” on actual data in a
safe environment.
Critical Incidence-based Learning (CIBL) occurs when learners engage in reflections on critical events
from their workplace.
Project-based Learning (PRBL) engages students in designing and creating products that meet authentic
needs.
No matter what the manifestation of the pedagogical design of an authentic environment, each has to pay
particular attention to how and why the situated opportunities thus created are authentic. For example, in PBL, a
problem consists of the information presented in the environment (I) and the feedback provided by the stories
guides the learner on what is successful action (A) through failure. The problems have to be consistent with the
goals of the learner (G) as well as the cognitive constraints (C). The DBPS simply adds additional sources of
information (I) that learner can access.
Critical Incidence-based Learning is particularly interesting in this context in that it is related to low-base rate
tasks (Johnson, Grazioli, Jamal & Zualkernan, 1992b). That is, environments where the incidence of situated
opportunities is very rare (e.g., earth quakes). In this case, the problem to be solved becomes mostly the
generation of an appropriate number of situated opportunities, so that adaptation can attain a high degree of fit.
Similarly, in CSBL, the emphasis is not so much on creating the right information (I). The appropriateness of
actions (A) is also not a primary issue. The emphasis really has to be on how well the fit can occur between the
learner’s cognitive constraints and learning styles as well as the goals.
Step 2: Construct architecture for the authentic environment
The primary objective of this step is to specify an architecture that provides appropriate support for the situated
opportunities outlined in the previous step.
The architecture of an authentic environment specifies the various components that must exist in a learning
environment or a computer manifestation of it. A general characterization of the constructivist learning
environments has been provided by Jonassen, Peck and Wilson (1998).
The components needed for such environments are:
¾
¾
¾
¾
¾
¾
Problem/project space. The learners are presented with an interesting, relevant and engaging problem. This
is simply the creation of a single situated opportunity.
Related Cases. When expecting learners to solve problems, they must be provided with a set of related
experiences on which the learner’s can draw. These represent a set of situated opportunities similar to the
one being presented.
Information Sources. Providing learners with information they need help with in a timely manner. This
simply stresses the information component (I) of the situated opportunity.
Cognitive (knowledge) construction tools. Tools that support the learner’s abilities to solve the tasks at
hand. These are a part of the physical environment (e.g., a paper and a pencil, a calculator, a utility program)
if the fit determines the learner’s cognitive constraints (C) need to be augmented to achieve an appropriate
fit.
Conversation (knowledge-negotiation) tools. Tools to support collaboration. These are a part of the
physical environment, if accessing information (I) or successful action (A) requires external conversations.
Social/contextual support. Physical, organizational, political and cultural aspects of the environment. This
can be primarily related to the motivation and goals (G) of the learner.
201
Step 3: Consider context of learning
The primary objective of this step is to ensure that the environmental constraints of the learning environment
have been considered for the architecture proposed in the previous step.
Authentic learning environments exist in a context. The environmental contexts of an intelligent tutoring system
as described by Kinshuk, Opperman and Russel (2001) can also be applied to an authentic learning environment.
The context is divided into seven categories.
¾
¾
¾
¾
¾
¾
¾
Student (natural abilities, learning styles and motivation)
Peers (interaction with fellow students)
Social environment (social values, institutional values, evolution of common metaphors)
Teacher (teaching styles, personality attributes)
Discipline (homogeneity, operational/conceptual, physical/virtual, teaching traditions, levels)
Characteristics of knowledge (operational, causal, contextual)
Characteristics of medium (hardware, software and communication capabilities).
From the perspective of the framework, these categories roughly map as shown in Figure 2.
Figure 2. Relationships between framework and environmental context
It is interesting to note that the teacher or the teaching style maps mostly to providing the available information
(what it provided to the student), successful action (guiding through assessment) and motivation. Most
traditional instructional theories such as Gagne’s nine steps (Gagne, 1985), John Keller’s ARCS model (Keller
& Suzuki, 1988) or Merrill’s ITT (Merrill, Li & Jones, 1991) can in fact be used to create these parts of situated
opportunities as they rely largely on information (I) and successful action (A) and are generally concerned with
how to enable a student to do something (as goals are prescribed by the teacher)
Step 4: Clearly specify what is learned?
The primary objective of this step is to understand and characterize the nature of adaptation that is expected to
emerge from the authentic activities being carried out in the learning environment.
One of the critical components of the framework is the nature of the adaptation that occurs as a result of
interacting with the information, and carrying out actions in the physical environment.
The adaptations that develop will be unique to a particular individual learner. While adaptation is an abstract
entity, it can be described. In the context of semantically rich problem solving domains, descriptions of these
adaptations can take the form of successful arguments generated by a learner. These arguments can be
constructed from five basic types of backings (Johnson, Zualkernan & Tucky, 1993)
202
¾
¾
¾
¾
¾
Type 1 – This is based on analytic truths; the interesting property is that the all actions preserve global
criteria of rationality such as consistency, soundness and completeness.
Type 2 – This is based on empirical judgments where the actions are constrained to consist of consensus
among groups of individuals.
Type 3 – This is based on complementary representations of a problem and actions are constrained by
agreement within complementary analytic representations and empirical judgments.
Type 4 – This is based on complementary representations of a problem and the actions are based upon
resolving conflicts within these.
Type 5 – This is based “systems of knowing” that lead the process such that actions are constrained by selfreflection.
The adaptation for a highly skilled individual can be described using a complex combination of these arguments
across diverse domains including medicine, chess, experimental design, VLSI manufacturing, and frauddetection (Johnson et al., 1993).
Bloom’s original description of the types of learning (Cognitive, Affective and Psychomotor) comes close to
serving as an appropriate language for a description of such adaptations (Bloom, 1956). For example, the
category of “synthesis” as described by Bloom (1956) consists of building a structure or pattern from diverse
elements. The keywords that describe this activity are combines, plans, creates etc. The adaptation for a skilled
individual engaged in a semantically rich domain such as statistical experimental design (Johnson et al., 1993),
however, contains many types of “synthesis”.
Figure 3. A description of an “adaptation” tied to a particular goal
For example, the individual constructs an initial conceptualization of the client’s problem, subsequently she
constructs a refined quasi-statistical representation (and re-representation) of the client’s problem, and then she
constructs an appropriate design type and finally a specific design.
The goal of carrying out each one of these “synthesis” steps actually are achieved by paying attention to very
different types of information and by using various types of qualitatively different backings. For example,
conceptualization of a client’s quasi-statistical representation of the problem is based on a Type 3 backing (see
Figure 3) while the construction of an appropriate design type is Type 4. Derivation of a particular design uses
Type 1 backing.
Hence the argument structure described in (Johnson et al., 1993) presents a refined language to describe specific
individual Adaptations for semantically rich domains as opposed to the classification based on keywords (e.g.,
combines, creates etc.) presented by Bloom (1956) to describe what is learnt.
In summary, the methodology applied to a particular instance consists of exploring and refining the four
dimensions in the context of the framework presented earlier.
A case study is presented next to show how the methodology is applied in the context of game-based learning.
203
Case Study – Just-In-Time Game-Based Learning
This section shows how framework and the methodology based on four dimensions of pedagogical design,
architecture, environmental context and what is learned was applied to the development of a just-in-time gamebased learning environment.
Game-Based Learning
Game-based learning is a particular form of incidental learning that occurs while the learner is engaged in an
activity that may not be directly tied to the task at hand. Schank and Cleary (1995) provide an example of such
learning where in order to teach grade students about geography, the students were asked to play a computer
game planning a driving trip to the city where their favorite team is playing. Incidental learning, in general (see
Marsick and Watkins (2001), for example), has been shown to be a significant mechanism of learning.
A detailed analysis of digital games for education is provided in Prensky (2004) that divides such games into the
different categories ranging from facts, skills, judgments etc. Attempts at building game-based learning
environments typically have fallen between two extremes.
On one extreme are games like Virtual-U (Virtual-U, 2005) that simulates working of a university to teach
administrators about strategy. In this instance, the operational rules as well as the constitutive rules (Salen &
Zimmerman, 2003) are embedded into the game (operational rules of a game are the surface rules while the
constitutive rules represent the deep mathematical structure that underlies the game). The end-user, however, has
a limited choice in which set of constitutive rules to use by selecting the scenario to be of a particular type of a
university (public vs. a private university, for example).
On the other extreme are game-engines (see Crookston (2005), for example) that use the cover-story of an
existing game format (corresponding to the operational rules of the game) such as the T.V. game show “who
wants to be a millionaire?” or more traditional games like a cross-word puzzle where one simply inserts the
syntactic elements of the particular domain in the game. For example, a game teaching biology is generated by
only using biological terms in a cross-word puzzle. At the level of constitutive rules, there is no deep logical
relationship between the game and the domain of learning. For example, in a cross-word puzzle, the two words
“bacteria” and “hormone” may appear together just because they have the letter “e” in common and not because
of their biological relationship.
The case study presented here applied game-based learning in the highly contextual problem solving domain of
software engineering. In this domain, complex software designs are constructed and re-constructed at a very
rapid pace. One critical problem of learning in these environments is to be kept updated with the continually
changing designs. These environments are particularly suited to automated just-in-time learning because of the
rapid rate of change and the immense time and cost involved in constructing any learning aids by-hand. In
addition, since these are semantically rich problem solving domains, the generic game engines that mostly
capture the operative (or surface) rules of a game (e.g., cross-word puzzle) may not be sufficient to capture the
richness of contextual learning that needs to take place.
The focus of study presented here was on software engineering design processes that typically produce byproducts like the UML diagrams (Jacobson, Booch & Rumbaugh,1999; OMG, 2005). An activity diagram is a
UML diagram that represents a flow of activities (see Figure 8, for a partial example).
This section presents an application of the methodology to the problem of constructing a constructivist e-learning
environment for building and using activity diagrams in a typical software engineering process.
Step 1: Complete pedagogical design
The first and the most important step of the methodology is pedagogical design. From the perspective of the
framework, each instance of a game-play is a “situated opportunity.” The key to designing a game-based elearning environment is to ensure that the adaptation that emerges as a result of playing the game will also lead
to successful actions in the software engineering domain. In order to do this, the goals and motivations of the
engineers have to be considered (G). Similarly, successful action in this instance consists of actions that will
lead to winning a game (A). In addition, the available information or cues provided in the game (I) should be
204
consistent with the learning styles and cognitive constraints (C) of a typical engineer. The pedagogical design is
carried out by considering the properties of the semantic and structural fit to determine what represents an
appropriate “situated opportunity” (i.e., I, A, C, G) in the game environment.
Semantic fit ties successful action to goals and motivation. Semantic fit, in part, determines the nature of
adaptation that will occur. In the case of software engineers using activity diagrams in their normal work
environment, an important determinant of successful action is their ability to predict the relationships between
the various activities. This is, in fact, one primary reason for an activity diagram to exist; knowing which
activity happens before which and under what conditions. For the game to lead to an appropriate adaptation (one
that would also lead to successful action in the actual environment), this meant that the successful action needed
to be tied to the ability of an engineer to predict the order and conditions of precedence of the various activities
in an activity diagram. The game, therefore, had to be designed to ensure that more successful the engineers
were at predicting relationships between activities, the greater the degree of their semantic fit.
Structural fit ties cognitive constraints to available information in the environment. In addition to a specific
activity diagram, information in the environment of a software engineer may consist of domain documents, other
colleagues and even computer programs. Therefore, the information (I) provided in the game should act as a
surrogate for the information in the real environment such that structural fit is high. Prior studies of software
engineer’s design activities in their environment suggest that this information is local in nature (Zualkernan,
Tsai, Jemie, Wen & Drake, 1992); due to cognitive constraints, engineers are only able to focus on “local”
aspects of an activity diagram. In order to be consistent with this assumption on the structural fit, therefore, it
was decided that games should only “encode” local information about an activity diagram. The “locality” of
information was implemented by providing limited threads of execution through the activity diagram itself. In
order to accommodate multiple learning styles of engineers, it was also decided that multiple game formats
should be provided.
Step 2: Construct architecture for the authentic environment
The second step of the methodology is to establish the architectural requirements for the environment. Once the
basic parameters of the particular game were established (i.e., what constitutes a “situated opportunity”), the
second dimension, architecture was used to further refine the design of this game-based learning environment.
The problem/project space describes the process by which a single “situated opportunity” is created. In gamebased learning, a situated opportunity typically consists of a single play of a game. In this context, because of
the nature of the problem (rapidly changing software designs), the process by which a situated opportunity is
created was considered at two distinct levels; the generation of a game for an arbitrary activity diagram and
various repeated plays of this particular game (Zualkernan & Parmar, 2004).
Related cases or ways in which a game was played successfully were also required to be a part of the
architecture. Similarly, the architecture also required that information (consisting of the actual activity diagram)
was also made available to the engineers. For analysis, cognitive tools such as paper or pencil or automated
analysis tools for activity diagrams were also required. In addition, user forums and real-time chat facilities that
allow software engineers to converse with fellow software engineers were also made a requirement. Finally, the
ability to group engineers to provide a social context built around the playing of these games was also mandated.
Step 3: Consider context of learning
Contextual analysis is used to ensure that the resulting architecture from the last step does not miss any
important part of its context of use. After establishing the basic architecture of this game-based learning
environment, the contextual analysis was carried out to further fine-tune the design of the environment. It was
recognized, for example, that available information (I) is tied to the student, his peers and the characteristics of
knowledge. However, the teacher dimension was missed and needed to be supported as well; therefore, the
dimension of a attaching a tutor was also added. Similarly, since goal and motivations are tied to peers and
social environments, it was decided to add a peer-to-peer challenge component to the game where an engineer
could challenge a fellow engineer with a particular game based on an activity diagram. Similarly, since the
game also had to run on mobile phones with fairly small screens, in order to be consistent with the cognitive
constraints, the number of activity diagrams appearing on the screen was reduced.
205
Step 4: Clearly specify what is learned?
The final and most crucial step of the methodology is to ensure that the adaptation that emerges as a result of
interaction with the e-learning environment is consistent with what is required for successful action in the real
environment. Since activity diagrams are semi-formal, the primary types of arguments constructed by software
engineers in a real environment are Type 1. This means that all their actions need to preserve some global
notions of consistency and soundness. In other words, if an engineer takes an action that indicates that activity A
(say, Pour Coffee) “happens before” B (say, Drink Coffee), the game needs to apply a formal rule to enforce this.
This implies that each game needs a model of the activity diagrams to make these determinations; it is necessary
for the game to determine what constitutes successful action for the engineer. In a real design context an
engineer may look at an activity diagram where activity A “comes before” B and activity B “comes before”
activity C and then take the action that activity A “comes before” activity C. The game must also make the
determination that in the presence of the information provided (A “comes before” B and B “comes before” C), A
“comes before” C constitutes successful action. The game simply needs to preserve the same notion of
“rationality” (Type 1, in this case) that connects information with a successful action. Practically this meant that
each game needed to maintain an internal model of the activity diagram.
Design
Based on the analysis presented above, four different forms of common game types were constructed for this
game-based e-learning environment. Each is briefly described below.
Tetris
The classical game of Tetris consists of a sequence of geometric figures falling down at various rates. The
learning version of Tetris was created by mapping the geometric constraints to the precedence relationship
between activities (Zualkernan & Parmar, 2004). So rather than matching the geometric constraints, the
engineers are asked to match the precedence relationships between falling activities. The information provided in
this environment, therefore, was a falling block with some precedence constraints. A player who was able to
match these precedence constraints achieves a high-score or a high semantic fit.
Figure 4. A Tetris-Like game automatically generated for activity diagrams
From the framework perspective, each situated opportunity in the Tetris-like game can be summarized as
follows:
I: Local precedence constraints between activities represented by falling blocks where each block has different
types of precedence relations (see arrows in Figure 4) attached to it.
A: Successful action consists of matching “local” precedence constraints (who comes before who?) in a very
short amount of time by rotating the falling block.
C: A small number of options of precedence constraints are provided and is suited to a visual learning style.
G: Match as many precedence constraints as possible in order to prevent the blocks from building up.
206
Snake
The game of snake consists of a snake that gets progressively longer and faster (and hence difficult to maneuver)
as it eats various bits of “food” that appear. The learning version for this game was created by mapping the
“eating” activity to the precedence relation and the length of the snake. If the player picks the wrong precedence
relationship, the snake gets longer, picking the right now, however, keeps the snake to be the same length.
Therefore, a player who is familiar with the precedence relationship will, in fact, achieve a higher score.
From the framework perspective, each situated opportunity in this snake-like game can be summarized as
follows:
I: Local precedence constraints between two activities represented by the head of the snake and the food (see
Food and Head in Figure 5).
A: Successful action consists of matching two precedence constraints (who comes before who?) in a very short
amount of time by eating the food or waiting until the undesired food (violating the precedence constraint)
disappears.
G: Match as many precedence constraints as possible in order to prevent the snake from getting larger.
Figure 5. A Snake-like game automatically generated from activity diagrams
Hexxagon
The Hexxagon game consists of a field of hexagons, where the player is allowed to move his marbles to a region
within green or yellow regions.
Green regions multiply the original marbles; while a move to yellow regions allow one to “kill” an opponent’s
marbles that happen to be in adjacent hexagons. All the hexagons are initially labeled as activities (see Figure
6). A move to a hexagon whose activity follows that activity in the current hexagon multiplies the number of
marbles. This rewards the player for knowing the precedence relationship.
Figure 6. A Hexxagon-Like game generated from activity diagram
From the framework perspective, each situated opportunity in the Hexxagon game can be summarized as
follows:
207
I: Local precedence constraints between two activities represented by the adjacency of hexagons (see hexagons
in Figure 6).
A: Successful action consists of matching two precedence constraints (who comes before who?) and deciding
which of the green or yellow move will result in better positioning.
G: Match as many precedence constraints as possible in order to kill the opponent’s marbles.
Space Invaders
The classical game of Space Invaders consists of a number of alien ships coming down from the sky. The
objective of the game is to shoot them down. The constitutive rule of this game is simply that enough “bullets”
will tear away and make an alien ship disappear. A learning version of this game is formulated by mapping the
precedence relation to the sequence in which the activities (that now represent alien ships) are targeted. The
activities have to be targeted in the right sequence of precedence to achieve a high score.
From the framework perspective, each situated opportunity in the Space Invader-like game can be summarized
as follows:
I: Local precedence constraints between two activities represented by the alien ships (see Figure 7).
A: Successful action consists of matching two precedence constraints (who comes before who?) in a very short
amount of time by killing only those aliens that represent activities that precede each other.
G: Match as many precedence constraints as possible in order to prevent the aliens from descending
Figure 7. A Space Invader-Like game generated from activity diagram
Implementation
All the four game types were implemented using Macromedia Flash (Macromedia, 2005; Neave 2005) and were
hosted within a modified version of the open-source Claroline Learning Management System (Claroline, 2005).
The learning management system provides the functionality of forums, groups, chats and messaging that can be
tied to a particular game. In addition, a manager can assign games to various players. Also included are
interaction between the engineer, his peers and a tutor. For example, the engineers can post on forums about tips
on how to play a particular type of game better. In addition, a tutor can post exercises so that the engineers can
learn the peculiarities of an activity diagram better before playing the game.
Since the games thus generated are SCORM-complaint (SCORM, 2005), the score of each student is
automatically tracked and delivered to the respective manager. Various resources related to the game including
the actual activity diagram used to derive the game is also made available as information (I) that can be readily
accessed by a learner while playing the game as shown in Figure 8. The player can pause the game, consult the
activity diagram and get back to playing the game.
208
Figure 8. Playing a SCORM-compliant game inside Claroline
Finally, to better support the peer-to-peer dimension of learning, the games were also implemented to run on
mobile phones supporting J2ME (Sun, 2005) and Flash Lite 1.1 (Macromedia, 2005). This allows players to
challenge other players on their own activity diagrams. For example, it is possible for a manager to send a game
to an engineer from Claroline (integrated with the Ozeki SMS Server (Ozeki, 2005)) or from another mobile
phone using the short messaging services (SMS). This score achieved by the challenged engineer is sent back to
the challenger via an SMS message.
Discussion
Game-based learning was specifically chosen as a case study for presenting this framework because it represents
an extreme case. Of all the pedagogical design paradigms of an authentic constructivist environment (i.e.,
problem-based learning, case-based learning etc.), game-based learning seems the most unobvious. For
example, software engineers do not really play games of the type presented in this paper. Therefore, one could
argue that since these games are not a part of the “cultural practice” of software engineers, learning
environments built around such games are not “authentic.” This view is essentially a simulation stance towards
“authentic learning” that assumes that an authentic learning environment ought to be a model or a surrogate of
the actual environment and therefore necessarily contain the complexity of the environment. Constructing a
complex environment, however, is not necessary for guaranteeing authenticity. For example, citing work habits
of apprentice tailors from Lave (1988), Brown et al. (1989) who point out:
“This is not to say that authentic activity can only be pursued by experts. Apprentice tailors …,
for instance, begin by ironing finished garments (which tacitly teaches them a lot about cutting
and sewing). Ironing is simple, valuable, and absolutely authentic.” (p. 36)
The framework presented here shows that not only can specific slices of authentic activity (like ironing, for
example) be isolated, but the “surface form” of the activity can be modified without sacrificing authenticity.
This is essentially what the case study in game-based learning shows. The key point is that the learning
environment needs to give rise to an appropriate adaptation under the mutual constraints of information, action,
cognitive constraints and goals. And even though the surface activity of game-playing seems like simple
repetition, when analyzed via the framework, this activity is authentic because the situated opportunities are
semantically similar to what one expects in a real environment of software engineers. Semantically similar
because the software engineer has the same goals (determining precedence relationships and higher motivation
because game-playing is fun), similar cognitive constraints (locality, in this case) and varying learning styles
(catered for by different game types) and finally, successful action in the game (higher score in the game)
corresponds to getting an appropriate answer in the real environment.
Another observation from the case study is that technology is required to implement much of the social
interactions (e.g., peer to peer interaction, sharing etc.) is already readily available in pre-packaged forms and
once the methodology is applied, the implementation is relatively straight forward.
209
Finally, the methodology is neutral in that it only supplies broad guidelines. For example, Bloom’s taxonomy
(Bloom, 1956) can be used to extend the low-level pedagogical design of the game-based learning environment
presented in the case study. The environment currently only supports learning of precedence relationships
mostly at “analysis” and “comprehension” level. However, the environment can easily be extended to higher
levels like “application”, “analysis” and “synthesis” by simply constructing a different characterization of the
situated opportunities.
In summary, the case study shows that the framework and the methodology enable a designer to build slices of
authentic activities at the semantic level. This methodology, therefore, does not dictate that a massive simulation
or reproduction of the real environment be constructed for all authentic learning environments (although, it does
not preclude these). This enables a designer to build authentic environments incrementally. In addition, the
methodology frees up the designer to employ novel forms (like game-based learning) while retaining
authenticity.
Conclusion
This paper presented a framework and a methodology for analyzing the development of authentic constructivist
e-learning environments for semantically rich problem solving domains. The development of an e-learning
constructivist environment has been conceptualized in terms of the creation of “situated opportunities.” These
opportunities have further been specified in terms of information available, successful action, cognitive
constraints/learning styles, and goals and motivation.
A methodology based on pedagogical design, architecture, context of learning and an analysis of what was
learned when developing such environments was also presented together with a successful application of this
methodology to create a rich just-in-time game-based learning environment.
Although the framework has currently only been applied in the context of game-based learning, this framework
is being used in building other types of constructivist environments such as problem-based learning. One
interesting outcome of this exercise has been that most technology required for such environments is readily
available as seen by the various types of open-source software used in the game-based e-learning environment.
What is required and needed are methodologies and frameworks to guide the development in novel instructional
paradigms such as game-based learning.
References
Anderson, J. (1990). The adaptive structure of thought, Hillsdale, NJ, USA: Erlbaum.
Bloom, B. S. (1956). Taxonomy of educational objectives (Handbook 1: The cognitive domain), New York,
USA: Addison-Wesley.
Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational
Researcher, 18 (1), 32-42.
Claroline (2005). Claroline : Open Source e-Learning, retrieved January 09, 2006 from http://www.claroline.net.
Crookston
(2005).
Flash
Learning
Games,
http://itc.umcrookston.edu/index.asp?section=flashgames.
retrieved
January
09,
2006,
from
Gagne, R. M. (1985). The conditions of learning and the theory of instruction, Chicago, USA: Harcourt Brace
College Publishers.
Gardiner, H. E. (1993). Multiple Intelligences: The theory in practice, New York, USA: Basic Books.
Herrington, J., & Oliver, R. (2000). An instructional design framework for authentic learning environments.
Educational Technology Research and Development, 48 (3), 23-48.
Ip, A., & Naidu, S. (2002). Experience-based pedagogical designs for e-learning. Education Technology, XLI (5),
53-58.
210
Jacobson, I., Booch, G., & Rumbaugh, J. (1999). The unified software development process, Addison-Wesley.
Johnson, P. E., Kochevar, L. K., & Zualkernan, I. A. (1992a). Expertise and fit: Aspects of cognition. In Pick, H.
L. Jr., Broek, B. V. & Knill, D. C. (Eds.), Cognition: Conceptual and methodological issues, American
Psychological Association, Washington, D.C., USA, 305-331.
Johnson, P. E., Grazioli, S., Jamal, K., & Zualkernan, I. A. (1992b). Success and failure in expert reasoning.
Organizational Behavior and Human Decision Processes, 53 (2), 173-203.
Johnson, P. E., Zualkernan, I. A., & Tucky, D. (1993). Types of Expertise: an invariant in problems solving.
International Journal of Man-Machine Studies, 39, 641-665.
Jonassen, D. E., Peck, K. L., & Wilson, B. G. (1998). Learning with technology: A constructivist perspective,
Prentice-Hall.
Keller, J. M., & Suzuki, M. K. (1988). Use of ARCS motivation model in courseware design. In Jonassen, D. E.
(Eds.), Instructional design for microcomputer courseware, Hillsdale, NJ, USA:Lawrence-Erlbaum, 34-40.
Kinshuk, Patel, A., Oppermann, R., & Russell, D. (2001). Role of human teacher in web-based intelligent
tutoring systems. Journal of Distance Learning, 6 (1), 26-35.
Lave, J. (1988). The culture of acquisition and the practice of understanding. IRL report 88-00087, Palo Alto,
CA: Institute for Research on Learning.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation, Cambridge, England:
Cambridge University Press.
Macromedia (2005). Adobe Website, retrieved January 09. 2006 from http://www.macromedia.com.
Marsick, V. J., & Watkins, K. E. (2001). Informal and incidental learning. In New directions for adult and
continuing education, 89 (Spring 2001), Jossey-Bass Publishers, 25-34.
Merrill, M. D., Li, Z., & Jones, M. K. (1991). Instructional transaction theory: An introduction. Educational
Technology, 31 (6), 7-12.
Neave (2005). Neave Games, retrieved October 17, 2005 from http://www.neave.com/games.
Oliver, R. (2001). Developing e-learning environments that support knowledge construction in higher education.
In Stoney, S. & Burns, J. (Eds.), Working for excellence in the e-economy, Churchlands, Australia: We-B Center,
407-416.
OMG (2005). OMG Unified Modeling Language Specification, Version 2.0, retrieved January 09, 2006 from
http://www.omg.org/technology/documents/formal/uml.
Ozeki (2005). Ozeki Website, retrieved January 09, 2006 from http://www.ozeki.hu.
Piaget, J. (1975). The construction of reality in the child, Ballantine Books.
Prensky, M. (2004). Digital game-based learning, McGraw-Hill.
Rumetshofer, H., & Woss, W. (2003). XML-based adaptation framework for psychological-driven e-learning
systems. Education Technology and Society, 6 (4), 18-29.
Salen, K., & Zimmerman, E. (2003). Rules of play: Game design fundamentals, MIT Press, Cambridge, Mass,
USA.
Schank, R., & Cleary, C. (1995). Engines for education, Hillsdale, NJ, USA Lawrence Erlbaum.
SCORM (2005). IMS Content Packaging Best Practice Guide Version 1.1.1 Update Specification, retrieved
January 9, 2006 from http://www.imsglobal.org/content/packaging/imscp_bestv1p1p1.html.
211
Simon, H. A. (1996). The sciences of the artificial, Cambridge, MA, USA: MIT Press.
Sun (2005). Java Platform, Micro Edition (Java ME), retrieved January 9, 2006 from http://java.sun.com/j2me.
Turvey, M. T., Carello, C., & Kim, N (1981). Cognition: The view from ecological realism. Cognition, 10 (1-3),
313-321.
Virtual-U (2005). Virtual-U Website, retrieved January 9, 2006 from http://www.virtual-u.org.
Vygotsky, L. S. (1980). Mind in society: The development of higher psychological processes, Cambridge, MA,
USA: Harvard University Press.
Zualkernan, I. A., Tsai, W. T., Jemie, I, Wen, I. C., & Drake, J. M. (1992). Object-oriented analysis as design: A
case study. International Journal of Software Engineering and Knowledge Engineering, 2 (4), 489-521.
Zualkernan, I. A., & Parmar, F. (2004). AGORA - An Engine for Automatic Generation of Game-Based
Incidental e-Learning Environments from UML Activity Diagrams. Paper presented at the 2004 International
Research Conference on Innovation in Information Technology, October 4-6, 2004, Dubai, UAE.
212
Zayim, N., Yildirim, S. & Saka, O. (2006). Technology Adoption of Medical Faculty in Teaching: Differentiating
Factors in Adopter Categories. Educational Technology & Society, 9 (2), 213-222.
Technology Adoption of Medical Faculty in Teaching: Differentiating
Factors in Adopter Categories
Nese Zayim
Department of Biostatistics & Medical Informatics, Akdeniz University, Medical School, 07059 Antalya, Turkey
Tel: +90 (242) 227 43 43
[email protected]
Soner Yildirim
Department of Computer Education & Instructional Technology, Middle East Technical University
Faculty of Education, 06531 Ankara, Turkey
Tel: +90 (312) 210 4057
Fax: +90 (312) 210 1112
[email protected]
Osman Saka
Department of Biostatistics & Medical Informatics, Akdeniz University, Medical School, 07059 Antalya, Turkey
Tel: +90 (242) 227 43 43
[email protected]
ABSTRACT
Despite large investments by higher education institutions in technology for faculty and student use,
instructional technology is not being integrated into instruction in higher education institutions including
medical education institutions. While the diffusion of instructional technologies has reached a saturation
point among early adopters of technology, it has remained limited among the mainstream faculty. This
investigation explores instructional technology usage patterns and the characteristics of medical school
faculty as well as contributing factors to IT adoption. The focus of the study was to explore the differences
between faculty members who have adopted new technology and those reluctant or resistant to IT adoption,
and to determine whether faculty characteristics contribute to the prediction of faculty adopter categories.
Faculties from the disciplines of basic and clinical science at a state university Faculty of Medicine were
surveyed to gather data concerning faculty characteristics, adoption patterns, perceptions of computer-use
self efficacy, the value of IT, barriers and incentives to adoption and preferences related to help and support
in technology adoption. The data analysis was based on Rogers’ theories of diffusion and adopter
categories. Significant differences were found between early adopters and the mainstream faculty in terms
of individual characteristics, adoption patterns, perceptions of barriers and technology learning preferences
The results indicated that computer use self efficacy and rank significantly contribute to the prediction of
faculty adopter group.
Keywords
Technology adoption, Diffusion of innovation, Adopter categories, Medical faculty technology use
Introduction
In the past few years, higher education institutions have invested heavily in infrastructure to support the diffusion
and adoption of technology (Green, 1999; Jacobsen, 2000). However, despite large investments by higher
education institutions in technology for faculty and student use, instructional technology is not being integrated
into instruction in higher education institutions including medical education institutions (Geoghegan, 1994;
Spotts, 1999; Surry, 1997; Albright, 1996; Carlile & Sefton, 1998). There are many reasons both technical and
societal, explaining why innovative technologies have not been widely adopted, however, the major reason for
this lack of utilization is that most university-level technology strategies ignore the central role that the faculty
plays in the process of change (Surry & Land, 2000).
The Association for Educational Communications and Technology (AECT) has defined instructional technology
(IT) as a complex, integrated process involving people, procedures, ideas, devices and organizations, for
analyzing problems and devising, implementing, evaluating and managing solutions to those problems involved
in all aspects of human learning (Seels & Richey, 1994). Despite the AECT definition of IT, in which the
emphasis is on IT rather than its’ products, many of the debates regarding the use of technology in education
continues to focus on products: computers, software, networks and instructional resources (Green, 2000).
213
Certainly, the use of an adequate technology infrastructure is a prerequisite of IT integration, but the major
challenge is to encourage the faculty to adopt these technologies once they are made available. Goeghegan
(1994) expresses this challenge as follows:
One of the most basic reasons underlying the limited use of instructional technology is
our failure to recognize and deal with the social and psychological dimension of
technological innovation and diffusion: the constellation of academic and professional
goals, interest, and needs, technology interest, patterns of work, sources of support, social
networks, etc., that play a determining role in faculty willingness to adopt and utilize
technology in the classroom.
Adoption of or hesitation to adopt new instructional technologies by the faculty involves a complex system
involving multiple variables. As stated by Spotts (1999), “…, the reality of instructional technology use is in the
relationship between the new instructional technologies and the faculty members’ individual and organizational
context and their personal histories” (p. 93-94).
Diffusion of Innovations
Studies of diffusion and adoption help to explain the what, where, and why of technology acceptance or rejection
in education (Holloway, 1996). Therefore, Rogers’ (1995) theory of diffusion of innovations provides a
theoretical framework for analyzing faculty technology adoption patterns.
Rogers (1995) defines an innovation as an idea, practice or object that is perceived to be “new” by the individual,
and diffusion as the process through which an innovation is communicated through certain channels over time
among the members of a social system. The innovation in the present investigation is represented by
instructional technology, including computer-based tools and processes, and diffusion is represented by the
extent to which the medical school faculty at a state university has adopted instructional technology in teaching
and learning.
According to Rogers (1995), individuals in a social system do not adopt an innovation at the same time, a
certain percentage of individuals are relatively earlier or later in adopting a new idea. Based on the
innovativeness criterion that the degree to which an individual is relatively earlier in adopting new ideas than
other members of a social system, the distribution of various adopter categories forms a normal, bell-shaped
curve that illustrates Innovator (2.5%), Early Adopter (13.5%), Early Majority (34%), Late Majority (34%), and
Laggards (16%) (Figure 1).
Figure 1. Adopter Categorization on the Basis of Innovativeness (Rogers, 1995)
Individuals who adopt an innovation at different points in diffusion process differ from one other in terms of
social and psychological characteristics (Rogers, 1995). Those characteristics determine the individuals’
willingness to adopt an innovation and their leadership functions. Some of the differences that have been cited
separating early adopters from the mainstream include:
¾
¾
¾
¾
Early Adopters
Favor revolutionary change
Visionary
Strong technology focus
Risk takers
¾
¾
¾
¾
Mainstream
Favor evolutionary change
Conservative
Problem oriented
Risk Averters
214
¾
¾
¾
Experimenters
Generally self-sufficient
Horizontally connected
¾
¾
¾
Want proven applications
May need significant support
Vertically connected
The differences between people who fall into Rogers’ Early Adopter and Early Majority categories create gaps
in motivation, expectations and needs. The literature on individual characteristics of the faculty indicated that
early adopters of instructional technology share common characteristics such as higher perceptions of efficacy
and expertise (Anderson, Varnhagen & Campell, 1999; Jacobsen, 1998; Lichty, 2000; Oates, 2001), risk taking
and experimentation (Oates, 2001), positive attitude toward technology ( Spott, 1999) and personal interest in
technology (Oates, 2001).
As stated by Bates (2000), “because of the central role that faculty members play in the work of the universities
and colleges, any change, especially in core activities such as teaching and research, is completely dependent on
their support” (p.95). Therefore, in order for large-scale technology adoption and diffusion to happen, it is
critical to understand and bridge differentiated needs and expectations of faculty members (Garofoli & Woodell,
2003). Thus, the purpose of the present study is to explore the differences between faculty members who are
open to, and those who are reluctant or resistant to IT adoption, and to determine if faculty characteristics
contribute to the prediction of faculty adopter categories. Rogers’ (1995) theories of diffusion of innovations
were used to analyze the differences in the IT adoption of the medical school faculty.
Method
The present investigation surveyed faculty members from basic and clinical science disciplines at a faculty of
medicine at a state university in Turkey. Information was gathered about technology use patterns, computer
experience and use of technology for teaching, perceived computer use self efficacy, perceived value of IT,
perceived incentives, and barriers. Survey items were adopted or selected from previous investigations of
faculty adoption patterns (Anderson, Varnhagen, & Campbell, 1999; Jacobsen, 1998) and Microcomputer
Utilization in Teaching Self-Efficacy Beliefs Scale (Enochs, Riggs, & Ellis, 1993).
The survey was distributed to 305 faculty members and complete data was obtained from 155 (50.3%)
participants (72.7% male and 27.3% female), holding various academic ranks (i.e. 32.7% professors, 19.6%
Assoc. Professors, 22.1% Asst. Professor and 25.6% others) having an average of 10 years of teaching
experience. While the average age was 41 years, the largest group (≈55%) was in the 31-40 age groups. The
majority of the respondents (76.8%) were instructors of clinical science.
Adopter Groups
In order to classify respondents into adopter categories (early adopters (EA), mainstream faculty (MF)), the
individual innovativeness scoring procedure developed by Anderson, Varnhagen and Campbell, 1999 was used.
The scoring procedure was developed on the basis of the assumption that EAs have come to use these
technologies earlier and have gained more expertise relatively to majority faculty (Anderson, Varnhagen, and
Campbell, 1999). A composite score was calculated for innovativeness of faculty by adding the self_rated
expertise level of each individual faculty member (i.e., 1 for Extensive, 2 for Good, 3 for Fair, 4 for Novice, 5
for None) indicated for each of the 11 types of computer software and tools The total possible range for
cumulative score for innovativeness is 11-55; the sample scores ranged from 13 to 51. Consistent with Rogers’
assertion that adoption of an innovation will be normally distributed, cumulative frequency of the scores on this
scale approach an S-shaped curve which lends confidence to assumption of normalcy.
By using Rogers’ (1995) adopter categories and innovativeness scores, 16% (n=25) of the respondents were
assigned to the Early Adopter group (EA) ( 2.5% Innovators+13.5% Early Adopters =16% EA), and 84%
(n=130) of the respondents were assigned to the Mainstream Faculty (MF) group( 34% Early Majority+34%
Late Majority+ 16% Laggards=84%MF) (Figure 2).
215
Figure 2. Distribution of Faculty Innovativeness Scores
Results
The EA group is more likely to consist of junior assistant professors (χ2(1, 155)=5.59, p=0.019), in the 20-40
age interval (χ2(1, 25)=4.840, p=0.028).
Although there are no significant differences between adopter groups in terms of computer ownership and
internet access at home and in the office, the EA group uses computers more often(χ2(1, 146)=11.34, p=0.001) .
Technologies Used in Teaching
Participants were asked to indicate which of the 12 instructional technologies they use in the teaching-learning
process. Early adopters significantly have used more technologies than Mainstream Faculty group (t(151)=2.841,
p<0.05 Ms 5.58 vs. 4.38), and it is likely that they have used course web pages (Pearson χ2(1, 153)=8.306,
p=0,009), web resources (χ2(1, 153)=7.018, p=0.018) and commercial educational software (χ2(1, 153)=22.077,
p=0.000) more than the Mainstream faculty. The proportion of technologies used by the adopter group is
presented in Table 1. These findings indicate that relatively new instructional technologies have diffused into the
early adopter group more than the mainstream faculty.
Table 1. Adopter Groups’ Technology Use
Early Adopters
%
Blackboard
62.5
Overhead
75.0
Slide Projector
75.0
Computer + Projection
100.0
Video
25.0
Sound
8.3
Special Laboratory
29.2
Course web sites
33.3
Web resources as a part of content
37.5
Commercial educational software
16.7
Word processors for course materials
45.8
Presentation software
50.0
Mainstream Faculty
%
67.4
69.8
76.7
94.6
19.4
3.1
20.9
10.9
14.7
0.0
31.0
29.5
Computer Use Self-Efficacy
Computer use self-efficacy is defined as an individual’s belief regarding their ability to use computer
competently (Compeau & Higgins, 1995). Previous research provides evidence that perceived efficacy regarding
216
computer use is a significant factor in an individual’s adoption decision (Lichty, 2000; Marcinkiewicz, 1994;
Compeau & Higgins, 1995).
In order to assess faculty self-efficacy in computer use in teaching, the 10-item “Microcomputer Utilization in
Teaching Efficacy Belief Scale” developed by Enochs, Riggs and Ellis (1993) and used in previous research
(Lichty, 2000) was used. Respondents were presented with a five-point scale (i.e. 5 for Strongly Agree, 4 for
Agree, 3 for Neutral, 2 for Disagree, 1 for Strongly Disagree) to indicate their level of agreement with each
statement about computer use. Internal consistency of this subscale yielded a coefficient alpha of .91. As
expected, EAs had significantly higher level competency of computer use than the MF group (t(144)=6.263,
p<0.01 Ms 44.68 vs. 37.38)
Perceived Value of IT
“The individual’s attitudes or beliefs about the innovation have much to say about his or her passage in the
innovation-knowledge process” (Rogers, 1995, p.167). To adopt an innovation, the individual must define the
innovation as relevant and useful in a specific situation. In order to gather data about faculty perception of the
value of IT in medical education, participants were asked to indicate their level of agreement on 10 items
regarding value of IT using five-point scale (i.e. 1 for Strongly Agree, 2 for Agree, 3 for Neutral, 4 for Disagree,
5 for Strongly Disagree). Although IT use in medical education is perceived to be valuable by the majority of the
faculty (Mean=11.75, SD = 3.4), EAs valued IT more than MFs significantly ( t(151)= -2 .681, p<0.01, Ms
10.12 vs. 12.07). The EA group had statistically stronger beliefs than the MF group that technology enables
instructors addressing different learning styles, using time effectively and increasing their productivity (Table 2).
Table 2. Significant Differences between Adopter Groups on the Perceived Value of IT
Items
t
df
p
Means
Technology enables me to address the different
-3.055
153
0.003
1.24 vs. 1.64
learning styles of students.
Using technology enables me to use lecture time
-2.357
153
0.020
1.28 vs. 1.61
efficiently
Using technology increases my productivity as
-2.357
153
0.020
1.28 vs. 1.61
an instructor
Perceived Barriers
In explaining the limited adoption of instructional technology among faculty members, Jacobsen (2000) states
that “the explanation for limited adoption may be found in the many barriers that still constrain use by
enthusiastic beginners” (p.2).
Table 3. Percentages of Agreement on Barrier Items Scale of Adopter Groups
Early
Mainstream
Barrier Items
Adopters
Faculty
%
%
Lack of computers for faculty members.
66.7
54.3***
Lack of printers and/or other peripherals needed to effectively
72.0
60.0***
use computers for teaching and learning.
Lack of computers for students
68.0
68.0***
Lack of support for supervising student computer use.
80.0
66.1***
No reward structure that recognizes faculty members for using
66.7
67.2***
technology in teaching and learning.
Inadequate financial support for the integration of instructional
54.2*
37.3
technology.
Lack or inadequacy of training opportunities for faculty
44.0
51.9**
members to acquire new computer knowledge and skills.
*** rated agree or strongly agree by over 50% of the both adopter groups
*rated agree or strongly agree by over 50% of EA
** rated agree or strongly agree by over 50% of MF
217
Participants were presented with a five-point scale (i.e., 1 for Strongly Agree, 2 for Agree, 3 for Neutral, 4 for
Disagree, 5 for Strongly Disagree) to indicate the level of their agreement with 20 barrier related items. An
estimate of the internal consistency of this subscale yielded a coefficient alpha of .86, which indicates that the
faculty responded consistently across the items.
Over 50% of both adopter groups agreed or strongly agreed that the lack of computers for individual faculty and
students, the lack of computer peripherals, the reward structure and the lack of support service for students were
barriers to the integration of technology into medical education. In addition to these barriers, 54.2 % of the EA
group agreed or strongly agreed that inadequate financial support was a barrier and 51.9% of the MF group
agreed or strongly agreed that the lack of training opportunities was an impediment to technology integration
(Table 3).
Although the adopter groups gave similar responses, the EA and MF groups had exhibited significant differences
in their perceptions about barriers to adoption (Table 4). Because the EA group has a relatively higher level of
technological knowledge and skills, and interest in technology use, as expected, the EA group rated knowledge
and skills barrier lower than the MF group. The EAs also disagreed highly than the MF group about the
statements that software and IT are unsuitable for their instructional needs. The EA group expressed a higher
level of agreement than the MF group regarding faculty members’ interest in using technology for instruction.
Table 4. Significant Differences Between Adopter Groups’ Perceptions About Barriers
Items
t
df
p
Means
I have lack of necessary knowledge and skills for
6,74
152
p < .001
4,64 vs.3,28
using technology effectively.
Available software doesn’t fit to my instructional
2,92
149
p < .05
4,24 vs. 3,75
needs.
Instructional technologies do not fit the course or
3,55
152
p < .05
4,48 vs.3,95
curriculum that I teach.
Faculty members are not interested in using
-2,43
151
p < .05
3,00 vs. 3,52
technology for instruction.
Perceived Incentives
The participants were presented with a five-point scale (i.e., 1 for Very Important, 2 for Important, 3 for Neutral,
4 for Not Important, 5 for Not Important At All) to rate the level of importance of various statements on
incentives for adoption. Over 90% of both adopter groups rated that among university policy and plans for
diffusion of IT, investments in infrastructure, training and support and financial support are very important
incentives for IT adoption. Reward structures and the reduction of teaching loads also were rated as important
incentives by over 60% of both groups.
Preferred Methods of Learning Concerning Technology and Support
In many research studies, resources for training and support were identified as important incentives to faculty
adoption of technology for teaching (Anderson, Varnhagen & Campbell, 1999; Green, 1999). Jacobsen (1998)
suggest that “Individuals tend to have preferred methods for learning more about technology (p.96). In terms of
methods for acquiring knowledge and skills about technology and support respondents rated their level of
preferences on 12 items using a five -point scale (i.e., 1 for Strongly Prefer, 2 for Prefer, 3 for Neutral, 4 for
Don’t Prefer, 5 for Don’t Prefer at All).
Over 90% of the both adopter groups rated online resources and printed materials as strongly preferred methods.
Because EAs have a higher level of expertise in technological resources, not surprisingly, EAs preferred online
resources more strongly (t(153)=-2.726, p<0.01 (Ms 1.24 vs. 1.64)) than MFs. While the EA group preferred or
strongly preferred workshops and presentations (84%), structured in service training (79.2%) and experimenting
alone (68%), MFs preferred structured in service training (88.3%), workshops and presentation (82.5%) and
experimenting alone (58.4%) in descending order.
Participants’ preferences of help and support are given in Table 5. Consistent with the Rogers’(1995) assertion
that early adopters’ interpersonal networks are more likely to be outside, EAs preferred colleagues at another
218
institution (t(146)=-2.807, p<0.01 (Ms 2.04 vs. 2.69)) and outside professionals (t(148)=-2.851, p<0.01 (Ms
2.08 vs. 2.71)) significantly stronger than MFs.
Table 5. Preferred Methods of Support By Adopter Groups
Early Adopters
Mainstream Faculty
Items
Experienced graduate students
Colleagues on campus
Colleagues at another institution
Outside professionals
Support service
Hot-line, or telephone assistance
One on one help
%
%
95.8
96.0
72.0
80.0
87.5
50.0
79.2
84.6
90.6
50.4
48.0
90.4
58.1
89.5
Relationships between Faculty Characteristics and the Adopter Categories
Logistic Regression analysis was conducted to determine whether a set of faculty characteristics (rank, sex, age,
discipline, teaching experience, self-efficacy, perceived value of IT) contribute significantly to prediction of
faculty members’ adopter categories. More than 30% of the variation in the Adopter Group variable explained by
the Logistic Regression model (Cox&Snell R2=0.307, Nagelkerke R2=0.520). Results of the stepwise logistic
regression analysis indicated that among the most powerful predictors of faculty adopter groups are rank and
computer use self-efficacy (Table 6). This model shows that faculty members whose ranks are lower than
professor and, faculty members whose self-efficacy beliefs are higher are more likely to be an Early Adopter.
Table 6. Results of Stepwise Logistic Regression
Beta
S.E.
z.stat
Wald
Variable
.930
2.57
6.609
-2.390*
Rank
-.375
.080
-4.68
21.769
Computer-use Self Efficacy
21.57
Constant
Model Chi-square=78.216, df=2, p<0.001
Cox&Snell R2=0.285, Nagelkerke R2=0.483
Note:*Significant at the .05 level.
a
Odds ratio associated with one unit increase in rank
b
Odds ratio associated with one unit increase in computer use self-efficacy
p
0.010
0.000
Exp(B)
0.092a
0.687b
The model explained more than 28% of the variation in the Adopter Group variable and was able correctly to
classify 47.8% of early adopters and 96.6% of mainstream faculty, for an overall success rate of 88.7%.
Discussion
The findings of this study provided additional evidence that early adopter and mainstream faculty have different
characteristics and different needs in technology integration. As suggested by Jacobsen (1998), understanding
the differences between early adopter faculty and mainstream faculty will help us build programs and encourage
faculty members to pursue the adoption of instructional technology.
Identifying the differences between early adopters and the mainstream faculty leads to the understanding that
different approaches are needed to bridge the gap between EA and MF groups in the diffusion of instructional
technology and to encourage EAs for their efforts. It is obvious that faculty support programs designed with a
“one-fit-all” approach fail to succeed.
The results of this study indicate that the early adopter faculty and the mainstream faculty in this study have
different needs in training and support. It seems that formal training programs do not appeal to early adopter
faculty members who have a level of expertise in technology use. In designing training programs, institutions
might consider gearing early adopters towards advanced topics and focusing on the specific needs of these
faculty members. By providing incentives such as release time for training, providing funds for developing
219
instructional materials, supporting symposia and conference participation, EAs must be motivated to continue
their focus on innovation, the re-invention of technology and to share their expertise with the mainstream.
Making innovative faculty members aware of each other and encouraging them to share resources and expertise
could be an effective way of increasing motivation and sustaining growth (Surry & Land, 2000).
The literature and the results of this study suggest that computer use self-efficacy belief of individuals is a
significant factor in their utilization of technology. From the standpoint of the self-efficacy theory, the ideal
method for developing teachers' self-efficacy for computer use would be to provide them with training and
support to work successfully with technology (Albion, 1999). In designing training and support programs,
institutions might consider a number of strategies that address self-efficacy perceptions of the mainstream
faculty members. Besides providing training opportunities, building comprehensive and systematic technology
support systems is essential to increasing the faculty member’s confidence with the use of technology in teaching
and learning.
Faculty development seemed to work best when the institution had a culture pervaded by the use of technology
and supported by a wide range of strategies (Bates, 2000). One of these strategies is an extensive investment in
infrastructure. Although ownership of computers for professional/home use is almost completely diffused among
the faculty in this study, most faculty members were dissatisfied with the current investment in technology and
distribution of available resources among departments. Therefore, to encourage the adoption and diffusion of
technology, the institutions’ investment in technology should be based on a long range technology plan driven
by the institutions’ overall vision and strategy for its teaching.
Similar to most higher education institutions, research publications are the predominant or even the only
criterion for appointment, tenure and promotion in medical institutions. Medical school faculty members,
especially young faculty members who have carrier concerns are unwilling to spend time and effort to develop
technology-based applications for teaching, although they are more familiar with technology than their senior
professors. The results from this study also indicate that the lack of a reward system that recognizes faculty
efforts in technology use for teaching is an impediment for wide-spread diffusion of instructional technologies.
Therefore, to motivate faculty members in technology use, intuitions should develop reward and incentive
systems that are linked to technology use in teaching and learning such as release time, funding to support the
development of technology materials and considering instructional activities in the promotion process.
Recommendations for Higher Education Institutions
Higher education institutions today are confronted with instructional technology innovation, which is
transforming the way in which faculty and students interact and the roles they take. If the goal of the higher
education institution is the integration of technology for a transformative change, then rather than the acquisition
of technology itself, there must be a clear focus on the faculty members who use technology. For large-scale
technology integration to occur in teaching, it is essential to understand and address differentiating needs of
faculty in faculty development and support systems.
The following suggestions are offered to higher education institutions to improve their faculty members’ IT
adoption for teaching and diffusion of instructional technology in medical education:
1. Develop a long-range technology plan driven by the institutions’ overall vision and strategy for its teaching.
2. Establish a promotion system that places a high value on teaching and the use of innovative teaching
methods.
3. Design faculty development programs considering the needs of different faculty member profiles.
4. Provide training programs not only on the technical aspects of technology, but also about the integration of
technology for teaching and learning.
5. Establish an instructional technology center in which faculty members can get help from and work together
with IT related professionals.
6. Provide systematic technical and professional support.
References
Albion, P. R. (1999). Self-Efficacy Beliefs as An Indicator of Teachers' Preparedness For Teaching With
Technology. In Price, J. D., Willis, J., Willis, D. A., Jost, M., & Boger-Mehall, S. (Eds.), Technology and
Teacher Education Annual 1999, Charlottesville, VA, USA: AACE, 1602-1608.
220
Albright, M. J. (1996). Instructional Technology and Higher Education: Rewards, Rights, and Responsibilities.
Keynote Address at the Southern Regional Faculty and Instructional Development Consortium, Baton Rouge,
LA, USA, February 5 1996, ERIC, ED392412.
Anderson, T., Varnhagen, S., & Campbell, K. (1999). Faculty Adoption of Teaching and Learning Technologies:
Contrasting Earlier Adopters and Mainstream Faculty. The Canadian Journal of Higher Education, 28 (2-3), 7198.
Bates, A. W. (2000). Managing Technological Change: Strategies for College and University Leaders, San
Francisco: Jossey-Bass.
Carlile, S., & Sefton, J. (1998). Healthcare and Information Age: Implications for Medical Education. Medical
Journal of Australia, 168 (7), 340-343.
Compeau, D., & Higgins, C. A. (1995). Social Cognitive Theory and Individual Reactions to Computing
Technology: A Longitudinal Study. MIS Quarterly, 23 (2), 145-158.
Enochs, L. G., Riggs, I. M., & Ellis, J. D. (1993).The Development and Partial Validation of Microcomputer
Utilization in Teaching Efficacy Beliefs Instrument in Science Setting. School Science and Mathematics, 93,
257-263.
Garofoli, E., & Woodell, J. (2003). The Faculty Development and the Diffusion of Innovations. Syllabus, 16 (6),
15-17.
Geoghegan, W. H. (1994).Whatever Happened to Instructional Technology. Paper presented at the 22nd Annual
Conference of the International Business and Schools Computing Association, July 17-20, 1994, Baltimore,
Maryland, USA.
Green, K. C. (1999). The 1999 National Survey of Information Technology in Higher Education. The Campus
Computing Project, retrieved November 29, 2005 from http://www.campuscomputing.net/summaries/1999.
Green, K. C. (2000). The Real IT Challenge: People, Not Products. Converge, January, retrieved November 29,
2005 from http://www.centerdigitaled.com/converge/?pg=mag&issue=1:2000.
Holloway, R. E. (1996). Diffusion and Adoption of Educational Technology: A Critique of Research Design. In.
Jonassen, D. H (Ed.), Handbook of Research For Educational Communications and Technology, New York,
USA: Simon& Schuster Macmillan, 1107-1133.
Jacobsen, D. M. (1998). Adoption Patterns and Characteristics of Faculty Who Integrate Computer Technology
for Teaching and Learning in Higher Education. Doctoral Dissertation. Educational Psychology, University of
Calgary, retrieved November 29, 2005 from http://www.acs.ucalgary.ca/~dmjacobs//phd/diss/
Jacobsen, D. M. (2000). Examining Technology Adoption Patterns by Faculty in Higher Education. Paper
presented at the ACEC2000: Learning Technologies, Teaching and the Future of Schools, July 6-9, 2000,
Melbourne, Australia.
Lichty, M. (2000). The Innovation-Decision Process and Factors That Influence Computer Implementation by
Medical School Faculty. Doctoral Dissertation, Instructional Technology, Wayne State University, UMI
ProQuest Digital Dissertations, 9966157.
Marcinkiewicz, H. R. (1994). Computers and Teachers: Factors Influencing Computer Use in the Classroom.
Journal of the Research on Computing in Education, 26 (2), 221-237.
Oates, K. D. (2001). University Faculty Who Use Computer Technology. Doctoral Dissertation, Instructional
Technology, Georgia State University, USA. UMI ProQuest Digital Dissertations, 3008105.
Rogers, E. M. (1995). Diffusion of Innovations (4th Ed.), New York, USA: Free Press.
Seels, B. B., & Richey, R. C. (1994). Instructional Technology: The Definitions and the Domains of the Field,
Washington, DC, USA: Association for Educational Communications and Technology.
221
Spotts, T. H. (1999). Discriminating Factors in Faculty Use of Instructional Technology in Higher Education.
Educational Technology and Society, 2 (4), 92-99.
Surry, D. W. (1997). Diffusion Theory and Instructional Technology. Paper presented at the Annual Conference
of the Association for Educational Communications and Technology (AECT), February 12-15, 1997,
Albuquerque, New Mexico, USA.
Surry, D. W., & Land, S. M. (2000). Strategies for Motivating Higher Education Faculty to Use Technology.
Innovations in Education and Training International, 37 (2), 145-15.
222
Nykänen, O. (2006). Inducing Fuzzy Models for Student Classification. Educational Technology & Society, 9 (2), 223-234.
Inducing Fuzzy Models for Student Classification
Ossi Nykänen
Senior Researcher, Tampere University of Technology, Digital Media Institute
Hypermedia Laboratory, P.O.Box 553, FI-33101 Tampere, Finland
Tel: +358 3 3115 3544
[email protected]
ABSTRACT
We report an approach for implementing predictive fuzzy systems that manage capturing both the
imprecision of the empirically induced classifications and the imprecision of the intuitive linguistic
expressions via the extensive use of fuzzy sets. From end-users' point of view, the approach enables
encapsulating the technical details of the underlying information system in terms of an intuitive linguistic
interface. We describe a novel technical syntax of fuzzy descriptions and expressions, and outline the
related systems of fuzzy linguistic queries and rules. To illustrate the method, we describe it in terms of a
concrete educational user modelling application. We report experiments with two data sets, describing the
records of the students attending to a university mathematics course in 2003 and 2004. In brief, we aim
identifying the failing students of the year 2004, and develop a procedure for empirically inducing and
assigning each student a fuzzy property "poor", which helps capturing the students needing extra assistance.
In the educational context, the approach enables the construction of applications exploiting simple and
intuitive student models, that to certain extent are self-evident.
Keywords
Fuzzy systems, Student modelling, Linguistic interfaces, Fuzzy queries and rules
Introduction
Modern information management systems enable the recording and the management of data using sophisticated
data models and a rich set of management tools. In the context of educational systems, the information typically
includes details about learning material, the tasks and the objectives, the course information, the contact
information, the teacher and the student profiles, and the information related to student assignments, the tests, the
grades, and other records. When perceived in the technical terms of the underlying information system, it soon
becomes very difficult to manage, integrate, and access different kinds of information. In this article, we seek
means to model the imprecision of information and simplify the access to information systems, in terms of fuzzy
modelling.
In brief, computer-based learning management systems seek to be equivalent, if not superior, to the traditional
learning systems (Darbhamulla & Lawhead, 2004). However, an increased technological potential does not
automatically mean better applications. For instance, while the standardisation of the application interfaces (see,
e.g., (Dodds, 2001)) certainly improves the development of learning systems, and it is possible to semantically
integrate different sources of information, say, by using standard general-purpose query languages (Boag et al.
2005; Prud'hommeaux & Seaborne, 2005), the engagement with the technical details may itself become an
obstacle. A data format and a technical vocabulary that suits the needs of technical storage, does not necessarily
suit the needs of the developers of the more abstract learning applications, not to mention the conceptualisations
of the teachers and the students. In particularly, while it is relatively easy to store information about students'
progress technically and to classify assignments as "easy", "intermediate", or "difficult" by hand, it is
surprisingly difficult to automate the process of classifying students with respect to these semantic labels in
terms of crisp computing. Assuming a good teacher and only a handful of students, this problem is largely
irrelevant. However, when the number of the students increases or they are out of reach, seeking computersupported means becomes interesting. In addition, the ability to capture the key domain objects using intuitive
expressions not only supports the activities of the teacher, but also makes it easier to implement and integrate
learning applications based on domain-specific design.
It is instructional to consider this problem from the perspective of domain, task, and user modelling (see, e.g.,
(Brusilovsky & Cooper, 2002)). In general, computer-aided teaching, learning, and collaboration systems (as
information systems) enable the construction of various kinds of decision-support systems that help organising
courses and adapting educational content to suit the needs of different kinds of students. At the heart of such
systems lies a student (user) model, that records and explains the progress of the students, to be exploited by the
learning environment. The basic requirement of student modelling is the ability to assign individuals and user
groups meaningful labels related to their characteristics and activities, to be exploited on the level of the reacting
223
application. Historically, the development of modelling has evolved from crisp to fuzzy models, eventually
taking context explicitly into account (Jameson 1995; Jameson 2001). Nevertheless, a good user model is
intuitive and simple, allows sharing domain knowledge, and matches the requirements of the modelling task.
The introduction of fuzziness typically aims managing imprecision (or vagueness) in applications. In short, the
existing fuzzy systems fall into two categories. While fuzzy logic or fuzzy expert systems consider fuzziness in
terms of fuzzy implication and a generalisation of crisp (two or finitely-valued) logic, fuzzy control systems aim
reproducing the behaviour of intuitive control rule groups by exploiting fuzzy models, i.e., computing with fuzzy
sets (Turunen, 1999; Jang & Sun, 1997; Hüllermeier, 2005; Cox, 2005). Further, the fact that fuzzy membership
functions appear often overly precise in applications has motivated the development and application of, e.g.,
interval-valued and type-2 fuzzy sets (Vaucheret, 2002; Mendel, 2001). In the context of educational
applications, the reported soft computing experiments include applications of Bayesian and Neuro-Fuzzy
methods (Jameson, 1995; Xenos, 2004; Stathacopoulou et al., 2005). These methods have been widely applied,
taking many aspects of education into account. The reported applications notably demonstrate analysing and
evaluating both students based on individual assessments, students' views, education quality, grades of journals,
and the performance of entire academic departments (Ma & Zhou, 2000; Liu & Maes, 2004; Suarez, 2003;
Nokelainen et al., 2001; Arriaga et al., 2005; Ibrahim, 2001; Turban et al., 2000; Lopes & Lanzer, 2002). The
claimed advantages of fuzzy modelling range from faithfully evaluating students' learning and cognitive abilities
to moving towards personalised education (Stathacopoulou et al. 1999; Weon & Kim, 2001; Kavcic et al., 2003).
In this article, we report an approach of implementing a predictive fuzzy system that manages capturing both the
imprecision of the empirically induced classifications and the imprecision of the intuitive linguistic expressions
via the extensive use of fuzzy sets. We describe a novel technical syntax of fuzzy descriptions and expressions,
and outline the related systems of fuzzy linguistic queries and rules. In the educational context, the approach
provides a concrete way of inducing intuitive semantic labels from the existing data archives, suitable for
capturing the domain objects (users, user groups, processes, tasks, and artefacts) using fuzzy linguistic
expressions. The basic idea is to describe the domain objects and the associated linguistic expressions in terms of
neutral fuzzy sets, subject of fuzzy queries and rules. For the benefit of a concrete discussion, we describe
implementing a novel predictive fuzzy system that demonstrates fuzzy modelling in the context of a realistic
application. We report experiments working with two data sets, describing the records of the students attending
to a particular university course of mathematics in 2003 and 2004 at the Tampere University of Technology,
Finland. In short, we aim predicting the poor students based on the archives, and semantically integrate this
information with fuzzy linguistic rules and queries.
The main contribution of this article is threefold: First, we introduce a design of a realistic predictive fuzzy
system that provides a basis for semantically integrating and encapsulating various kinds of information systems.
Second, we develop the syntax of fuzzy expressions by introducing the notions of fuzzy linguistic constructors
and pure linguistic concepts, and describe the related rule and query systems. Third, we provide a concrete
bridge between the two complementary aspects of fuzziness in educational design, by syntactically integrating
the imprecision due to lack of information (prediction accuracy) with the imprecision due to use of informal
language (fuzzy linguistic expressions).
The rest of this article is organised as follows: Section 2 describes the empirical data sets and Section 3
demonstrates the potential of predicting the failing students. Section 4 presents the design of predictive fuzzy
systems and Section 5 outlines the definitions related to fuzzy queries and rules. Section 6 briefly considers the
applicability of model-free methods, and finally, Section 7 concludes the article with few notes.
Empirical Data
The success user modelling applications is determined by the quality of the user data. In the context of learning
management applications, the backbone of modelling may be economically compiled from the formally recorded
achievements. Our experiments were conducted using the accumulated data sets from the well-established basic
course Engineering Mathematics I, during two successive years 2003 and 2004.
The data describes the performance of the individual students in terms of 22 attributes: points from 13
assignments (0-10, 'x' denotes a missing value), four mid-term exams (two attributes of 0-6 points each, 'x'
denotes a missing value), and the final grade (0-5). The final grades were compiled using the attributes and
scaled according to the final statistics. Compiled into chronological order, this approach provides a vector of 22
224
attributes for each student ([a1, a2, a3, e1, e2, a4, a5, a6, a7, e3, e4, a8, a9, a10, e5, e6, a11, a12, a13, e7, e8, g ]).
Considering ongoing courses, the attribute vectors become available gradually.
The data includes 358 student instances considering the year 2003, and 311 instances considering the year 2004.
This provides two data sets D2003 = [αij], i = 1, 2, ,..., 358, j = 1, 2, ..., 21, and D2004 = [βij], i = 1, 2, ,..., 311, j = 1,
2, ..., 21, accompanied with the final grades [gi], i = 1, 2, ,..., 358, and [hi], i = 1, 2, ,..., 311, respectively. We can
use this information to define a training data set for the concept failing student, [g'i], where g'i = 1 when gi = 0
and g'i = 0 otherwise (define [h'i] accordingly). The training data sets [αij|g'i] and [βij|h'i] may now be used for
recognising the failing students. In particularly, the construction of the data sets enable using the first data set
D2003 as training data for predicting the second data set D2004.
100
120
90
100
80
80
60
Students
Students
70
50
40
60
40
30
20
20
10
0
0
0
1
2
3
4
5
0
1
2
3
4
5
Grades
Grades
(a)
(b)
Figure 1. The grade histograms of 2003 and 2004
Figures 1a and 1b depict the grade frequencies of the years 2003 and 2004. Of course, the data from the year
2003 does not completely characterise the year 2004, even if we assume that the course format remained
unchanged. In principle, any change in the course agenda makes it hard to use the data from the previous year for
the benefit of the next year. Changes might notably include (and partly did include) change of topics, providing
extra tutoring for the poorly performing students, extra tests and questionnaires, and simply showing statistics
like Figure 2 to the students (please see below).
Using Training Data in Applications: Making Predictions
Considering applications that rely on user modelling in the educational domain, perhaps the most important use
case is classifying the students based on their observed and expected performance. This yields categories of
fuzzy user groups providing decision-support information for various applications. For instance, consider the
expression "student is very poor" in the context of the following informal rule as a part of the adapting
application:
If a student is very poor and lives far from the campus,
1) then the need for extra attention is high.
In order to establish rules, the related fuzzy terms must be defined. In this article, we assign students (as
identifiable domain objects) fuzzy descriptions which are to be associated with fuzzy linguistic constructors (as a
sort of hedges) in rules and queries. Let us first briefly consider the predictive component of the process of
inducing fuzzy descriptions.
In our application, it is possible to establish a straightforward procedure for recognising the failing students
using a simple domain-specific design, simply by assuming average performance: A student fails the course if
she does not accomplish at least 40% of the assignments or gets less than 24 points from the mid-term exams
(the point limit may decrease based on the final statistics). Additional requirements include attending to at least
four computer exercises and completing a preliminary test. The additional requirements were not used in the
predictions in this study. Accompanied by the fact that the grades were scaled based on the final statistics, this
actually makes the prediction problem somewhat harder.
225
1,2
Accuracy
1
0,8
2003
2004
0,6
0,4
0,2
21
19
17
15
13
11
9
7
5
3
1
0
Attributes
Figure 2. Prediction accuracy of the years 2003 and 2004
Figure 2 depicts a demonstration of using the series of the first 1, 2, ..., 21 attributes of D03 and D04 for predicting
the concept failing student by simply assuming average performance. In the experiment, the data was divided
into k sets, according to the number of the attributes used in the prediction. The experiment shows that a naïve
classifier, relying upon domain-specific information and simple design insight, manages to predict, or recognise,
the failing students with a relatively high accuracy.
Informally speaking, Figure 2 provides a clear message for the students struggling to pass courses. When grades
are considered, failing performance can be reasonably well recognised after mid-course. Of course, this data
alone can not tell what to do in order to improve the situation, expect for the "work harder" signal. In addition,
when separated from the more decisive contextual characteristics, the recorded points and the mid-term exams
do not necessarily faithfully reflect the students' skills nor understanding. Considering fuzzy systems, however,
the experiments seem to support the rationale of implementing predictive fuzzy models, since the trend of the
prediction accuracy is clearly positive.
Implementing a Predictive Fuzzy System
Assume we wish to assert rules and perform queries on top of a database, using fuzzy linguistic expressions. For
instance, a teacher might want to establish heuristic rules like (1) and search for the students needing extra
attention. Further, assume all the information is not readily available and we have to rely on generalisations
learned from the historical archives. For instance, the points from assignments become available only during
courses. In practice, we face this task during the internal design of an adapting educational system.
Trend of the
Prediction
Accuracy Accuracy
IF s.a is rather A AND s.b is very B
THEN s.c is very C
...
Attributes (k)
Training Data
α1,1 α1,2 α1,3 ...
α2,1 α2,2 α2,3 ...
...
g1
g2
Fuzzy Rules
Predictive
Model
notAtAll notVery moderately rather very
Definitions of the Fuzzy
Linguistic Constructors
Fuzzy Descriptions
Fuzzy Knowledge Base
Observed Instance Data
s1: β1,1 β1,2 ... β1,k ? ?
s2: β2,1 β2,2 ... β2,k ? ?
...
?
?
si :
poor
Application
Default Knowledge Base
Figure 3. The basic components of a predictive fuzzy system
Figure 3 depicts the basic components of a predictive fuzzy system. At the heart of the systems lies the
predictive model which assigns domain objects (e.g. students si) fuzzy descriptions, based on the available
226
training data and the observed instance data from the application. The fuzzy descriptions add to the default
knowledge base (e.g. a default thesaurus defining the basic concept taxonomies and the terminological
synonyms), providing a basis for interpreting the fuzzy rules and queries. The linguistic component is defined by
identifying the linguistic concepts to which the descriptions apply (e.g. poor), and establishing the fuzzy
linguistic constructors (e.g. rather), to be used in the rules and queries. In brief, applications use the predictive
fuzzy system for performing fuzzy inference and making fuzzy linguistic queries. This might also include, e.g.,
interpreting fuzzy SQL queries (see, e.g., (Cox, 2005)). More complex applications might in addition explicitly
control the default knowledge base and the definitions of the fuzzy linguistic constructors, in terms of
bootstrapping the predictive fuzzy system with application-specific defaults (e.g., modifying the fuzzy linguistic
constructors upon context).
Let us next consider the task of establishing the fuzzy linguistic expressions for fuzzy linguistic rules and queries.
For technical purposes, let us equate, in the context of students attending a course, the concepts failing and very
poor. Effectively, this provides a fuzzified notion that enables capturing the students that are "close to" failing.
Turned around, a student that passes courses with (nearly) maximal grades is not at all poor.
Technically speaking, we would thus like to assert, e.g., the following kinds of expressions concerning students:
T = { not at all poor, not very poor, moderately poor, rather poor, very poor }. In a typical fuzzy application,
these labels would be modelled in terms of a linguistic variable, using the elements of T as the term set (see, e.g.,
(Jang & Sun, 1997; Mendel, 2001; Cox, 2005)). Alternatively, however, the expressions include internal
structure: a constructor part, and a concept part. The constructor resembles the notion of a hedge or a modifier
while the concept part identifies the (pure) linguistic concept that defines the type of the particular expression.
For instance, instead of asserting
s is RatherPoor,
(2)
we may assert
s isRather Poor.
(3)
The subtle syntactical difference becomes noteworthy when considering the constructors in the context of crisp
classes. Two practical consequences are particularly significant: First, when compared to (2), separating the
constructor and the concept part in (3) reduces the number of the semantic labels required in applications since
the constructors may be associated with several concepts. Second, since the shape of the associated fuzzy set
does not have to follow the limitations of simple functional hedges (e.g., concentration for "very"), the
constructors may be defined empirically. In particularly, this includes the case of taking carefully into account
the actual use of particular semantic labels in applications (about survey-based terms, see Mendel (2001)).
notAtAll
notVery
moderately
rather
very
(1, 1)
Coupling
(0, 0)
Degree of Association
Figure 4. A fuzzy linguistic constructor sequence
Figure 4 depicts a fuzzy linguistic constructor sequence that models the constructor terms. In short, a constructor
may be perceived as a predicate that assigns a relationship between an subject and an object (see (3)), in terms of
a (type-2) fuzzy set. This provides a proper generalisation of the is-a relationship or a crisp membership function
and is suitable for describing the relationship between any domain objects and the selected linguistic concepts
(e.g., students and the concept poor). Using fuzzy sets in modelling thus allows capturing imprecise descriptions
which establishes an extension to description logic (see (Baader et al., 2002)). A singleton fuzzy set coincides
with the plain (scalar) degree of fuzzy membership while fuzzy sets of other shapes enable modelling, e.g., the
imprecision of the fuzzy definition itself. It is worth emphasising that while in this article we use fuzzy linguistic
expressions in the syntax of the rules and the queries, they also enable recording imprecise assertions as well227
established statements. However, the considerations related to the assertion context are not trivial and are not
discussed here.
Let us then address the question of adequately modelling the semantic descriptions of the domain objects in the
context of the course application. In brief, we wish to assign students a fuzzy property poor, using a predictive
model. We do this by assigning each student identifier a fuzzy description that associates the identifier with the
linguistic concept poor in terms of a type-1 fuzzy set (see Figure 5 below). In queries and rules, we may then
compute with these properties using the related term poor (establishing the type of the expression) and a
linguistic constructor, e.g., "moderately" in expressions (establishing the meaning of the constructor term as a
fuzzy set).
(1, 1)
c ~ Centroid of
Association
Coupling
p ~ Imprecision
c
(0, 0)
p
Degree of Association
Figure 5. A fuzzy model for a student being poor
We say that the fuzzy set depicted in Figure 5 is neutral. This is due to the fact that the domain of the
membership function is normalised to [0, 1] and is not assigned any particular datatype information. Datatype
information, however, might be required for establishing the definition via bindings to explicit quantitative data
(e.g., predicted points of a course, distance in meters, etc.). Neutral fuzzy sets provide an intuitive explanation of
the concepts imprecision (or expectancy, modelled by p), association (centroid of which is denoted by c),
coupling (the max height of the membership function), precise, and crisp. In particularly, a type-1 fuzzy set is
called precise if it is defined by a singleton membership function (p=0). Precise fuzzy sets for which c = 0 or c =
1 are called crisp. Note that the definitions of association and coupling are not traditionally introduced in terms
of type-1 fuzzy sets; they become meaningful only when interpreted in the context of the following rule and
query semantics (or in the context of type-2 fuzzy sets).
2
1,8
1,6
1,4
1,2
1
0,8
0,6
0,4
0,2
0
21
19
17
15
13
11
9
7
5
3
2003
2004
1
Average Error
The experiment yielding Figure 2 provides a basis for modelling students being poor in terms of two
observations. First, the predicted or the projected performance enables the definition of an appropriate semantic
label in terms of a recognition task. Second, the precision of the assigned semantic labels is not a constant: it
changes according to the available information or according to the estimated accuracy of the predictions. As a
consequence, we may capture students throughout courses with a fixed set of semantic labels that get more
precise as more information becomes available. In other words, the fuzziness of the application to certain extent
decreases in time.
Attributes
Figure 6. The average error of predicting the exact grades
228
It seems obvious that we may try to categorise students directly by predicting the grades. As suspected, assuming
average performance provides a concrete method for doing this. Figure 6 demonstrates an example of trying to
predict the exact grades. The average errors from using the data of 2003 and 2004 follow a similar pattern. It
seems that the difference can be explained by the statistical corrections of the grades. After the second mid-term
exam, the grades can be predicted with the average error of 0.8. Indeed, the experiments (see Figures 2 and 6)
seem to demonstrate that the naïve prediction algorithm provides systematic results and that we may use the
archives for estimating the prediction errors reliably.
Considering the application, we may describe the students of the year 2004 using the fuzzy property poor, based
on the above definitions. Thus, as the course proceeds, we receive more information about the students (in terms
of attributes) and may classify the students more accurately. In brief, the application involves two kinds of
imprecision. The first flavour of imprecision is due lack of information. The second is due choice. Roughly
speaking, a student may change her projected trajectory by studying harder (or less harder). However, when
considering getting a grade via the mid-term exams, the window of influence obviously gets narrower as the
course proceeds.
We will next describe a simple algorithm that can be used to associate each student a fuzzy description. It might
be helpful to imagine that we assign each student s a triangular neutral fuzzy set Ãs of the type poor. The shape
of the related membership function μs can be described in terms of two parameters, centroid of the degree of the
association, s.c, and imprecision, s.p: μs (x) = μs (x; s.c, s.p). By letting u = s.c - s.p - ε, v = s.c, and z = s.c + s.p
+ ε, 0 < ε << 1, we get the conventional definition (Jang & Sun, 1997):
μs (x) = triangle(x; u, v, z) = max ( min ( (x-u)(v-u)-1, (z-x)(z-v)-1 ), 0), x ∈ [0, 1].
(4)
Let s.x ∈ X = { 0, 1, ..., 54 } denote the number of points student s will earn from the mid-term exams and the
exercises. By doing sufficiently exercises, students may earn max 54 - 48 = 6 bonus points to be added to the
results of the mid-term exams. In practice, s.x = s.x(k) where k is the number of the attributes available during
the course. Thus, s.x is a projected value, based on observing the performance of the past and assuming an
average performance in the future. Define a function s.c: X → [0, 1], for modelling the (centroid of the) degree
of association, associating the student s with the concept poor:
s.c(s.x) = max( 0, min( 1, (24 - (s.x-24) ) / 24 ) ).
(5)
Let Error(k) denote the error of the prediction of the grades during the previous year(s), using k attributes (see
Figure 6). Define a function s.p: K → E ⊂ ℜ, where k ∈ K = { 1, 2, ..., 21 } is the number of the attributes
available, and E denotes the set of errors, for modelling the imprecision of the association:
s.p(k) = Error(k) / 5.
(6)
We can now assign each student s a pair (c, p)s which models her being poor. Figure 6 depicts the development
of the imprecision as more information becomes available. Intuitively, the pair (c, p)s may be interpreted as a
neutral type-1 fuzzy set of certain shape (for triangular functions, consider (4) and Figure 5). The shape,
however, must be decided by the application designer.
The definitions (5) and (6) establish the induced fuzzy properties. For instance, the fuzzy set depicted in Figure 5
suffices modelling a student s when s.c = 0.8 and s.p = 0.2. Considering the binding, this happens when s.x =
28.8 and Error(k) = 1.0.
Fuzzy descriptions provide intuitive labels of data in terms of the fuzzy property poor. As an example, Figure 7
depicts classification trajectories describing six particular students that took the year 2004 course with the
grades 0, 1, 2, 3, 4, and 5. In brief, the trajectories denote the association of the students being poor, suitable for
fuzzy modelling. The imprecision associated with the fuzzy models may be read from the Figure 6, using the
errors of the year 2003. In other words, the classifications get rather precise around mid-course. Intuitively, the
analysis reveals that the student s3 seems to improve her performance during the course while the student s1 is
doing worse and worse as the course proceeds. Of course, more information is needed for explaining why the
behaviour occurs.
Other fuzzy properties may be modelled and studied in a similar fashion. For instance, a fuzzy property passive
can be induced from the attendance statistics. Examples of completely crisp properties include the number of
credit units, sex, age, etc. Finally, students might also be explicitly asked further attributes for various properties
229
that can help categorising them. For instance, learning style, interest towards the subject, estimated effort, time to
travel to the campus, studying at home, and being active or passive in the course. As the user models become
more refined, it is possible to model and interpret students' behaviour more faithfully. Note however, that these
extra sources of information inevitably introduce new sources of fuzziness as well, in particularly the
imprecision related to communication. Nevertheless, using neutral fuzzy sets for modelling these properties
provides a well-established reference system for various kinds of applications. The chief benefit is the ability to
model a wide range of fuzzy properties in terms of uniform descriptions. From the perspective of fuzzy systems
and applications, this helps integrating information from various sources. Associated with the idea of linguistic
constructors, the approach provides a concrete way to describe domain objects using a relatively small set of
intuitive linguistic concepts and expressions.
Association of Being Poor
1,2
1
s0
s1
s2
s3
s4
s5
0,8
0,6
0,4
0,2
21
19
17
15
13
11
9
7
5
3
1
0
Attributes
Figure 7. The classification trajectories of five particular students
Notes About Implementing Fuzzy Linguistic Queries and Rules
In the previous section, we demonstrated the construction of a predictive fuzzy model yielding a simple
knowledge base of fuzzy descriptions. However, the model and the rationale behind the associated technical
definitions can not be appreciated unless the related query and the rule applications are outlined as well. In order
to illustrate the technical use cases, let us next very briefly reconsider fuzzy properties in the context of linguistic
expressions, namely fuzzy linguistic queries and rules (see (1)).
A typical use case of educational decision-support systems might be retrieving a list of students that are likely to
fail the course, i.e. a list of poor students as a fuzzy user group. Consider the following informal fuzzy linguistic
query:
Find students that are at least rather poor and very passive.
(7)
The expression (7) can be formalised in terms of conjuncted match patterns where match patterns may include
an associated modifier part (at least, around, at most).
Formally speaking, the expression (7) establishes a sequence of two match patterns <Aj> = <atLeast(rather),
very> (where, in general, j = 1, 2, .., m) now associated with the fuzzy properties <pj> = <poor, passive>.
Assuming each student s = 1, 2, ..., n is appropriately associated with the related fuzzy properties <vsj>, j = 1, 2,
..., m, we may compute the match for each student and rank the students according to the matches.
Perhaps the most straightforward method for computing fuzzy matches is due to aggregating the firing strengths
computed from the matching sequences of the fuzzy sets. Let Aj and vj denote two neutral fuzzy sets, associated
with the fuzzy membership functions μAj: [0, 1] → [0, 1] and μvj: [0, 1] → [0, 1]. Define
wj = maxx μAj(x) ∧ μvj(x).
(8)
This definition demonstrates linearity which is useful in implementations. We say that wj denotes the firing
strength of <Aj, vj> and define the fuzzy match of <Aj, vj> as:
230
fMatch( <Aj, vj> ) = minj <wj>.
(9)
In other words, the fuzzy match is a value between 0 and 1, suitable to be used to order the students based on the
query. There exists other strategies for computing fuzzy matches as well, e.g., aiming to preserve the aspect of
imprecision of the matches. For brevity, we ignore the other definitions.
Fuzzy linguistic queries are useful but do not provide good syntactic means for using the information further.
Sometimes it is useful to assign students new fuzzy properties based on the existing ones. For instance, one
might wish to assert the following simple fuzzy linguistic rule:
IF a student is at least rather poor AND very passive,
THEN she is problematic.
(10)
It is easy to see that fuzzy queries establish a special case of (simple linguistic) fuzzy rules. For instance, query
(7) might be equivalently modelled in terms of assigning each student a new fuzzy property, problematic. The
key difference, however, is that the new property problematic can be utilised in further queries and rules, to be
aggregated with other properties of the same type. With certain restrictions, simple fuzzy linguistic queries
provide the semantics of simple fuzzy linguistic rules. This is due to the fact that it is possible to interpret the
fuzzy match as a fuzzy set; in our simple example, a fuzzy singleton would do.
In general, more complex fuzzy rules may be implemented in terms of (control) rule groups. With certain
assumptions, rule groups may be modelled using (slightly modified) Mamdani fuzzy systems (see (Jang & Sun,
1997; Cox, 2005)). The key differences between simple fuzzy linguistic rules and rule groups lie in descriptional
complexity and coupling. Rule groups enable the modelling of more complex relationships than simple linguistic
rules, but by default, the coupling (max height) of the implied fuzzy properties might decrease during the
inference (outputting non-normal fuzzy sets). This causes certain difficulties related to interpretation in
applications. For brevity, we will not consider rule groups further in this article.
From Domain-Specific Design to Model-Free Methods
There are several ways to implement predictive fuzzy models. In particularly, even if applying domain-specific
design (e.g., assuming average performance) may provide the best results, it is not always feasible nor necessary.
In principle, there exists a wide range of machine learning, data mining, and exploration methods suitable for the
task (Russel & Norvig, 1995; Mitchell, 1997; Cox, 2005).
We describe elsewhere a study and a method for inducing membership functions of fuzzy sets empirically upon a
training set of positive and negative examples (Nykänen, 2004). In short, the study provides a method that
generalises concept learning to include learning fuzzy concepts. The basic idea is to train a redundant Decision
Tree (DT) to classify data according to a goal attribute that denotes the related crisp concept, eventually
representing the positive end of the bipolar fuzzy concept. The membership grades of the instances are then
computed from the topology of the tree, based on the depth of the tree and the number of attribute tests required
for recognising a negative instance. Assuming the crisp concept can be learned well, the method allows
assigning objects fuzzy membership degrees with an information-theoretic explanation.
Let us next briefly describe an approach for implementing a model-free fuzzy system. A decision tree was
trained for each j = 1, 2, ..., k data sets, using [αij | g'i], i = 1, 2, ..., 180 as the training data for predicting [h'i], i =
1, 2, ,..., 311, based on the instance vector [βij], i = 1, 2, ,..., 311. The available training data was simply split into
half, in order to minimise the negative effect of overfitting. Figure 8 depicts an example of learning the concept
failing student for the year 2004 with a rudimentary ID3 decision tree learning algorithm (for details, please see
(Mitchell, 1997)). The experiment shows that even if blind concept learning is indeed possible (using a
transparent DT learner), the naïve classification method clearly outperforms it.
Thus, while the performance of recognising the failing students is better than chance, the prediction accuracy
does not seem sufficient for inducing good descriptions. The situation might be improved by encoding the data
differently, strengthening the algorithm, and introducing appropriate domain heuristics (Murthy, 1997). Of
course, the other more powerful concept learning algorithms, such as neural networks or other equally expressive
statistical methods, might perform better. It is worth noticing, however, that by definition, no statistical learning
algorithm is able to demonstrate positive performance in every domain (Schaffer, 1994).
231
1,2
Accuracy
1
0,8
Decision Tree
Naive Classification
0,6
0,4
0,2
21
19
17
15
13
9
11
7
5
3
1
0
Attributes
Figure 8. A comparison of the prediction accuracy
Conclusions
We have reported a design and a demonstration of a predictive fuzzy system that enables the description of the
domain objects with fuzzy descriptions, suitable for fuzzy linguistic rules and queries. We have discussed the
method in the context of an educational application, providing a way to capture students with fuzzy descriptions
based on accumulative attribute sequences. We have developed the notion of neutral fuzzy sets, and introduced a
system of fuzzy linguistic constructors, to model the structure of fuzzy linguistic expressions in rule and query
expressions and logical statements. In brief, this construction relies upon the notion of pure linguistic concepts,
providing a fuzzy extension to the relationship.
The design of our system demonstrates two pivotal aspects of fuzziness in applications: While we welcome
fuzziness in linguistic expressions because it enables, e.g., writing useful queries using intuitive expressions, we
would nevertheless appreciate our knowledge base to be completely precise. For instance, we would appreciate
an exact, a priori characterisation of the poor students – which is of course impossible because such data is
simply not available.
Motivation for our work is due rather practical considerations. As more information becomes available in
applications, it gets difficult to set up user models and formulate expressions solely using the crisp design
properties and the technical terms. Developing means to capture domain objects with fuzzy descriptions provides
a concrete alternative that helps managing the built-in complexity in applications. Using intuitive semantic labels
and expressions in modelling should also help making more transparent systems, thus supporting the activities of
the end-users more openly. This process may be perceived as an activity of constructing (fuzzy) linguistic (enduser) interfaces. From the end-users' point of view, the main contribution thus lies in the potential of using
intuitive expressions, without the need of adopting expert user interfaces, or assuming a strict terminology and
in-depth understanding about the encapsulated data model. Note that in order to simplify the discussion we have
ignored, e.g., the issues related to contextual interpretation (e.g., bootstrapping of the constructors) and heuristic
modelling (selection of the labels) in this treatment.
The quality of our fuzzy system is determined by the (induced) fuzzy model and the rule heuristics. For instance,
the early classifications are rather imprecise. However, since this imprecision is transparent, the approach
manages to escape several problems related to crisp modelling. In particularly, the system does not force crisp
nor overly precise fuzzy classifications, and the "minor" errors in the classifications are not severe, due to fuzzy
rules and queries. The approach is quite general and suits the needs of various application domains that require
inducing fuzzy classifications and decorating archives with intuitive labels. In fact, we have developed the notion
of our predictive fuzzy systems in the context of general-purpose Semantic Web technologies (Nykänen, 2005).
Finally, we wish to emphasise that it is not clear whether the introduced semantic categories should be made
publicly available, even if it might support, say, the creation of certain kinds of student groups. Indeed, being
"accurate" or not, people might find certain semantic labels uncomfortable or even stigmatising. In turn, this
highlights the subtle issues related to user modelling in the fundamentally social context of humans.
232
Acknowledgements
The author wishes to thank the teachers of the course Engineering Mathematics I, in particular Dr. Lasse
Vehmanen, for the opportunity to use the anonymous records of the years 2003 and 2004 in this study.
References
Arriaga, F. de, Alami, M. El., & Arriaga, A. (2005). Evaluation of Fuzzy Intelligent Learning Systems. In
Méndez-Vilas, A. & González, J. A. M. (Eds.), Proceedings of m-ICTE2005, June 7-10, Spain, retrieved
November 24, 2005, from http://www.formatex.org/micte2005/200.pdf.
Baader, F., Calvanese, D., McGuiness, D. L., Nardi, D., & Patel-Schneider, P. F. (2002). Description Logic
Handbook, Cambridge University Press.
Boag, S., Chamberlin, D., Fernández, M. F., Florescu, D., Robie, J., & Siméon, J. (2005) (Eds.) XQuery 1.0: An
XML Query Language, W3C Working Draft 04 April 2005, retrieved November 24, 2005 from
http://www.w3.org/TR/2005/WD-xquery-20050404/.
Brusilovsky, P., & Cooper, D. W. (2002). Domain, task, and user models for an adaptive hypermedia
performance support system. In Proceedings of the 7th International Conference on Intelligent user interfaces,
San Francisco, California, USA: ACM Press, 23-30.
Cox, E. (2005). Fuzzy Modeling and Genetic Algorithms for Data Mining and Exploration, USA: Elsevier.
Darbhamulla, R., & Lawhead, P. (2004). Paving the way towards an efficient Learning Management System. In
Seong-Moo Yoo, S-M. & Etzkorn, L. H. (Eds.), Proceedings of the 42nd annual Southeast regional conference,
Huntsville, Alabama, USA: ACM Press, 428-433.
Dodds, P. (2001). Sharable Content Object Reference Model (SCORMTM) Version 1.2: The SCORM Overview,
Advanced Distributed Learning, retrieved November 24, 2005 from http://www.adlnet.org.
Hüllermeier, E. (2005). Fuzzy methods in machine learning and data mining: Status and prospects. Fuzzy Sets
and Systems, 156 (3), 387-406.
Ibrahim, A. M. (2001). Assessment of Distance Education Quality Using Fuzzy Sets Model. In Proceedings of
the SEFI Annual Conference, Copenhagen, 12-14 September, 2001 (in CD).
Jameson, A. (1995). Numerical Uncertainty Management in User and Student Modeling: An Overview of
Systems and Issues, User Modeling and User-Adapted Interaction, 5 (3-4), 193-251.
Jameson, A. (2001). Modelling both the Context and the User, Personal and Ubiquitous Computing, 5 (1), 2933.
Jang, J-S. R., & Sun, E. M. (1997). Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning
and Machine Intelligence, USA: Prentice Hall.
Kavcic, A., Pedraza-Jimenez, R., Molina-Bulla, H., Valverde-Albacete, F. J., Cid-Sueiro, J., & Navia-Vazquez,
A. (2003). Student Modelling Based on Fuzzy Inference Mechanisms. In Zajc, B. & Tkalcic, M. (Eds.),
Proceedings of the EUROCON 2003, Ljubljana, Slovenia, 379-383.
Liu, H., & Maes, P. (2004). What would they think? a computational model of attitudes. In Vanderdonckt, J.,
Nunes, N. J. & Rich, C. (Eds.), Proceedings of the 9th International Conference on Intelligent User Interfaces,
Funchal, Madeira, Portugal, USA: ACM Press, 38-45.
Lopes, A. L. M., & Lanzer, E. A. (2002). Data envelopment analysis - DEA and fuzzy sets to assess the
performance of academic departments: a case study at Federal University of Santa Catarina - UFSC, Pesquisa
Operacional, 22 (2), 217-230.
233
Ma. J., & Zhou, D. (2000). Fuzzy Set Approach to the Assessment of Student-Centered Learning, IEEE
Transactions on Education, 43 (2), May 2000, 237-241.
Mendel, J. M. (2001). Uncertain Rule-Based Logic Systems: Introduction and New Directions, USA: Prentice
Hall.
Mitchell, T. (1997). Machine Learning, USA: McGraw-Hill.
Murthy, S. K. (1997). On Growing Better Decision Trees from Data. Doctoral dissertation, University of
Maryland, retrieved June 21, 2005 from http://www.cbcb.umd.edu/~salzberg/docs/murthy_thesis/thesis.ps.gz.
Nokelainen, P., Silander, T., Tirri, H., Nevgi, A., & Tirri, K. (2001). Modeling Students' Views on the
Advantages of Web-Based Learning with Bayesian Networks. In Ruokamo, H., Nykänen, O., Pohjolainen, S. &
Hietala, P. (Eds.), Proceedings of the 10th International PEG Conference, June 23-26, 2001, Tampere, Finland,
101-108.
Nykänen, O. (2004). Fuzziness as a Recognition Problem: Using Decision Tree Learning Algorithms for
Inducing Fuzzy Membership Functions. In Zanasi, A., Ebecken, N. F. F. & Brebbia, C. A. (Eds.), Data Mining
V, UK: WIT Press, 143-153.
Nykänen, O. (2005). Implementing Semantic Web Agents that Learn Upon Experience: A Machine Learning
Module for the CWM Rule System. In Guimaraes, N. & Isaias, P. (Eds.), Proceedings of the IADIS International
Conference on Applied Computing, 22-25 February, 2005 Algarve, Portugal, 409-416.
Prud'hommeaux, E., & Seaborne, A. (2005). SPARQL Query Language for RDF, W3C Working Draft 19 April
2005, retrieved November 24, 2005 from http://www.w3.org/TR/2005/WD-rdf-sparql-query-20050419/.
Russel, S., & Norvig, P. (1995). Artificial Intelligence: A Modern Approach, USA: Prentice-Hall.
Schaffer, C. A. (1994). Conservation Law for Generalization Performance. In Proceedings of the International
Conference on Machine Learning, San Mateo, Ca., USA: Morgan Kaufmann, 259-265.
Stathacopoulou R., Magoulas G. D., & Grigoriadou, M. (1999). Neural network-based fuzzy modeling of the
student in intelligent tutoring systems. In Proceedings of the INNS-IEEE International Joint Conference on
Neural Networks, Washington, USA, July 1999, retrieved December 5, 2005 from
http://www.dcs.bbk.ac.uk/~gmagoulas/JCNN0679.PDF.
Stathacopoulou, R., Magoulas, G. D., Grigoriadou, M., & Samarakou, M. (2005). Neuro-fuzzy knowledge
processing in intelligent learning environments for improved student diagnosis. Information Sciences, 170 (2-4),
273-307.
Suarez, J. (2003). Student evaluation trough membership functions in CAT systems. Revista Mexicana De
Fisica, 49 (4), 371-378.
Turban, E., Zhou, D., & Ma. J. (2000). A Methodology for Evaluating Grades of Journals: A Fuzzy Set-based
Group Decision Support System. Paper presented at the 33rd Hawaii International Conference on System
Sciences, January 4-7, 2000, Maui, Hawaii.
Turunen, E. (1999). Mathematics Behind Fuzzy Logic, Heidelberg, Germany: Physica-Verlag.
Vaucheret, C., Guadarrama, S., & Muñoz. S. (2002). Fuzzy Prolog: A Simple General Implementation using
CLP(R). Paper presented at the 9th International Conference on Logic for Programming Artificial Intelligence
and Reasoning, October 14-18, 2002, Tbilisi, Georgia.
Weon, S., & Kim. J. (2001). Learning Achievement Evaluation Strategy Using Fuzzy Membership Function.
Paper presented at the 31st ASEE/IEEE Frontiers in Education Conference, October 10-13, 2002, Reno, NV,
USA.
Xenos, M. (2004). Prediction and Assessment of Student Behaviour in Open and Distance education in
Computers Using Bayesian Networks. Computers & Education, 43 (4), 345-359.
234
Suhonen, J. (2006). Book review: Using Technology in Teaching (William Clyde and Andrew Delohery). Educational
(Book Review)
Reviewer:
Jarkko Suhonen
Senior Lecturer
Department of Computer Science
University of Joensuu, Finland
[email protected]
Textbook Details:
William Clyde and Andrew Delohery
http://yalepress.yale.edu/yupbooks/book.asp?isbn=0300103948
Yale University Press, New Haven and London
ISBN: 0-300-10394-8
2005, 256 pages + includes CD-ROM
Overview
The book “Using technology in teaching” is about using common technology to support everyday teaching
activities. The authors of the book are William Clyde, dean of academic technology (finance teacher), and
Andrew Delohery, director of the Learning Center (English composition teacher), both at Quinnipiac
Univerisity, Hamden, Connecticut, United States. The aim of the book is to give practical ideas to instructors or
teachers for enhancing their teaching with technology. The authors have divided the book into nine chapters
organized around instructional goals and commonly encountered tasks and problems in teaching. Each chapter
presents a different instructional activity common in many educational contexts, such as distribution of the
course materials to the students, collaboration as a way to improve learning and learning through experiences.
The chapters introduce several technological solutions to support the presented activities. The solutions are
mainly based on technological tools that are widely used and easily available, such as emails, word processors,
and Course Management Systems (CMS).
Chapter Summary
Each activity (e.g. chapter) is presented through several cases or practical scenarios which are common in
everyday teaching. The following example is a scenario where students are contacting the instructor between
classes (related to the communicating with students activity): “You have a big exam coming up in your class.
You expect several lastminute questions as your students prepare for the exam. Unfortunately, you must be at a
conference the two days before the exam and will miss office hours both days (p.1).”. The authors then present
how to use email, threaded discussion and chat in this scenario. All the chapters in the book are organized in a
similar way. Each scenario within a chapter is presented separately (or in some cases two scenarios are presented
together). First, the authors present a non-technological solution to the scenario’s challenges. Secondly, usually
several technology-based solutions with relation to the traditional solution are presented. Finally, potential
pitfalls of the technological solutions are discussed.
Chapter 1 deals with the need to communicate with students during a course. The authors stress, for instance, the
value of between lectures communication. Scenarios within the chapter include issues such as making changes to
lectures and students submitting course work. Technological solutions based on common email programs,
threaded discussions (within a CMS tool), chat and web postings are presented. Chapter 2 covers the activity of
distributing course materials to students during a course. Scenarios within the chapter consist of cases such as
distributing the course syllabus, providing access to extra materials and providing graphics, videos and audio
materials to students. The authors present, for instance, how to send a course syllabus written with a word
processor to students via email. The authors also show how to use hyperlinks to provide supplementary material.
Chapter 3 includes instructions on how to support collaboration among the students. The scenarios in the chapter
deal with issues such as facilitating group work and fostering peer feedback. The authors present, for instance,
235
how to use discussion forums to organize peer learning activities. Other solutions presented in this chapter are
based on chat, annotation and change tracking features in word processors to give precise feedback.
Chapter Four of the book explores the possibilities to support learning through experience. At the beginning of
the chapter, the authors give a short introduction on how a teacher can find technology tools for supporting
“virtual” experiential activities. The authors present a set of tools that they have found from MERLOT
(Multimedia Educational Resource for Learning and Online Teaching) database. The following tools, for
instance, are presented to give an idea of what virtual environments can offer: A.D.A.M virtual learning system
for learning human anatomy, Sniffy, The Virtual Rat for learning the idea of conditioning and Chemland for
exploring the field of chemistry. Some of the presented tools are freely available for everyone. In Chapter 5, the
authors show how to use hypertext to build and present a course with the Strategic Planning model framework.
Hypertext is used, for instance, to link course activities and assessment to course objectives. The authors also
show how to use hypertext to provide detailed information about different aspects of the course to the students.
In Chapter 6, different solutions are discussed to improve students writing activities. The authors present, for
instance, how to use commenting and highlighting functions in Word to give feedback to students.
Technological solutions to detect plagiarism are also discussed. The theme in Chapter 7 is to improve students
research skills. The authors acknowledge that there is a crowing need for students to be able to find and evaluate
information on their own. They explain, for instance, how to use search engines, online databases and virtual
libraries to find relevant literature. Chapter 8 deals with technical solutions to use assessment and feedback to
enhance students learning. The authors present, for instance, how teachers can use technology to provide
targeted feedback and follow-up instructions to the students with word processors and CMS tools. In the final
chapter, the authors show how to use hypermedia to gather course materials in the course. The authors also
discuss how students can use technology (mainly web searches) to find relevant information for a project work.
At the end of the book, there is an appendix that includes a short introduction to copyright laws in U.S.A. The
appendix also includes a list of plagiarism detection services available in the net. The final part of the book is a
glossary including all the technical terms used in the book.
Evaluation
The authors take an interesting approach by presenting technological solutions through practical cases and
scenarios. This gives a contextual nature to the theme of the book; how to apply technology to support teaching
activities. The authors have very positive attitude throughout the book. One can sense that they really appreciate
the possibilities of technological solutions. The book includes lots of practical examples and solution alternatives
to show that technology can be applied by teachers who master the basic technology. The technology does not
exists on its own right, but it is applied to overcome real problems and challenges in teaching. The real need and
meaningful use of technology is emphasized throughout the book. All the concepts and terms are explained
carefully. All new terms are underlined like links in webpages and there is a separate explanation box for the
term in the same page. Furthermore, the authors have gathered a comprehensive glossary of relevant
technological terms at the end of the book. The text was also easy to read and follow. The authors use
nontechnical style avoiding confusing ICT jargon. Various snapshots taken from different applications and tools
clarify the authors message. The structure of the book is clear because all the chapters follow the similar
structure as explained earlier. However, because same technology is applied in several scenarios there is some
repetition. In my opinion, Chapter 9 does not include any new perspectives. The authors could have distributed
the content of the chapter to other chapters.
The main shortcoming of the book (to a person working in educational technology with computer science
background) is that it does not progress from commonly used tools, such as word processors, emails and CMSs,
to more advanced technologies. This is, however, acknowledged by the authors already in the introduction part
of the book. “This book is not for the technology innovators, nor it is for the skeptics. It is for the 75 to 80
percent in the middle: the mainstream faculty who are using some technology..”, authors point out in the very
first page of the book. For me, the most interesting part of the book was Chapter 4. It included several examples
of more advanced technologies, such as tools for experimental learning (e.g. virtual reality), role playing via the
net, and problem solving (for instance Mike's Bikes – Advanced; simulator for experiencing the creation of
business strategies). The chapter also includes a short introduction to the content of MERLOT database for
searching meaningful tools within different disciplines.
I also found some minor problems and improvement areas in the book. First, the amount of figures is even too
high. Although the figures support the text nicely, the authors could have decreased the amount of snapshots at
some parts of the book. This would have given the opportunity to deepen the discussion to more advanced use of
236
the technology. I also found that the authors explained some basic functions of the tools too carefully, such as
saving a picture from a website. Some figures are also blurred (e.g. Figure 4.11, Figure 4.2a-b), it was difficult to
see the content of the figure clearly. Furthermore, in some figures the authors use annotation to clarify the
meaning of some functions. The black text inside the annotations is placed in a dark background making it hard
to read (e.g. Figure 6.2, Figure 6.3). Although every technological solution includes a short discussion on
potential pitfalls, the authors could have deepen the discussion in this section. In many cases, they mention that
the main problem is that students are not able use or do not know how to use the presented technology. For
instance, in Chapter 8 the authors explain that the main pitfall with online surveys is that students do not know
how to access or use the available survey tool (p.192). I would have expected more discussion about technology
per se. For instance, in my experience the discussion forums in many CMS tools are often limited. They do not
support, for example, graph-like presentation of the discussion threads. In some cases, it is also difficult to give
precise feedback using the discussion forum. We have found, for instance, that commenting a piece of
programming code with a common discussion forum can be a problem. It would have been useful to know, for
instance, how the authors would develop these tools to make them even more useful or what needs cannot be
fully met with available technology. The supplementary CD-ROM was aimed to improve the understanding of
technology presented in the book. It included video snapshots of tools discussed with audio explanation. In the
CD-ROM, the authors present, for instance, how to set up discussion threads in the Blackboard CMS tool. In my
opinion, the CD-ROM does not provide much extra value to the content of the book.
I would recommend this book to a teacher who has not been working with the technology, but would like to
enhance his/her teaching through the technology. Furthermore, the book might be used in teacher training to
show how to apply technology in everyday teaching context The aim of the book is to give readers ideas of
possible technological solutions, and to encourage them to look for appropriate tools in one’s own discipline. In
my opinion, the book succeeds in this goal. If you already master the basic technology or work with educational
technology, this book is not for you. At least for me, the book did not provide much new ideas or knowledge.
The authors also take a contact teaching perspective, there is no discussion about possible problems and
challenges in distance learning (although many of the presented solutions can be easily applied in the distance
learning context). The book also serves a good reference in basic educational technology courses as an example
of utilizing common technology in everyday teaching.
237
Malinski, R. (2006). Book review: Unlock the genius within; neurobiological trauma, teaching, and transformative learning
(Daniel S. Janik). Educational Technology & Society, 9 (2), 238-240.
Unlock the genius within; neurobiological trauma, teaching, and
transformative learning
(Book Review)
Reviewer:
Richard Malinski
Ryerson University, Canada
[email protected]
Textbook Details:
Unlock the Genius Within: Neurobiological Trauma, Teaching, and Transformative Learning
Daniel S. Janik, MD, PhD
Rowman & Littlefield Education
Paper 1-57886-291-4 September 2005 230pp
http://www.rowmaneducation.com/ISBN/1578862914
Introduction
There has been a great expansion over the last decade of neurobiology through the use of progressively more
sophisticated imaging technology. With these imaging techniques clearer links are being drawn between our
physical, rational, and emotional functioning and specific and/or multiple brain locales. This book surveys some
of this recent research, draws together insights from the author’s medical and educational experience, and
outlines a foundational learning theory. Because of this breadth of scope, the book is both exciting and
frustrating to read, not to mention review.
The author sets out a daunting set of goals. Specifically he states that this book is;
¾ an exploration of the principles underlying exceptional, effective, traumatic learning;
¾ a sketch of neurobiological theory that makes sense of it all;
¾ a description of hypothetical deductions which with attendant research should refine the theory and
method; and
¾ the development of a framework within which basic principles of effective, non-traumatic learning may
be successfully applied. (Janik, 10)
This tantalizing set of goals should attract educators interested in theoretical discussions, language teachers
because of the English as a second language (ESL) context, and teachers searching for process and technique to
help their classes.
Content
‘Unlock the genius within’ has, after the preparatory sections, ten chapters with a number of binding threads
which run them. The chapters are devisable into three major groupings; supporting material, theory, and
promotion.
The supporting material is in the first seven chapters. This covers discussions of the need for ‘Yet another
teaching theory’ (a chapter title), the author’s idea that ‘It all begins with traumatic learning,’ the significance of
‘What the body tells us,’ the importance of ‘What machines are saying about us,’ the integrative approach of
‘The neurobiological way,’ and the value of a neurobiological theory basis for learning that is non-traumatic,
non-violational, and is direct (scientific) and founded on physical (neurobiological). The author weaves an
integrated philosophical and physical net to contain his neurobiological theory of learning. This reviewer found
these chapters full of intriguing details of brain functioning, insights into trauma cases from the author’s medical
experience, and contrarian views of traditional education. The technical jargon and the lack of clear definitions
may be a hurdle for some readers. While much of the material is not new, the comments on education and the
suppositions on the physicality of learning are intriguing and fascinating.
238
It is in this supporting group of chapters that four main threads or themes become evident. First, right from the
beginning the author pleads for a return to the Socratic method of teaching or the use of a mentorship approach
in learning. This method contrasts with the Platonic approach which is ideational and currently the mainstay of
our schools. The author pleads that this is unfortunate for our students because while it is effective it is
traumatic! The author’s disdain for traditional teachers and teaching seems overly harsh and not likely to
engender a second look.
Second, the author draws numerous links between the physical workings of the brain viewed through imaging
technology and learning. Much of this is reported in the literature but some seem like leaps of faith. For
instance, literature links myelination and the thickness of myelin with intelligence but not specifically with
learning. What direct learning techniques would be most beneficial during the periodic process of myelination
and/or the types of myelin present? Do these periods commence at the same time for everyone? Practical
examples and some definitions would have been useful here. Another instance concerns birth. One might agree
that the birth event is a traumatic experience but what is actually learned by the infant and how do we know, as
the author suggests, that the thalamic gateway is active in learning at that moment? In a later chapter the author
states that ‘…learning after birth is driven first and foremost by the raw power and form of the traumatic birth
event’ (p 141) but doesn’t elaborate what this power and form are. Some of the difficulty here may be a result of
semantics. The author does not define learning or thalamic learning or explain how one might evaluate or
understand what is learned or what is occurring. As a result the reader might be at a loss as to what is actually
being suggested.
Third, the main premise of the author is that understanding trauma and its significance to learning is essential if
we are to break with our traditional traumatic teaching approach. The author makes it clear that traumatic
events produce learning that results in triggers and associations that can be destructive to the individual. The
strength with which the author calls for a change to this traditional traumatic teaching seems overdone and
counterproductive. The author suggests that teachers are not sufficiently learned in the workings of the brain to
make this trauma-teaching association but if they were they would cease and desist from their approach.
Fourth, the author talks about mentorship as a viable alternative to today’s teaching environment. His view
might be encapsulated in the desire to move from a sage-on-the-stage approach in which teachers simply dump
the structured information into their students (the empty vessels they are!) to a guide-on-the-side in which the
mentor lets them find their own way but is there to provide assistance when asked.
The first seven chapters can be seen as an attempt to provide background information in traumatic learning and,
to promote imaging technology as a useful tool in understanding the development of the brain with its particular
responses to learning situations, and to foster the author’s practical, physical view of effective learning.
The second unit in this book consists of the eighth chapter, ‘Neurobiological learning’ and contains the kernel of
the author’s view. The author defines two key terms and sets out seven tenets or laws. First, neurobiological
learning (NL) is at the core of the author’s thesis and ‘…occurs in discrete steps, phases or levels, beginning with
object-data and proceeding through association, symbolism, interpretation, and cognition to metacognition and
possibly beyond.’(p141). Second, transformative learning (TL) is learning that is ‘volitional, curiosity-based,
discovery-driven, and mentor-assisted’ (p144). The author maintains that ‘The physical evidence speaks
strongly in favor of the existence of a single unifying theory of neurobiological learning that in its application
follows two pathways – traumatic and transformative learning’ (p 146). We are to avoid traumatic learning with
all is negative triggers and associations, and strive for TL. These are not particularly new ideas. In essence, we
need to understand what learning is with NL being another tool helping us approach learning.
The author then provides a set of seven tenets which together provide the ‘natural laws’ of neurobiological
theory and method and transformative learning. Many of these would be recognized and understood by good
teachers anywhere. First, we must assist learning by demonstrating curiosity, discovery, and self-discovery and
not just explaining how to learn. Second, using TL techniques creates self-sustaining success and interest in
learning within the student. Third, effective learning is not localized in classrooms as schooling at home or
distance learning can show and so we need to think in broader terms about learning. Fourth, effective learning
results when students noticing something of interest bring it into central focus where they can discovery and
reflect on it. The mentor’s role is to provide a resource rich environment and the occasional guidance for the
student as he or she progresses through the levels of NL as (noted above). Fifth, building upon such tools as
listening, language, and visualization the use of rhythm (music) and rhythmic patterns (stages in TL and/or
cycles of brain development) can promote structure and success in learning. Sixth, mentors are at the core of the
239
tenets. This tenet links closely with the others in emphasizing the resources richness of the learning environment
and the laissez-faire approach of TL. At the same time there is a need for mentors (teachers don’t seem to have
this ability!) to be aware of and assist the learners to sense their feelings, be aware of their cyclic rhythms and
internal time consciousness. Seventh, this last states that ‘neurobiological method underlies all teaching and
learning methodologies’ and ‘reasons, rules, meanings, sense, and truth are defined by the learner.’ Here again
is a call to understand brain development cycles and merge this with TL techniques to allow, as constructivism
suggests, for the individual to build his or her knowledge.
This chapter is the summation of the exploring, sketching, and describing as well as the development of a
framework within which to work. The two definitions and the seven tenets integrate many concepts and
processes with which some traditional teachers are already familiar but the integration of neurobiological
components with learning should prove new and insightful to many more.
The third and last group of chapters, chapter nine and ten, consists of the author’s assessment of the success of
his approach in teaching ESL at his college and an invitation to further discussion. The chapters might be called
promotional because of the laudatory nature of the comments about the college and the enthusiasm being built
for NL and TL training and certification through the ESL college. There is some description of the use of the
author’s method at the college but insufficient for any real understanding of techniques and how they might
apply to other situations.
Concluding comment
The author alludes to the need for further work in solidifying the links between learning and brain imaging and
in firming up his tenets. These refinements are essential to help explain how NL and the TL techniques
introduced can be used by others. How might one determine the rhythmic patterns of students and once this was
done determine what strategies, tactics, and data-objects might be appropriate? How this might be done for
many students as opposed to those one or two mentored learners would be most useful. Resource richness is
always a concern of teachers but in the case of TL mentors what resources need be in place? These are issues
that the author needs to spend a great deal of time detailing. He has the odd example but much more needs
doing. The author’s next book which deals with the application of NL/TL to distance education needs to focus
more on tactics and concrete illustrations.
The overall result of all this material in these few pages is a very choppy writing style with the focus jumping
back and forth over the major themes all the while loading on more and more information some as factual
material and other as supposition. There are many terms that need definition and/or concrete illustrations. The
index is overly complex and not without error. There is some summary and direction at the ends of chapters but
the reader would have benefited from a variety of graphics and/or summarizing lists as guides. While the
information is rich the visual component is not!
There is a sense that the natural laws of NL/TL, as drawn from the author’s experience of trauma cases and ESL
and his outlining the links between brain activities and learning language, are universal and can be applied
broadly. Is this approach tenable? The extrapolation from trauma cases with their negative learning triggers
and associations through brain imaging with its views of learning and from traditional education to the
application of NL/TL in ESL are fascinating and worth discussion. However, can we rely, as the author does,
on these as essential links and applicable to learning in general? Is trauma in our education system as negative
and pervasive as the author details? Is the work on ESL applicable to other types of learning? Are the tenets
proposed within NL/TL sufficiently robust to be the basis of a learning theory that makes sense of it all?
240

Complete Issue in PDF

Transcription

Similar documents

fuzzy`s pit box lounge

Learning Styles

learner services guide

IB learner profile IB learner profile IB learner profile IB learner profile

Ben Vaske – Unique approach of basic skills

From click`n go to click`n stay

ELIZABETH SCHEU AND KNISS View ISO 9001:2008

DeskStore Product Data Sheet - Top-Tec

IB INQUIRER

Sin título-1