Conceptual Architecture including Component Evaluation

Transcription

Project Number 027023
APOSDLE: Advanced Process Oriented Self-Directed Learning Environment
Integrated Project
IST – Technology enhanced Learning
Conceptual Architecture including
Component Evaluation
Deliverable D 2.12
Due date
Actual submission date
2010-02-28
2010-02-26
Start date of project
Duration
2006-03-01
48
Revision
Organisation name of lead contractor for this deliverable
UT – University of Twente
Project co-funded by the European Commission within the Sixth Framework Programme (2002-2006)
Dissemination Level
PU
Public
PP
Restricted to other programme participants (including the Commission Services)
RE
Restricted to a group specified by the consortium (including the Commission Services)
CO
Confidential, only for members of the consortium (including the Commission Services)
Final
D2.12 – Conceptual Architecture including Component Evaluation
Disclaimer
This document contains material, which is copyright of certain APOSDLE consortium parties and may
not be reproduced or copied without permission. The information contained in this document is the
proprietary confidential information of certain APOSDLE consortium parties and may not be disclosed
except in accordance with the consortium agreement.
The commercial use of any information in this document may require a licence from the proprietor of
that information.
Neither the APOSDLE consortium as a whole, nor a certain party of the APOSDLE consortium warrant
that the information contained in this document is capable of use, nor that use of the information is
free from risk, and accepts no liability for loss or damage suffered by any person using the information.
This document does not represent the opinion of the European Community, and the European
Community is not responsible for any use that might be made of its content.
Imprint
Full project title:
Advanced Process-Oriented Self-Directed Learning Environment
Title of work package:
WP II: Self-Directed Learning
Document title:
Conceptual Architecture including Component Evaluation
Document Identifier:
APOSDLE-D02.12-UT-ConceptualArchitecture3
Work package leader:
UT
List of authors:
Henny Leemkuil & Robert de Hoog (UT) (Editors.)
Ton de Jong, Jose Kooken, Wilco Bonestroo (UT)
Stefanie Lindstaedt, Tobias Ley, Günter Beham (KC)
Mario Aehnelt (FHG)
Andreas Faatz (SAP)
Luciano Serafini (FBK)
Helmut Leitner, Barbara Kump, Viktoria Pammer (TUG)
Administrative Co-ordinator:
Harald Mayer
Scientific Co-ordinator:
Stefanie Lindstaedt
Copyright notice
© 2006-2010 APOSDLE consortium
Document History
Version
Date
Reason of change
1
2010-01-04
Document created, continuous updates
2
2010-02-22
Final version after internal review
3
2010-02-25
Updates by FHG
4
2010-02-26
Final version submitted to EC
© APOSDLE consortium: all rights reserved
page
ii
Executive Summar y
This deliverable gives an overview of the overall APOSDLE design approach to work-integrated
learning support. For an in-depth discussion of the work-integrated learning paradigm related to
informal learning theories please refer to the integrated Deliverable D II.8 & D III.5 APOSDLE
Perspective on Self-Directed Work-integrated Learning. This deliverable focuses on the approach to
design support mechanisms for work-integrated learning as prototypically implemented within
APOSDLE Prototype 3 and on empirically investigating several features and support mechanisms
included.
Specifically this deliverable addresses three major APOSDLE design concepts: models, user‟s work
context and user profiles; integrated work, learn, and cooperation support; and the knowledge artefact
lifecycle. We apply a battery of semantic and scruffy approaches to each of these concepts in
order to design learning support flexible enough to cope with challenges of work-integrated learning.
By doing so, we draw from approaches to context-aware systems, user modelling, adaptive system
design, learning and instruction methods, recommendation systems, retrieval and clustering
algorithms as summarized in the following:

APOSDLE models and their relationships
Several models are needed to enable APOSDLE to deliver the needed support for selfdirected work-integrated learning. This are respectively the domain model (refers to the worklearn domain concepts), the task model (refers to the tasks that are performed in the worklearn domain) and the learning goal model (refers to generic learning goals that can be linked
to tasks). Each of the models is described in more detail and methods and techniques for
modelling are explained. Also attention is paid to the crucial role these models play in the
APOSDLE approach.

User’s work context and user profiles
APOSDLE relies on the automatic identification of a user‟s current work task and relevant
domain concept. This information is utilized to on the one hand trigger a number of work, learn
and cooperation support mechanisms (see below). On the other hand this information feeds
into a user profile designed as a layered overlay of the domain model. User profile services
then allow us to infer the user‟s knowledge levels, identify learning goals and compute
prerequisite relations, etc. In addition the notion of a knowledge indicating event is explored as
a means for unobtrusive profile adjustment and a proxy for outcome assessment.

Integrated work, learn and cooperation support
Based on a user‟s current work task and/or topic and on her knowledge levels APOSDLE
provides a number of recommendation and support mechanisms for learning and cooperation.
These support mechanisms utilize on the one hand the underlying semantic models (domain,
task, learning goal) and on the other hand advanced retrieval and clustering algorithms based
on text and multi-media data. Support is embedded in instructional guidance that intends to
help in performing key learning functions in self-directed learning. Cooperation is facilitated by
recommendations of knowledgeable persons, context preservation during cooperation and
providing communication channels. Outcomes of cooperation activities can be stored and
accessed, thus contributing to knowledge creation and knowledge distribution.

Knowledge artefact lifecycle
Since APOSDLE re-uses organizational content (text as well as multi-media content) for
supporting work-integrated learning this content needs to be turned into knowledge artefacts.
We employ advanced retrieval and clustering algorithms based on text and multi-media data in
order to do much of this automatically. In addition, the user is empowered to provide feedback
and create new knowledge artefacts which over time improve the automatic mechanisms.
page
iii
Several of the designs were empirically investigated in nine component evaluation studies, covering
automated user work context detection, methods for assessing user levels and the validity of these
levels, the acceptability of the privacy concept, the effect of manually created learning paths, the
difference between several ways to show the domain in terms of work time speeding up, the usability
and acceptability of scripted communication, the added value of multimedia and the relative
performance of similarity measures for retrieval of resources. Together these studies provided
valuable insights into the effectiveness and acceptability of these designs.
page
iv
Table of Contents
Executive Summary ..............................................................................................................................iii
Table of Contents ...................................................................................................................................v
1 Introduction........................................................................................................................................1
1.1
1.2
1.3
1.4
Purpose of this document .........................................................................................................1
APOSDLE Scope and Boundaries: Focus on Prototype 3 .......................................................1
Reader Guidance ......................................................................................................................1
Relationship to other Deliverables ............................................................................................2
2 APOSDLE Overall Approach ............................................................................................................3
2.1
2.2
APOSDLE approach: Work-integrated learning support ..........................................................3
Scruffy technologies to enable work-integrated learning ..........................................................4
2.2.1
2.2.2
2.3
Domain, task, learning goal model and their relationships .......................................................5
2.3.1
2.3.2
2.3.3
2.3.4
2.3.5
2.3.6
2.4
Context awareness and context determination .......................................................................... 9
User profile and user context representation ............................................................................. 9
User profile services .................................................................................................................. 9
Work-integrated learning support ............................................................................................10
2.5.1
2.5.2
2.6
Domain model ........................................................................................................................... 5
Task model ................................................................................................................................ 6
Learning goals model: A competency-based approach for formalizing learning goals and
prerequisite knowledge .............................................................................................................. 6
APOSDLE knowledge base ....................................................................................................... 6
Where and why do we need models in APOSDLE? .................................................................. 6
Challenges for modelling in APOSDLE ..................................................................................... 8
Context awareness and user profile services ...........................................................................9
2.4.1
2.4.2
2.4.3
2.5
Technology challenges .............................................................................................................. 4
APOSDLE scruffy approach ...................................................................................................... 4
Supporting learning from documents ....................................................................................... 10
Supporting cooperation processes .......................................................................................... 11
Knowledge artefact lifecycle ...................................................................................................11
2.6.1
2.6.2
2.6.3
2.6.4
What is an APOSDLE knowledge artefact? ............................................................................. 11
Access to knowledge artefacts ................................................................................................ 12
Creation of knowledge artefacts .............................................................................................. 12
Retrieval of knowledge artefacts .............................................................................................. 12
3 Domain, Task, Learning Goal Model and their Relationships ....................................................13
3.1
Domain model .........................................................................................................................13
3.1.1
3.1.2
3.2
Task model ..............................................................................................................................15
3.2.1
3.2.2
3.2.3
3.3
Challenges............................................................................................................................... 15
The APOSDLE Approach ........................................................................................................ 15
Workflow approach – critical reflection and choices in an organizational setting ..................... 16
Conclusions - recommendations for the conceptual architecture ............................................ 16
Conclusion ............................................................................................................................... 17
Learning goal model: a competence-based approach for formalizing learning goals and
prerequisite knowledge ...........................................................................................................17
page
v
3.3.1
3.3.2
3.3.3
3.3.4
3.3.5
3.3.6
3.4
Challenges for realising adaptivity in work-integrated learning with APOSDLE ....................... 17
The conceptualizing of learning goals in APOSDLE ................................................................ 19
Role of Learning Goals in APOSDLE ...................................................................................... 20
The learning goal model and its theoretical background: competence-based knowledge
space theory ............................................................................................................................ 21
Using the learning goal model in APOSDLE............................................................................ 24
Comparison of the APOSDLE Modelling Approach to Standard Industrial Competency
Management Practice .............................................................................................................. 25
The APOSDLE knowledge base .............................................................................................29
3.4.1
3.4.2
3.4.3
3.4.4
Challenges............................................................................................................................... 29
Knowledge acquisition ............................................................................................................. 30
Knowledge Integration ............................................................................................................. 32
Where else has something like this been used........................................................................ 33
4 Context Awareness and User Profile Services ............................................................................34
4.1
Context awareness and context determination .......................................................................34
4.1.1
4.1.2
4.1.3
4.1.4
4.1.5
4.2
User context representation ....................................................................................................44
4.2.1
4.2.2
4.2.3
4.2.4
4.2.5
4.3
Challenges............................................................................................................................... 34
Theoretical background ........................................................................................................... 35
APOSDLE approach ................................................................................................................ 36
New challenges and ideas ....................................................................................................... 37
Component evaluation: context detection ................................................................................ 38
Challenge ................................................................................................................................ 44
Naturally Occurring User Actions as Knowledge Indicating Events ......................................... 45
Properties of APOSDLE user profile ........................................................................................ 46
Implementing KIE in a Learning System .................................................................................. 48
Relevance Feedback ............................................................................................................... 51
User profile services ................................................................................................................52
4.3.1
4.3.2
4.3.3
4.3.4
Challenge ................................................................................................................................ 52
Theoretical background and APOSDLE approach .................................................................. 52
Component evaluation: identifying user levels ......................................................................... 55
Component evaluation: the privacy concept ............................................................................ 59
5 Work Integrated Learning Support ................................................................................................64
5.1
Supporting learning from documents ......................................................................................64
5.1.1
5.1.2
5.1.3
5.1.4
5.1.5
5.1.6
5.1.7
5.1.8
5.1.9
5.1.10
5.2
Challenges............................................................................................................................... 64
Background and similar problems ........................................................................................... 65
APOSDLE approach ................................................................................................................ 66
Supporting preparatory Functions ........................................................................................... 66
Supporting executive Functions ............................................................................................... 74
Supporting closing Functions................................................................................................... 82
Remaining problems – new challenges ................................................................................... 83
Component evaluation: learning paths .................................................................................... 84
Component evaluation: the relationship tool ............................................................................ 91
Component evaluation: user profile and MyExperiences ......................................................... 98
Supporting cooperation processes .......................................................................................102
5.2.1
5.2.2
5.2.3
5.2.4
Challenges............................................................................................................................. 102
Cooperative Activities in APOSDLE ...................................................................................... 103
Contextualized Cooperation Framework................................................................................ 105
Component evaluation: scripted communication ................................................................... 122
page
vi
6 Knowledge Artefact Lifecycle ......................................................................................................141
6.1
6.2
What is an APOSDLE knowledge artefact? ..........................................................................141
Access to knowledge artefacts .............................................................................................141
6.2.1
6.2.2
6.2.3
6.2.4
6.2.5
6.2.6
6.3
Creation of knowledge artefacts ...........................................................................................146
6.3.1
6.3.2
6.3.3
6.3.4
6.4
Challenges............................................................................................................................. 141
Theoretical Background ......................................................................................................... 142
APOSDLE Approach ............................................................................................................. 142
Where else has something like this been used and proven to work ...................................... 143
New challenges and ideas to solve them............................................................................... 144
Component evaluation: multi-media ...................................................................................... 144
Challenges............................................................................................................................. 147
APOSDLE Approach ............................................................................................................. 147
Why did we choose this approach? ....................................................................................... 149
Retrieval of knowledge artefacts ...........................................................................................150
6.4.1
6.4.2
6.4.3
6.4.4
6.4.5
6.4.6
6.4.7
Challenges............................................................................................................................. 150
APOSDLE approach .............................................................................................................. 151
Why did we choose this approach? ....................................................................................... 152
Theoretical Background ......................................................................................................... 153
Where else has something like this been used and proven to work ...................................... 154
Component evaluation: similarity measures .......................................................................... 155
7 References .....................................................................................................................................160
8 Glossary .........................................................................................................................................172
page
vii
page
viii
1 Introduction
1.1 Purpose of this document
APOSDLE can be characterized by the following three aspects which pose a number of research
challenges:

Real time learning – APOSDLE aims at supporting the knowledge worker in learning situations
within her current work task. Learner support needs to be adapted to a user‟s work context
and her experiences, and be short, and easy to apply.

Real computational environment – APOSDLE aims at providing a variety of tools which are
integrated seamlessly within her desktop and allow one-point access to relevant back-end
systems of her organization (via some intelligent middleware). Tools need to be inconspicuous
and easy to use.

Real content – APOSDLE aims at dynamically creating learning content out of resources from
the underlying organizational memory (which originally were not intended for instruction).
Resources need to be compared, analyzed and retrieved based on their relationships to each
other and the user context.
APOSDLE addresses these three challenges from two perspectives:
(1) Learning Perspective: Development of a paradigm for work-integrated learning
For an in-depth discussion of the APOSDLE learning perspective please refer to integrated
Deliverable D II.8 & D III.5 APOSDLE Perspective on Self-Directed Work-integrated Learning
(2) Design Perspective: Application of semantic and scruffy technologies for the design of
support mechanisms for work-integrated learning
This deliverable addresses two main topics:

the design perspective of APOSDLE, the approaches taken when designing support for workintegrated learning.

the research perspective, investigating specific design features of the 3 Prototype in order to
obtain insight into their performance, quality and user acceptance
rd
1.2 APOSDLE Scope and Boundaries: Focus on Prototype 3
This deliverable is an updated version of the previous two: D.II.3 Conceptual Framework and
Architecture Version 1 and D.II.7 Conceptual Framework and Architecture Version 2. Compared with
the second version this deliverable is extended with several component evaluations.
1.3 Reader Guidance
In order to convey the different aspects of such a complex, interdisciplinary research activity to the
reader we have chosen the following structure: Chapter 2 gives an overview of the overall APOSDLE
design approach and provides short overviews of the individual chapters and their interrelationships.
Chapters 3-6 are the core of the deliverable. They address three major design concepts in APOSDLE:
context of the knowledge worker and its handling through a user profile; support for working, learning
page
1
and cooperation; and the knowledge artefact lifecycle. In each of these chapters component
evaluations are reported.
Chapter 7 contains all the references used in this deliverable and Chapter 8 provides a snap shot of
our terminology. Since the terminology keeps evolving based on our growing understanding of the
research issue itself as well as of the different research fields involved, we have set up a Wiki System
which contains our terminology and which is updated regularly by the responsible partners. Please
refer to https://aposdle.itc.it/glossary/index.php/Main_Page to view the current version of the glossary.
1.4 Relationship to other Deliverables
This deliverable is directly related to the deliverables listed below and already partly mentioned above:

D.II.3 Conceptual Framework and Architecture Version 1

D.II.7 Conceptual Framework and Architecture Version 2 (this deliverable is an updated
version of this one)

D II.8 & D III.5 APOSDLE Perspective on Self-Directed Work-integrated Learning (contains
the main theoretical background to the self-directed work integrated learning approach)

D I.7 Privacy in APOSDLE (contains information about how APOSDLE handles privacy
issues)
Several other deliverables are referred to in the text, but the ones above are most closely related.
page
2
2 APOSDLE Overall Approach
This chapter gives a brief overview of the APOSDLE approach. The three major design concepts in
APOSDLE (context of the knowledge worker; support for working, learning and cooperation; and the
knowledge artefact lifecycle) are then detailed in the chapters 3-6 below.
2.1 APOSDLE approach: Work-integrated learning support
Work-integrated learning happens very frequently during social interaction while knowledge workers
collaborate on digital artefacts or communicate aspects of these artefacts. The role a knowledge
worker embodies in social interaction, is subject to continuous change: at one point in time, a
knowledge worker acts as a learner, at another point in time, the same knowledge worker herself acts
as an expert (teacher) depending on her expertise with regard to the subject matter at hand (Lave &
Wenger, 1991). Hence, when we learn there is always explicitly or implicitly some teaching involved. In
the case of formal training we usually encounter one teacher or trainer who conveys the content to be
learned. But in other situations – such as reviewing code – this teaching role is not so obvious to the
expert herself. In the following we will refer to such episodes as work-integrated teaching.
The key distinction of the APOSDLE approach, as compared to more traditional (e)Learning
approaches, is that APOSDLE will provide integrated support for the three roles a knowledge worker
fills at the professional workplace: the role of learner, the role of expert, and the role of worker.
Learner Support. APOSDLE provides learners with support for self-directed exploration and
application of knowledge. This is done within their work environment such that learning takes place
within the learner‟s current work context. APOSDLE provides learners with guidance through the
available knowledge by applying novel learning strategies. Content from knowledge sources are
presented to learners, even if the content provided has originally not been intended for learning.
Expert Support. APOSDLE acknowledges that most effective learning transfer happens during
communication, cooperation and social interaction. APOSDLE lowers the hurdles for knowledge
workers to informally convey knowledge via their computational environment in that it captures the
context of the creation, evolution and usage of artefacts. APOSDLE enriches artefacts with context
information and thereby allows artefacts to be turned into true learning artefacts (contextualized
cooperation).
Worker Support. APOSDLE tightly incorporates learning and teaching episodes into the work
processes in that it takes care of several aspects of workers‟ work contexts, such as a worker‟s
knowledge state, work situation, and application domain. Workers are provided with context sensitive
knowledge, thus raising their own awareness of learning situations, content, and people that may be
useful for learning. APOSDLE enables workers to access content from several diverse knowledge
sources without having to change the work and learning environment.
To work, learn and teach efficiently and effectively, a knowledge worker must be provided with optimal
guidance to manage the large variety of knowledge artefacts available in the corporate information
infrastructure. Therefore, the seamless integration of the underlying information spaces into an
integrated semantic knowledge structure is of paramount importance. APOSDLE will therefore create
such an infrastructure (referred to as the APOSDLE platform) to support the integration of the three
roles.
page
3
2.2 Scruffy1 technologies to enable work-integrated learning
2.2.1
Technology challenges
Typically eLearning systems are a wonder of carefully designed content, fine-granular models,
interdependencies and hand crafted metadata: The learning domain is broken down into meaningful
learning units or modules which encompass concepts, facts and processes. They entail fine granular
learning information, exercises, tests, etc. Each of these units is carefully designed using a multitude
of different media appropriate for the learning type and learning purpose the unit is serving. A
dependency structure identifies prerequisites and post-conditions. Based on the units, learning paths
(courses) can be created by instructional designers taking into account the target group as well as
preferred didactical aspects. In order to allow for improved personalization, a multitude of metadata is
attached to the units. In addition, eLearning systems provide detailed user models which allow for the
representation of different learning levels in the different areas, learning preferences, etc. Tutors and
teachers are represented in order to allow students access to expert help. Not to speak of class and
lecture management, simulation and games, etc.
In short, one is faced with a thoroughly designed network of interrelated pieces which need to be
artfully concerted to deliver a meaningful learning experience to the user. Reflecting on these
properties, one can easily understand why eLearning content is expensive to create, requires lots of
(metadata) standardization, and also requires a lot of organizational structure.
In contrast, new learning approaches such as work-integrated learning (see Lindstaedt & Mayer, 2006,
for possible scenarios) and organizational learning put one requirement in the centre of attention:
Flexibility. Being closer to the application of knowledge (rather than to the internalization of
knowledge) such approaches critically rely on providing always the newest available content in ever
changing learning situations. While in traditional course-oriented eLearning one could still manage the
large amount of design work (also because the learning domains stayed rather stable), this is not the
case any more in these new settings. Here we have to strive for the best possible available learning
information instead of striving for the best designed eLearning content.
Thus, in such situations it is simply impossible to create and maintain such a carefully crafted network
of interdependent learning pieces and structures. Instead, we have to move towards embracing
approaches which enable us to best deal with change – while at the same time accepting their side
effects such as a lower level of accuracy, likelihood of errors and not always optimal instructional
design.
With APOSDLE, we present possibilities of moving away from the pure and neat approaches of
instructional design (based on hand crafted verified formal models) to the application of scruffy
technologies (hybrid approaches which also take context into account) to enable work-integrated
learning. The “intelligence” within such systems may be “seen as a form of search and as such not
perfectly solvable in a reasonable amount of time” (Gigerenzer & Todd, 1999).
2.2.2
APOSDLE scruffy approach
The foundation for the APOSDLE approach is to not rely on specifically created (e)Learning content,
but to reuse existing (organizational) content which was not necessarily created with teaching in mind.
We tap into all the resources of an organizational memory which might encompass project reports,
studies, notes, intermediate results, plans, graphics, etc. as well as dedicated learning resources (if
available) such as course descriptions, handouts and (e)Learning modules. The challenge we are
1
The term scruffy stems from Artificial Intelligence where a distinction is made between neats and scruffies. Neats consider that
solutions should be elegant, clear and provably correct. Scruffies believe that intelligence is too complicated (or computationally
intractable) to be solved with the sorts of homogeneous system such neat requirements usually mandate. The distinction was
originally made by Roger Schank in the mid 70s to characterize the difference between his work on natural language processing
from the work of John McCarthy, Alan Newell and others whose work was based on logic.
page
4
addressing is: How can we make this confusing mix of information accessible to the knowledge worker
in a way that she can advance her competencies with it?
A frequently travelled path (also within eLearning systems) is the creation of fine-grained semantic
models which allow for the categorization and retrieval of such resources. But as we discussed above,
the creation of such models, their maintenance and the annotation of resources with their concepts
prove prohibitive in a dynamic environment. Thus, the APOSDLE approach is a hybrid one:
complementing coarse grained semantic models (maintained as much as possible automatically, see
below) with the power of diverse associative methodologies, improved over time through usage data
and user feedback (collective intelligence).
Here the models play two roles: serving as initial retrieval triggers and providing the basis for simple
inferences and heuristics to interpret user interactions. A disadvantage of this approach is that
“statements” made by the system such as “this resource helps you to understand the concept of use
case modelling” or “this person has expertise in use case writing” rely on empirical observations with
no claim to accuracy. However, users have become increasingly accustomed to this concept through
their usage of (internet) search engines. Also, obsolete models do not provide any added value and
additionally are in danger of providing a false sense of security.
The APOSDLE approach is to apply a battery of advanced scruffy technologies to bridge the gap
between coarse grained models and fine grained learning needs. The ultimate goal of this research is
to minimize or at best fully eliminate the need for formal models. This will also significantly reduce the
amount of human effort needed to create eLearning systems.
2.3 Domain, task, learning goal model and their relationships
Chapter 3 explains the three model structures and their relationships which provide the basis for
reasoning within APOSDLE:

Domain model – provides a representation of the learning domain in OWL format (OWL is a
Web Ontology Language)

Task model – provides a representation of the work tasks to be supported in OWL format

Learning goal model – provides a mapping between domain concepts, tasks and general
learning goal types
2.3.1
Domain model
The purpose of the Domain Model described in Section 3.1 is to provide a semantic and logic
description of the work domain which also constitutes the learning domain of an APOSDLE
deployment environment. The domain is described in terms of concepts, relations, and objects that are
relevant for this domain. Technically speaking the Domain Model is an ontology that defines a set of
meaningful terms which are relevant for the domain and, which are used to classify and retrieve
knowledge artefacts.
These concepts are also used for semantic annotation of documents (or parts thereof). In the following
we will refer to the combination of a document (or part thereof) together with one or more domain
concepts as a knowledge artefact (see Section 6.1). This annotation process can either happen
automatically (using text based classification algorithms Section 6.3.2.2) or manually (Section 6.3.2.1).
These semantic annotations are used later within the Associative Network (see Section 6.4) to retrieve
relevant knowledge artefacts.
page
5
2.3.2
Task model
The objective of the Task Model described in Section 3.2 is to provide a formal description of the tasks
the knowledge worker can perform in a particular domain. The Task Model identifies and groups tasks,
that is, working steps, and their interdependencies and determines a formalization of patterns and
procedures occurring in a business domain. The very core of a process model is a control flow. For the
sake of consistency with the Domain Model, we have also translated the control flow into an OWL
ontology.
This formal description is used in various ways within APOSDLE. One aspect is the task prediction
(see Section 4.1), which needs a set of predefined tasks. Another important aspect is the dependant
task-competence mapping forming the learning goal model (see Section 3.3).
2.3.3
Learning goals model: A competency-based approach for formalizing learning goals
and prerequisite knowledge
The Learning Goal Model described in Section 3.3 establishes a relation between the Domain Model
and the Task Model. It maps tasks of the Task Model to concepts of the Domain Model. A learning
goal describes knowledge and skills needed to perform a task, with respect to a certain topic in the
Domain Model. In other words, each learning goal refers to one topic in the Domain Model.
This formalism is necessary for a number of functionalities provided by the APOSDLE User Model
Services (see Section 4.3). For example, it enables the determination of user skills from past task
executions (see People Recommender Service in Section 5.2), or the identification of a user‟s learning
need within a certain task (see Learning Need Service in Section 3.3). Within APOSDLE, the
formalisms employed for achieving these functionalities are based on competence-based knowledge
space theory. The usage of competence-based knowledge space theory has several advantages for
work-integrated learning environments. One such advantage is that the mappings afford the
computation of prerequisite relationships between learning goals. This allows us to identify learning
goals which should be mastered by the user on the way to reaching a higher level learning goal.
2.3.4
APOSDLE knowledge base
This section described the APOSDLE Knowledge Base (AKB) which contains the three different
models briefly discussed above, and the meta-model schema interrelating them. Within this AKB each
model is stored in a separate module. Additional components are devoted to contain the interrelations
between the models as specified within the meta-model schema. The resulting AKB thus is a modular
ontology represented in the OWL ontology language.
2.3.5
Where and why do we need models in APOSDLE?
Figure 2-1 below illustrates the information flow through APOSDLE and the key knowledge services
provided.
page
6
Figure 2-1: Information flow in APOSDLE
Based on low level system events the Context Monitoring Daemon (see Section 4.1) detects the
current task a knowledge worker operates in (steps 1 and 2 in Figure 2-1). The CMD is a background
service installed on the client computer. The CMD gathers low-level system events and maps them to
tasks from the task model using machine learning algorithms. The goal of the CMD is to automatically
identify the current user‟s work task.
The detected current task of the user is then stored in the User Profile of the user which is accessed
using the User Profile Service (UPS) (steps 3 and 4 in Figure 1). The UPS is part of the APOSDLE
Platform running on the server side. It serves as a repository for user-related information and as an
engine enabling the system to infer information about the user. The UPS utilizes a history-based user
profile representation where activities of users are stored together with a timestamp. Based on this
history of user activities the UPS infers information about the user as described in more detail in
Section 5. The user is able to overrule the learning goal suggested by the UPS (step 5 in Figure 2-1).
Based on the selected learning goal a search is triggered in the User Profile Service (UPS, see 4.3)
(step 6 in Figure 2-1). The UPS identifies APOSDLE users whose knowledge level of the topic
selected (part of the learning goal) is equal or higher to the one of the user. The user has then the
opportunity to select one or several of these “knowledgeable people” and initiate the cooperation
wizard.
Based on the selected learning goal a search is triggered engaging the Associative Retrieval Service
(ARS, see 6.4) (step 8 in Figure 2-1). The ARS is used for context-based retrieval of resources for
work-integrated learning. It incorporates semantic similarity between concepts in the domain model,
content based similarity between knowledge artefacts and semantic annotations of knowledge
artefacts.
To get a better understanding of a user interaction with APOSDLE the following scenario is helpful: A
knowledge worker is working in some task (1). In order for APOSDLE to start delivering useful
information and support for the user, the task must be detected (2). This can be done either manually
or automatically. In both cases, the task must have been modelled in the task model.
page
7

If the knowledge worker manually selects the task (s)he is in, the tasks must be described
verbally in an intelligible way. Also, the task must be at a granularity level that makes sense to
the user, that is, neither very specific nor highly abstract.

If the task is detected automatically, the task again must be at a fitting granularity level, but
this time it must “make sense to the detection algorithm”. This is given, for example if the task
can be clearly expressed by a set of detectable human-computer-interactions.
To each task a set of learning goals are related (3). These learning goals are modelled in the learning
goal model. Obviously, it is necessary, that to each task in the task model at least one learning goal is
related, otherwise, the information flow within APOSDLE “breaks” here.
The set of learning goals required by the selected tasks are ranked (4) according to various criteria.
One criterion is prerequisite relations between learning goals (see Section 2.3.3). In order to derive
prerequisite relations between different learning goals, it is necessary that some learning goals are
associated with more tasks. For ranking of learning goals it is therefore advantageous if there are
many tasks but few learning goals.
The knowledge worker manually selects one learning goal (5). In order to be able to do so, the
learning goal must be intelligible to the user. The learning goal consists of a learning goal type and a
domain model element. The domain element is modelled in the domain model. This means that the
learning goal types must be well described, the domain model element must be well described
verbally and the semantic of a learning goal must be clear (that is, what does the combination of such
a learning goal type and a domain model element mean).
The APOSDLE system recommends other APOSDLE users whose knowledge in the topic in question
(part of the learning goal selected) is greater or equal to the knowledge level of the initiating user (6).
The user has then the opportunity to choose one or several users from the recommendation list and
initiate cooperation via the cooperation wizard. The wizard is initialized by the context information of
the initiating user, this includes: the topic and/or task detected, the learning goal selected, the
document currently active, the recommended knowledge artefacts and a suggestion for a
communication medium that could be used.
The APOSDLE system also offers resources that are related to the selected learning goal (8). The
associative retrieval uses the domain model element-part of the learning goal to search for resources
based on an associative network that connects similar domain model elements and similar documents
(see Section 6.4.2). The associative network assumes correct semantics from the domain model
(especially with respect to the hierarchical relations). The associative network uses different similarity
measures depending on the domain model‟s characteristics. In general, a more complex structure of
the domain model allows the application of more different similarity measures, which can improve the
retrieval of relevant knowledge artefacts.
2.3.6
Challenges for modelling in APOSDLE
The challenges related to the role of models within APOSDLE mainly refer to the problem of
conceptually integrating the three separate models each having their own representation, as well as
integrating the “semantics of APOSDLE”, that is the way APOSDLE interprets the models‟ meanings
given the intention of the models.

Tasks, domain model elements and learning goals need to be modelled at some
intermediate level of granularity. How this level of granularity can be decided upon more
precisely, is a question for research.

At various points in the information flow described in Figure 2-1, different parts of
APOSDLE have potentially conflicting requirements on the models. For example,
whereas for learning goal ranking, many tasks and few domain model elements would
be advantageous, knowledge artefact retrieval only makes use of the domain model
elements, and so fewer domain model elements lead to a potentially less differentiated
view on the knowledge artefacts.
page
8
2.4 Context awareness and user profile services
User context determination plays a crucial role in the overall APOSDLE approach. In order to be able
to provide the user with information, learning material and links to people relevant to her task at hand,
the system needs to identify the work task reliably. In addition, user activities in the work-integrated
learning system can be monitored in order to update the user profile, more in particular their levels of
competence.
2.4.1
Context awareness and context determination
Section 4.1 explains and stresses the importance of context awareness for work-integrated learning
systems in general and APOSDLE in particular. We start with providing the reader the vision of context
aware systems, its origin, and current developments in this field. Specifically a number of context
aware systems are discussed. The APOSDLE approach to context awareness in based on a common
architecture by Baldauf et al. (2007) which includes the use of agents in three layers separating the
detection of context, planning and action based on context.
The goal of task prediction is to know the active task of the user at any point in time. A task is defined
as a unit of work consisting of activities to reach a certain goal. The problem of task prediction is seen
as a machine learning problem. When first using the prediction system, it is untrained and the user
needs to specify the task s/he works on from a predefined list of business tasks (by manual selection).
During the work process a context monitoring component logs any desktop events reflecting the user‟s
actions. These include keyboard presses, application launches, full texts in documents, etc. As soon
as sufficient classified events are gathered, the system trains a ML model of the user‟s work task in
the application domain. The optimal result is achieved when the user continues to work and she does
not need to manually notify the system of task switches anymore. The task predictor automatically
classifies the active tasks using continuously recorded event streams (automated selection).
Whenever the classification engine detects a change in tasks, the work-integrated learning
environment displays a new list of associated learning resources and suitable experts relevant for the
detected work task.
2.4.2
User profile and user context representation
APOSDLE stores user related context information in digital user profiles. These profiles are used for
maintaining the user‟s usage history and current context with respect to their personal work-, learningand cooperation-related experiences. The APOSDLE User Profile is an overlay of the topics in the
Domain Model. Whenever a user executes a task within the APOSDLE environment the counter of
that task within her User Model is incremented. APOSDLE uses data stored in the user profiles for
adapting its support to the users‟ needs and requirements. Based on the user profile data,
recommendations are computed aiming at supporting the users‟ learning goal attainment, the
preparation of the retrieval of resources (Section 6.4), the creation of learning paths (Section 5.1) and
cooperation activities (Section 5.2).
An important issue is how user actions during working with the system can be used to update
competence levels in the user profile. Given the nature of APOSDLE, being domain independent, and
the prospective deployment context where testing knowledge as is done in formal learning settings is
not feasible, more indirect and unobtrusive indicators are needed. These indicators are called
knowledge indicating events or KIE. Several options for how to use which KIE for updating
competence levels are explored.
2.4.3
User profile services
The component responsible for operations upon user profiles is the User Profile Service (UPS). The
UPS‟ functionality is made accessible to other parts of the APOSDLE system via four core types of
services.
page
9
Logging services are responsible for updating the User Profile with new observed usage data, and
thus provide the basis for all other services. Sensors within the APOSDLE environment send detected
user activities (such as task executions, cooperation events) to Logging Services to be added to the
user model.
Production services make the stored usage data available to other (client) services within the
APOSDLE environment. Based on the specific requirements of the client, production services filter or
aggregate usage data – they provide specialized views on the usage data.
Inference services process and interpret usage data to draw conclusions about different aspects of
users, such as levels of knowledge. Inferences are then utilised to adapt the functionality of the
service itself, or by providing the outcome to other services.
Control Services provide ways to control usage data stored in the user model. Controlling usage data
is important for handling privacy issues and imprecise usage data collected in the user model.
2.5 Work-integrated learning support
At the core of the APOSDLE system is supporting learning at the workplace. However, the goal of
APOSDLE to be a domain (work-learn) independent generic system, makes this goal much more
difficult to attain than in domain specific e-Learning systems. This means that beforehand the topics
are not known and, furthermore, the types of material that are available are also unknown. Therefore it
is impossible to generate specific (sequences of) instructional events (based on instructional design
models) that should lead to a specific learning goal. However, we still have to create a situation in
which a person is stimulated to learn something. The solution could be to embed parts of existing
documents in a general context that contains elements to support self-directed learning. In addition,
outcome assessments in terms of more or less formal tests are not possible in the APOSDLE context.
First, because devising test-like assessments when the learning domain is next to impossible. Second,
because a self-directed workplace learning context rarely aligns with the notion of testing and
examination-like activities.
st
To deal with this challenge learner support is available throughout the system and not, as in the 1
nd
and 2 Prototypes, limited to one learning event generated by a learning tool. More in general, it is
arranged round documents retrieved from the document base (see Chapter 6) and cooperation with
co-workers and experts.
2.5.1
Supporting learning from documents
This support is based on the learning functions around self-directed learning as proposed by Simons
rd
(2000). For a subset of learning functions the 3 APOSDLE prototype provides support. This support
consists of:

Using instructional relevant annotations of knowledge artefacts to improve the fit between
learning and the presented material

Creating learning paths consisting of topics the learner has not yet mastered and that are
closely related in the domain model

Providing hints and related engagement activities hat promote active processing of learning
material

Suggestions for using other features of the APOSDLE system to expand their learning or to fill
in remaining gaps

Presenting structured overviews of the current knowledge levels of a learner for tasks and
topics.
page
10
Some learning functions refer to outcome assessment, but this turns out to be a very difficult issue in
rd
domain independent systems. The solutions selected for the 3 Prototype are the KIE or Knowledge
Indicating Events (see Section 2.4.2 and 4.2) and self assessments.
2.5.2
Supporting cooperation processes
Cooperation support within APOSDLE (see Section 5.2) focuses specifically on communication and
cooperation support in learning situations. That is, it is not intended to support the entire palette of
possible user-to-user interactions, but aims to investigate which interactions are especially important
during a learner-expert exchange.
rd
The focus for the 3 Prototype is on:

Recommending other persons (peers and knowledgeable persons) in the organisation that
can help a knowledge worker/seeker when she encounters a learning need.

Recommending the appropriate communication medium or media for the cooperation that can
be used in a synchronous and a-synchronous context.

Detecting the context of the knowledge seeker and making this context explicit to the
cooperation partners during the cooperation process.

Scripting parts of the communication in terms of making clear the problem the knowledge
seeker is facing, in order to improve the smoothness of the interaction

Store and make accessible to other people transcripts of cooperation processes that can
contain valuable knowledge, thus contributing to organisational knowledge distribution.

Provide the actual communication facilities (like Chat and e-mail) depending on the
organizational context where a tailored version of APOSDLE will be deployed.
The approach to these issues is based on the notion of different cooperation spaces and a
cooperation model.
2.6 Knowledge artefact lifecycle
Chapter 6 describes the lifecycle of knowledge artefacts: first a document enters the APOSDLE
system as a repository object (for example, during system instantiation, during nightly updates, after
communication events). Next, the entire document, or parts thereof, are turned into knowledge
artefacts by manually or automatically attaching two types of metadata: domain concept(s) present
and material use of the knowledge artefact. Finally, the knowledge artefacts are related to each other
and retrieved via an associative network. Mechanisms are introduced on how user feedback
mechanisms during the different steps can be utilized. The individual steps are described in more
detail below.
2.6.1
What is an APOSDLE knowledge artefact?
Section 6.1 gives a formal definition of what we in APOSDLE refer to as a “knowledge artefact”.
Informally speaking, a knowledge artefact is a document or part thereof together with two types of
metadata: the learning domain concept addressed/described within the document (piece) and the
material use type of the document (piece). Optionally a knowledge artefact can also have a free text
comment associated with it.
page
11
2.6.2
Access to knowledge artefacts
Section 6.2 addresses the challenges APOSDLE faces when providing access to a number of different
back-end systems of the organization. The two main challenges are unified access and privacy
aspects, the latter are discussed in detail in Deliverable D VI.9.
Based on technologies like GRID computing, APOSDLE allows homogeneous access to all
documents stored within the attached back-end systems via a Data Object Repository. Within this
repository each document is represented by a repository object. This repository object can be
understood as a document representation enriched with metadata. They are stored in a database
within the APOSDLE platform for further access by other APOSDLE modules. The repository manager
deals with the repositories within one APOSDLE system.
2.6.3
Creation of knowledge artefacts
One essential step in the APOSLDE knowledge artefact lifecycle is how knowledge artefacts are
created. One approach to creating knowledge artefacts is to do this in a collaborative fashion, where
users create knowledge artefacts by annotating documents with a domain concept and/or a material
use as described below. Due to the huge amount of knowledge items targeted by APOSDLE, a single
user or a small user group will not be capable to annotate all knowledge artefacts. Therefore, tools
must support a collaborative annotation so that users can share their knowledge about knowledge
artefacts with each other.
However, manual annotation is a labour intensive task and we do not necessarily expect manual
annotations to provide training data covering all necessary concepts for automatic classification. To
overcome this natural limitation and to bootstrap an annotation free repository, APOSDLE takes use of
linguistic and machine learning approaches to automatically assign domain concepts and material
uses to knowledge artefacts.
2.6.4
Retrieval of knowledge artefacts
In order to provide powerful, intelligent retrieval mechanisms to the work integrated learning support
tools the APOSDLE approach includes an associative network. This associative network implements
heterogeneous retrieval mechanisms: semantic retrieval (based on learning domain concepts) is
seamlessly integrated with a variety of similarity-based retrieval mechanisms. This has the advantage
of on the one hand providing the learning activities with exact matched materials and on the other
hand also providing more in-exact similarity-based results for work and learning support. In addition,
the fact that associative networks can “learn” based on changing the edge weights, is used by
APOSDLE to incorporate implicit as well as explicit user feedback.
The challenges identified in this chapter stress this particular use of the associative network in
APOSDLE. An additional challenge is the sparse annotation of knowledge artefacts within the
repository, since annotating documents or parts thereof with metadata is an effort intensive process. In
APOSDLE we have taken two approaches to solve it. On the one hand we employ automatic
annotation mechanisms as much as possible. On the other hand, we acknowledge that not all
knowledge artefacts will have high quality metadata attached. By utilizing text-based and multi-media
based similarity metrics, APOSDLE is able to identify knowledge artefacts which are similar to a set of
previously retrieved knowledge artefacts (retrieved based on domain concept(s)) and to identify
domain concepts which are similar to the original one.
page
12
3 Domain, Task, Lear ning Goal Model and
their Relationships
This chapter explains the three model structures and their relationships which provide the basis for
reasoning within APOSDLE:

Domain model – provides a representation of the learning domain in OWL format

Task model – provides a representation of the work tasks to be supported

Learning goal model – provides a mapping between domain concepts, tasks and general
learning goal types
We present the structures of the models, explain their roles (Sections 3.1, 3.2 and 3.3) and
interdependencies (Section 3.4), and discuss the choice of modelling languages. In deliverable DVI.8
Application Partner Specific Models the reader will find the specific models created by the four
application partners. These specific models utilize the structures presented here. In addition, in order
to create these specific models the application partners employed the integrated modelling
methodology. It was created in APOSDLE in order to streamline, simplify and support the modelling
activities of the application partners and is documented in Ghidini et al., 2007.
3.1 Domain model
The objective of the domain model is to provide a semantic/logic description of the learning/work
domain of an APOSDLE deployment environment. The domain is described in terms of concepts,
relations, and objects that are relevant for this domain. Technically speaking, the domain model is an
ontology that defines a set of meaningful terms which are relevant for the domain and which are used
to classify and retrieve the knowledge artefacts (see also Chapter 6). Figure 3-1 provides an example
of a domain model of APOSDLE (concepts, sub-concepts and properties) and shows it's usage for the
classification of knowledge artefacts.
page
13
Domain Model
Figure 3-1: The domain model and the relation with resources and knowledge artefacts
In Figure 3-1, the upper part shows the domain model for one of the application partner‟s domains.
The model is a hierarchical structure (left hand part) and contains relation types (right hand part). The
red lines show how concepts and relations from this model are related to documents and knowledge
artefacts.
page
14
3.1.1
Challenges
The research area of Knowledge Representation and reasoning provides a set of mature of-the-shelf
approaches and tools for the representation of a domain. In APOSDLE we build on top of these
results, focusing more on the problem of generating a good representation of the domain rather than
developing new representation paradigms. So the main research challenge in APOSDLE concerns
supporting the construction of appropriate domain models. The parameters that must be considered in
order to build an appropriate model are the following:

Level of abstraction. The same domain can be described at different levels of abstraction.
Choosing the right level of abstraction is an essential issue to guarantee the effective usage of
the domain model. If the level of abstraction is too high, then learning about too generic
concepts is in many cases useless. Furthermore, finding enough abstract description of a
certain concept is quite often very difficult. At the opposite side, if the domain model is too
detailed, then learning about one particular aspect could require one to learn about many
irrelevant details. In defining the right level of abstraction, one should take also into account
the material available in the documents and the necessary effort for annotation. Again, a finegranular domain description necessitates a detailed annotation of resources in order to make
use of them. APOSDLE offers, via the associative network (see Section 6.4), the possibility of
annotation at some intermediate level.

Coverage. The domain model should completely describe all aspects of a learning domain.

Reusability. The construction of a domain model is very expensive in terms of human
resources. So it's important to build models that can be reused in many different situations.

Robustness to revision. The models constructed should be robust for small changes and
adaptations. Indeed, in the course of usage of the APOSDLE platform it will happen that the
domain changes or it becomes necessary to represent a new part of the domain. This will
require updating the domain model. Such an update should be done with minimum effort and
minimal impact on other parts of the system.

Background knowledge. The knowledge about a specific domain is typically composed of
specific domain knowledge that is a characteristic of the single domain, and some background
knowledge, which is more general knowledge common to a large set of domains. For
instance, in formalizing the RESCUE domain, the notion of an i* model is domain specific and
concepts such as, user, application … are generic enough to be applicable to a wide range of
domains. The research challenge here is to make available as many pre-compiled background
knowledge as possible, and to allow a smooth integration with the domain specific knowledge.
3.1.2
The APOSDLE Approach
We based our representation on semantic web technologies. Domain models are mainly represented
using OWL language (Web Ontology Language) which is based on Description Logics (Baader et. al,
2003). This approach allows expressing classes, properties and instances and axioms among them.
This is standard methodology of the semantic web.
3.2 Task model
The first version of the conceptual architecture contained a detailed discussion of task modelling in the
light of the requirements of APOSDLE prototype 1. YAWL seemed to be a good tool to model our work
processes. In the following paragraphs, we will critically review the need for process modelling
languages in the light of the modelling as done in APOSDLE prototype 2. At first glance, workflow
modelling languages seem to be a good alternative to model our application domain, work-integrated
learning at the workplace. However, after a brief introduction of YAWL we will summarize why YAWL
(Van der Aalst & ter Hofstede, 2002) – at least as a tool environment - does not fit our needs while
page
15
modelling the E-Learning tasks of four different application partners. We will conclude with a
discussion of two approaches to modelling.
3.2.1
Workflow approach – critical reflection and choices in an organizational setting
YAWL (Yet Another Workflow Language) is a workflow language based on Workflow patterns. A
workflow can be defined as a composite set of tasks that comprise coordinated computer-based and
human activities (Lei & Singh, 1997; Leymann, 2006). A workflow model or schema is a formal
representation of work procedures that controls the sequence of performed tasks and the allocation of
resources to them (Bardram, 1997; Oberweis, 2005). The scope of workflow languages is an improved
efficiency. The automation of business processes often results in the elimination of many unnecessary
steps. Furthermore, an improved management of business processes leads to better process control
and flexibility. A better control over processes enables their re-design in line with changing business
needs.
As described in the first version of the conceptual architecture, YAWL seemed to be a good tool to
model our work processes. In the following paragraphs we will discuss why we reached the limits of
YAWL in real work-integrated learning settings, that is, applying it in the contexts of our application
partners.
Usability
The usability of the applied tool was poorly rated by the application partners. Although there is a
graphical interface, the user interface cannot be compared with other tools for task modelling.
Complex processes lead to a confusing and unclear arrangement of different icons. In addition, the
training period to efficiently use the YAWL editor is too long. Modelling for work-integrated learning will
mainly be done by domain experts with little background knowledge in modelling tools, therefore
YAWL does not seem to be suitable for our purpose.
Scope of the tool
YAWL was mainly written to model rather straightforward business processes. Processes involving
work-integrated learning seem to be much more informal than those in some sectors. Parallelization
and jumps can only be cumbersomely constructed in YAWL. Because those unusual constructs occur
frequently in our domain, they result in increasing modelling effort and confused models.
Incompatible Versions:
During the different phases of our project, several models were developed. Incompatible versions
complicated the process. To run different models in all versions of our prototype, they had to be
migrated. This fact is probably a result of the fact that YAWL is not driven by a standardization
organization, in contrast to, for example, BPEL.
3.2.2
Conclusions - recommendations for the conceptual architecture
For our conclusion on a workplace-based learning system we briefly point to two directions to replace
the initial approach, which was to map a YAWL task model and an OWL domain model. The first
approach would be keeping and restricting YAWL as a tool (but not as a format) and just train the
application partners for a minimal set of functionality. The other one would be replacing the whole
YAWL-environment by an environment capable of OWL, which we decided to do for APOSDLE
prototype 3.
Transformation approach
A solution to cope with the drawbacks from the former section could be to mainly use the YAWL
modelling environment merely as a drawing tool for task-subtask structures. The resulting XML should
then be transformed to OWL matching the OWL-based APOSDLE knowledge base. The advantage
page
16
would be producing an XML, that is, machine-processable output, already in the informal definition of
a task model. However, the YAWL environment would have to be changed and restricted to keep the
system from failures in the formal checks of the YAWL models. If we abstract from that property, which
is rather YAWL-specific, the transformation approach would also apply, if the YAWL environment
would be replaced by another workflow environment, for example a BPEL environment. We
recommend this approach to organizations, which want to re-use a given workflow or to formalize the
resulting APOSDLE workflow further in an (APOSDLE-external) workflow management system.
OWL-approach
The alternative solution is to model the task-subtask structures on the basis of OWL. For the tools
support this means a possible choice between wiki-based technology (as applied in APOSDLE) and
ontology editors as for example Protégé. The modelling means for tasks and subtasks come along
with the part-of relation. The logical and sequence-specific information has to be captured by a
numbering scheme, which is similarly intuitive as numbering sections and subsections in a text
document. Such a numbering scheme is currently under development and experimentation and will be
explained and detailed in the deliverable on the integrated modelling methodology. Nevertheless, the
nd
2 Prototype showed, as an emergent property of the task modelling process, that the users
introduced numbers to groups and order their tasks during informal modelling. We took this as a clear
hint, that usability, tool-related frictions and the versioning problems as explained in the previous
section could dealt with by starting formal modelling by means of OWL – that means let the knowledge
engineers model their task models “as similarly as possible” to a domain model and step back from
introducing a highly formal workflow language at all.
3.2.3
Conclusion
During the course of the APOSDLE project, we will follow the second approach, as the workflow
models come “from scratch” in the sense of already formalized input – and there is no requirement
demanding, that the models have to re-enter a workflow management system. For knowledgeintensive work and the resulting workflows this seems to be the best alternative. However, for each
new organization eliciting APOSDLE requirements, the transformation approach/opportunity should
also be taken into account. The most interesting open research question would then be how for
instance the processes in knowledge-intensive work, such as software engineering, could be extracted
from existing systems, which go beyond workflow management in less knowledge-intensive work. This
question can only be tackled by a set of empirical studies involving (to keep the frame of the software
engineering example), for example bug tracking systems, electronic tools for Scrum and lightweight
workflow tools like MS Project.
3.3 Learning goal model: a competence-based approach for formalizing
learning goals and prerequisite knowledge
3.3.1
Challenges for realising adaptivity in work-integrated learning with APOSDLE
One of the main goals of APOSDLE is to provide a user with learning content in a highly adaptive
manner. For the selection of learning content, the APOSDLE environment should take into account the
actual learning need of a knowledge worker. The actual learning need shall be determined by
requirements of a task at hand, and by a knowledge worker‟s existing knowledge and skills, which has
originated from previous learning and working experiences.
For realising this kind of adaptivity, two types of models have to be realized. First, information about a
user in terms of his or her existing knowledge and skills needs to be stored. In adaptive learning
research, the model in which this kind of information is maintained has been called the user model
(also student model, see Murray, 1999; Albert et al., 2002). To realize adaptivity to a full extent, the
user model has to be updated according to the user‟s learning progress while engaged in the use of
the system. In case of APOSDLE, the user model is stored as part of the user profile (see s 4.1.1 and
page
17
4.3), and the update is one of the services performed there. Second, in order to adapt the provided
learning content to the tasks that have to be performed in the work process, a second model needs to
define a learner‟s learning goals in terms of the knowledge and skills that are required for performing
each task. This second model is the learning goal model and provides a mapping between the task
model (see 3.2) and the knowledge and skills needed to perform the tasks (learning goals).
Looking for a common conceptual basis for these two models, we chose a competence-based
approach in the context of APOSDLE to address the challenges mentioned above. The use of
competencies has often been advocated as a way to deal with the challenges in workplace learning
(Green, 1999; Lucia & Lepsinger, 1999) for several reasons. Competencies are being used to more
closely relate learning to organizational requirements such as organizational goals, or task
requirements. Putting personal competencies in the centre of professional education seems necessary
as the content of tasks is changing so rapidly that requirements cannot be defined in detail. The shift
to competencies is therefore not a fashionable hype, but a necessity for organizations to cope with
uncertainty (Weinert, 1999).
Another reason why competencies have received significant attention is technical in nature. The fact
that competencies can be used as an abstraction to describe several different types of objects (such
as persons, tasks, content objects and others) with the same vocabulary, make them a good
candidate for reducing the complexity by introducing a semantic layer (Hockemeyer et al. 2003; Sicilia,
2005a,b). However, this use of the concept has lead to a diversity of definitions and conceptualizations
discussed in more detail below.
Though the concept of competency is of research interest in several different scientific disciplines (for
example, organizational psychology, educational sciences, management), the term lacks a standard
definition. In the management literature, for instance, competency can be a trait of an individual, a
group or an organization (see, for example, Rumsey, 1997). A related definition speaks of “core
competencies” (Prahalad & Hamel, 1990), and refers to the unique capabilities of a business
organization to deliver products and services, giving it its competitive edge on the market. A rather
psychological definition has been brought up by Boyatzis (1982), who defines a competency as “an
underlying characteristic of a person in that it may be a motive, trait, skill, aspect of one‟s self image or
social role, or a body of knowledge which he or she uses” (p. 21). This is rather typical for the way
competencies have been conceptualized in the area of job analysis where a major effort has been
undertaken to describe job requirements in standard terms (for example, as KSAOs, see below) in
order to establish a common language for all Human Resource processes (like recruiting, selection,
training and appraisal) (Green, 1999; Lucia & Lepsinger, 1999). In the educational sector,
competencies are seen as a way to focus curriculum development more on generalizable proficiencies
and abilities rather than on specific and isolated skills (van Assche, 2007). Finally, existing
conceptualizations of competencies have been taken up in the Information Systems field either to
support existing HR Information Systems (Schmidt & Kunzmann, 2006) or in the area of eLearning
(Sicilia, 2007).
Not surprisingly, this diversity in the adoption of the concept of competencies has lead to a plethora of
definitions. In a review, Weinert (1999) concludes that “in all of these disciplines, competency is
interpreted as a roughly specialized system of individual and/or collective abilities, proficiencies, or
skills that are necessary or sufficient to reach a specific goal” (p. 4).
These conceptual problems come along with several requirements on modelling learning for
APOSDLE. In order to allow for adaptive learner support, the model needs to have an interface with
the user profile present in the learning environment. Furthermore, in order to allow for adaptive
support with regard to a task at hand, there has to be a link to the task model of the APOSDLE
environment. As content has to be retrieved that takes into account the learning need of a user, an
association with the domain ontology is required. Predicting a worker‟s performance in a “new” task
requires inferences about prerequisite relationships among tasks. Computing optimised learning paths
requires prerequisite relationships among different learning goals to be achieved.
page
18
3.3.2
The conceptualizing of learning goals in APOSDLE
Looking at the requirements of a competence-based approach for APOSDLE, it became clear that a
much narrower conceptualization would suffice, and hence a narrower scope should be devised for
the model. First, as the main target is to detect a knowledge worker‟s learning need for a task at hand,
and devise appropriate learning goals, the concept of a learning goal was established as the central
concept around which adaptivity should be realized. A learning goal is understood as a discrete unit of
knowledge or skill which is needed to perform a task, and which a knowledge worker attains at a
certain point in time by performing learning activities or attains by practicing in tasks and becoming
proficient in them.
Second, the scope of learning in APOSDLE is limited to learning at work, which also determines the
scope of the learning goals modelled in APOSDLE. Which ones to include and exclude has to be fine
st
tuned with the learning support provided (see Section 5.1). Also it has been shown (see the 1
Workplace Learning Study by de Hoog et al., 2006) that learning events should be concise and cover
a time range of minutes to hours rather than days to weeks, which again is an indication of a finer
granularity. A further limit on the scope of learning is an emphasis within APOSDLE on domain
specific learning activities, rather than generic (soft) skills.
As we are seeking a competence-based approach for devising and structuring learning goals, we are
next reviewing several conceptualizations of competencies from which we then derive a conception of
learning goals for APOSDLE.
Because of their great popularity, various approaches have been brought up to categorise and
organize work-related competencies. For instance, New (1996) proposed a “three-tier model” of
managerial competencies. By means of a case study, van den Berg (1998) suggested to classify
competencies into Knowledge, Behavioural Style, Cognitive Capacity, and Personality. Within another
conceptualization, the KSAO approach (see for example, Schippman et al., 2000; Lievens et al., 2004)
led to categorizing competencies into Knowledge, Skills, Abilities, and Other characteristics. The
KSAO classification has also been used for structuring competencies in the o*net database on
occupational information (National O*NET Consortium, 2005).
All these approaches have in common their high degree of abstraction, which is not desirable in the
APOSDLE context for the abovementioned reasons. Since learning artefacts are retrieved by means
of a domain ontology, the abstract concepts of the competency model would have to be linked to
concrete domain model elements, which constitutes an additional potential source of errors for
modelling. One way to overcome these difficulties has been suggested, for instance, by SánchezAlonso & Frosch-Wilke (2007). The authors present a way for integrating competency-related
concepts in domain ontologies. In APOSDLE, we want to go one step further. We argue that all
learning goals, by definition, have to be related to the domain model. In other words, each element in
the domain model is related to a (different) learning goal. However, an ontology in itself, when it only
formalizes factual (or declarative) knowledge about a domain, is not sufficient for deriving learning
goals. Instead, learning goals refer to a cognitive action. They describe a state of a person in which he
or she is able to do something: perform a task, solve a problem or perform some kind of mental
operation. In short, learning goals need to describe the way declarative knowledge is used or applied.
This is in line with conceptualizations which connect semantic models to a competency
conceptualization. For example, Pernici et al. (2006) describe skills in terms of how domain knowledge
is being used: a skill is a knowledge object plus an action verb.
Preferably, this second component of a learning goal (the action verb) is derived from a general
instructional theory and should be independent of a specific learning/work domain. Anderson &
Krathwohl (2001) have presented such a conceptualization which we adopt. We use elements of the
cognitive process dimensions of Anderson & Krathwohl (see Section 5.1.4 for more details) remember,
understand, apply, and create) to establish the procedural component of a learning goal which we will
call its learning goal type. Taken together, the combination of one domain model element and one
learning goal type constitutes one learning goal. This conceptualization is in line with recent
developments in other FP6 projects, such as the iClass project (see, for example, Albert et al. 2007 or
the Calibrate project (van Assche, 2007).
page
19
Summarizing: As we are aiming for a narrower scope in terms of the units of knowledge to be
acquired, we have devised the concept of learning goal as the main unit of analysis. This will be
detailed in the subsequent sections. In order to also be consistent with the broader competency
conceptualizations mentioned in the review above, we will be seeking to link our conceptualization to
these approaches. A first attempt in doing so is presented in Section 3.3.6 where we compare our
approach to standard industrial competency management practice.
3.3.3
Role of Learning Goals in APOSDLE
Learning History
The role of learning goals within the APOSDLE approach is shown, in a simplified way, in Figure 3-2.
Task
Learning Goal
has engaged in
has engaged in
Learning Need
User
wants to perform
Task
Learning
Content
receives
has
requires
is linked to
Learning Goal
is related to
Domain
Concept
Figure 3-2: The role of learning goals within the APOSDLE approach
Performing a task is the main objective of a knowledge worker. This may lead to a learning need
associated with the task he or she wants to perform. The task requires certain knowledge and skills
which are modelled as discrete learning goals. The mapping between tasks and learning goals is
formalized in a Learning Goal Model. Given a valid mapping, and given a valid learning history in
terms of learning goals the learner has engaged with in the past, APOSDLE can detect the learning
need of the worker in terms of the learning goals he or she needs to attain. Each learning goal is
related to an element in the domain model (domain concept), for each of which (ideally) learning
content is available. This way, APOSDLE should be able to retrieve knowledge artefacts tailored to the
learning need of a user, taking into account both the learning need as specified by the task at hand
and the learning history of a worker.
More concretely, we are using the learning goal model for three purposes. First, based on the
assumption that in order to perform a task successfully, the user needs knowledge about all learning
goals related to one task, we regard the past task engagements of a user as implicit information about
his or her knowledge and skills instead of an explicit (self-) assessment. This procedure is termed
event-based learning history (see 3.3.5). Second, we are determining the learning goals of a user for a
task at hand in terms of knowledge and skills he or she has to acquire by performing a learning need
analysis. Third, we are computing an optimised sequence, a learning path, for attaining learning goals
page
20
taking into account a prerequisite relation on the set of learning goals (this will be described in Section
5.1.4).
3.3.4
The learning goal model and its theoretical background: competence-based knowledge
space theory
In order to compute a task-based learning history, and a learning need analysis, a formal model is
needed that allows for inferences about what knowledge is required for a certain task, and which
learning goals a user may potentially have when engaged in the task. Given such a model,
conclusions can be drawn from a worker‟s past task engagements about his or her most likely state of
knowledge. Given the knowledge state of a worker and the knowledge requirements of a task at hand,
a discrepancy can be identified and instructional support can be initialised. Moreover, prerequisite
relationships among learning goals need to be represented in the learning goal model in order to
derive individualised learning paths.
One approach for dealing with prerequisite relationships among learning goals has been described in
the context of the Intelligent Guide system (Khuwaja et al., 1996), a competency-based curriculum
sequencing technology for educational purposes. In the Intelligent Guide system, domain concepts are
linked via nodes in a network. The core of the system is the pedagogy engine, which is responsible for
interpreting the knowledge state of the learner. Depending upon his or her state of interaction, it
determines a learning path in order to attain a certain learning goal. The behaviour of the pedagogy
engine depends on the domain knowledge in the knowledge network, the user‟s actions, an
assessment of the user‟s knowledge state, and the pedagogy knowledge represented as rules in the
engine.
Recently, several models for ontology-based learning systems were proposed for competence-based
technology-enhanced workplace learning. For instance, Sicilia (2005a) introduces the basic elements
of an ontology of competency, where prerequisite relationships between knowledge elements,
attitudes, skills and competency elements are explicitly modelled. Similarly, Schmidt & Kunzmann
(2006) propose to model competencies hierarchically decomposed into other (sub-) competencies
(subsumption relation). Within the latter two conceptualizations, a user‟s learning history in terms of
knowledge and skills he or she has acquired is stored in a user model (see, for example, Albert &
Mori, 2001). In a concrete learning situation, the user model has to be related to a model of learning
goals for the learning domain. Moreover, besides the mapping between learning goals and their
prerequisites, a mapping between learning goals and work tasks or job situations has to be
established for taking into account the user‟s work context. Otherwise, the suggested learning path
would be independent from the task at hand. Taken together, a learning goal model needs to connect
the working domain (tasks) with the learning domain (learning prerequisites).
Ley et al. (2005) have suggested Competence-based Knowledge Space Theory as a framework to
formalise learning goals and their connection to workplace performance for work-integrated learning.
With the Competence-based Knowledge Space Theory, Korossy (1997) has introduced an extension
of Doignon & Falmagne‟s Knowledge Space Theory, which has been developed in the 1980s and 90s
as an attempt to model a person‟s knowledge as close as possible to observable behaviour (Doignon
& Falmagne, 1985, 1999). This was developed as a qualitative set-theoretical approach for the exact
testing of a learner‟s knowledge in a technology-enhanced learning environment. Knowledge Space
Theory has been applied in adaptive testing and tutoring scenarios and systems (ALEKS Corp., 2003;
Hockemeyer et al., 1998). One fundamental idea of Knowledge Space Theory is that a person‟s
knowledge in a certain domain can be described as the set of problems this person is able to solve.
This set of problems constitutes the person‟s performance state. The second fundamental assumption
of Knowledge Space Theory is that prerequisite relationships exist between the problems. This comes
along with two implications that are relevant for APOSDLE. First, not every combination of problems
constitutes a feasible performance state. The collection of all feasible performance states is called a
performance space. Second, from a person‟s performance concerning one problem, it is possible to
infer his or her performance in other problems.
page
21
In an attempt to develop Knowledge Space Theory further, Korossy suggested that in addition to the
set of problems, one should look at the set of elementary competencies that are needed to solve the
problem. This would generate information about the reasons for different levels of performance, and
thereby help to suggest instructional interventions. Similar to the set of problems, elementary
competencies are also structured in a competence space which results from a prerequisite relation on
the set of competencies.
The relation between the two sets (problems and elementary competencies) is formalised by an
interpretation function which maps each problem to a subset of competence states which are
elements of the competence space. This subset of competence states contains all those competence
states in each of which the problem is solvable. The competence state in which a person has to be
minimally to solve a certain problem is termed the competency interpretation of a problem. The
interpretation function induces a representation function which assigns to each of the competence
states all problems which are solvable in that competence state. Which problems are solvable is
determined by the interpretation function. If a knowledge domain is structured by a competence space,
a performance space, an interpretation function, and a representation function, we call the model a
competence-performance structure.
Competence-based Knowledge Space Theory has been applied in technology enhanced learning
solutions. For example, Hockemeyer et al. (2003) assigned “competencies required” and
“competencies taught” as metadata to a collection of learning objects. Thereby, prerequisite structures
could be derived for the eLearning content which allow for adaptive tutoring. New course content could
easily be integrated, as metadata was only held locally.
The learning goal model in APOSDLE - which could be termed as a task-learning goal structure - is
formalized according to the principles of a competence performance structure in the following way.
The elements of the two sets are the tasks the knowledge worker has to perform in the workplace and
the learning goals which describe the knowledge and skills needed to perform these tasks. We
assume both prerequisite relations on the set of tasks and on the set of learning goals. The mappings
between the two sets are called the interpretation and representation function. A feasible set of
learning goals is called a knowledge state and it induces a task representation which gives all tasks
that can be performed in that knowledge state.
A brief formal example for a task-learning goal structure in a knowledge domain encompassing the set
2
of tasks {T1, T2, T3, T4} and the set of learning goals {K1, K2, K3, K4} can be found in Table 3-1.
This table shows the interpretation function, the representation function, the learning goal
interpretation of each task, the feasible knowledge states, and the matching task representations of
the example structure.
Interpretation and Representation Function of the Task-Learning Goal Structure
Representation Function
Knowledge State
Interpretation Function
{ }
{K1}
{K3}
{K1, K3}
{K1, K2, K3}
{K1, K2, K3, K4}
Learning Goal Interpretation
X
X
X
{K1}
X
X
X
{K3}
X
X
{K1, K2, K3}
X
{K1, K2, K3, K4}
Task
T1
X
T2
X
T3
T4
Task
Representation
{ }
{T1}
{T2}
{T1,T2}
{T1,T2,T3}
{T1,T2,T3,T4}
Table 3-1: Interpretation and Representation Function of the Task-Learning goal Structure (Example)
2
Note that usually there are more tasks than competencies in a learning domain.
page
22
The column “Learning goal Interpretation” indicates for each task the knowledge state a person has to
be minimally in to be able to perform the task. For example, in order to perform the task (T3), the
knowledge state {K1, K2, K3} is required. Crosses specify the interpretation function in the table by
assigning to each task all knowledge states in which the task is solvable. The task (T4), for instance, is
only solvable by a person who is in knowledge state {K1, K2, K3, K4}. Task (T2) is solvable in
knowledge states {K3}, {K1, K3}, {K1, K2, K3}, and {K1, K2, K3, K4}. The row “Task Representation”
shows for each of the knowledge states all tasks that are solvable in that knowledge state. For
example, a person who is in knowledge state {K1, K3} is able to solve the set {T1,T2} of tasks.
The prerequisite relation and a task-learning goal structure can be visualised by means of a HasseDiagram (Figure 3-3). Interpreting lines going from bottom to top leads to prerequisite relations. In the
prerequisite relation on the set of learning goals, for instance, (K1) is a prerequisite for (K2) and (K4).
Task - Learning Goal Structure
Prerequisite Relation
(Learning Goals)
{T1, T2, T3, T4}
K4
{K1, K2, K3, K4}
… Learning Goal
… Prerequisite Relationship
(reflexive, transitive)
K2
{T1,T2,T3}
{K1, K2, K3}
K1
K3
{T1, T2}
{K1, K2}
Prerequisite Relation
{T1}
{T2}
{K1}
{K3}
(Tasks)
… Task
T4
{}
… Prerequisite Relationship
(reflexive, transitive)
{}
T3
… Task Representation
… Knowledge State
T1
T2
… Prerequisite Relationship (reflexive, transitive)
Figure 3-3: Task-learning goal structure, prerequisite relations on the sets of tasks and learning goals
Another advantage of Competence-based Knowledge Space Theory is that it incorporates a model of
the knowledge domain and a model of the individual learner. Moreover, a task-learning goal structure
can be combined with a domain ontology and there are various possibilities for evaluating the validity
of a task-learning goal structure in a learning domain.
For creating the learning goal model, we have developed a tool the TACT (Task Competency Tool)
which uses the domain ontology and the task model as input structures, provides ways to link the two
by mapping the elements from the two sets, and generates prerequisite relation as an output. Short
descriptions of previous versions of the TACT have been given in Deliverable DI.3 Integrated
Modelling Methodology and in Deliverable DP2 Second Prototype. A detailed description of the TACT
tool for APOSDLE‟s third prototype is given in Deliverable DI.6 Integrated Modelling Methodology.
page
23
3.3.5
Using the learning goal model in APOSDLE
As mentioned above, in the scope of APOSDLE, we use Competence-based Knowledge Space
Theory for three practical purposes: for building a task-based learning history, for performing learning
needs analysis, and for computing learning paths. Below these will be described in more detail.
Building an event-based learning history. Competency testing has a long tradition in business and
organisational psychology, and several approaches were proposed for testing a worker‟s work related
knowledge and skills. Most of them require significant efforts and have a rather low validity, since very
often the test situation and the test material have a moderate validity for predicting concrete task
performance (see, for example, Arvey & Murphy, 1998). Our approach for assessment will be less
ambitious. As the main purpose for building a user model is not to perfectly diagnose the user‟s
knowledge and skills but only to arrive at the best fitting learning goal for a specific task and a specific
user, we will restrict ourselves to the rather heuristic approach of an event-based learning history. The
learning history for each user consists of a collection of user actions (for example, performing a task)
related to the domain concepts the user was engaged with in the past. From these user actions, we
approximate the most likely knowledge state of the user concerning a certain domain concept (for
more details see 4.3).
Due to the direct link of tasks and learning goals in Competence-based Knowledge Space Theory,
there is an opportunity to approximately assess a worker‟s minimally needed knowledge state by
reviewing his or her past task performance. The rationale behind this approach is one that regards the
application of knowledge in performing a task as a valuable learning experience. This type of
workplace learning has been discussed as informal and experiential (see, for example, Garavan et al.,
2002).
For approximating the minimal knowledge state, we look at the set of tasks the worker has engaged
with in his/her past use of the system. From the learning goal interpretation of each of these tasks, we
infer this state. Several algorithms are conceivable for this inference. The easiest approach is to
regard the minimal knowledge state of a person as the union of the learning goal interpretation of all
tasks he or she has engaged with in the past. For instance, if a worker in the example of Table 3-1 has
performed the tasks (T1) and (T3), the set {K1, K2, K3} would be regarded as his or her minimal
knowledge state. The appropriateness of an algorithm for building a task-based learning history
depends on the validity of the learning goal model and on other properties of the learning domain,
such as the number of learning goals or the number of tasks.
In the second APOSDLE prototype (see Ley et al., 2008a), we have based the building of the user
model solely on information on past tasks performed (a task-based learning history). While there is
some evidence that in fact most learning at the workplace is connected to performing a task, and that
task performance is a good indicator for available knowledge in the workplace, this restriction to tasks
performed certainly limits the types and number of assessment situations that are taken into account.
It is evident that a user‟s knowledge and skills do manifest themselves through other types of user
interactions with the work-integrated learning system. For example, a user who seeks help while
performing a task might be in a different knowledge state than a user who provides help to others.
Additionally, the tasks a user performs may be driven by organizational constraints or simply by task or
job assignments, and may therefore only draw a partial picture of the knowledge and skills a user has
available.
Won & Pipek (2003), for example have introduced what they called “competence-indicating events”
that are used to create awareness of a users competencies in a computer-supported cooperative work
scenario. These authors allow users to flexibly construct hypotheses about certain events (for
example, “User A answered many questions in a Java a newsgroup”) and their relation to fields of
expertise of the users (for example, “User A is an expert on Java”). In the context of adaptive
eLearning, the approach of evidence-bearing events was suggested to maintain user models based
on various user interactions with the system Brusilovsky (2004).
In a similar manner, we have developed algorithms for making inferences about which knowledge and
skills a worker has available by tracking his or her interaction with the APOSDLE system. We are
page
24
calling these events “knowledge indicating events”. For instance, the participation in a chat about a
certain topic, or the publication of a document could be such competence-indicating events.
A more detailed discussion of the events that are collected in the user model and the inferences that
are drawn from these can be found in s 4.1.1 and 4.3.
Performing learning needs analysis. In order to provide the best-possible support for a worker who
tackles a certain task, the worker‟s learning need has to be specified. For this purpose, we employ a
learning needs analysis in which the requirements of a task in terms of knowledge and skills are
compared with the knowledge state of a worker. Again, several algorithms might be useful in this
context. The easiest one would be to take the difference set between the learning goal interpretation
of the “target” task, and the current knowledge state of the worker. In the example of Table 3-1, if the
target task is (T4), and the set {K1, K3} constitutes the worker‟s current knowledge state, the set {K2,
K4} represents the worker‟s learning need. Each element of the learning need then refers to one
learning goal.
Computing learning paths. One of the main assumptions in technology enhanced learning derived
from pedagogical principles and theory is that the learning process can be improved through guidance
(Schmidt, 2005). The concept of a prerequisite relation in the Competence-based Knowledge Space
Theory has direct implications for the best-possible sequence for learning goals to be achieved.
Returning to the example from above, if the set {K2, K4} constitutes the learning need of a learner, he
or she should satisfy (or reach the associated knowledge state) (K2) before (K4), since (K2) is a
prerequisite of (K4).
Reflecting a user’s learning history. In contrast to quantitative measurement approaches, competencebased Knowledge Space Theory is based on a qualitative approach for assessing knowledge. This is
especially applicable for our purposes as we are trying to suggest the best next learning activities,
rather than placing learners along some continuum of competence. An analysis of the knowledge
indicating events that can possibly be gathered in the APOSDLE system (see 4.3), suggests that the
types of events differ with respect to the orientation that users have when interacting with the system.
For each topic in the domain ontology, we differentiate between a learning (for example, consuming
learning content), a working (for example, producing documents) and a knowledge transfer (for
example, being contacted in a cooperation episode) orientation. In our understanding these different
orientations indicate different qualitative levels of knowledge and skills in use. Similarly, qualitative
differences in the use of knowledge and skill have been suggested, for instance, by Dreyfus & Dreyfus
(1986) and Eraut (1994) who identified five stages of skill development (Novice, Advanced Beginner,
Competent, Proficient, and Expert). In extending the learning history to encompass these qualitative
levels, we have created a visualization of a users learning history that is based on the collected data
as a personal reflection tool. Again s 4.1.1 and 4.3 should be consulted for details about collected
events and algorithms.
3.3.6
3.3.6.1
Comparison of the APOSDLE Modelling Approach to Standard Industrial Competency
Management Practice
Purpose of the Comparison
The purpose of this section is to compare the competence-based approach using learning goals as
the unit of operation as used in APOSDLE to standard practice of competency management in
industry settings. The need for such a comparison was prompted by the first APOSDLE review in
which it was recommended to consider whether the use of competencies as the basis of discussion on
employees‟ learning needs fit into the employers existing use of competencies for job descriptions,
personal learning plans, appraisal or career development,
This section makes a first step into this direction by comparing the use of competencies in APOSDLE
to standard industrial practice, and by giving an outlook on the future developments in this important
field of research.
page
25
3.3.6.2
Procedure
The following documentation was analyzed for performing the comparison: An excerpt from the
deliverable “D6.1 Requirement analysis and specifications: Industrial use cases” as well as the
deliverable “D6.2 Industrial use case: Design and modelling of the specific metadata and models” both
3
from the LUISA project and authored by Anne Monceaux and Joanna Guss from EADS-CCR.
3.3.6.3
Results
The analyzed documentation contains information about the conceptualization of the competence
domain within the Airbus division at EADS with some concrete examples of positions and required
competence profiles. The documentation also contains descriptions of a use case specifying how the
approach is being applied at Airbus. Although being specific for Airbus, the approach taken there
corresponds to several other industrial cases the author is aware of, such as those described in Wöls,
Kirchpal & Ley (2003), Hiermann & Höfferer (2003) and Pernici et al. (2006). As such the EADS case
can be seen as a good example of standard industrial practice in Competence Management.
Dimensions
considered in
the analysis
APOSDLE Approach
EADS Approach
Purpose of
Modelling
Building a task-based learning history
Performing task-based learning need
analysis
Devising learning paths as sequences
of learning goals
Supporting work-integrated learning
Performing job based learning needs
analysis
Devising training measures
Supporting competence appraisal in
annual performance review sessions
Granularity of
Learning objects
Granularity of learning objects
corresponds to Aggregation Level 1 or
2 in IEEE LOM (IEEE, 2002): Raw
media and fragments, and collections
of fragments.
Granularity of learning objects
corresponds to Aggregation Level 4 in
IEEE LOM (IEEE, 2002): Courses and
collection of courses.
Granularity of the
concepts used
Learning Goals: Single elements of
knowledge or skill needed to perform a
task and which can be acquired in a
work-integrated fashion.
Knowledge state: A collection of
knowledge and skills together with the
ability to apply them in a number of
tasks.
Knowledge & Skills: Knowledge and
know-how that need to be
demonstrated for a competence
Competence: Ability to apply a number
of activities, ability to apply knowledge
and skill
Job vs. Task
Based Analysis
The task performed in the workplace is
the point of departure, irrespective of
the position or job in which it is
executed.
The job or “profession” is the point of
departure in modelling and is used to
model devise the learning need.
Separation of
competence and
performance
Clear conceptual differentiation
between task performance and
knowledge. Knowledge and skills can
be utilized in a number of tasks.
One to one correspondence between
activities and skills. Skills seen mainly
as abilities to execute certain activities.
3
http://www.luisa-project.eu/www/
page
26
Dimensions
considered in
the analysis
APOSDLE Approach
EADS Approach
Proficiency
Levels
Qualitative approach that takes into
account application of knowledge in
different contexts.
Employing a scale (1-5) for
Competencies, as well as skills and
knowledge, as an attempt to measure
knowledge in a context free fashion.
Table 3-2: Overview of differences between the APOSDLE and the EADS approach
Six dimensions were considered in the comparison. The dimensions constitute important
characteristics of modelling in general and for competency modelling in particular. They are not
independent from each other. In each of the dimensions, marked differences in the approaches have
been identified. Table 3-2 gives an overview of the differences found.
Purpose of Modelling. Clearly, the purpose of the APOSDLE approach is narrower in focus than a
standard competency management approach. While work-integrated learning is only one aspect of
standard HR processes, it also has certain special requirements, namely (1) that it is more fine grained
in nature than in usual job based learning, and (2) that it is more strongly embedded and driven by
task executions. These two issues will be covered in more detail below, as they explain two of the
main differences found. Furthermore, the two last differences analyzed (3 and 4) relate to a more
general difference in the philosophy and history of the two approaches. These will be discussed
subsequently.
(1) Granularity of learning objects and granularity of the concepts used. A clear difference was found
in the granularity of the targeted learning objects or knowledge artefacts. The finer granularity needed
for work-integrated learning needs to be reflected by the granularity of the concepts used. As a result,
the main unit of analysis in the APOSDLE approach is on the level of units of knowledge and skills, not
encompassing the broader aspects of competencies. However, in line with research and practice, we
see a competency composed of a collection of related units of knowledge and skills together with the
ability to apply them in work-related tasks. In contrast to many approaches where the distinction
between these two levels is not made explicitly, the APOSDLE approach has a very clear conceptual
model to deal with this. A knowledge state (consisting of a collection of learning goals) and the
associated task representation (consisting of a collection of tasks) can be seen as the equivalent to a
competency in the sense described above.
The example given in Table 3-3 illustrates this by comparing one knowledge state from the APOSDLE
learning goal model of Prototype 1 with a part of a position profile in the EADS Competency Model. In
the EADS model, a competency requires a set of knowledge and skills. In the APOSDLE model, a
knowledge state is composed of a set of learning goals as well as a set of associated tasks. The
examples in the table are not equivalent, as different domains have been modelled. They are given
here to illustrate the possible linkage of concepts used in standard HR competency management and
in the APOSDLE approach.
(2) Job vs. Task Based Analysis. Standard HR practice is usually job based. This relates both to
modelling (for example, bundling knowledge and skills and competencies in a job driven fashion), and
to learning (for example, deriving competency gaps from a position profile). This is the practice
reflected in the EADS model. Table 3-3 shows that the model has specified professions which entail
concrete positions (which in turn can be held by particular persons). A profession consists of a
consistent group of activities and is composed of competencies, skills and knowledge which are
needed in that position. Position profiles can then be compared to employee profiles to devise optimal
learning opportunities.
In APOSDLE, the point of departure is a task, irrespective of the position in which it is undertaken. In
our view, the more dynamic organizational settings in knowledge work require that we employ a more
flexible approach than a job based approach is able to provide. Of course, a grouping of fine level
concepts (such as learning goals or knowledge & skills) to form more abstract concepts (such as
page
27
competencies or knowledge states) is always possible. As we have demonstrated in the example
above, such a grouping is not constrained by jobs or positions but can be formed more flexibly by
grouping any collection of tasks in the task model to form a viable knowledge state. As in the EADS
case a position is defined to encompass a set of activities (tasks), it is certainly possible to group
those activities to form a position profile.
APOSDLE Model (Prototype 1)
EADS Competency Model
Job: RESCUE Requirements Engineer
Profession: Requirements Based Engineering
Position: Engineer - Engineering Requirements
Knowledge State
Learning Goals:
Competency: Processes Method & Tools
Management
Knowledge & Skills:
 2. Knowledge about how to organise an
acquisition programme
 Design processes
 3. Knowledge about the Activity Model and
the activity descriptions
 Knows definition of the norms meth
 8. Knowledge about the domain and the
environment of the system
 Knows recommendations
 10. Basic technical understanding
 Support the teams
 Manages development environments
 36. Ability to apply available outputs to Use
Case Specification
 37. Knowledge of domain lexicons
Tasks:
 7.1 Gather input (HAM, SGM, Creativity
workshop output, precis, UC level
requirement)
 7.2 Design a domain lexicon
 7.3 Identify concepts relevant to Use Cases
 7.4 Identify Key Ideas
Table 3-3: Comparison between a knowledge state from the APOSDLE learning goal model of Prototype 1
and one part of a position profile in the EADS Competency Model
(3) Separation of competence and performance. Such a separation is not clearly foreseen in the
EADS approach, where a “skill” is defined as the “ability to perform a particular task”, and hence
defines a one-to-one correspondence. The approach we are following in APOSDLE allows us to keep
track of the task context in which certain knowledge has been applied in the past. As a unit of
knowledge (or learning goal) is mapped to a number of tasks, a user model may keep track of the
various tasks during which a learner has applied that knowledge before.
(4) Proficiency Levels. Similar to the previous point, the learning goal modelling in the APOSDLE
approach follows a different philosophy than standard competency modelling practice. Knowledge
Space Theory, as the underlying framework, follows a qualitative measurement approach and
presents a sharp departure from traditional parametric measurement approaches in the social
sciences (on which standard competency management practice is usually based). Again, an important
point here is to express expertise in terms of the (qualitative) task context in which knowledge has
been applied, rather than in terms of a context free measurement scale. This does not preclude that
certain scales can not be used in the future. However, these would need to describe differences in
expertise in qualitative ways rather than quantitatively. Furthermore, the APOSDLE approach is by no
means committed to a deterministic approach.
page
28
3.3.6.4
Discussion and directions for future research
Summarizing the comparison: what we found were differences in terms of granularity (1) and the
general organizational approach (2). We have shown a clear conceptual schema of how these two
issues will be dealt with by aligning the APOSDLE discourse with that employed in standard
competency management practice. It needs to be further researched if the alignment we are seeking
is actually feasible and beneficial. This would require finding ways to (a) align knowledge states and
competencies, and to (b) align the job based approach with the task based approach. The EADS case
within the APOSDLE project is a good opportunity to continue with this research and answer the
questions raised above.
We have further found two differences which relate to the conceptual differentiation between
competence and performance (3) and qualitative vs. quantitative measurement approaches (4). As
these are the core of the underlying conceptual model, we will continue to research ways for
integrating this into industrial practice.
3.4 The APOSDLE knowledge base
In a nutshell, the APOSDLE Knowledge Base contains all the necessary information about the tasks a
user can perform in a certain organisation (the task model), the learning goals required to perform the
tasks (the learning goal model), a description of the application domain of the organisation (the
domain model), and finally, specific APOSDLE categories used to classify tasks, learning goals and
learning material, that are exploited by the learning support in APOSDLE to produce the learning
material and suggestions (the APOSDLE categories).
3.4.1
Challenges
The APOSDLE knowledge base is to provide a machine processable integrated theory (= knowledge
base) of the application domain in which the APOSDLE platform supports work-integrated learning.
This implies that at every new installation of APOSDLE one needs to specify a new domain model,
task model and learning goal model. APOSDLE provides for this a modelling methodology and a set of
modelling tools that support the methodology.

The first objective is to provide a methodology and a set of tools for acquiring knowledge for
the AKB (APOSDLE Knowledge Base). To successfully achieve this objective, we have to
support domain experts in eliciting their knowledge in a logic-based model without requiring
them to be expert logicians. Furthermore, we have to exploit as much as possible those
company resources, such as a list of key terms, classification schema, database schema, and
other schemata, which are used by the companies to organize their document archive, data
and activities. These schemata constitute the result of an analysis and a conceptualization of
the domain, and encode domain knowledge in a quasi-logical format. For this purpose in
APOSDLE we have developed a collaborative modelling methodology based on Web 2.0 tools
(semantic media wiki) widely described in Christl et al. (2008) and Rospocher et al. (2008).

The second objective is integrate in the AKB different types of knowledge, such as the
procedural knowledge encoded in the task model, the knowledge about competences of
workers, and knowledge about the domain models. The main difficulty here is to establish the
connection among the different types of knowledge. For instance, stating that a certain task
needs some competences and that a task will deal with a certain concept. For this purpose we
have defined the APOSDLE metamodel described in Rospocher et al. (2008).

The third goal is to provide a set of services on top of the AKB for the APOSDLE learning
support and the APOSDLE classification tool. These knowledge services should be able to
answer both logical queries, that is, queries that imply some form of logical reasoning in the
models, and non-logic based queries, that is, queries that implies some form of analogical,
statistical, and similarity based reasoning on the knowledge. Again the problem is being able
page
29
to reason about different, but related, logics (that is, the logic of processes, the logic of the
competency model and the logic of the domain model). Implementation of such services is
based on the notion of semantic matching which are described in Scheir et. al (2007) and
Scheir et. al (in press).
We will describe the above challenges and the solutions provided in the following subsections
3.4.2
Knowledge acquisition
The problem of knowledge elicitation has been around since the beginning of Artificial Intelligence. In
the early years, to build expert system human experts were asked to express the knowledge about a
certain domain into machine processable languages such as rules. Recently, with the advent of the
semantic web, we have seen a major shift on how knowledge should be expressed. In particular logic
has become the basic tool to express machine processable knowledge, thanks to its declarative
fashion, its unambiguous meaning and the availability of efficient reasoning tools. This shift makes the
process of knowledge elicitation more complex than in the past, as the meaning of logical formulas
and the effects of axioms is much more complex than simple activation rules. Domain experts,
nowadays, need more support than in the past in the process of knowledge elicitation. Furthermore,
the recent development of the web (in particular WEB 2.0) highlights the importance of the social
aspects of knowledge. In our setting, this means that the elicitation of knowledge is not a process that
is performed by a single individual in isolation, but is more a social negotiation process. In this
negotiation process, the members of a community (the group of people who share and use a common
piece of knowledge for their activity) should converge to a common representation of the shared
knowledge.
In the area of knowledge engineering there are several well established methodologies and tools for
knowledge elicitation. One of the best known methodologies is the CommonKADS methodology
(Schreiber, et. al, 2000). This methodology was especially designed for building expert systems.
Recently a large number of methodologies and tools has been developed for many aspects of the
complex process of building and maintaining ontologies. For instance DILIGENT (Vrandecic, et al.,
2005) is a methodology focusing on the evolution of ontologies and in particular on users and their
usage of the ontology, instead of on the initial design, thus recognizing that knowledge is a tangible
and moving target. HCOME (Kotis et al., 2006) supports the development of ontologies in a
decentralized fashion, which in turn supports the creation of an agreed ontology between a group of
people in a community. These are only some examples of knowledge acquisition methodologies. The
paper by Cristani and Cuel (2005) provides a survey of proposals for ontology creation methodologies.
All these tools, however, could not be simply adopted in the APOSDLE project, as these
methodologies need the development of a set of intelligent tools that supports the automatic
performance of their steps.
For the APOSDLE prototypes 1 a more traditional approach was used. A person knowledgeable about
the domain constructed the domain model which was represented in the Protégé language. For
prototype 2 and 3, inside the project we have developed a collaborative methodology, based on an
extension of Semantic MediaWiki (http://meta.wikimedia.org/wiki/Semantic_MediaWiki) called MOKI
(for Modelling viKI) (http://moki.fbk.eu/) The methodology is described in more detail in the Deliverable
D 1.3 (Integrated Modelling Methodology. Below we give a brief overview of the methodology.
The APOSDLE Integrated Modelling Methodology guides the process of creation of the application
domain dependent parts of the APOSDLE Knowledge Base.
The methodology consists of five distinct phases, which cover the entire process of model creation,
from the initial selection of the application domain, to its informal, and then formal, specification, and
finally to the validation and revision of the final Knowledge Base:
 Phase 0: Scope & Boundaries. In this phase the scope and boundaries of the application
domain are determined and documented. The first step of this phase was to use questionnaires
to elicit the main tasks and learning needs of the different Application Partners in order to
identify candidate application domains for learning (also called learning domains). The
page
30
candidate application domains were then discussed and the final domain was decided upon,
and briefly documented. The key aspect of this phase is to support the Application Partners to
identify a learning domain appropriate for the "learn @ work" approach taken by APOSDLE.
 Phase 1: Knowledge Acquisition. The goal of this phase is the acquisition of knowledge about
the application domains that have to be formalised and integrated in the APOSDLE knowledge
base. The proposed methodology aims to extract as much knowledge as possible from both
domain experts and available digital resources identified by the Application Partners. The
elicitation of knowledge from domain experts is based on well known techniques like interviews,
card sorting and laddering, already introduced in literature, while the extraction of knowledge
from digital resources is based on algorithms and tools for term extraction described in (Scheir
et al., 2007). The key aspect of this phase is twofold: first, the methodology has to support an
effective and rapid knowledge acquisition from domain experts, who are often rarely available
and scarcely motivated towards modelling; second the methodology has to ease the process of
modelling by reusing knowledge already present in digital format in the organisation.
 Phase 2: Informal Modelling. The goal of this phase is to start from the knowledge elicited in
Phase 1 and provide an informal but structured and rather complete description of the different
models which will constitute the APOSDLE knowledge base: the Domain model ( 3.1), the Task
model ( 3.2) and the Learning goal model ( 3.3). The descriptions of the informal models are
obtained by filling pre-defined templates provided in a Semantic Wiki. The use of a Semantic
Wiki allows describing the elements of the different models in an informal manner using natural
language. However, at the same time it allows to structure the descriptions so that they can be
easily (and often automatically) translated into formal models, without forcing the Application
Partners to become experts in the formal languages used to produce the formal models. The
key aspect of the informal modelling is to provide the Application Partners with tools that
support informal modelling – and thus hide the complexity of formal languages – but at the
same time are not simple textual editors and provide some typical facilities of modelling tools,
such as the possibility to have a (possibly graphical) overview of the entire model.
 Phase 3: Formal Modelling and Integration. In this phase the informal descriptions of the task
model and of the domain model are transformed in formal models. The current version of the
methodology supports the automatic translation of the domain model in an OWL ontology, and a
manual translation of the task model in an OWL representation. Additionally, the information
about the learning goals contained in the informal models is automatically extracted and used
by the Task And Competency Tool (TACT for short), to help the Application Partners to formally
specify learning goals. The output of this phase is a first version of the APOSDLE Knowledge
Base. The key aspect of this phase is to simplify the transformation from informal to formal as
much as possible to avoid duplication of work and to make full use of the rich informal models
produced during phase 2.
 Phase 4: Validation & Revision. In this phase the APOSDLE Knowledge Base is evaluated and
possibly revised. The current version of the methodology provides the support for checking
automatically, via SPARQL queries, different aspects (properties) of the APOSDLE knowledge
base and of its single components. The results of these checks are evaluated and used to
revise the knowledge base, if needed. The key aspects of this phase are first to provide an
accurate set of tests for the APOSDLE knowledge base, and second to allow the domain
experts to validate the models obtained. This second aspect is particularly challenging as the
domain experts usually are not familiar with formal models.
This methodology has been followed by the Application Partners to build their specific APOSDLE
Knowledge Bases.
page
31
3.4.3
Knowledge Integration
The basic idea of knowledge integration in APOSDLE is based on federated, diverse knowledge
bases. In this federation, each type of knowledge (knowledge related to working, knowledge related to
learning, and knowledge related to the domain) is stored in the homogeneous format of the Web
Ontology Language (OWL). The integration is done separately in what we can define as the metamodel schema (see Figure 3-4).
Knowledge base – Working
Knowledge Base – Learning
Federated Knowledge Bases
Conceptually integrated via the
APOSDLE meta-model
Search
Knowledge Base – Domain
Figure 3-4: Knowledge bases conceptually integrated via the APOSDLE meta-model
The meta-model schema is an ontology represented in OWL. The meta-model schema is a formal
representation of the APOSDLE meta-model, which forms the conceptual basis for integrating the
federated knowledge bases. The conceptualization of this approach is shown in Figure 3-5.
Figure 3-5: The approach for knowledge integration in the AKB
page
32
3.4.3.1
Learning goal model – Task model
As was defined in 3.3 the notion of a learning goal is intrinsically connected with the ability to perform
a certain task. That is, a task is an activity that requires the mastering of certain learning goals. The
mapping between tasks and learning goals is a binary relation, which we denote with “requires”
between tasks and learning goals. Formally:
Requires Task LearningGoals
This simple definition raises the question: How does the specific structure of the task model and the
domain model affect this relation? For instance, if task T is, say, “Define Simulation Objective and
Requirements” and is composed four sub-tasks (for example, Identify and understand customer
needs, Elaborate Simulation objectives, Specify Simulation Requirement and Specify Simulation
Requirements), what is the relation between the learning goals of the sub-tasks and the more general
task? The inheritance pattern, as present in the concept/sub concept relation, is not so trivial, as in
some cases sub-tasks can be delegated and therefore one does not need the specific knowledge for
the subtasks. In APOSDLE we have tackled this problem by disallowing any propagation of learning
goals through tasks.
3.4.3.2
Learning goal model – Domain model
The basic idea is that learning goals of the user should be linked to domain concepts (see Section
3.3.3). For instance, if in the domain model we have the concept of “interview” then the learning goals
of a knowledge worker could be “being able to perform an interview” or “knowing what is an interview”.
The idea in APOSDLE is that these learning goals can be pursued by extracting learning material from
the knowledge artefact which is related to such a concept (see Section 5.1). The ontology itself, with
its axioms, metadata and comments, can also be exploited for extracting useful learning material. For
example, if a knowledge worker needs to know what is “Progressive Abstraktion” (example taken from
the domain of a German partner) then, by accessing the domain model (s)he can obtain learning
content like:

A “progressive abstraction” is a “Kreativitätstechniken” (showing the concept of which it
is a subconcept)

A “progressive abstraction” is a method of systematically specifying problems.
Assessing core questions of a problem with a systematic approach. (showing the
definition of the concept)

“Progressive abstraction” in English is Progressive Abstraction (showing it’s translation
into another language)
3.4.3.3
Task model – Domain model
The issue of combining procedural knowledge (described in the task model) with domain knowledge
(described in the domain model) is not novel, but it is becoming a real issue in the web-services
research area. However, there is not a standard approach to combine knowledge about processes
(tasks) with knowledge about objects (concepts). The language OWL-S (Martin et al., 2004), is an
extension of OWL for the specification of processes. Being an extension of the language for specifying
APOSDLE processes, it will be used for the integration of the process model with the domain model in
APOSDLE (see also Section 3.2).
3.4.4
Where else has something like this been used
Domain modelling via an ontology is a standardized methodology nowadays and most applications
that support semantic information retrieval are based on ontologies. Similarly, the classification of
documents (or any other set of object, like pictures, movies, etc.) with respect to an ontology is also a
standard practice in the semantic web. In the APOSDLE project we also use parts of the domain
model as learning material.
page
33
4 Context Awareness and User Profile
Ser vices
This chapter explains and stresses the importance of context awareness for work-integrated learning
systems in general and APOSDLE in particular. We start with providing the reader with the vision of
context aware systems, their origins, and current developments in this field. Specifically a number of
context aware systems are discussed. The APOSDLE approach to context awareness is based on a
common architecture developed by Baldauf et al. (2007) which includes the use of agents in three
layers separating the detection of context, planning and action based on context. It also details the
challenges of user context determination. The goal is to identify a user‟s current work task based on
the user‟s interactions with the system (key strokes, mouse movements, applications used, etc.) and
the metadata and content of the resources accessed (mail messages, documents, links to people,
etc.).
User context determination plays a crucial role within the overall APOSDLE approach. In order to be
able to provide the user with information, learning material and links to people relevant to her task at
hand, the system needs to identify the work task reliably. The identified task then updates the user
profile (Section 4.3) and causes several activities to be started pro-actively: re-computation of a user‟s
learning goal and learning goal histories (Section 3.3), retrieval of knowledge artefacts relevant to the
learning goal (Section 6.4), finding people relevant to the learning goal (Section 5.2), and the dynamic
creation of learning support (Section 5.1). The results are displayed in a resource list and a people list.
APOSDLE also stores user related context information in digital user profiles. These profiles are used
for maintaining the user‟s usage history and current context with respect to their personal work-,
learning- and cooperation-related experiences. The APOSDLE approach differentiates between four
forms of user related data: user data, usage data, inferred data, and environment data. This layering
of user profile information allows us to clearly separate between factual information and assumed
information about the user. The component responsible for operations upon user profiles is the User
Profile Service (UPS). The UPS‟ functionality is made accessible to other parts of the APOSDLE
system via a set of services. Examples are services for logging users‟ activities and services
computing queries for resources, which optimally match a user‟s current learning need. The UPS‟
contextualized services will be described (see Section 4.3).
These features were partly investigated in three component evaluations:

Context detection: the relation between acceptance and efficiency of a simple context
detection environment.

User context identification: which algorithm performs best for identifying user levels using user
behaviour.

Privacy: what is the opinion of users and experts about socio-technical and legal aspects of
APOSDLE privacy approach.
4.1 Context awareness and context determination
4.1.1
Challenges
In this the foundations behind the APOSDLE task detection engine, which is one of the major research
challenges to face if you want to support unobtrusive real-time task-based context-aware learning, are
explained. First, we introduce the concept of context awareness and explain possible context
architectures and then we detail the context-based task detection engine in APOSDLE.
page
34
4.1.2
Theoretical background
The vision of ubiquitous computing and context-awareness was developed by Xerox PARC researcher
Mark Weiser in the late eighties and early nineties of the previous century. In his landmark article “The
Computer for the 21st Century” (Weiser, 1991) starts with the following sentences: “The most profound
technologies are those that disappear. They weave themselves into the fabric of everyday life until
they are indistinguishable from it”. The terms context-awareness and context-aware applications were
introduced a few years after Weiser‟s vision was published. Context-awareness was defined in several
publications, most prominently in (Schilit et al., 1994; Brown, 1996; Abowd et al., 1997; Dey & Abowd,
1999; Schmidt et al., 1999).
All definitions have in common that information from the users‟ or the entities‟ environment is acquired
and used to trigger certain actions in the system. Using such context-information, systems adapt to a
given situation to improve the user‟s interaction with the system. Moreover, systems can trigger
actions autonomously. Finally, context can be attached to information to make it easily retrievable.
Once again, the use of context information in novel applications needs to be embedded in the way
humans live. The earliest context-aware applications had a strong focus on location. Want et al.
(1992) and Harter et al. (1999) presented the Active Badge system: it used ultrasound beacons to
determine the position of active badges. By coupling the system to a telephone exchange, it allowed to
route calls to the phone next to its recipient, even when the recipient was not sitting at his desk. The
system was extended with mobile devices: the ParcTabs, an early PDA-like device (Schilit, 1995;
Want et al., 1995). Brown (1996) presented stick-e notes: notes that are activated by context. As soon
as a certain context, such as location, time, or others, becomes active, stick-e notes are delivered to
the user. They envision the use of such notes for tourist guide applications, mobile workers, making
surveys, and control panels (a note is activated close to the device that is to be controlled). Dey and
Abowd (2000) described a similar system for context-aware reminders as well as many early projects,
which focused on mobile or wearable devices.
Much work on the management of notifications can in fact be considered as context aware. Horvitz et
al. (2003), for example, presented the BestCom system that allows managing telephone calls in an
office setting, depending on context factors such as, whether a conversation is taking place in the
office or not. Microsoft‟s ongoing MyLifeBits project (Gemmel et al., 2002) aims at collecting and
storing any digital information about a person, but leaves the annotation to the user. People do forget
things, even if they are deemed important. Czerwinski and Horvitz (2002) studied how (quickly) users
forget important events in a desktop-based computing environment. They aimed at identifying
important events automatically and providing memory prostheses for important computing events:
Beyond Capture & Access, a context aware application can support a user in performing a specific
task. Therefore, Dehn and Van Mulken (2000) defined user interface agents as follows: “Interface
agents are computer programs that aid a user in accomplishing tasks carried out at the computer,
such as sorting email, filtering information and scheduling meetings. These agents differ from
conventional computer programs in that they can act autonomously on behalf of the user, that is,
without requiring the user to enter a command or click a button whenever she wants the task to be
carried out. In addition to autonomy, a characteristic of intelligent agents is their ability to perform
tasks delegated to them in an intelligent, that is, context- and user-dependent way.” Current research
is particularly strongly geared toward user interfaces in which communication between humans and a
computer is partly mediated by this type of interface agents (as originally introduced by Maes, 1994).
Classic texts on user interface agents comprise a collection of key papers (see Cassell, 2000; Koda &
Maes, 1996; Rist et al., 1997; Dehn & Van Mulken (2000) for a thorough review of evaluation
techniques for embodied conversation agents). Well-known examples of interface agents are
intelligent tutoring systems and context sensitive help systems, such as Selker‟s COACH (Selker,
1994) or the Remembrance Agent (Rhodes, 1996). The latter is an autonomous interface agent that
reminds the user of relevant files stored on the user‟s local disk, thus "remembering" relevant material
that the user has already seen. This was later expanded (Rhodes & Stamer, 2000) to explicitly make a
contextually aware associative memory, for instance, to make literature suggestions based on the
browsing behaviour of a web-user. Liebermann and Selker (2000) explained the use of context in user
interfaces by the following: “Each application can be thought of as establishing a context for user
page
35
action. The application determines what actions are available to the user and what objects can be
operated on. Leaving one application and entering another means changing contexts. The user gets a
different set of actions and a different set of objects. Each tool works in only a single context and only
when that particular application is active. Any communication of data between one application and
another requires a stereotypical set of actions on the part of the user (copy, switch application, paste).”
The problem is that keeping track of contexts and acting upon them can be tedious to manage in the
real world, and software agents are used in this case to keep the interaction as simple as possible and
manage context for the user. The primary job of a user interface agent in such cases is thus to
understand the intent of the user. In such systems, the user can use the interface without explicitly
taking notice of the agent; the agent can display suggestions, or directly manipulate objects in the
interface, based on context as implicit input collected from the user. After these first publications
stressed the importance of agents to manage context, many layered context-aware architectures and
frameworks have evolved during the last years
For key research on these architectures we refer to Baldauf et al. (2007), who collected a large
number of framework and middleware examples, and identified a common architecture in modern
context-aware applications by design analysis (see Figure 4-1). They also stressed the use of agents
in three layers or modules that separate the detection of context (perception), planning (reasoning),
and action based on context. This separation of detecting and using context is mainly utilized to
improve extensibility and reusability of systems. Blackboard systems (Engelmore & Morgan, 1988) are
often applied in multi agents systems. Winograd (2001) contains several pointers as to what the
blackboard approach‟s characteristic weaknesses might be for use in agent-based context aware
systems: since it is loosely coupled it pays a price in communication efficiency. On the other hand,
ease of configuring, simplicity and robustness are its benefits.
Context can be seen as a restricted set of important information and constraints in a specific situation.
The restriction is based on a goal aimed at. Wessner (2005) qualifies a computer system as context
aware if it supports the users by providing relevant information or services taking into account context
information for the selection process. Dey and Abowd (1999) underlined that the relevance of the
regarded services or information is also strongly connected to the users‟ current tasks.
4.1.3
APOSDLE approach
We base our approach on the Baldauf et al. (2007) architectural paradigm (see Figure 4-1), since this
is the one most widely used in related work. For task detection we implement layer 1 and 2. Layer 3 is
later used for recommending suitable learning resources with regard to the current work task of the
learner.
page
36
Figure 4-1: 3-Layer Context Architecture (from Baldauf et al., 2007)
In order to support the recommendation of suitable learning resources and many other aspects of an
interwoven learning paradigm, the work integrated learning system needs to be aware of a user‟s
current work task. This information can be seen as a prerequisite for finding suitable resources or
cooperation experts and is to be retrieved automatically and unobtrusively, using low-level context
information as indicators. The applicability of machine learning (ML) algorithms to this problem is an
APOSDLE research question. The goal of task prediction is to know the active task of the user at any
point in time. A task is defined as a unit of work consisting of activities to reach a certain goal. The
problem of task prediction is seen as a machine learning problem. When first using the prediction
system, it is untrained and the user needs to specify the task s/he works on from a predefined list of
business tasks (by manual selection). During the work process a context monitoring component logs
any desktop events reflecting the user‟s actions. These include keyboard presses, application
launches, full texts in documents, etc. As soon as sufficient classified events are gathered, the system
trains a ML model of the user‟s work task in the application domain. The optimal result is achieved
when the user continues to work and she does not need to manually notify the system of task switches
anymore. The task predictor automatically classifies the active tasks using continuously recorded
event streams (automated selection). Whenever the classification engine detects a change in tasks,
the work-integrated learning environment displays a new list of associated learning resources and
suitable experts relevant for the detected work task.
4.1.4
New challenges and ideas
Preliminary experiments have shown low recall for infrequent tasks and especially low precision for
tasks at a high granularity level (that is, very general tasks like “Working on X”). Future research
efforts will continue to be dedicated to increase both the precision and the recall of the APOSDLE task
prediction. More elaborated methods for feature engineering and aggregation will be tested for this
purpose.
Another research challenge is the evolution of the task prediction model. In order to support changing
work processes we need to build incremental and continuously learning algorithms that take
descriptive task executions by knowledge worker into account to improve the prediction model.
page
37
4.1.5
Component evaluation: context detection
This section describes the general setup of a component evaluation of context detection in APOSDLE.
The research question is: How is the interrelation between acceptance (i.e., usability) and efficiency
(i.e., ability to speed up knowledge work) of a simple context detection environment?
The work originates from task 1.2. and is dedicated to context detection mechanisms. Our study
reflects an in-house experiment at SAP Research CEC Darmstadt and consisted of two phases:


In phase 1 we collected training and evaluation data. We used these data to improve our
system and find the best machine learning algorithms.
In phase 2 we tested the improved system.
We will sketch these phases, discuss the measurements and draw conclusions for the introduction of
context detection in an organization – in general, along with tasks with models and along with domain
ontologies characterizing the organization.
4.1.5.1
Phase 1
We defined 12 tasks, which have been performed by 20 participants from the SAP Research CEC
Darmstadt (mainly postdoc-level researchers and PhD candidates) during the first phase without any
contextualized support. The tasks are condensed versions of typical work a researcher has to carry
out in the context of the industrial research projects, transfer projects and program activities at SAP
Research in Darmstadt (explanation in italics):












create a presentation on Generics in Java (preparing technical slides)
leave request (interacting with a typical SAP tool)
update the SRN ( the SAP Research Knowledge Representation Tool) page of your project
distribute presentation slides (find the right people and their full names to send slides to)
visualization of quantitative research results (MS Excel-style)
translation of an executive summary (a typical task as SAP is a bilingual company)
code development: Hello World class in Java (very simplified programming task)
create a handout (for a presentation)
create a UML-diagram (very simplified)
budget calculation (no obligatory tool )
software update (non-automatic software update of one tool)
inventory update (modelled as an interruption of another task)
To avoid correlations between the tasks, the participants received the tasks in a random order. During
the data collection phase, these tasks were carried out in limited time and with some hints on
supporting material (e.g. the presentation slides). We collected the data in a database and labelled it
according to the task for which it was collected. The offline analysis phase 1 showed the following
candidates for our combined analysis in phase 2:



Task detection (transactional mechanisms): operating on slices of the event stream captured
by desktop sensors, a hybrid approach using voting between the decision tree ID 3 (Quinlan,
1986), naïve Bayes (Dumais, Platt, Heckerman & Sahami, 1998), Euclidean distance
(Agrawal, Faloutsos & Swami, 1993), Irep (Fürnkranz & Widmer, 1994) and SMO(128) (Platt,
1999) algorithms, outperformed these single algorithms. Annotation of a document as being
relevant for a task preceded the first phase.
Topic detection (informational mechanisms): we applied string matching on a list of keywords
characteristic for the tasks from the list above and the tagged material was automatically
drawn from Wikipedia, that is referring to pages with the respective tags.
Detection of a matching document without background models (navigational mechanisms):
We can create a navigation graph, like the one in Figure 4-2, from the navigation objects.
page
38
Each navigation object is represented by a node with directed edges to all navigation objects
that occurred as a successor of it in the history. On this graph we can use partitioning
algorithms and propose all navigation objects in the actual partition that have not been
accessed in the actual session. Phase 1 showed the most effective results for navigational
mechanisms, which are based on the paradigm of spreading activation (Anderson, 1983) on a
navigational graph.
Figure 4-2: Navigational graph enabling algorithms without task or domain model
4.1.5.2
Phase 2
In this phase we let 15 participants from the SAP Research CEC Darmstadt (again mainly postdoclevel researchers and PhD candidates) perform the tasks a second time. This time they were
supported by our context detection system. We let them perform the tasks again in random order
many weeks later to reduce the effect of the participants‟ memory on how the tasks are performed – a
quite realistic condition with support from our system. In comparison to P3, we used a very simplified
user interface (see Figure 4-3) only showing documents stemming from task detection, topic detection
and a machine learning algorithm directly guessing documents and URLs as navigational targets from
the event stream, to collect information about how our system helps them to fulfil the task faster and in
a more easy way.
By design, the users in phase 2 had as easy access to the tools for the tasks (left hand pane in Figure
4-3, for instance MS Excel) as in phase 1 in order to enable work without context detected support.
page
39
Figure 4-3: Screenshot of the simplified UI – the right hand sidebar showing three potential recommended
lists of documents from topic detection (with a topic as an informational goal), purely unsupervised
mechanisms based on the users click stream (a document or URL as a navigational goal) and task
detection (with a task as a transactional goal)
4.1.5.3
Evaluation of phase 2 and discussion
We used a connection of the users‟ actions to a system clock to measure the amount of time needed
per task in phase 1 and phase 2.
Table 4-1 shows a detailed time analysis for the tasks and the relative amount of time gained (green)
or lost (red) with or without machine learning support.
page
40
Table 4-1: Speed measurements, relative speed-up (green) or slow-down (red) of users
The users became 30% faster on an average over all tasks. Intuitively, this number is more than the
speed-up expected by a pure learning effect for the same tasks without tool support, given a set of
alternating and once interrupted short tasks, which resemble a heavy workload.
The two tasks speeded up most were routine tasks (Update SRN and Software Update) with a
relatively low involvement of personal creativity. However, despite the fact that these were routine
tasks, their repetition is frequent enough to still offer enormous room for automated support as in our
study, for example, by just presenting the right entry point to the system to be updated. Tasks like
Visualisation Results and the ones on Prototype Development or Translation involve more creativity,
but still show that support by context detection pays off, at least in our lab setting.
There was one user (User 2), who was very much distracted by the recommendation system – at least
in a positive sense, as s/he liked the system very much and started playing with it. This attitude seems
to be an outlier in comparison to the other fourteen users. Another outlier is the leave request task,
with relatively many users taking more time to perform with context detection support. This lacks an
explanation so far. None of the 15 users encountered particular bad suggestions by the system
regarding this task – and even 43% of the users found particularly good recommendations for that
task. Figure 4-4 shows the overall satisfaction with the combination of tools as determined by a
questionnaire administered after the sessions.
page
41
Figure 4-4: Results of the questionnaires
A surprising result, shown at the top of Figure 4-4, is the overall opinion about the intrusiveness of the
system. As many as 57% of the users did not feel it to be intrusive, a reason to follow to the
presentation of documents and not tasks and topics, such as in APOSDLE P3 and similar systems as
the Microsoft Office help. As the design of the UI came more or less on the fly and just piggy-bagged
on the scientific approach to make the three different flavours of context detection (navigational,
transactional, informational) comparable for the lab study, we see a very interesting opportunity for
further UI design around the smoothly morphing list of document hits regularly updated by the context
detection. Further research in that direction, drawing a distinction between the current metaphors
“pop-up of suggestions” or “click to get a context-based suggestion” seems to be very valuable.
page
42
Although the system was not felt to be very intrusive (at least by a minority of the users perceived it as
such), a strong majority (93%) of the users were aware of the fact, that different kinds of context
detection were designed to support them. The recommendation of documents from the spreading
activation algorithm turned out to be most valuable from the users‟ perspective. We also related this
question to the click behaviour, which showed that these documents from the navigational mechanism
were also selected most often – in contrast to an average selection (and an average judgement in the
questionnaires) of documents from the task detection and almost no response to the topic detection.
This clearly differs from the results in the summative evaluation of APOSDLE (see deliverable D.6.12),
where the topic detection was favoured above the task detection. The outcome also depends on the
quality of the models and annotation. In the component study, the linking to Wikipedia was perceived
as too general, even for “non-office” tasks, where a source of encyclopaedic knowledge might be
expected to be useful. The other way round, navigational mechanisms which do not depend on
models, supported the users in our study – but have the conceptual drawback of only referring to
documents, which were already at least touched by the users (thus triggering the question, how new
knowledge can be transferred to the individual user, a question of true knowledge transfer in the
organization).
Finally, the system was perceived as helpful, which we conclude from the combination of the two facts
that it caused a speed-up in tasks and that most of the users claimed, that they would use it again.
The questions which remain open are less about the algorithms, but more on the organizational side
and UI side:


How to effectively and efficiently connect models to the context detection and how to
complement this with the more personal navigational mechanisms?
Which metaphor is best suited for presenting the continuous flow of context-dependent
suggestions?
Outlook: Contribution to organisational context detection and modelling strategy
We identified the best machine learning algorithms for task detection, topic detection and model-free
context detection. Contextualised support seems to be helpful along with the right metaphor for the UI
and is – when going beyond the helpful reconstruction of personal document streams and piles –
strongly dependent on the investment in the modelling. Thus the main follow-up of our study is, that
we are now in the position to suggest a way of context detection intertwined with modelling, which
slightly deviates from the strategy taken in APOSDLE P3. The key is human activity monitoring: we
suggest to apply context detection already in the very early phases of modelling by navigational
mechanisms individually and to complement this by task models and ontologies, which in some sense
(for example, by pooling search terms) are seeded in a de-central way in the organization. The
purpose is knowledge exchange and transfer – and the lack (for example, situations, where we can
automatically determine, that navigational context detection does not suffice plus some easy
mechanisms to determine what is missing) of it should determine the shape of the domain and task
model. This differs from the APOSDLE P3 strategy of interviewing experts and opens the door to a
more decentralized and continuous way of relating detection mechanisms and modelling. However,
the knowledge engineer and the domain expert are still needed to work on the seed we mentioned as
a result of human activity monitoring and thus the later phases of the integrated modelling
methodology remain valid.
page
43
4.2 User context representation
In order to make APOSDLE adaptive to the user‟s task and his or her knowledge and skills, the
system needs to "know" what the user is able to do and what she is not able to do. Otherwise, the
information that is suggested to her in a certain situation would not take into account any user-specific
pre-knowledge or experience and therefore will be the same for each and every user. Therefore, the
adaptive system contains a user model, which can be understood as a representation of “the
knowledge about the user, either explicitly or implicitly encoded, that is used by the system to improve
the interaction” (Kass & Finin, 1988, p.6). APOSDLE stores user related context information in digital
user profiles.
Providing adaptive functionality requires that the user model translates user behaviour into
assumptions about the user (Benyon & Murray, 1993), such as his or her knowledge and skills, or his
or her perceived goals (for example, performing a task at hand). As a consequence, the role of the
user profile is twofold. First, it must infer user knowledge and skills, and her goals (for example, in
terms of a task she wants to perform) from system interaction. Second, based on the user‟s
knowledge and skills, the user profile needs to provide adaptive functionality which supports her in
achieving her goal (for example, performing a task at hand). Based on the user profile, data
recommendations are computed aiming at supporting the users‟ learning goals attainment (See
Section 3.3), the preparation of the retrieval of resources (see Section 6.4) and cooperation activities
(see Section 5.2). The component responsible for operations on user profiles is the User Profile
Service (UPS). The UPS‟ functionality is made accessible to other parts of the APOSDLE system via a
set of services. Examples are services for logging users‟ activities and services computing queries for
resources, which optimally meet a user‟s current learning need. Storage and manipulation of user
related information, of course, raises critical issues concerning privacy and ethics in general. These
issues are tackled by giving users a large amount of control over their own personal data. This section
will describe how the APOSDLE UPS deals with digital user profiles of knowledge workers.
4.2.1
Challenge
For any adaptive system, a number of design and usability challenges have to be tackled in order to
not to nullify the benefits of the adaptation to the individual user, namely the challenges of
Predictability and Transparency, Controllability, Unobtrusiveness, Privacy, and Breadth of Experience
(for a compact and comprehensive summary see Jameson, 2003). All these issues need to be taken
into account when designing any kind of adaptive (learning) system. In our view, the issues of
Unobtrusiveness, Controllability and Privacy constitute the hardest challenges for a work-integrated
learning system.
In line with Jameson (2003), with the term (un)obtrusiveness, we refer to the extent to which the
learning system places demands on the user‟s attention, thereby reducing her ability to concentrate on
her primary tasks. Many sources for obtrusiveness are located in other system components than the
user profile, e.g. the presentation of distracting animated characters, or of pop-ups that prevent the
user from working and that require user action in order to disappear. The main source of
obtrusiveness stemming from the user model is related to the ways in which information about
individual users is acquired and maintained during the process of learning. This is necessary in order
to provide appropriate learning content over time. There are several ways to diagnose and maintain a
user profile. The knowledge represented in the user profile can be elicited explicitly from the user but it
can also be acquired implicitly from inferences made about the user (Benyon & Murray, 1993).
In our view, explicit acquisition involves some co-operative behaviour of the user, such as answering
relevant questions (for example, self-assessment) or performing tests. In systems that support
learning, it is often natural to administer tests of knowledge or skill (see for example, Jameson, 2003;
Guzman & Conejo 2004). The main advantages of testing are that it can be used in many domains
and it is easy to implement. However, testing cannot easily be applied for work-integrated learning
system for several reasons. Firstly, these systems aim at seamlessly integrating working and learning
– having to perform a test after each learning situation disturbs the worker and has been met with
page
44
considerable resistance in previous APOSDLE prototypes. We have similar experiences with selfassessment approaches, which are in line with Carroll and Rosson (1987), who found that users are
unlikely to engage in additional efforts even when they know that they would benefit in the long run.
Secondly, a work-integrated learning system is especially promising for ill-structured learning domains
(such as requirements elicitation or innovation management). As the tasks are realistic work tasks of
knowledge workers, tests cannot easily be designed, nor can the task outcomes be easily evaluated
and interpreted, as there is usually not one correct solution but several possible solutions for one and
the same task.
Other explicit ways for filling and maintaining the user profile of learning systems would be selfassessment, peer-assessment, or supervisor-assessment. During our research, we have learned that
self-assessment is not very reliable, and therefore should not be the only basis of the user profile. On
the other hand, peer- and supervisor-assessment are costly and not practicable in work-integrated
learning systems, among others for ethical reasons and privacy. The situation is even more difficult
because, as mentioned above, it is necessary to update the user profile continuously, which would
require continuous test and evaluation, or self-, peer-, or supervisor-assessment. For all these
reasons, we were seeking intelligent ways to initialize and update a user profile for a work-integrated
learning system without being too invasive (for example, by asking users questions all the time, or
letting them do tests).
4.2.2
Naturally Occurring User Actions as Knowledge Indicating Events
A number of interesting approaches have been suggested in other adaptive systems than learning
systems. For instance, researchers interested in adaptive hypertext navigation support have
developed a variety of ways of analyzing the user‟s navigation actions to infer her interests or to
propose navigation shortcuts (see for example, Goecks & Shavlik, 2000). Schwab & Kobsa (2002)
came up with an unobtrusive approach for user learning interest profiles implicitly from user
observations only. The problem is that such approaches cannot easily be re-used for adaptive workintegrated learning. From a psychological perspective, the concept of “interest” which shall be
“diagnosed” with the approaches presented above refers to an attitude of a person which is
considered to be rather stable over time. In psychological terms, interest constitutes rather a trait of a
person than a state. In contrast, a learning system aims at providing learning content which fits the
needs of the user in a certain competence state (his or her knowledge and skills). Typically, the
processing of the learning content changes the user‟s knowledge and skills. As the adaptive workintegrated learning system aims at continuously modifying a user‟s knowledge and skills, that is,
exactly those characteristics that should be diagnosed with the adaptive learning system, an approach
is needed which can handle these inherent dynamics of the user model.
Within APOSDLE, we tackle the challenge of user profile maintenance by observing naturally
occurring actions of the user (Jameson, 2003) which we interpret as knowledge indicating events
(KIE). KIE denote user activities which indicate that the user has knowledge about a certain topic. In
APOSDLE‟s second prototype, we have based the maintenance of the user profile solely on
information on past tasks performed (task-based knowledge assessment). The algorithms for
maintaining the user profile and the ranking of learning goals were based on competence-based
knowledge space theory (see 3.3) which is a framework that formalizes the relationship between overt
behaviour (for example, task performance) and latent variables (knowledge and skills needed for
performance). While there is some evidence that in fact most learning at the workplace is connected to
performing a task, and that task performance is a good indicator for available knowledge in the
workplace, this restriction to tasks performed certainly limits the types and number of assessment
situations that are taken into account. It is evident that a user‟s knowledge and skills do manifest
themselves through other types of user interactions with the work-integrated learning system. For
example, a user who seeks help while performing a task might be in a different knowledge state than a
user who provides help to others. Additionally, the tasks a user performs may be driven by
organizational constraints or simply by task or job assignments, and may therefore only draw a partial
picture of the knowledge and skills a user has available.
page
45
With APOSDLE‟s third prototype, we will go one step further and look at a variety of additional
potential KIEs. Examples include executions of tasks which involve a certain topic, communication
with other users about that topic, or the creation of documents which deal with that topic. KIE thus are
based on usage data ( 4.2.3). Our approach goes into a similar direction as Wolpers et al. (2006), who
suggested using attention metadata for knowledge management and learning management
approaches. This idea is similar to the approach of evidence-bearing events (see for example,
Brusilovsky, 2004). So far the approaches of attention metadata and evidence-bearing events have
been discussed from a rather technical point of view, and how the approaches, once implemented,
could be used in different settings. With our approach to KIE, we extend the technical perspective by
taking into account the entire process of implementing the KIE approach in a work-integrated learning
system, from identifying a potential KIE to their evaluation.
The idea for knowledge-indicating events (KIE) has been derived from the observation that tests are
not the only thing that humans (e.g. teachers) rely upon when they build their “opinion” about the
knowledge and skills of another person, that is, their cognitive “user model”. Rather, they look at the
behaviour of the person in many different situations they observe what the person is able to do without
help, and what he or she needs help for. From the behaviour of the person in its entirety, his or her
knowledge and skills are inferred, learning needs are detected, and possibly learning interventions are
suggested.
This observation is in line with what Eraut & Hirsh (2007) termed learning trajectories. The basic idea
of learning trajectories is that progress in a worker‟s performance is reflected by different types of
quantitative and qualitative changes in the worker‟s behaviour. According to the authors, “[…] some of
these types of progress could be described as doing things better, some as doing things differently
and some as doing different things” (Eraut & Hirsh, 2007, p12). Thus, we argue that qualitatively
different behaviour related to one and the same domain concept, may indicate different levels of
knowledge about that domain concept. By observing the worker‟s behaviour, we therefore can draw
conclusions about the worker‟s underlying levels of knowledge and skills. This is also a general idea
that relies on a common distinction made in Psychology where manifest behaviour (for example,
answers to test questions) is related to latent constructs (for example, knowledge or skills). That way,
manifest behaviour functions as an indicator for latent characteristics, and the latent level in turn is
used to predict or explain past and future behaviour.
For all these reasons, with APOSDLE‟s third prototype we look at a variety of user interactions with the
system. These different types of actions are tracked as potential knowledge indicating events. We
define KIE as traceable naturally occurring interactions of a user with an adaptive work-integrated
learning system that potentially allow inferences on the user‟s knowledge and skills. From the different
types of events, different levels of knowledge or skills are inferred. For a specific learning situation, the
different levels then are used for suggesting appropriate learning content.
In 4.2.4, the process of identifying KIE for a learning system to their evaluation will be described in
more detail. Before that, in 4.2.3, we will outline the structure of the APOSDLE user profile.
4.2.3
Properties of APOSDLE user profile
The theoretical background of our work with respect to user profile properties is influenced by two
streams of research. The first stream is research on user context on a general level (see, for example,.
Dey et al., 2001, Kobsa et al., 2001, Oppermann, 2005, Wessner, 2005), the second is research on
user modelling (see, for example, Kay 1995, Fink & Kobsa 2002, Fink 2004, Heckmann et al., 2005,
4
5
6
Heckmann 2005, Howes & Smith, 2006, but also FOAF , IEEE PAPI Learner , IMS LIP or
7
UbisWorld ). User context research traditionally differentiates between three forms of user related
4
http://xmlns.com/foaf/spec/
5
http://edutool.com/papi/
6
http://www.imsglobal.org/profiles/index.html
7
http://www.ubisworld.org/
page
46
data: user data, usage data and environment data (see Kobsa et al., 2001). User data comprises data,
which is related to a user personally. Examples are record data (for example, name, and address),
characteristics such as gender, age and income, users‟ knowledge and previous experiences but also
the users‟ preferences and goals. Usage data comprises all forms of users‟ activities, which can be
observed such as selective actions, ratings, confirmatory actions (for example, purchase, print, save)
or usage regularities. Environment data is usually made up of data related to the software environment
(for example, operating system, browser and other applications used), the hardware environment (for
example, display device) and the user‟s location.
The APOSDLE approach is based on the classification outlined above but adapts it slightly. First, we
aim at arriving at a clearer distinction between factual information and assumed information about the
user. Second, we aim at providing a sound basis for identifying those forms of user activities, which
shall be managed by the UPS in the form of usage data. The APOSDLE approach differentiates
between environment data, user data, inferred data and usage data. The respective semantics of
these types of data are presented below.
For the APOSDLE UPS, environment data is mainly made up of influences, which are external to the
user, but still have an impact on the user‟s work execution. For the APOSLE User Profile Service the
environment is the user‟s computational work environment plus the supporting infrastructure. The
essential parts of the environment are modelled formally and models are mapped onto each other
where this is feasible. The models and mappings are abstracted through the APOSDLE meta-model.
The meta-model forms the conceptual basis for the environment data-part of the APOSDLE user
profiles (Ulbrich et al., 2006). The meta-model is directly related to the models for the domain
knowledge, tasks and competencies (described in Chapter 3) and the mappings defined between
them. The meta-model is formally represented as ontology in the form of the meta-model schema
(see 3.4.3).
In APOSDLE user data is understood to consist of facts about the user (for example, name, address,
contact details etc.), whereas more uncertain information is referred to as inferred data. An example of
uncertain information is data, which represents a user‟s goal with respect to the future attainment of
learning goals. Inferred data allows finding ways of dealing with heuristics and assumptions. User data
on the other hand comprises all known facts about a user, which only change slowly over time, if they
change at all. Examples include date of birth, name, organisational unit and so forth.
Usage data comprises information coming from observations of the user‟s behaviour during work
execution. Examples for this are users‟ previous work experiences or past cooperation with others. It is
crucial, to make a clear distinction between forms of usage, which are relevant to the domain in
question (that is, work-integrated learning) and usage which has no meaning for the domain. Usage
data, as we envision it, thus reside on a more abstract level than on the level of operating system
events such as mere keystrokes or mouse clicks (Budzik & Hammond, 2000; Budzik et al., 2001;
Maus, 2001; Rath et al., 2007, Lokaiczyk et al., 2007). Usage data is of great importance as it
provides the foundation to infer new data about users. Figure 4-5 gives a brief, graphical summary of
our taxonomy about user profile data. In this figure the user profile data is arranged in circular layers.
The innermost layer consists of the factual personal information (user data). The second layer (usage
data) covers, what can be observed about a user during her work execution. The third layer (inferred
data) is computed from observations stored in the usage data. The outermost layer (environment data)
is actually not directly related to the user profile as such nor is it stored or maintained within the UPS.
Nevertheless, environment data has a certain impact on the user profiles and therefore needs to be
considered.
page
47
Facts about the user, e.g.
Name
Address
Email address
User’s behavior during work, e.g.
Tasks executed
Collaborations participated
Further information inferred from usage data,
e.g.
Potential future competency development
Potential collaboration partners
Data external to the user but still having an impact,
e.g.
Domain model
Task/Competency mappings
Environment data
Inferred data
Usage data
User data
User Profile Data
Figure 4-5: User profile data arranged in layers as implemented for the APOSDLE user profile
We chose this approach because it provides us with a clear distinction between factual knowledge,
observations, mere assumptions and influences from the user‟s environment. Each type of data can
be treated in such a way, that its specific characteristics are sufficiently respected. Factual user8
related data is stored within a directory service such as LDAP . Observations about a user are stored
as lists of entries reflecting a user‟s usage history. Inferred assumptions are not stored as such, but
computed on demand. Environment data is stored in domain-dependent models outside the UPS. This
allows for greater flexibility and separation of concerns.
Similar approaches have been proven to work, especially in the domain of recommender systems
9
(Montaner et al., 2003). Probably the most well known examples is Amazon . Amazon stores for each
user personal information (user data), a history of purchased and rated items (usage data),
recommendations (inferred data) and additionally holds a huge taxonomy of its products and services
(environment data). In the domain of educational systems, ELM-ART (Brusilovsky et al., 1996) and
especially ELM-ART II (Weber & Brusilovsky, 2001) follow a similar approach. Users are modelled
using different layers, where all layers are basically consistent with APOSDLE‟s usage data and
inferred data. ELM-ART II keeps track of visited learning units and learned units (usage data) and
known and inferred units (inferred data). Another educational system is KBS Hyperbook (Henze &
Nejdl, 2001). Here the user profile is basically constructed from concepts of a knowledge base
(environment data). User models are fed into a Bayesian network, which calculates probabilities of a
user‟s knowledge state with respect to a given concept (inferred data).
4.2.4
Implementing KIE in a Learning System
In order to introduce the KIE approach in an adaptive work-integrated learning system, several
conceptual decisions have to be made. First, it needs to be defined, what the KIE should achieve, for
instance with respect to accuracy of the user model. Further, the desired granularity (number of levels)
in which the knowledge and skills of the users should be represented needs to be defined. For
inferring the knowledge levels from KIEs a two-step procedure is proposed. First, a mapping from KIE
8
http://tools.ietf.org/html/rfc4510
9
http://www.amazon.com
page
48
to knowledge levels has to be established. This can either be done manually or automatically. Second,
the knowledge levels inferred from single KIEs need to be aggregated. For aggregation we further
need to define a time span over which events are taken into consideration for the inference of the
current knowledge level and an aggregation mechanism to infer from the detected KIEs over the time
span to the knowledge level.
Our approach envisages that the latter two, namely the mapping of KIE to levels of knowledge and
skills, and the aggregation over time for diagnosing the knowledge and skill level of a user, are defined
by a human operator. Machine learning approaches for automatic classification were not taken into
consideration for pragmatic reasons. Such approaches require a substantial amount of high-quality
training data at the design stage which typically are not available in work-integrated learning domains.
The mappings and aggregations are based on theories and assumptions and therefore need to be
empirically evaluated, and iteratively refined. The evaluation of the mapping and the algorithms
therefore is an inherent part of the KIE design cycle. Figure 4-6 schematically depicts the steps for
introducing KIE in an adaptive learning system that will be detailed subsequently.
Identifying
Potential KIE
Defining the
Scope and
Purpose of the KIE
Mapping KIE to
Knowledge Levels
Designing
Aggregation
Mechanisms for
Knowledge Levels
Testing and
Improving the
Mapping and
Aggregation
Mechanisms
Defining Desired
Levels of
Knowledge
Figure 4-6: Process of introducing KIE in an adaptive work-integrated learning system
Defining the Scope and Purpose of the KIE
In this first phase it needs to be defined what should be achieved with the user model, and what are
the limitations of the KIE approach for the learning system at hand. Within this phase, the potential KIE
need to be detected, and the desired levels of knowledge to be diagnosed based on the KIE need to
be delineated. These two design decisions are strongly interwoven because the variety of KIE
available, limits the number of potential qualitatively different knowledge levels. For instance, if only
few activities can be performed with a system, the number of levels that can be distinguished by
observing those few different types of user actions is limited. Moreover, pragmatic considerations have
to be taken into account: for example, if there is only one type of instructional support conceivable (for
example, someone is provided with the definition of the topic she should learn about), it is not of much
use to depict the knowledge of the users using five qualitatively different levels (for example, from
Beginner to Expert).
Defining Desired Levels of Knowledge
In the first place, the decision how many and which different levels of knowledge are to be diagnosed
with KIE should be a conceptual one. However, as suggested above, the number of knowledge levels
that can be distinguished with KIE is limited by the number of KIE available in the system. The
definition of knowledge levels and the identification of KIE are iterative processes which are mutually
influencing each other.
page
49
Based on the desired functionality of the adaptive work-integrated learning system (for example, how
many qualitatively different types of learning support should be provided based on the knowledge
levels) the first step for implementing the KIE approach is to define the levels of knowledge that are to
be diagnosed with the system. A number of frameworks have been suggested which might serve as a
starting point for the definition of desired levels of knowledge. For instance, Dreyfus & Dreyfus (1986)
suggested five levels of knowledge, Novice, Advanced Beginner, Competent, Proficient, and Expert.
Eraut & Hirsh (2007) suggested a typology of so called learning trajectories. The authors describe
learning trajectories for six different areas of work: task performance, awareness and understanding,
personal development, working with others, role performance, knowledge of the field, decision making
and problem solving. Each of these types of learning trajectories might serve as a starting point for
implementing the KIE approach in an adaptive work-integrated learning system. Another well-known
approach based on Bloom's taxonomy of educational objectives (Bloom, 1956) has been brought up to
date by Anderson & Krathwohl (2001). The authors describe six different cognitive process
dimensions, namely Remember, Understand, Apply, Analyze, Evaluate, and Create, which can be
interpreted as six different levels of knowledge about a certain topic which build upon each other. For
instance, if someone is able to evaluate certain software, she must be able to analyze that software.
Anderson & Krathwohl‟s approach is quite popular, even though the hierarchical nature of their
cognitive process dimensions was criticized in the past. Besides these rather complex frameworks
several less sophisticated approaches are conceivable. For instance, imagine that the adaptive
system shall only distinguish if a user has knowledge about a certain topic or not. In this case the
system only needs two levels of knowledge. Once the knowledge levels are identified they have to be
described comprehensively.
Identifying potential KIE
Starting from the conceptual decision of how many different knowledge levels should be identified by
the system, the KIE need to be defined. The definition of KIE happens both theory-driven (top-down)
and system-driven (bottom-up).
Top-down definition of KIE. Based on the description of each knowledge level user actions are
identified that indicate that the user has that knowledge level about a certain topic. For the abovementioned approaches (Dreyfus & Dreyfus, 1986, Anderson & Krathwohl, 2001; Eraut & Hirsh, 2007)
theoretical descriptions of the different levels serve as a basis for the identification of potential KIE.
The challenge is to break down a rather broad description of a wide range of behaviours into concrete
actions that users might show in interaction with the adaptive work-integrated learning system.
Bottom-up definition of KIE. Besides the theory-driven identification of KIE, further possible KIE are
identified by walking through the entire functionality of the adaptive system (for example, using mockups). All actions of a user that might serve as indicators for a certain level of knowledge are collected.
Examples include contacting a person about a topic, or annotating a document with a topic or being
involved in a learning hint with a note (see Section 5.1.5.2). Because the user model is designed as an
overlay of the domain model, only such user actions can serve as KIE which can be related to one or
several topics in the domain model.
Mapping KIE to Knowledge Levels
As a next step, a mapping of KIE to knowledge levels must be established. We argue that this
mapping does not necessarily have to be 1:1, that is, possibly one KIE can be related to several
knowledge levels. Obviously it makes no sense to define such KIE that are related to all of the
knowledge levels, as the purpose of the KIE is to distinguish between knowledge levels.
It is crucial that the mapping between KIE and knowledge levels is based on assumptions that can be
open to empirical investigation. For instance, one such assumption might be: If somebody is asked by
another person about a certain topic, he or she might be a knowledgeable person for that topic.
Making the mapping empirically testable allows us improving and refining the KIE approach.
Designing Aggregation Mechanisms for Knowledge Levels
After having identified the knowledge levels and the KIE related to these levels, the mechanisms for
maintaining the user model need to be defined. Imagine, for instance, a user, for whom a number of
KIE are tracked all of which indicate different levels of one and the same topic. How do we know the
page
50
precise knowledge level of the user? Should we diagnose the knowledge level for which the user had
the most KIEs, or the knowledge level for which he or she had the most recent KIE? Should we take
the highest knowledge level for which the user has a KIE? Obviously a wide variety of rules for
diagnosing the knowledge level of a user are conceivable. At this stage, the decision which rule to
apply is based on theoretical assumptions which have to be tested empirically and revised if
necessary. Assumptions might be of the type of the following example: The knowledge level of a user
is the highest knowledge level for which the user has shown a KIE.
The aggregation mechanisms can for instance use this simple maximum detection (most frequently
performed expert events), or more complex, decide the final current knowledge level on the basis of
the distribution over knowledge levels. A distribution based mechanism could for instance infer the
following: Over the time span the user has performed many expert events, but nearly as many
beginner events, which indicates an advanced knowledge level.
Testing and Improving the Mapping and Aggregation Mechanisms
Up until this stage, both the mapping of KIE and the rules for maintaining the user model are based on
theoretical assumptions and considerations. In order to improve the validity of the user models, the
mapping of KIE and knowledge levels and the maintenance mechanisms need to be evaluated and
refined.
For testing and improving the KIE and the maintenance mechanisms (a) the “real” knowledge levels of
users need to be known, that is, an external criterion is needed, and (b) a number of users of different
knowledge levels about the topics have to use the system over a certain period of time. The latter is
important the users must have the opportunity to show the different actions when interacting with the
system.
Testing hypotheses concerning the mapping between KIEs and knowledge levels.
Testing hypotheses concerning the mapping between KIEs and knowledge levels means to compare
KIEs with external criteria for knowledge levels, for example, stemming from self-assessment.
In order to ensure that the KIEs are useful for distinguishing among the different knowledge levels,
research questions of the following type have to be answered: Do KIEs of type x occur more
frequently for persons with knowledge level X than for persons with knowledge level Y? Such research
questions are then answered by means of frequency analyses, cross-tables, or by the paradigm of
feature selection. The idea here is to identify these KIEs that distinguish best between users of
different “real” knowledge levels (external criterion). For finding statistically significant differences, for
2
example, χ analyses can be applied.
Testing hypotheses concerning the update mechanisms of knowledge levels based on KIEs.
A correct mapping of KIEs and knowledge levels constitutes a necessary pre-condition for
investigating the question, if the mechanisms for maintaining the user model based on KIEs are valid.
Testing hypotheses concerning the update mechanisms for knowledge levels means to compare
automatically detected knowledge levels with external criteria for knowledge levels. The associated
research question is: Do automatically diagnosed knowledge levels correspond to “real” knowledge
levels of users, that is, an external criterion? One possible means to test for statistical significance
would be contingency analyses, that is, computing the agreement of the automatically diagnosed
levels with “real” knowledge levels.
4.2.5
Relevance Feedback
Another way to deal with updating knowledge levels without the need for detecting KIE, is relevance
feedback. For the third prototype we investigate how to integrate user feedback about knowledge
levels into the user profile. Feedback about knowledge levels promises to be well suited for improving
user profiles as they provide direct feedback on the quality of inferences about users. To gather user
feedback users will be presented with a visualisation about automatically calculated knowledge levels.
This visualisation should help them to get an overview about the current state of their user profile, and
aims to provide users with an easy way to change knowledge levels concerning topics which do not
match from their perspective.
page
51
4.3 User profile services
User Profile Services (UPS) are responsible for operations upon user profiles described in the
previous sections. With UPS, we refer to all kinds of WIL functionality that maintains and utilizes the
data stored within the user profile. We chose a service oriented architecture (SOA) approach based on
10
the OASIS reference model . This is for several reasons. Firstly, the paradigm of SOA allows us to
split a kaleidoscope of adaptive functionality into different sub-groups (services) that can be used
independently from each other. Secondly, services can easily be integrated in existing applications
which make them especially attractive for the situation of work-integrated learning. Thirdly, services
are formally described which provides an overview of service functionality, protocols, etc. Eventually,
existing services can be used for implementing new services (service mesh-up).
4.3.1
Challenge
The UPS makes the contents of the user profile and results from computations available to the other
parts of the APOSDLE system. Their functionality includes logging users‟ activities, predicting the
knowledge worker‟s performance in a (novel) task, detecting his or her current learning need,
computing queries for resources, which optimally match the learning need, or recommending an
optimized learning path. The latter has been seen as a crucial function in the context of workintegrated learning and will be described in more detail in Section 5.1. Data from user profile needs to
be analysed and made available by the UPS in such a way that other parts of APOSDLE can adapt
their own contribution to the users‟ current needs in order to support learning in the best possible way.
4.3.2
Theoretical background and APOSDLE approach
The theoretical background of our work basically stems from research on systems that are designed to
collect, process and make available user profile data through a set of user-focused services. A service
is crafted towards an actual, concrete need of a knowledge worker or an application supporting the
knowledge worker. The selection and design of services has been informed by research in
recommender systems (Montaner et al., 2003, Middleton et al., 2004, Adomavicius et al., 2005,
Adomavicius & Tuzhilin 2005), user context lifecycles (Kobsa et al., 2001, Teltzrow & Kobsa, 2004),
knowledge management (see, for example, Maurer & Tochtermann, 2002), the semantic desktop
(see, for example, Sauermann et al., 2006) and social semantic desktop (see, for example, Decker &
Frank, 2004a,b). According to research on recommender systems (Montaner et al., 2003) and the
user context lifecycle (Kobsa et al., 2001) user-focused services are divided into services for acquiring
user related data (acquisition or profile generation and maintenance), inferring further information from
it (secondary inference or profile exploitation) and producing a result, which is to be presented to the
user (production). Kay and Kummerfeld (2006) additionally suggest including further services, which
allow for larger user control over their own data (control).
The APOSDLE approach rests upon the approaches illustrated above. We chose this approach
because we pursue the goal of providing a user-focused emergent system. Emergent means that the
UPS not only collects data but also offers advanced computations and inferences over the data
11
collected . From a conceptual viewpoint, for this inference services are required. Offering acquisition
services too, is a rather natural consequence, given that data, which are to be processed, need to be
acquired at some point in time. Production services for presenting external actors with results from
computations and control services for user control round off the classification of service types.
Logically, the separation between acquisition services, inference services, production services and
control services appears to be plausible. Figure 4-7 gives an overview over the different service types
which we use in APOSDLE. In the following, each of these service types will be described together
with its realization in APOSDLE.
10
http://docs.oasis-open.org/soa-rm/v1.0/soa-rm.pdf
11
In this context Gruber (2007) writes about the differentiation between „collected intelligence‟ and „collective intelligence‟.
page
52
User input
Presentation to user
Logging
(collecting usage data)
Control
(access to
usage data)
Production
(for presentation to the
user)
User Profile Services
Inference
(exploitation of user profile data)
Usage data
(history-based models)
Figure 4-7: Types of services of the APOSDLE UPS
Logging services are responsible for updating the user profile with new observed usage data ( 4.2.3)
and thus provide the basis for all other services. Sensors within the learning environment (possibly
from many different applications) send detected user activities (such as task executions, cooperation
events) to Logging Services to be added to the user profile. Pre-processing of incoming user activities
is handled here. This could involve the transformation of user activities into a format required by the
user profile, or enriching incoming data with timestamps and other system related information. The
APOSDLE system implements two different Logging Services. The Work Context Logging Service is
dedicated to collect executions of tasks corresponding to the task model (delivered from the task
detection agent). Logging information consists of a user identifier, a task identifier and an optional
timestamp (depends on privacy settings). The second logging service, Resource Activity Logging
Service collects all activities related to resources presented to users. Such actions are reading
documents, engaging in learning activities, or contacting another user.
Production services make the stored usage data available to other (client) services within the learning
environment. Based on the specific requirements of the client, production services filter or aggregate
usage data– they provide specialized views on the usage data. For example, one such service could
produce a list of all tasks executed by one user. The receiving client could then provide visualizations
of task executions over time. Views also offer a way to retrieve usage data associated with a specific
enterprise model. Besides providing predefined views filtering usage data, production services could
also allow to query the user profile with individual parameters. In order to allow users to examine the
information work-integrated learning services have gathered, APOSDLE offers two Production
Services. The Usage Data History Service delivers a history of task executions and all resource-based
actions. The output of this service is basically a history of all events including all KIEs ( 4.2.2). Another
feature is that relations between events are also preserved. It provides a way to visualize which steps
users have taken when doing a certain task. It features also the links to outputs generated by
Inference Services (for example a ranked list of learning goals inferred by the Learning Need Service).
How this linked usage data can be utilized to explain and visualize system behaviour is further
described by García-Barrios et al. (2008). The Evaluation Service is another kind of Production
Service. It is specially designed to export different aspects of usage data for evaluation outside the
APOSDLE system. In APOSDLE this service generates files containing detailed information about
task executions, system usage, and information from inference services.
Inference services process and interpret usage data to draw conclusions about different aspects of
users, such as levels of knowledge. Inferences are then utilised to adapt the functionality of the
service itself, or by providing the outcome to other services. The user profile allows generating
inferences in different ways. Heuristics could be directly applied on usage data to generate
aggregated information about users. Exploitation of usage data with regard to formal models or a
page
53
hybrid approach by combining heuristics with organisational models, could also lead to inferences.
Within APOSDLE, as one of the most important Inference Services the Learning Need Service allows
to compute a learning need for a user. Its design is driven by the goal to support knowledge workers
based on their knowledge level. A user‟s learning need is inferred in three steps. Starting with the
user‟s current task, the user profile is queried to retrieve the learning goals required for this task. This
set of learning goals is then used to query the user profile for corresponding knowledge levels. Step
two calculates the knowledge gap between the knowledge levels required by the task and the
knowledge levels the user has achieved so far. The last step utilizes the knowledge levels to rank the
calculated learning goals of step two, and thus generates the final learning need. The less experience
a user had with a learning goal (low knowledge level), the higher the rank of the learning goal. The
„most required‟ learning goal is therefore listed on the top. The learning need is used by the APOSDLE
system in two ways. An application running in the working environment of the user visualizes the result
as a ranked list. Users can choose a learning goal which invokes a Retrieval Service to find resources
relevant for this learning goal. The Learning Need Service also provides other services with current
knowledge levels of users. This feature is utilized for example by the service described below as basis
for its inference. The People Recommender Service aims at finding people within the organization
which have expertise related to the current learning goal of the user. This service provides similar
functionality as the expert finding systems described by Yimam-Seid & Kobsa (2003). Users
specialised in certain topics are represented in the User profile with high knowledge levels for these
topics. Other users can now individually be provided with colleagues having equal or higher
experience. Compared to the MetaDoc system, this service uses a more dynamic way of identifying
experts. Knowledgeable users are always identified compared to the knowledge of the user who will
receive the recommendation. To infer knowledgeable users, the People Recommender Service
utilises the Learning Need Service to retrieve knowledge levels for all users. The next step removes all
users with lower knowledge levels compared to the user receiving the recommendations. The
remaining users are then ranked primarily according to their knowledge levels. The most
knowledgeable user will be ranked highest. The service can be configured to also use the availability
status of users as ranking criteria. This setting allows recommending only users currently available.
Control Services provide ways to control usage data stored in the user profile. Controlling usage data
is important for handling privacy issues and imprecise usage data collected in the user profile. Privacy
issues could be addressed by applying certain privacy policies of organizations to usage data. An
example would be a policy about data retention, demanding the deletion of usage data after a certain
period of time. The aspect of imprecise data can be addressed by presenting users with an overview
of usage data associated with them. Based on this overview users could then use a control service to
manually delete or modify usage data. APOSDLE implements two Control Services. The Usage Data
Control Service allows users to modify and delete any usage data. APOSDLE Clients present users
with a task history provided by the Usage Data History Service, and invoke the Usage Data Control
Service to delete task executions selected by users. A dedicated privacy component (part of the
APOSDLE server) also accesses this service to enforce certain privacy policies on usage data.
Figure 4-8 presents an overview of APOSDLE User Profile Services and how data is exchanged with
the User profile and corresponding APOSDLE Client applications.
page
54
Figure 4-8: Interaction of APOSDLE User Profile and User Profile Services with APOSDLE Client
Applications
Similar approaches have been proven to work in knowledge management (KM) on a conceptual level.
Maurer and Tochtermann (2002) have introduced a model of KM systems. This model defines
functionalities of KM systems and classifies these functionalities. The model differentiates between
acquisition services, which rely on explicit input of information and those, which rely on implicit input.
Additionally, production services are subdivided into services, producing results on demand (for
example, based on an explicit query) and services, producing results autonomously. The model also
makes a difference between inference services creating “new knowledge” based on observations of
user behaviour as opposed to services creating “new knowledge” based on existing knowledge.
Compared to the APOSDLE UPS, this approach lacks the control services. Further similar approaches
have been proven to work in recommender systems (Montaner et al. 2003, Perugini et al, 2004).
12
13
14
Systems like GroupLens , Ringo/Firefly , LikeMinds , (again) Amazon and others, infer similarities
between users from users‟ past behaviour. Nevertheless, for the time being, our research has
revealed that the separation of service types and the classification of actual services according to
service types are new and original contributions of the APOSDLE UPS. We did not identify further
approaches, which chose to follow this approach to the same extent as the UPS.
4.3.3
Component evaluation: identifying user levels
The User Profile Service (UPS) is the software component, which maintains the user profiles of
APOSDLE users and provides different user related services (for example, document recommender
service, people recommender service) to the other parts of APOSDLE (client and server). In a
nutshell, the UPS has two goals: First, the diagnosis of a user‟s knowledge level (beginner, advanced,
expert) in each topic in the learning domain, and second, the diagnosis of changes in the knowledge
level (maintenance).
To keep the User Profile up-to-date, APOSDLE implicitly collects data by observing the user
interactions with the system, or explicitly by requesting the feedback directly from the User. In order to
keep the efforts of the users involved in user model maintenance to a minimum, the UPS tries to
12
http://www.grouplens.org/ - a news recommender systems
13
A music recommender system now owned by Microsoft
14
A clickstream analysis system now owned by Macromedia
page
55
reduce the explicit input data and focuses on the implicit data collected by observation of the user
activities. These user activities are introduced in APOSDLE as Knowledge Indicating Events (KIE).
To provide generic capabilities of the UPS, we followed a configurability approach. One configuration
consists (a) of a mapping of each KIE to a certain knowledge level (in the system known as Chain
Model) and (b) one inference algorithm. Since there are multiple inference algorithms and chain
models, the goal is to find out which of these combinations are best suited for the APOSDLE system.
To do so, “real” usage data is needed which can be used to benchmark different configurations.
Because we wanted to calibrate the UPS before real users were using it, we introduced a simulation
framework with the goal to create/simulate the real usage data. Data should always be simulated for
one knowledge level (BEGINNER, ADVANCED or EXPERT). Normative behaviour was varied, that is,
the extent to which a person showed KIE‟s that were related to his or her knowledge level. For
instance, if the behaviour of an EXPERT was simulated and 90% out of the simulated user actions
(KIE) were “EXPERT actions”, we speak about a simulation of an EXPERT with 90% normative
behaviour.
The following research questions should be answered with the simulation study:





4.3.3.1
Q1-Are there differences in detecting the knowledge level (BEGINNER, ADVANCED or
15
16
EXPERT ) between the percentages of normative behavior ?
Q2- How many simulation steps (events) do algorithms need to identify the correct (simulated)
level?
Q3- How long does it take for different configurations (algorithm plus chain model) to detect
modifications in the user level?
Q4-Is there a difference in the correctness of inferences when using different chain models
(with the same algorithm) in the inference process?
Q5- Which algorithm shows the best results for the given chain model? In other words, are
there differences between different algorithms when using the same chain model?
Method
We used a specially designed simulation framework to simulate user behaviour in the system. The
created data is called usage data and it is used to benchmark our configurations. For each simulation
trial, the input parameters must be defined based on the research question to be answered.
Concretely, the input parameters were (i) the user level to be simulated, (ii) the percentage of
normative behavior, (iii) the number of events to be simulated, (iv) the number of users to be simulated
and (v) the configuration to be used (Chain Model and Algorithm). Basically, three algorithms exist
which differ in terms of logics, namely FREQUENCY, WEIGHTING, and EVENT_WEIGHTING.
With our simulation, two aspects should be taken into account, that is the question of how well an
inference mechanism is able to detect the knowledge state of a user, and the question of how well an
inference mechanism is able to detect modifications in the knowledge level of a user. Imagine, for
instance, that a person was “Beginner” in a topic for one year. Then, the person becomes “Advanced”.
But when will the UPS realize that the person now is ADVANCED? If all events a person has ever
shown are taken into consideration, detecting the modification in the level might take quite long.
Therefore, each of the three algorithms is implemented in two ways: DEFAULT meaning that all
recorded
events
are
taken
into
account
by
the
inference
mechanism,
and
RECENT_NUMBER_OF_EXECUTIONS meaning that only a predefined number of the recorded
15
These are the levels wich indicates the knowledge of the user in the system based on his/her behavior in the system. For
example, the BEGINNER persons are often requesting assistence from the system or other knowledgeable persons in system.
Such a knowledgeable person would be EXPERT in this case.
16
Normative behavior means the extent to which simulated user behavior corresponds to the simulated knowledge level
page
56
events are taken into account by the inference mechanism. Combining these two elements we get the
following six inference algorithms:






FREQUENCY
WEIGHTING
EVENT_WEIGHTING
FREQUENCY_RECENT_NUMBER_OF_EXECUTIONS
WEIGHTING_RECENT_NUMBER_OF_EXECUTIONS
EVENT_WEIGHTING_RECENT_NUMBER_OF_EXECUTIONS
In order to simulate realistic sequences of events which may also occur when users interact with
APOSDLE, certain interface constraints were taken into account. Such interface constrains were, for
instance, that the event editing an annotation must always be preceded by the event opening a
document because without an open document, no annotation can be edited. This simulation mode
was called “simple constraint mode”. The simulation framework works upon input data and provides
the output which is then used to answer our research questions. The output data is created as a .csv
file and then imported into the SPSS program for the visualization. These data allow us to visually
compare the results. The comparison of the results is based on two types of measurements: (a) the
number of correct inferences after one simulation trial and (b) the mode of the inferred level after a
simulation trial (which level was most often inferred by the algorithm for a simulated knowledge level
and percentage of normative behaviour).
4.3.3.2
Results
The results of the simulations provided us with the answers to the research questions:
• Q1-After conducting a series of simulations with different knowledge levels, it was obvious that the
correct inference system needs at least 60% of the normative behaviour regardless of the simulated
level, chain model and algorithm. This means, that if 60% of a user‟s activities are related to one of the
three levels, APOSDLE will be able to detect that level.
• Q2- Simulations showed that, given 60% of the normative behaviour, on average, the algorithms
need 40-50 events to make a correct inference. The answer to the question of how many steps do
algorithms need to converge to the right level provides us with the answer to how many recent
executions do we need to have for the RECENT_NUMBER_OF_EXECUTIONS type algorithms.
Because on average, the algorithms need 40-50 events to make a correct inference, also the number
of events that should be the basis for the inference should be limited to the past 40-50 events.
• Q3- Here we tried to simulate the learning process of a user, meaning that the user actions are
changing over time as a consequence of learning (Beginner->Advance->Expert) or knowledge ageing
(Expert->Advanced->Beginner) process. The goal was to compare the two types of the inference
algorithms (DEFAULT vs. RECENT_NUMBER_OF_EXECUTIONS). It was obvious that the DEFAULT
type algorithms would have difficulties in inferring correctly when the behaviour of the user is changing
over time. The reason was that the records about the previous behaviour had too much influence on
the inference algorithms. Figure 4-9 shows that the DEFAULT Algorithm was not able to recognize the
ADVANCED level user behaviour at all (the gap between 50 and 140 events) and had problems
detecting the EXPERT level as well. After applying the RECENT_NUMBER_OF_EXECUTIONS
algorithm, the problem was solved. This algorithm used the last 40 events recorded for the inference
and provided a very good result. It needed about 35-40 events to infer the correct knowledge level of
the user.
• Q4- Two different chain models were used in our simulations. The obvious question was which one
of those two is better suitable for APOSDLE. The first model included 14 KIEs, whereas the second
one comprised only 9 KIEs. They differ not only in number of KIEs, but also in terms of their mapping
to EXPERT levels. The difference in mapping was relatively small and the main question was whether
the number of KIEs could be reduced so that the inference process was simplified, without losing the
page
57
preciseness of the inference. After conducting a simulation, it was obvious that although the reduced
Chain model needed a few steps extra to make correct inferences, it was precise enough.
• Q5- After conducting numerous simulations with different configurations, we managed to narrow
down the number of candidate configurations (algorithm plus chain model) for the UPS inference
process. After seeing that DEFAULT type inferring algorithms are not good for inferring the user
knowledge level in cases when the user changes his normative behaviour, it was obvious that the final
solution would be one of the RECENT_NUMBER_OF_EXECUTIONS type of algorithms. And indeed,
we
managed
to
distinguish
two
of
them.
These
are:
the
FREQUENCY_RECENT_NUMBER_OF_EXECUTIONS
algorithm
and
the
EVENT_WEGHTING_RECENT_NUMBER_OF_EXECUTIONS algorithm. With very small differences,
both algorithms showed good results. Therefore these two algorithms were seen as top candidates for
APOSDLE.
Figure 4-9: Simulation of 50 events with “Beginner” level followed by 50 events with “Advanced” level
and 50 Events of “Expert” level. The percentage of normative behaviour was 60% in all cases
4.3.3.3
Conclusion
One of the first principles in the decision making process for one of the configurations was to reduce
the complexity as much as possible. The clear results of the simulations were four configurations. Both
chain models showed good results and the best algorithms were the FREQUENCY and the
EVENT_WEIGHTING algorithm from the RECENT_NUMBER_OF_EXECUTIONS type. Led by our
principle of choosing the simplest solution, the 9-Event Chain model and the simplest algorithm,
namely the FREQUENCY_RECENT_NUMBER_OF_EXECUTIONS was the preferred one.
Even though the generalization of the outcomes is limited by the fact that a simulation can never
replace a real user study, we still believe that the simulation provides us with answers to the very basic
page
58
issues. It is possible however that when the real usage data are available, some other (more complex)
configuration may lead to better inferences.
4.3.4
Component evaluation: the privacy concept
The APOSDLE Privacy Concept (APC) provides an extensive set of innovative features to solve very
critical legal, socio-technical and purely technological aspects of adaptive SDWIL-based systems like
APOSDLE. Such systems combine and follow distinct complex strategies, such as fine-grained user
profiling techniques, real-time monitoring of user behaviour, cooperative activities of communication,
and user-driven resource sharing. Thus, APC is a rather complete as well as a very flexible privacy
framework for personalisation-pertinent systems.
For that purposes, APOSDLE’s Privacy Concept (APC) considers the complexity and adaptability of
the APOSDLE system as well as its broad areas of practical applicability in companies, and embraces
seven key concepts (see Figure 4-10: APOSDLE Privacy Concept; the chapter numbers in the inner
circle refer to chapters of Deliverable D 1.7 Privacy in APOSDLE).
Figure 4-10: APOSDLE Privacy Concept
In general terms, the APC consists of three interrelated, highly relevant privacy perspectives:

socio-technical and legal aspects

technical aspects

user-centric design aspects
page
59
In APOSDLE, the first issue is solved with the development of a general legal framework (called
Adaptable Privacy Statement, APS). The disparity of legal regulations for privacy protection in the
European countries, combined with the diversity of applicable privacy policies in companies, makes it
very difficult (or rather impossible) to provide an ultimate solution for an adaptive SDWIL-based
systems like APOSDLE. What is needed is some flexible solution that connects what is lawfully
stipulated, what is stated in the privacy statement, and what really happens within the business and its
capabilities. The solution should be transparent and simple, and further, provide real mappings
between user‟s privacy preferences and system‟s behaviour. APOSDLE solves these problems with a
template-based APS, which is the result of two influences: (i) the most relevant findings of well-known
organisations and experts, called Topics in the Personal Information Life Cycle (Collection, Retention,
Disclosure, Transparency, User Control, Maintenance, Security, and Further Use), and (ii) Roger
Clarke‟s Privacy Statement Template (a generic document for privacy statements).
Figure 4-11: Privacy Enhancement Technologies in APOSDLE
From the technical perspective, our security-related solution approach within APC focuses on three
issues: (a) the APOSDLE system supports identity management by means of Pseudonymity, (b)
communication is secured with the standardised Web Service Extensions (Policy, Security, and
Reliable Messaging) and (c) real-time monitoring of user interactions at workplace is solved by the
smart, user-controlled, non-privacy-intrusive Context-Aware APOSDLE Features which do not record,
transfer or disclose any private information. In addition, our system utilises a centralised, serviceoriented, standardised policy-based engine called the Privacy Enhancement Service (PES) to ensure
a privacy-protected access control and filtering of personal information flow (see Figure 4-11).
Regarding the third perspective of our APC, understanding and retracing the privacy-related adaptive
behaviour of the APOSDLE system as well as giving the user control over many relevant privacyrelated preferences, is solved through many features of designing Usable Privacy for all customisable
privacy options in a central user interface, e.g. predefined privacy levels, fine-grained options to
customise preferences, making combinations of privacy preferences reusable and retraceable by
means of a privacy repository and a privacy history, a mapping of the terms given in the Adaptable
Privacy Statement to the configurable options in order to better retrace the legal implications of
page
60
customisations, and a visualisation based on Sankey diagrams of the information flow within
components of the system with respect to each privacy preference (see Figure 4-12).
Figure 4-12: Practical Implications of Privacy Issues in APOSDLE
The following aspects of the APC have been covered in the component evaluation:


Socio-technical and legal aspects: comprehensibility of the APOSDLE Privacy Statement
(APS), compatibility of APS with privacy policies of companies
User-centric design aspects: comprehensibility of APOSDLE‟s privacy behaviour, Usability of
APOSDLE‟s privacy settings
The technical aspects of the APC (privacy enhancement technologies used in APOSDLE, their
functionality and how they were integrated into the APOSDLE system) have not been addressed in the
component evaluation.
4.3.4.1
Method
The APOSDLE privacy component evaluation has been accomplished by use of questionnaires
presented to APOSDLE end users on the one hand, and with interviews with IT experts responsible
for privacy on the other hand.
The following list should give the reader an idea about the kind of questions asked in the APOSDLE
end user questionnaire:

Is the APOSDLE Privacy Statement (APS) coherent and understandable?
page
61




Is the relation between APS and the privacy behaviour and functionality of the APOSDLE
system clear?
Is it clear what the privacy settings (privacy section in the APOSDLE client Preferences tab)
mean?
Is it clear how changes in the privacy settings of APOSDLE influence the functionality and
privacy behaviour of the APOSDLE system?
Do you realize that changes in the privacy settings may break the agreement made by
accepting the APS?
The following list should give the reader an idea about the kind of questions asked in the interviews
with the IT experts:


4.3.4.2
What are the experiences with your employees regarding dealing with and understanding of
privacy statements?
How does the idea of an adaptable privacy statement, like the APS, fit into the company‟s
privacy policy?
Results
The following list summarizes the results of the inquiries and interviews regarding the APOSDLE
privacy component:








4.3.4.3
Users of software systems are increasingly sensitised when privacy (data privacy as well as
private sphere) is an issue.
Users are not very interested in privacy statements that come with a software system: they
read them only sketchily, if at all, and sometimes they accept them blindly.
Sometimes users of software systems do not understand the content of privacy statements
they have to accept (use of legal jargon and legal terminology).
Users of the APOSDLE system liked the tighter connection between the regulations of the
APOSDLE Privacy Statement and the functionality of the system.
The user interface and the manifold possibilities to define the privacy behaviour of the system
in the privacy settings of the APOSDLE client have been judged as too complex.
The implications on the privacy behaviour of the system when changing the privacy settings
was not always clear to users.
The users liked the presentation and visualisation based on Sankey diagrams to better trace
the legal implications of privacy customisations.
IT experts liked the idea of an adaptable privacy statement as implemented by APOSDLE‟s
APS.
Conclusion
The main conclusions of the APOSDLE privacy component evaluation are:



Employees and especially knowledge workers as defined in the APOSDLE project context ,are
increasingly sensitive whenever there is information collected about their operations and
behaviour at the workplace. They do not like the idea, that there is software installed on their
computers at the workplace that monitor what they do and feel very uneasy about it, even if
they benefit from this kind of software systems. Employees appreciate at least a minimum
sphere of personal privacy at their workplaces.
Employees and especially knowledge workers as defined in the APOSDLE project context,
appreciate to have as part of their deployed software some kind of control centre to control
and understand privacy-related behaviour of the software.
It is very difficult to describe and characterise the functionality (and how something works) of
complex, dynamic software systems like APOSDLE. This holds particularly true for the privacy
related behaviour of such systems: how do privacy statements relate to the privacy behaviour
of the software and how do changes in the privacy settings influence the privacy behaviour of
the software.
page
62

Extraordinary effort is necessary to design and implement user interface components
applicable to present and describe complex and dynamic systems like the privacy related
behaviour of APOSDLE.
page
63
5 Work Integrated Lear ning Suppor t
In this chapter we will explain how APOSDLE will provide integrated support for the three roles a
knowledge worker fulfils at the professional workplace: the role of worker, the role of learner, and the
role of expert. Support is offered in several ways. First of all, by giving the worker access to (parts of)
documents that are relevant for the task at hand and for the learning goals that can be derived from
the task and the learning history of the worker. Second, by embedding these (parts of) documents in a
context that should support active engagement with the materials, as well as reflective thought.
Thirdly, by facilitating cooperation and communication with co-workers that have expertise with the
task at hand or that have (mastered) the same learning goals. Section 5.1 deals with the first two ways
to support learning, Section 5.2 is devoted to cooperation and communication for learning.
In addition four component evaluations were performed related to different aspects presented in this
chapter:
Learning paths: is learning more effective when learners manually create a learning path, compared to
when they receive an automatically created learning path?
Context visualisation: is there an effect on the time needed for task completion of using either a
context visualisation tool, a process and a tree based tool?
User profile and MyExperiences: do the levels of expertise of a user as represented in APOSDLE,
correspond to the user‟s “true” levels of expertise in each topic?
Scripted communication: does the cooperation script on a macro level guides knowledge seekers and
knowledgeable persons through the overall contextualized cooperation process
and does the cooperation scripts on a micro level helps knowledge seekers and knowledgeable
persons to individually use each process phase as efficient as possible for problem solving and
learning?
5.1 Supporting learning from documents
Presenting the right information at the right time can help a knowledge worker that has problems in
performing a specific task, to complete this task, but this does not necessarily mean that (s)he has
also learned something. For learning to take place the user has to actively process the information
presented and give meaning, adjust existing cognitive models or develop new models or schemata.
This idea is stressed in the instructional design theories of Merrill (2001) who states that an
instructional strategy has at least two phases: presentation/demonstration and application/practice. In
this section we will elaborate on this and describe how the APOSDLE system can support selfdirected learning at the workplace.
It should be emphasized that compared with the first version of this document, the notion of a (single
isolated) learning event presented by the system to the learner, has disappeared. In line with the
notion that self-directed learning can take place at any time and any moment, learning support is now
provided in different “corners” of the system. This section serves to describe them together, but they
are no longer located in one single feature such as a learning event.
5.1.1
Challenges
Instructional design generally starts with an analysis of learning needs which leads to a specification of
learning goals. Next, the level of prerequisite knowledge is determined and, based on this, learning
materials are selected and activities are specified that should enable the learner to reach the learning
goals. These activities are evaluated to give the learner feedback on his learning process.
page
64
The challenge we face is that the APOSDLE system should be applicable across different domains.
This means that beforehand the topics are not known and, furthermore, the types of material that are
available are also unknown. Therefore it is impossible to generate specific (sequences of) instructional
events (based on instructional design models) that should lead to a specific learning goal. However,
we still have to create a situation in which a person is stimulated to learn something. The solution
could be to embed parts of existing documents in a general context that contains elements to support
self-directed learning. In addition, outcome assessments in terms of more or less formal tests are not
possible in the APOSDLE context. First, because devising test-like assessments when the learning
domain is next to impossible. Second, because a self-directed workplace learning context rarely aligns
with the notion of testing and examination-like activities (see also Section 4.2). Apart from the solution
proposed in Section 4.2 (knowledge indicating events) some kind of self-assessment is possible.
5.1.2
Background and similar problems
Simons (2000) introduces a theory of self-directed learning that is based on three types of learning
functions: preparatory, executive and closing functions. Learning functions are psychological functions
to carry out before, during and after learning by a learner alone or with the help of outsiders, like
teachers, fellow students, computers or bosses. Within each type he distinguishes cognitive, affective
17
and meta-cognitive functions. Below the cognitive and meta-cognitive functions will be presented .
Simons lists the following preparatory functions:
 Finding missing prior knowledge
 Getting an overview of knowledge, skills and attitudes to be learnt
 Mobilizing prior knowledge and skills for the task at hand
 Finding connections between prior knowledge and new information and skills
 Orientation on learning goals
 Choice of learning goals and sub goals
 Orientation on learning strategies
 Choice of learning strategy
 Planning of time, sequence and places for learning
Executive learning functions are:
 Selecting information
 Thinking about information
 Coming to conclusions and own opinions
 Formulating conclusions verbally
 Getting overview
 Practicing and applying
 Getting an overview over application conditions and possibilities
 Monitoring learning processes
 Monitoring learning outcomes
 Testing progress
17
The affective functions will be left out as this is outside the scope of APOSDLE. However, when dealing with the interaction
between people in Section 5.2, these aspects will be implicitly touched upon.
page
65
 Diagnosing causes of failures and problems
 Repairing
 Reflecting on the learning process
Closing learning functions are:
 Summing up new knowledge and skills
 Thinking about future use and transfer conditions
 Evaluating learning process
 Evaluating learning outcomes
 Reflecting
In traditional learning settings, a teacher or other person takes care of most of the learning functions.
However, self directed learners should be able to perform these functions themselves. In many cases
people will not be able to do this by themselves and some kind of support is needed. In the context of
APOSDLE, supporting self-directed learning is even more difficult and challenging because not much
is known about the learner-worker, the work-learning domain and the (learning) material that can be
made available. As a consequence, traditional instructional design models are not well suited,
because it is difficult to foresee what kind of learning need will arise, what type of knowledge is
involved and what the level of prerequisite knowledge is that the learner already has mastered and
what the fitting information and knowledge to be provided should be. The learning content is not made
available by a knowledgeable instructor, as is mostly the case in a “formal” instructional context.
Since the content of materials that are available is unknown, we can only focus on the learning
functions. This can be done by creating (small) tools that support these functions. For instance tools
that help learners in getting an overview of the content to be learned, tools that help to plan learning,
tools that support the selection of information etc. Furthermore, this can be done by implementing
learning hints that encourage active engagement (supported learning activities) with the materials that
are selected and encourage reflection.
5.1.3
APOSDLE approach
We will organize the description of learner support along the lines of the learning functions proposed
by Simons (2000) which have been described in the previous section. Simons makes a distinction
between preparatory, executive and closing functions and we will use this distinction to describe
support in more detail in the next sections.
5.1.4
Supporting preparatory Functions
Important aspects of the preparatory functions are the identification of missing knowledge and related
learning needs and goals, getting an overview of the content to be learned, and planning of learning.
We will discuss how these could be supported.
5.1.4.1
Identification of missing Knowledge and related Learning Needs
Several studies investigated what self-directed learners actually do in the workplace when they learn.
Collin (2006) concludes that research of workplace learning suggest that much learning (and thus also
self-directed learning) at work derives its purpose and direction from the goals of the work itself. In the
first APOSDLE Workplace learning study De Hoog et al. (2006) also found that workplace learning is
strongly driven by work tasks. However, learning driven by curiosity is also present. This confirms the
typology that Houle (1961) formulated. Houle proposed that there are three different categories of
learners:
 those who are pragmatically Goal Oriented, who have in mind a very specific learning outcome,
and usually desist as soon as their objective is reached;
page
66
 those who are Activity Oriented and whose participation is related to some purpose other than
the ostensible objective of the activity (such as a social purpose or simply to obtain a formal
qualification); and
 those who are Learning Oriented and who simply enjoy it, and see learning as an aim in itself.
The starting point of a self-directed learning process is a knowledge need of the knowledge worker. A
learning need arises when knowledge workers are in doubt about the knowledge that is needed to
give meaning to an event. They acknowledge that there is a gap between the knowledge needed and
knowledge they possess. These feelings of uncertainty are converted into concrete subjects or
questions that lead to searching for information or knowledge (Choo, 1998). From a learning need,
more concrete learning goals can be derived. As Merriam and Cafferella (1991) comment, this means
of conceptualizing the way we learn on our own is very similar to much of the literature on planning
and carrying out instruction for adults in formal institutional settings. It is represented as a linear
process. However, adults do not necessarily follow such a well defined set of steps, but are far more
liable to chance and circumstances. Spear and Mocker (1984) found that “self-directed learners, rather
than pre-planning their learning projects, tend to select a course from limited alternatives which
happen to occur in their environment and which tend to structure their learning projects”, or, in other
words, an event or phenomenon triggers learning.
According to Gery (1991) and Choo (1998), people often have specific questions or requests that
come to mind when faced with performing new or complex tasks. For instance, questions like: “What
must I do? How do I do it? Am I doing it right?”, or requests like: “Show me…”. The information type
associated with a given question or request can reasonably be defined. One way of supporting
learners with the preparatory functions of self-directed learning could be to identify a set of relevant
questions and requests and a set of related information types. By presenting these questions and/or
information types to learners, one can help them in their orientation and choice of learning goals. This
is similar to the approach followed by Anderson and Krathwohl (2001) who developed a taxonomy of
learning goals which are subsequently used for assessment purposes. However, linking these goals to
instructional (that is, learning supporting) activities is far from clear. As they say on p.257 (under the
header of “Unsolved problems”): “The linkage between objectives and instruction needs further study.
....specifying a learning objective does not automatically lead to a prescribed method of instruction”.
Having such a link in a system like APOSDLE is important as, especially in the context of documents,
we cannot rely on the presence of an “instructor” to figure out during the learning process what fitting
learning support could be.
To bridge this gap in APOSDLE, we opt for a generic categorisation of information types/materials that
could be used to specify and limit the type of content that should be presented to learners (with a
specific question). There are some categorisations available (based on projects like LOM, Ariadne,
and SCORM), but these are rather low in meaning from a learning perspective. A categorisation based
on the one developed in the IMAT project (de Hoog et al., 2002) will be used. IMAT stands for
Integrating Manuals and Training. This was an EU funded project in which fragments extracted from
maintenance manuals, were classified in categories like: definition, overview, example, assignment,
guideline, how-to, summary, etc. For instance, the learner with the question “How do I do it?” could be
referred to fragments from documents that are labelled with categories like guideline or how-to.
Information needs and material resource types
Based on an analysis of what knowledge workers do, as was done in the second workplace learning
study (Kooken et. al., 2007), four basic information needs can be distinguished that each lead to more
specific needs and specific types of resources needed. These will be described below. The
combination of a topic and a task sets a learning goal (see 3.3). The four types described below refer
to four of the six cognitive processes that are described in the Anderson and Krathwohl (2001)
taxonomy (remember, understand, apply and create).
page
67
 A knowledge worker has no or a very limited knowledge about a topic that she encountered
when performing a task and wants to have a basic understanding of it, or wants to check
whether her basic knowledge is accurate (up-to-date). Specific needs:

Give me an introductory text about the topic.

Give me a definition,

Give me an example.
 A knowledge worker has a basic understanding of a topic, but she still has questions like: OK, I
know what it is, but how does this work? Why should I do it (in a certain way)? How did this
happen? Why did this happen? She searches for explanations that help her to answer these
questions. Or she just wants to know more about the topic to be able to understand things she
reads in documents, or to be able to communicate about the topic with co-workers or to be able
to generate new ideas Therefore she searches for information that contains background
information, historical data, trends and developments, relationships with other domain elements
etc. Specific needs:

Give me an explanation

Tell me more about the topic.
 A knowledge worker wants to know what the (next) steps are in a procedure or a well defined
task that she has to perform, but that she is not able to carry out without some guidance. She
searches for information that tells her which steps there are and in which order they have to be
completed. This information is like a recipe or prescription. Furthermore, she likes to have an
example or demonstration of the procedure. Specific needs:

Tell me how to do

Give me an example

Show me, give a demonstration
 A knowledge worker has to perform an ill defined task to produce something that is not clearly
defined but has some constraints (for example: a plan, an agenda for a meeting, a design) and
wants to know what she has to keep in mind when performing such a task. She searches for
lessons learned by others like guidelines, checklists, templates, examples and/or constraints
that give some structure in performing the task without giving a recipe or prescription. Specific
needs:

Give me a guideline

Give me a checklist

Give me a template

Give me an example

Show me a constraint.
The specific needs refer to resource types (in this document also referred to as material use (MU)
types or knowledge artefacts) that the knowledge worker would like to have access to. A more detailed
description of these types is given below.
Introduction
A piece of text or video that introduces basic features or aspects related to a domain
element (in general a concept from the domain under consideration) without going into
details. It functions as a kind of advanced organiser for information available in other
(parts of) documents or audiovisual material.
page
68
Definition
Formal description of a domain element that lists its characteristics or properties and that
enables learners to identify whether an instance is an example of the domain element or
not, and that enables learners to distinguish the domain element from a related/similar
domain elements.
Example - what
A description of a specific instance of something that can be identified as a typical or less
typical example of a more general domain element. For instance a case description. An
example is more specific than the abstract domain element that is part of the domain
model.
For example, a sparrow and a pigeon are typical examples of the concept “bird” and an
ostrich and a penguin are less typical examples. But still they all are examples of the
same concept.
Example - how
A description of a specific procedure or of a specific way a task can be carried out that
can function as a kind of frame of reference (model) for a person that has to perform this
procedure or task. The learner should be able to imitate the example.
How do I
Statements that specify how a certain domain element should, or could be applied (for
instance: a description of steps). Or a piece of text that gives job-aids that help a learner
in performing a certain well defined task or procedure. By following the information given
the learner is sure to reach a certain end product.
Checklist
A list with related elements that serves as a kind of memory aid. People can check
whether each of the elements is present or is satisfied/met. The order of the list does not
necessarily have a time sequence. Mostly answers on checklist questions are simply
“yes” or "no” Sometimes a set of open ended questions can also be seen as a checklist.
Demonstration
A piece of audio-visual material that illustrates how something should/could be done. This
typically has to do with well defined procedural tasks. This is only for audio-visual
material, for textual material the “How do I” material use is applicable.
Explanation
Statements referring to a certain domain element that clarify why something is done, why
something is as it is, how something works. The difference between "how do I" and
"explanation" is that the former refers to procedural (how) information and the latter to
conceptual (why) information. Procedural information is typically identified by the
description of steps or sequences to reach a certain goal. Conceptual information is an
in-depth explanation that goes beyond the simple definition/description of a concept.
There can also be an explanation for a procedure in the sense that it justifies why the
procedure is a good one (before applying paint you first have to sand the surface,
because if you don‟t do this the paint will not stick and the result will stay uneven)
Explanations are characterized by the presence of terms like: because (of), due to, for
reasons of, as a consequence
page
69
More about
Statements referring to a certain domain element that give more information about the
domain element, without giving a definition of the domain element or without a (step-wise)
description of the procedure or an explanation. For instance, statements with background
information, historical data, trends and developments, relationships with other domain
elements etc.
For example, more about “credit crisis” would provide information about when the crisis
started, who are hit by the effects, what measures are taken to deal with it.
Template
Statements that enforce a kind of standardization in the design of something. A template
has certain fixed elements but also open slots. Examples of templates are forms that
must be filled in.
Guideline
A statement that gives a directive/line of action; gives advice or summarises a lesson
learned by others that could be of value when performing an open ended ill defined task.
Following the guideline does not necessarily mean that a certain end product will be
reached but gives some structure to performing a certain task. Mostly a single or a few
lines which contain one indication what an actor should or should not do or take or not
take into account
Guidelines are characterized by terms like: take into account, be aware of, take into
consideration, be cautious, make sure that, take care of, ignore, do, don‟t, dangers, bear
in mind, pitfalls, be alert, consider, can be done, should be done, tip.
Constraint
A piece of text that contains restrictions or limitations that are applicable to an open
ended task or the result of such a task and that limit the degree of freedom to perform a
task, but that do not give a recipe to perform the task. Constraints are often based on
regulations, budgets, or wishes of a client.
A typical example of a constraint is that the price of a house that an architect designs for
you should not exceed 400.000 Euro.
In practice this means that the APOSDLE system, based on a selected task and a learning goal that is
associated with topics related to this task, will search for documents that contain knowledge artefacts
(documents and fragments of documents) that have been labelled with material uses that are related
to this specific learning goal. For instance, for a topic with the learning goal “basic understanding of ..”
knowledge artefacts with the label “definition”, “example” or “introduction” will be selected. Figure 5-1
gives an example of how this might look like on the screen of an APOSDLE user. In the left hand side
of the screen tasks and topics (with related learning goal types) can be selected. Based on this
18
selection descriptions of resources are presented in the right hand side of the screen .
18
In the example presented in Figure 5-1, a document is selected with the material use “main points”. This is not correct
because this material use type is not associated with the learning goal type “understand”. Anyway, these “use” indications will
not be present in the 3rd Prototype.
page
70
Figure 5-1: Example of a selection of documents with additional information
5.1.4.2
Getting an Overview of the Content to be learned and planning of Learning
Next to the setting of learning goals in the first phase of learning, it is important for learners to get an
overview of the content to be learned to be able to plan learning. This is especially true for curiosity
driven learners and to a less extent for learners with a very specific learning goal, unless this learning
goal is more complex to achieve and comprises several sub-steps. As stated above, these learners
often will stop the learning process as soon as their specific goal has been reached and therefore
extensive planning is not necessary. However, for finding missing prior knowledge, mobilizing prior
knowledge and finding connections between prior knowledge and new knowledge and skills, it still
would be helpful for learners to have an overview of the relevant parts of the underlying knowledge
domain. The activation of prior knowledge positively influences learning (Merrill, 2002) because
learning is an active process in which new knowledge is integrated in existing cognitive structures.
Piaget and Inhelder (1973) speak of assimilation of new knowledge in the existing knowledge
structures and accommodation of existing knowledge structures to make place for new information.
Learning in which new information is integrated into existing structures is called meaningful learning,
because the new information is somehow related to existing knowledge. This is different from rote
learning. In rote learning new knowledge is not integrated in the existing structures, but it is stored
isolated from those structures. Therefore, knowledge gained through rote learning is easily forgotten,
while knowledge gained through meaningful learning is remembered better.
To support learners, the main domain elements and their relationships could be described or
visualised so that they are ready for inspection. An example of such a representation of the RESCUE
domain containing the SR model is given below in Figure 5-2.
page
71
Dependencies
Model
Actors
I-Star Model
Use Case
SR Model
SD Model
Figure 5-2: Representation of domain elements
From Figure 5-2 it can be derived that the SR model and the SD model both contribute to the I-Star (i*)
model. We can use the domain model (see Section 3.1) to create learning paths that can support
learners.
Planning of learning: setting learning paths
Curiosity driven learners often do not have specific information or learning needs, but have a more
general need to know more about a topic or (part of) a knowledge domain or to learn to perform a
complex task. For them it is important to plan learning (to set out a learning path that should lead to
the general learning goal). To be able to do this, they need to know which topic or knowledge is
prerequisite knowledge for other topics and which topics are related to the topic identified as important
to learn. Giving the learners insight in the domain model can facilitate the planning of learning. Apart
from giving learners insight in the domain, it is also helpful to give them insight in the relationships
between tasks that they already can perform and those that they have to learn to perform and domain
elements that are related to these tasks. In traditional instructional design (for regular courses) an
analysis of these elements would form the basis for the sequencing of instructional content and
actions. An example of a functionality that will support these functions is presented in Figure 5-3. In
this figure a small example is presented that refers to the knowledge domain of statistics. In the upper
part of the diagram the domain model is represented and in the bottom part elements of the task
model are represented. Furthermore relationships between elements within and between these
models are shown.
page
72
Figure 5-3: Example of a tool that gives insight in the domain and task model.
Based on the task and domain model suggestions could be provided to the knowledge workers to help
them set their learning paths. Learning paths can be set by selecting topics from the domain model or
from a list of topics that are related to a certain task. The APOSDLE system subsequently can
propose a sequence in the selected topics based on information about prerequisite knowledge
relationships, information from the user profile (see Section 4) and/or general heuristics based on
instructional strategies. An example of such functionality is given in Figure 5-4. At the left hand side of
the pop up window is a list of topics that are preselected for the learner based on the task and domain
model and her level of competence from the user profile (these are in the tab proposed topics) or a list
with all topics (without pre-selection by the system). The user selects a topic from this list and adds it
to the learning path. When the user has selected several topics the APOSDLE system will put these in
a sequence (the proposed learning path) which is displayed at the right hand side of the window. This
sequencing is based on the domain model. The learner can save this learning path so it can be
retrieved and accessed later.
Another way to support workers in planning of learning could be to give them access (via a search
functionality) to learning paths that have been defined by co-workers or knowledgeable persons and
that have been saved in the system and have been made public (by the author).
page
73
Figure 5-4: Example of the learning path editor functionality.
5.1.5
Supporting executive Functions
Important executive functions are: selecting information, actively processing the information,
monitoring of learning and reflection. Since these last two are also important closing functions, we will
discuss them in Section 5.1.6 that focuses on those functions.
5.1.5.1
Selecting Information
A common strategy to help find the right knowledge is by annotating documents or parts of documents
and to use these annotations to find relevant documents. The annotations are based on elements of
the relevant knowledge domain. However, from a learning perspective presenting a document that
contains information about a topic that is relevant to the learner is not sufficient. The fragments with
information that are presented should not be too large, because this can cause cognitive processing or
information overload that can impede learning (Mayer & Moreno, 2003). Especially for goal-directed
learners, it is important that these chunks of information have a direct relationship with their specific
learning goal. As stated above, these learning goals often are based on a specific question or
information need. This means that apart from domain related annotations, also annotations should be
used that are based on types of information needs. These are based on the types of material uses that
were discussed in the previous section, like definition, explanation or guideline.
When a learner selects a specific resource from the list of selected resources (see Figure 5-1) a tool
(the APOSDLE reader) will be opened that displays the content and that shows which parts of the
content are annotated (and with which labels). This is illustrated in Figure 5-5. When the learner
selects a specific knowledge artefact at the right hand side of the screen, the document in the left
hand side will scroll to and highlight that specific piece of text in the document. This means that the
learner does not have to search the document for herself but will be presented with well delineated
chunks of information in a meaningful context (the chunks are not isolated from the rest of the
document). This contextual information is important because much knowledge is very context specific.
page
74
Furthermore, the learner can see which parts of the document have also been annotated. This can
help him find related content without having to read the entire document.
Figure 5-5: Example of the APOSDLE reader.
Self-directed learners can also be supported in their selection process by providing them with
information about the quality of the information (sources) that were retrieved. This can be done by
implementing a kind of rating system which users can use to express their judgement about the quality
of the piece of information that is found by the system. The aggregated judgements of other users can
be very informative and can guide the learner in selecting resources when more than one resource is
available.
Another way to give learners indirect information about the quality of the resource is to show who was
the source of the annotations that are associated with the selected part of the document. This gives
the learner the possibility to answer questions like: Is it a trustworthy source? Do I often
rd
agree/disagree with this source? All of these elements will be implemented in the APOSDLE 3
Prototype (see Figure 5-1 and Figure 5-5).
5.1.5.2
Actively Processing Information
Presenting the right information at the right time can help a knowledge worker that has problems in
performing a specific task to complete this task. However, this does not necessarily mean that the
knowledge worker has learned something in the sense that she has changed existing knowledge
page
75
structures or has acquired new structures that enable her to perform the task in the future without
support. This also is acknowledged by some researchers who focused on workplace learning. Billett
(1996a, 1996b) concludes that, although resource-based learning materials have an important part to
play in the development of workplace knowledge, they need to be utilised in conjunction with guidance
available through interaction with others in the workplace. McKavanagh (in Smith et al. 2002) indicates
that effective conceptual learning presupposes forms of interactivity that include discussion, action
learning, and questioning.
Self directed learners can be supported by accompanying selected information with support that aims
at:

Thinking about information

Coming to conclusions and own opinions

Formulating conclusions verbally

Getting an overview

Practicing and applying

Getting an overview over application conditions and possibilities

Testing progress – self assessment
Support for actively processing the information presented is also available in the APOSDLE reader by
means of the “Learn more” tab (at the right hand side of Figure 5-6). This tab gives access to several
support features that can enhance learning, like a description of the topic and references to specific
learning hints (these will be displayed in a new pop-up window, see Figure 5-7) and co-workers (see
also 5.2).
page
76
Figure 5-6: Example of the APOSDLE reader with additional support for learning and references to
learning hints.
Learning hints can be contemplative in nature (without observable output), but can also be aiming at
explicit outcomes that can be observed and assessed by others. In the latter case, the hints will
contain an engagement activity with the opportunity for the user to enter information in specific input
fields (see Figure 5-7).
page
77
Figure 5-7: Example of a learning hint with a note facility.
Four types of hints can be distinguished based on the work of Simons (2000). These are described
below. To make these categories of hints more specific, we related them to measurable action verbs
for generating performance objectives based on Bloom‟s learning taxonomy (1956) and to cognitive
processes that are used in the taxonomy of Anderson and Krathwohl (2001).
 Hints aiming at activation of existing knowledge structures and at relating new information to
these structures, and at consolidation of new knowledge. Process: remember.
 Hints aiming at comprehension of new information. Processes: interpret and explain.
 Hints aiming at practicing and applying new skills and information. Processes: exemplify,
classify, and execute.
 Hints aiming at (re)structuring of new information, accommodation of existing knowledge and
evaluation of the value of the new knowledge. Processes: summarise, infer, compare, organize,
check, and differentiate.
Meaningful combinations are made between processes and resource types (not every combination is
meaningful from a learning perspective). In some cases the name of the processes (as presented
above) is changed because it is not very informative for the learner. For each combination a hint is
formulated that contains two elements (see the pop-up window in Figure 5-7):
 An engagement activity that states what a learner could do to process the information.
page
78
 A rationale that states why it is considered useful to engage in this activity.
The content of these hints is adjusted to the specific material resource type associated with a learning
goal (need). This means that hints that accompany an example are (slightly) different from hints
related to a guideline or a constraint. In Table 5-1 examples are given of the type of hints that are used
for learning support.
Material resource type
Definition
Hints
Help me to remember this definition
What
Try to relate the definition to other things you already know about this
topic.
Why
This can make it easier to retrieve the definition when you need it in the
future.
-What
Retrieve the relevant definition from your memory.
Why
Reproducing the definition can make it easier to use it directly when it is
needed in the future.
How can I illustrate this definition
What
Give an example that fits the definition.
Why
By giving an example that fits the definition you can find out whether you
understand the definition and are able to use the characteristics in the
definition to identify an example.
How can I categorize this definition
What
Does your point of interest belong to the category given in the
definition?
Why
By doing this you can check whether the characteristics given in the
definition are clear and sufficient to determine if other cases match the
definition.
How can compare this definition with my task or area of interest
What
Compare the given definition to a definition of a related topic and try to
identify differences and similarities.
Why
By doing this you can discover whether you understand the differences
between these two definitions.
-What
page
79
Hints
Compare the given definition to your point of interest and try to identify
differences and similarities.
Why
By paying attention to the differences and similarities between your
point of interest and the given definition you can discover whether you
understand the differences between them.
Help me to inspect this definition on contradictions
What
Check whether the elements of the definition are clear, consistent and
unambiguous. If so, state why. If not, state why not.
Why
By doing this you can make sure that the definition enables you to
unambiguously determine if something fits the definition or not.
How to …
Help me to remember this procedure
What
Try to find a memory aid that helps you to remember the procedure.
Why
By doing this, it is easier to retrieve the procedure when you need it in
the future.
-What
Retrieve the relevant procedure from your memory.
Why
By retrieving it, you will be able to apply that procedure in the future
without the need to look it up again.
How can I understand this procedure better
What
Try to rephrase the presented procedure in your own words.
Why
By describing it in your own words you can discover if something is still
unclear in the procedure.
How can compare this procedure with my task or area of interest
What
Compare the given procedure to a procedure of a related topic and try
to identify differences and similarities.
Why
By doing this you can discover whether you understand the differences
between these two procedures.
-What
Compare the given procedure to your task and try to identify differences
and similarities.
page
80
Hints
Why
By paying attention to differences and similarities between your task and
the given procedure, you can better decide if the procedure is applicable
to your task or not.
How can I implement this procedure
What
Give an example how you could do it, by carrying out (part of) the task.
Why
By following the steps that are needed to carry out (part of) the task, you
can make sure that you do them in the right way and the right sequence.
Table 5-1: Examples of hints that can be presented when interacting with material resources
The engagement activities are accompanied by an additional component: a suggestion that draws a
learner‟s attention to other elements of the APOSDLE system that could support learning, like: “You
could try to use the APOSDLE Browse facility to find more about this topic” or “You could try to use the
Relationship tool to find more about related topics” or “You could use APOSDLE Search to find more
documents or colleagues who know more about this topic.” These also contain a what and a why
component. These additional suggestions will be available by means of a separate tab in the Hints
window. In some cases no additional suggestions are available because no meaningful activities could
be formulated.
An example of additional suggestions for the second element in the table above (How can I illustrate
this definition) could be:
What:
You could try to use the APOSDLE Browse facility to find examples related to your topic.
Why:
This could help you to check whether your example matches with (one of) the examples
found.
The hints clearly try to engage learners in self assessment, a topic that will be elaborated below.
5.1.5.3
Testing progress
An important executive function is the testing of progress. An important prerequisite for this is
feedback on activities. Therefore it is important to get feedback on the answers users give to
questions and tasks that are specified in the hints. Users can give their answers in the input fields (the
notes) that are available. However, evaluating the content of these input fields by the system is
impossible because of the generic architecture of the system. In advance it is not known what the
content of the documents is that are displayed (only their material use type). Therefore the questions
and tasks in the hints can only be very general and hint at the elements presented above. This makes
it impossible to formulate content related evaluation criteria that can be used by the system to assess
the quality of the input. As a consequence this cannot be used for outcome assessment and profile
adjustment.
This implies that other types of feedback mechanisms have to be implemented. One way to obtain
feedback is to consult co-workers who are more experienced (knowledgeable persons) and to discuss
the content of the selected information and the learning hint with them (see Section 5.2). Another way
is to compare the results of an engagement activity with the results of others who have also done the
activity and to discuss with one or more of them one‟s own answers. This is a kind of self assessment.
page
81
To make this assessment possible the notes of others who accessed the same learning hint (and the
same material use type) are made available on request to the learners (see Figure 5-7).
In addition, the methods described in Section 4.2 can be used for another kind of “testing” progress.
5.1.6
Supporting closing Functions
Important closing functions are: evaluation and reflection. Dewey (1933) already stated that
“experience plus reflection equals learning”. Reflection plays an important role in finding out what was
learned and what should still be learned. When non reflective learning occurs, mostly the learners are
not aware that they have learned something and knowledge is often implicit and difficult to verbalise.
This means that learners can apply the knowledge in situations that resemble the situation in which
they have learned it, but have difficulties in explaining what they did (and why they did it) to other
people. For example: when typists were given caps for typewriter keys, they could not arrange them in
the proper configuration, yet they all could type rapidly and accurately (Norman, 1988). Furthermore,
learners often have difficulties in transferring implicit knowledge to other contexts.
Norman (1988) describes two modes of cognition: an experiential mode and a reflective mode, “These
two modes do not capture all of thought, nor are they completely independent”. The experiential mode
is one of perceptual processing, it is a pattern or event driven activity. It requires some thought but the
information processing is data driven and reactive. According to Norman, this mode leads to an
accumulation of facts, it reactivates information that is already present in the memory system and it
leads to a tuning and shaping of knowledge structures already available. Reflective thought requires
the ability to store temporary results, to make inferences from stored knowledge and to follow chains
of reasoning backward and forward, sometimes backtracking when a promising line of thought proves
to be unfruitful. The use of external aids facilitates the reflective process by acting as external memory
storage, allowing deeper chains of reasoning over longer periods of time than possible without the
aids. Effective reflection requires some structure and organization and is greatly aided by systematic
procedures and methods and the aid of other people.
Dialogue and discussion are important means to enhance reflection because in these processes
people have to make their thoughts explicit and can compare their ideas with those of others and this
can enhance self-evaluation. Furthermore, in these processes feedback can be given on activities that
learners were engaged in. Other people are essential for providing feedback, because a system like
APOSDLE cannot give content related feedback because beforehand it is not known what the content
of the information is (only their resource type is known).
To stimulate and enhance reflection when dealing with material resources, hints can be provided (see
previous section) that can promote dialogue and discussion, but that also promote reflective thought
by the learner.
Another way to support reflection on learning is to give the learners an overview of their learning
experiences and their knowledge levels. An example of such functionality is displayed in Figure 5-8.
This can also support the preparatory functions that have been described in 5.1.4.
page
82
Figure 5-8: Example of a learning experiences functionality (MyExperiences)
In Figure 5-8 a learner can see the knowledge levels for the different tasks and concepts which are
part of the domain model. The different levels are set apart by using different basic colours. Inside
each task field the level of knowledge for topics is indicated by different grades of the basic colour.
This allows the learner to inspect her “competence map” and find out where lacking knowledge is
present.
5.1.7
Remaining problems – new challenges
Although the general approach described above is clear and operational in the third prototype, there
still remain problems that have not been solved yet. These will be elaborated below.
Selecting knowledge artefacts based on criteria
In many cases more than one instance of a certain knowledge artefact may be available. In that case
rules must be available to determine which of these instances will be used in a template. Criteria have
to be defined for this selection process based on a quality rating by experts or users.
Selecting knowledge artefacts based on quality means that all annotated artefacts also have to be
rated on a quality scale. This is a very labour intensive process that is difficult to do automatically. An
alternative procedure is to rate documents, but this means that every knowledge artefact (as a part of
that document) will get the same quality rating. This could mean that a very good definition or example
that is present in a document with a low quality rating, will never be retrieved.
An alternative would be to start with random selection and have users rate the quality of artefacts. This
(average) rating will be used in the future to select knowledge artefacts. However, user ratings could
page
83
be unreliable because users, who obviously lack certain knowledge, have to rate artefacts related to
that knowledge. In the third prototype we will experiment with a rating procedure.
Keeping track of learning materials accessed
The APOSDLE system selects and presents the best fitting documents and knowledge artefacts for a
certain topic and learning goal. This means that the content in principle will be the same every time it
is generated if there are no new documents and knowledge artefacts available since the last session
and the learning goal and topic are still the same. To prevent this, the system should keep track of the
learning experiences and not present the same references to documents and artefacts that have been
processed in a previous session concerning the same learning goal. This means that the user
competency profile should include not only a learning experiences section (see Figure 5-8) that logs
which topics have been accessed but that also keeps track of the knowledge artefacts that were
accessed.
5.1.8
Component evaluation: learning paths
Self-regulated learners plan, monitor, and evaluate their own learning process. APOSDLE‟s Learning
Path (LP) component provides support for the first step: planning. Although planning is generally
considered an important aspect of effective learning, prior research shows that learners do not always
perform such regulative activities spontaneously. Furthermore, planning is a difficult process and
learners sometimes need support. Therefore, the Learning Path component was developed to
stimulate and support planning.
To support planning, the LP component uses information from APOSDLE‟s underlying models. With
this information, the LP could potentially construct individualized LPs automatically. If APOSDLE
would completely take over the planning process from learners, this could have an impact on the
learning process. For example, in a previous study, we examined the effects of planning on knowledge
(Bonestroo & De Jong, 2009). In that study, participants performed planning activities and their
knowledge was measured directly after planning, but before the actual learning phase had started. We
found that participants had more knowledge about a learning domains‟ structure when they had
actively created their own plan, compared to when the computer tool had suggested a plan. We found
no differences for factual knowledge.
In the current evaluation, we wanted to test whether the positive effects found in our previous study
would also lead to positive learning effects when the LP component is embedded in a larger learning
environment. Whereas our previous study focussed on planning only, the current evaluation focuses
on effects of planning and the subsequent performance phase of learning.
In APOSDLE, learners create LPs by hand (see for an example Figure 5-4). In this evaluation we
wanted to find out whether learners who actively create a LP and use that path for learning gain more
domain knowledge (factual and conceptual knowledge) and more structural knowledge (knowledge
about the structure of the learning domain) in the learning process, compared to learners that receive
a plan that was automatically generated by an LP component. Furthermore we wanted to explore
effects of creating a LP on task load and applied learning processes. Therefore, we stated the
following research question and three sub questions:

Main Question: Is learning more effective, when learners manually create a LP, compared to
when they receive an automatically generated LP?
o
Sub Question 1: Do learners gain more factual, conceptual, and structural knowledge
when they manually create a LP, compared to when they receive an automatically
generated LP?
o
Sub Question 2: Does it lead to a higher task load when learners manually create a
LP, compared to when they receive an automatically generated LP?
page
84
o
Sub Question 3: Does it lead to the use of more effective meta-cognitive learning
strategies when learners manually create a LP, compared to when they receive an
automatically generated LP?
Type of evaluation
This evaluation is a follow up of a previous study performed within the APOSDLE project (Bonestroo &
De Jong, 2009). We used a lab study with students and analysed log data from the lab studies. A
within-subjects experimental design was used for this evaluation. Each participant worked with the two
tools described below. Both tools could be used with two learning domains. To compensate for carryover effects, the order of the tools and domains was counterbalanced. The resulting four combinations
are presented in Table 5-2.
First Trail
Second Trail
Group
Tool
Domain
Tool
Domain
1
CG
A
LG
B
2
CG
B
LG
A
3
LG
A
CG
B
4
LG
B
CG
A
Table 5-2: Overview of groups in evaluation design
Dependent variables in this evaluation were knowledge, task load, regulative processes, and
preferences. Three types of knowledge were measured: factual, conceptual and structural knowledge.
In our evaluation, we did not use a pre-test because it might interfere with the effects of the planning
tools. A pre-test not only activates prior knowledge, it might also make learners more aware of their
shortcomings and therefore change their learning strategies. Furthermore, because the data was
analyzed within subjects, we did not have to compensate for individual differences such as prior
knowledge.
The application domain was Statistical Data Analysis (SDA). Eighty-three behavioural science
students participated in the study, 18 males and 65 females. Their average age was 19.49 years (SD
= 1.63). Participants received credits for participating. Participants were randomly allocated to one of
the four groups described in Table 5-2. All participants worked with both tools.
To run the sessions in the computer lab, a mock-up of the system was developed that could run in a
web browser. The mock-up contained a tool to create LPs and that could show the domain model in a
graphical way. We compared two tools: the computer generated the plan (CG-tool) and learners
generated plans (LG-tool). The two tools were identical, except the CG-tool provided the learners with
a predetermined plan.
The contents of the domain were adapted to match the knowledge level of the participating students.
Two topics within SDA were used: measures of central tendency and dispersion, and correlation and
regression. For every topic we identified 25 key concepts and for every concept, a page with learning
material was composed. Every page had approximately 250 words and 1 or 2 images illustrating the
concept. Learning material was taken from two introductory statistics books. We used the first two
chapters from Discovering Statistics using SPSS (Field, 2009) and the first two chapters from the
Dutch version of Introduction to the Practice of Statistics (Moore & McCabe, 1997). To create the
graphical overviews, we identified the relationships between the concepts in such a way that they
were in line with the instructional material. Then, we ordered the concepts so that they were in line
with the recommendations described by O‟Donnell, Dansereau, and Hall (2002).
page
85
5.1.8.1
Measurements
Three types of knowledge were measured: factual, conceptual, and structural knowledge. Two
versions were developed for every knowledge test: one for the central tendency domain and one for
the correlation and regression domain.
Factual knowledge addressed isolated statements about the learning domain. Factual knowledge did
not require understanding of the material. An example of a factual knowledge test item was: „There are
four quartiles‟ (correct answer: false). Factual knowledge was measured with 16 items for each
domain. Participant could indicate whether a given factual statement was either correct or not. The
score was the number of correct items and ranged from 0 to 16.
Conceptual knowledge addressed the connected and integrated knowledge about the learning
domain. Conceptual knowledge required understanding of the material. To measure conceptual
knowledge, translated items from the fourth version of the Comprehensive Assessment of Outcomes
in Statistics (CAOS) and from the Artist database (https://app.gen.umn.edu/artist) were used. Those
items were developed to measure how well students understand important basic statistical concepts.
The CAOS items have shown adequate reliability, validity, and discriminative power (delMas, Garfield,
Ooms, & Chance, 2007). For each learning domain, 26 multiple choice items were constructed. The
score for the participants was the number of correct items and could range from 0 to 26.
Structural knowledge addressed the information that was presented in the graphical overviews. We
defined structural knowledge as knowledge about the relations in the graphical overview. Structural
knowledge was addressed with closed items that addressed the structure of the graphical overview.
For every item, participants had to indicate whether there was a relation between two concepts from
the graphical overview. In the example shown in Figure 5-9, there is a relation between Method 635
and Neurovation (and between Neurovation and Method 635), but not between Neurovation and MindMapping, and not between Method 635 and Mind-Mapping. There were 32 structural knowledge items
per topic. Again, the score was the number of correct items and could range from 0 to 32.
Neurovation
Mind-Mapping
Methode 635
Figure 5-9: Example of Graphical Overview
Perceived knowledge was measured by asking participants to indicate from which tool they learned
most. The following two questions were used „From which way of working do you learn most about the
domain contents?‟ and „From which way of working do you learn most about the structure shown in
the graphical overview?‟ For both questions, participants could select: manually, automatic, or nodifference.
Task load
To measure the task load, an adapted version of NASA‟s Task Load Index (TLX) was administered
twice. The TLX was developed by Hart and Staveland (1988) and combines characteristics of the task,
and of the conditions under which it was carried out (for example, time pressure) into one score. The
original version uses 6 scales and 15 pair wise comparisons between the scales. In this experiment,
the scale „physical demand‟ was removed from the test because physical demands were not relevant
for this study. Thus, the TLX was used with 5 remaining scales (mental demand, temporal demand,
performance, effort, and frustration level) and 10 pair wise comparisons between those scales.
Learning Processes
To analyze the behaviour of the participants, three distinct processes were identified: planning time,
reading time, graphical overview time. The planning time, was the time participants worked on creating
the planning. Planning was stopped when the created plan was approved by the tool, and the
participants started traversing the plan. Once a participant had started traversing the plan, they could
page
86
not go back to the planning phase. During traversal participants could either read the instructional
material (reading time) or study the graphical overview (graphical overview time). The times that
participants spent on these processes were stored in log files.
Several learning processes were measured using self-reporting questions. The following four topics
were tested with three questions: clarity of the learning goal, the activation of prior knowledge,
difficulty of creating plans, and whether the use of software supported made the planning easier. The
items were based on the learning strategies scale of the Motivated Strategies for Learning
Questionnaire (MSLQ) and more specifically on the cognitive and meta cognitive strategies within the
MSLQ (Pintrich, Smith, Garcia, & McKeachie, 1993). Participants indicated whether they agreed with a
given statement on a 5 point Likert scale. For example, to measure the difference in prior knowledge
activation participants received the following three statements: „my prior knowledge was activated
when working with the LG-tool‟, „my prior knowledge was activated when working with the CG-tool‟,
and „my prior knowledge was better activated when working with the LG-tool, compared to the CGtool‟. In the „result‟ section, the scores on the first two questions are compared and the score on the
last question is presented. Furthermore participants received a number of single statements and were
asked whether they agreed with the statement or not. Table 5-3 contains those (translated)
statements.
#
Statement
1
I had more time for studying when I worked with the CG-tool.
2
If you create a LP, you already learn from planning.
3
When I created the LP myself, I had more control over the learning process,
compared to when the computer created it.
4
I rather select the learning goal myself, and not have it selected by the computer.
5
When I study, I prefer to have control over the learning process, compared to giving
the computer control.
6
By actively creating a planning, it is easier to understand information, compared to
when the computer created the plan.
7
I would be useful to have a planning tool for difficult topics.
8
I would be useful to have a tool with a graphical overview for difficult topics.
9
Due to planning, I had too little time left for learning.
Table 5-3: Statements
Preferences
To measure the preference for the tools, participants were asked to indicate which tool they would
prefer to use in two situations. One item was: „If you would have to learn a topic yourself, which tool
would you prefer?‟ The other was: „If you were to create a LP for someone else, which tool would you
prefer?‟ For both items, participants could select: the LG-tool, the CG-tool, or no preference.
5.1.8.2
Procedure
Participants were tested in the computer laboratory in groups of approximately 15 people. A session
took 105 minutes to complete. The sessions started with a demonstration and explanation of the tools.
Participants were told that they had to construct LPs and that they had to complete tests afterwards.
Then they individually followed a tutorial and performed a practice session with the practice domain. In
the tutorial, participants created some LPs as an exercise. They also completed knowledge tests
about the practice domain, so they knew what kind of questions to expect.
After the introduction, participants worked on the first assignment. Depending on the condition,
participants received one tool with one of the two learning domains. After the plan, either constructed
page
87
by participant or by the computer, was approved, the tools arranged the learning material according to
the plan. Participants then used the created plan to deal with the learning material. Once they had
started with the learning material, they could not go back to the planning phase. After 18 minutes, the
TLX was administered. Next the conceptual knowledge test, structural knowledge test, and factual
knowledge test were administered. The completion of the tests took about 20 minutes. However, there
was no strict time limit for the completion of the tests.
When all the tests were completed, participants started on the second assignment using the other tool
and the other topic. This again took 18 minutes, followed by the same test types as in the first run.
After all the tests were completed, participants completed the forms with the statements about the
learning processes and preferences.
5.1.8.3
Results
Process Measures
The tool influenced the times that participants used different tools of the learning environment. The
mean times are shown in Table 5-4. We performed three separate univariate repeated measures
ANOVAs and used a Bonferroni correction to control for familywise errors. Results showed that
participants spent more time on planning with the LG-tool, F(1, 82) = 116.34, p < 0.05, spent more
time on reading the learning material with the CG-tool, F(1, 82) = 75.03, p < 0.05, and spent more time
on studying the graphical overview after the planning phase with the CG-tool, F(1, 82) = 19.10, p <
0.05. On the self-reporting statements they reported that they had more time for studying the learning
19
material when they used the CG-tool, CI.95 = [3.25, 4.24] , and they (slightly) agreed with the
statement that due to the planning activities, they had too little time to study the learning material, CI.95
= [3.02, 3.55].
LG-tool (s)
CG-tool (s)
M
SD
M
SD
Planning *
334.86
223.30
53.12
96.72
Studying Graphical Overview *
141.71
136.86
219.74
172.61
Reading Learning Material *
636.80
219.42
840.56
170.33
* comparing times for LG-tool and CG-tool at α = 0.001
Table 5-4: Average times working with different parts of the tools in seconds
Self-regulation
Based on the self-report questions, participants indicated that the goal of learning was clearer when
they used the LG-tool (M = 3,67, SD = 0.95) than when they used the CG-tool (M = 2.86, SD = 0.99),
t(82) = 4.73, p < 0.05. Furthermore, participants agreed with the statement that the goal of learning
was clearer when working with the LG-tool, CI.95 = [3.22, 3.92].
Participants reported that their prior knowledge was better activated with the LG-tool (M = 3.19, SD =
0.92) than with the CG-tool (M = 2.60, SD = 1.07), t(82) = 3.69, p < 0.05. Furthermore, they agreed
with the statement that their prior knowledge was better activated with the LG-tool, CI.95 = [3.25, 4.05].
Participants thought that they had more control over the learning process with the LG-tool than with
the CG-tool, CI.95 = [3.24, 3.80].
19
The results from the self-reporting items on the learning process are interpreted as interval data. If the interpretation of the
scale is not explicitly described, the scales range from 1 = totally disagree, through 3 = neutral, to 5 = totally agree. For these
items, we present a ninety-five percent confidence interval for the mean, CI.95 = [lower bound, upper bound]. We interpreted that
participants disagreed with the statement, if the upper bound of the confidence interval was lower than 3, and that they agreed
with the statement when the lower bound was higher than 3.
page
88
Perceived Knowledge
Participants were asked whether they learned more while working with the LG-tool, the CG-tool, or
whether there was no difference between the tools. Most participants thought that they gained more
structural and domain knowledge while working with LG-tool. Table 5-5 gives an overview of the
responses.
LG-Tool
CG-tool
No difference
Structural Knowledge
59 (71.1%)
24 (28.9%)
0 (0%)
Domain Knowledge
56 (67.5%)
27 (32.5%)
0 (0%)
Table 5-5: Perceived knowledge gain
Participants agreed with the statement that it was easier to understand new knowledge when they
used the LG-tool, compared to using the CG-tool, CI.95 = [3.21, 4.53]. They also agreed with the
statement that they gained knowledge from the active planning process itself, CI.95 = [3.21, 5.02].
Knowledge
In the experiment, every learning domain had its own tests. One knowledge test addressed the
„correlation and regression‟ domain and one the „measures of central tendency and dispersion‟
domain. To compare these scores, the results were first normalized and the resulting z-scores were
used in the analyses. We analyzed the knowledge tests with a repeated measure multivariate analysis
of variance. We used the order of the tools and the order of the domains as between-subjects factors.
A multivariate repeated measures ANOVA was performed to compare the knowledge scores. Using
Pillai‟s trace, there was no significant main effect of the tool on the knowledge tests‟ scores, V = 0.04,
F(3, 76) = 0.96, p > 0.05, and no interaction effect between the order of dealing with the domains and
the order of the used tool, F(3, 76) = 0.91, p > 0.05. There was an interaction effect between the type
of tool used and the order in which the tools were used, V = 0.12, F(3, 76) = 3.36, p < 0.05. Separate
univariate analyses of this interaction effect showed that the effect was significant for structural
knowledge, F(1, 82) = 8.87, p < 0.05, but not for factual knowledge, F(1, 82) = 1.56, p > 0.05, nor for
conceptual knowledge, F(1, 82) = 0.59, p > 0.05. The interaction effect showed that participants
gained more structural knowledge working with the first tool they used when compared to the second
tool, independent of which tool that was.
LG-Tool
CG-Tool
Score
M (SD)
z score
M (SD)
Score
M (SD)
z score
M (SD)
Factual Knowledge
10.59 (2.20)
-.03 (.94)
10.69 (2.54)
.02 (1.06)
Conceptual Knowledge
12.06 (3.72)
-.01 (.97)
12.10 (3.74)
.01 (1.03)
Structural Knowledge
19.12 (3.39)
.04 (.97)
18.82 (3.61)
-.05 (1.03)
Table 5-6: Knowledge test scores
Further analysis showed that participants who explicitly reported that they had gained more structural
knowledge while working with the LG-tool, did not score higher on the structural knowledge test
F(1,58) = 2.72, p > 0.05. Likewise, participants who thought that they had learned more conceptual
knowledge with the LG tool actually did not score higher on the conceptual knowledge test, F(1, 55) =
0.54, p > 0.05. Remarkably, they did score higher on the structural knowledge test, F(1, 55) = 4.72, p
< 0.05.
page
89
Task Load
There was no main effect for the self-reported task load, F(1, 82) = 0.69, p > 0.05, and there were no
interaction effects between the tool order, F(3, 79) = 2.65, p > 0.05, and the domain order, , F(3, 79) =
1.25, p > 0.05. Further analysis revealed that when participants used the LG-tool, there was a positive
relation between the time participants needed to create the plan and the perceived task load, r = 0.22,
p < 0.05. This is a small positive effect, indicating that the longer it took to create an approved plan,
the higher the self-reported task load.
Tool Preferences
Participants were asked which tool they would prefer in two hypothetical situations. In one situation
they were learning a domain themselves and they would use the LP for their own learning. In the other
situation someone else was learning a domain and the participants were asked to make a planning for
the other person. Table 5-7 shows their responses. The table shows that most participants preferred
to use the LG-tool, when the LP was for themselves, and preferred the CG-tool when the LP was for
someone else. To measure the difference between the answers, we scored the selection for the LGtool as 1, the selection for the CG-tool as -1, and the selection for no preference as 0. Using a paired
samples t test, there was a significant difference between the preference participants had for their own
learning (M = 0.25, SD = 0.88) and for the learning of others (M = -0.43, SD = 0.87), t(82) = 5.3, p <
0.05.
LG-tool
CG-tool
No preference
Plan for yourself
45 (54.2%)
24 (28.9%)
14 (16.9%)
Plan for someone else
21 (25.3%)
57 (68.7%)
5 (6.0%)
Table 5-7: Tool preferences
Participants thought it was useful to have graphical overviews for difficult learning domains, CI.95 =
[3.22, 4.57] and to have planning support tools, CI.95 = [3.22, 4.52]. Furthermore, they preferred to
have control over the learning process, compared to giving the computer control, CI.95 = [3.24, 4.25].
They also agreed with statement „I‟d rather determine the learning goal myself, than having a
computer do that for me‟, CI95 = [3.20, 4.19].
Participants thought that the creation of a plan was easier with (M = 2.51, SD = 1.08, 1=very easy and
5=very difficult) than without the software, M = 3.18, SD = 0.84, t(82) = 4.70, p < 0.05. Furthermore,
participants agreed with the statement that it was easier to create the plan with the software than
without it, CI.95 = [3.25, 4.48].
5.1.8.4
Conclusions
In this section we first address the sub questions stated in the introduction and conclude with
answering the main question.

Sub Question 1: Do learners gain more factual, conceptual, and structural knowledge when
they manually create a LP, compared to when they receive an automatically generated LP?
The main result of this evaluation was that although most participants thought that they had learned
more while working with the LG-tool, this was not confirmed by their scores on the knowledge tests.
There was no effect on structural, factual, or conceptual knowledge. Even participants, who explicitly
indicated that they gained more structural knowledge with the LG-tool, did not score higher on the
corresponding knowledge test. Although SRL theories (for an overview see, Puustinen & Pulkkinnen,
2001; Zimmerman & Schunk, 2001) state that the preparatory phase is important for learning, we did
not find positive effects of active planning on the learning outcomes.
Results from our previous study (Bonestroo & De Jong, 2009) showed that learners gained more
structural knowledge when they actively created LPs. However, results from the current evaluation
show that when the LP component is embedded in a learning environment and the component is used
page
90
in the learning process, the effect is not observable anymore. Thus, the initial gain in structural
knowledge (found in our previous study) did not lead to more conceptual knowledge (found in this
evaluation). A possible explanation for this effect can be derived from the process measures. We
found that participants who used an automatically created path gave more attention to the higher level
information in the graphical overview in the performance phase of the learning process.

Sub Question 2: Does it lead to a higher task load when learners manually create a LP,
compared to when they receive an automatically generated LP?
There was no difference in the task load of both tools. In our previous study, we found an increased
task load for the active planning and we expected the same outcome for this evaluation. The fact that
there was no significant difference in task load, could indicate that the planning phase of the learning
process does not heavily contribute to the total cognitive load of the learning process, or that the
processing of learning material also contributes to the overall load, resulting in a similar total load.
Moreover, because the TLX also takes frustration into account, studying learning material without
knowing the overall structure of the learning domain might also have increased the task load.

Sub Question 3: Does it lead to the use of more effective meta-cognitive learning strategies
when learners manually create a LP, compared to when they receive an automatically
generated LP?
Manually creating a LP leads to the use of meta cognitive learning strategies. Participants indicated
that the LG-tool evoked more self-directed learning process than the CG-tool. They indicated that the
learning goal was clearer, that their prior knowledge was better activated, and that they had more
control over the learning process with the LG-tool. As these processes are all associated with positive
processes for learning, it would be reasonable to expect that the actual learning outcomes would
improve. However, we did not find such differences in learning outcomes. Prior research shows that
these processes influence learning. Therefore, a possible explanation could be that self-reporting
questions do not correlate with the actual performed learning processes. It is conceivable that learners
did not accurately reported what they actually did.

Main Question: Is learning more effective, when learners manually create a LP, compared to
when they receive an automatically generated LP?
In this evaluation, we wanted to examine effects of active planning on the preparatory and the
performance phase of the learning process. Results of the evaluation show that participant thought
that they had performed more learning activities when they actively created LPs. They reported that
their prior knowledge was better activated, and that the goal of learning was clearer. Overall,
participants thought that they had gained more structural and domain knowledge when they actively
created the plans. However, objective knowledge test showed no difference for the two tools, and it
seems that there is some kind of planning illusion. Participants think that planning is positive for
learning, but in fact, there is no difference.
Summarizing, although we found no differences for learning outcomes, participants thought that they
had learned most from actively creating LPs. They indicated that they had more control when they
actively created the LP and that they preferred to have control over the learning process. Furthermore,
participants thought it was useful to have graphical overviews for difficult learning domains, and to
have planning support tools. Moreover, when they were to learn new domain, they would prefer to
actively create a planning with the tool over having the tool creating the planning for them. For
APOSDLE this implies that both options, creating a learning path by the user and creating a learning
path by the system, should be available.
5.1.9
Component evaluation: the relationship tool
The research reported in this section focuses on the contextualized visualization tool or relationship
tool. In this case contextualized visualization means to visualize data in a way that not only one datum
itself is represented but also additional information to this datum. If one visualizes, for example, a task
page
91
structure it might be useful to know experts for the tasks. Thus tasks and experts are simultaneously
shown by the contextualized visualization.
The visualization tool has been evaluated in a user study. This study evaluated the following aspects
of the system:
 How much time does a knowledge worker need to complete his / her tasks with and without a
contextualized visualization tool?

What is the impression of the quality of the visualization tool?

Does the visualization tool meet the requirements?

Are tree-like visualizations suited for process data?

Are the non-technical requirements met?
To evaluate these questions a user study with qualitative as well as with quantitative aspects has been
carried out. For the quantitative aspect, the time has been measured that users need to carry out
certain tasks with and without the visualization tool. In addition, a questionnaire with several multiple
choice questions was used. The questionnaire was based on a standard known as ISO 9241-110.
This standard focuses on ergonomic requirements in human-machine-interaction. Based on this
standard, the questionnaire contained questions related to topics like adequacy for a certain task, self
descriptiveness, meeting certain expectations, fault tolerance, etc. For these topics the questionnaire
contained multiple choice questions. For each question four answers were possible. The answers
followed the schema “not helpful”, “a bit helpful”, “helpful”, and “very helpful”. The concrete
formulations of the possible answers were adapted to the specific questions. The questionnaire also
contained two free text fields allowing to obtain feedback concerning the satisfaction of the users in
order to measure the qualitative aspects.
The user study consisted of three parts:
1. a pre-study for testing the concept of the main evaluation,
2. a main evaluation,
3. and a second, smaller, evaluation for a specific part of the user study.
5.1.9.1
Pre study
The pre study aimed on testing the concept of the main evaluation. Ten persons were asked to
perform three tasks. In all three tasks the participants had to navigate through a net structure and to
select a certain node from this structure. In the first task, the net structure was visualized in a process
based view, in the second task it was visualized in a tree structure and in the third task the relationship
tool was chosen.
The participants were split into two user groups, each group consisting of 5 persons. One group did
the tasks with a contextualized visualization and the other group received a visualization without
context enrichment. Main topic of all tasks was the navigation through a net structure: In the first task
in a process view, in the second task in a tree view, and in the third task with the relationship tool.
Results of the pre study
Time needed to fulfil the tasks:
page
92
Process view
Tree view
Relationship tool
Group 
41,2
27,4
30,2
Group 
34,2
23,2
34
Table 5-8: Average time needed to perform the tasks in seconds
Group  used contextualized visualizations. If one compares group  and group  one can see that
using contextualized visualizations leads to slightly better results for process view and tree view. For
the relationship tool the not contextualized view is a little bit better. This is probably caused by the fact
that two persons, who were working with the non-contextualized view, already knew the relationship
tool. Thus they were able to work faster. If the results from these persons are not taken into account
the contextualized view leads to better results for the relationship tool as well. (Remark: all results are
not significant as the number of users are quite small.)
User satisfaction:
The user satisfaction was surveyed with a questionnaire. Most of the users working with a
contextualized visualization stated that this kind of visualization is useful. They said they would work
with it in the future. Most users needed more explanations for the relationship tool than for the process
view or the tree view. The reason for this might be that most users know process views or tree views.
But they do not know the relationship tool. Therefore they needed explanations for it.
Summary
The pre study gave a first positive trend for contextualization. Thus a second, more comprehensive
user study was carried out.
5.1.9.2
Main evaluation
Setup of the user study
In this part of the evaluation, a user study with a larger user group with a realistic scenario has been
carried out. To find a representative user group, 30 persons were selected. All users have an
academic background (students and PhDs in computer science or similar sciences and other scientific
staff members) to obtain a representative sample for the target group – the knowledge worker in a
software developing, “industrial-scientific” environment.
The participants were asked to carry out particular tasks in two variants of a role play situation: one
variant was a run through the scenario without support of the contextualized visualization. The second
variant was a run with tool support by networked contextualized visualization as represented by the
relationship tool. The starting situation for both variants was the same: The test person was asked to
act like a new employee in a company, who has to do certain tasks:

Task 1: Find the name of a colleague who can help you.

Task 2: Suggest a colleague who can do a publication. Take into account the workload of your
colleagues.

Task 3: Find the file name of a specific form.
To be able to compare both variants of the role play, similar kinds of tasks in both variants have been
chosen. But the content of the tasks was a little bit different, in order to get different solutions for
variant A and variant B. This eliminated the risk that participants learned from one variant and thus
were able to answer the second variant based on what they learned before.
page
93
We split the group of users into two smaller groups of 15 persons per group. Both user groups did
both variants. In both variants the time to complete the task has been measured (quantitative study)
and the users had to answer a questionnaire, where the users are asked about helpful features and
obstacles of the relationship tool (qualitative part).
We have chosen a cross evaluation that is, a combination of between-groups and within-groups
design. This gave us the advantages from both designs: smaller user groups (from within-groups)
while making sure that the side effects of learning from the first task for the second task were limited or
completely eliminated (from between groups).
For the user groups this means that every participant has to complete the tasks twice – once with our
contextualized visualization system (see Figure 5-10, see also Figure 5-3) and once without. To
eliminate learning effects, the order is changed, so 15 users started with our system (variant A) and
the other 15 without the contextualized visualization system (variant B). In this way we could observe
a learning effect in the measured time, if it occurs.
Figure 5-10: Screenshot of the visualization component
As every user runs through both variants (with and without our system), we change the tasks slightly.
This way, we tried to avoid that a user remembers the right answer from the first run and simply
answers from memory.
To accomplish the user study, a typical workplace of a knowledge worker has been simulated: To work
on the task the user was seated at a table with a PC and some print out material on it. On the PC the
software needed for the evaluation was installed. But only in variant A users were allowed to use the
contextualized visualization system.
To simulate phone calls, users were allowed to talk to the test supervisor in order to tell him / her to
whom they would like to talk to. The supervisor then acted as this person and answered the questions
of the test person. As this is faster than a “real” phone call, a penalty of 10 seconds was added.
(Compared to the time normally needed to reach someone by phone the 10 seconds are quite small:
page
94
Normally you need some time for greeting and explaining, why you are calling. In the user study this
was simulated by the penalty.) Users received descriptions of both variants including the tasks they
had to carry out in each variant.
Figure 5-11: Organization of the main study
Figure 5-11 shows how the main study was organized: In the first part of the user study, every
participant received an oral introduction. Next the participants got time to read the task description.
After this, the order was changed with each run of the study so that one half of the participants started
with version A and the other half with version B. This made it possible to balance the learning effects
that could occur when not changing the order. The user study ended with filling in a questionnaire.
During the execution of variant A and variant B, the time for the execution of both variants was
recorded. To do this measurement an Excel sheet was used. The Excel sheet provided a start button.
When a test person started with a task, he or she pressed the start button. This caused that the actual
time was recorded. The test person then accomplished the task. Each task ended in answering a
question. To do so, the test person had to choose from a list of possible answers presented in the
Excel sheet. For this action, again the actual time was recorded. By calculating the difference between
the start time and the end time, the time needed to perform the task was determined. (This can be
seen in Figure 5-12).
Figure 5-12: Measuring the time
Results of the user study
Speed
Figure 5-13 shows an overview of the time needed for each task with a contextualized visualization
(variant A) and with a visualization without contextualization (variant B). In Table 5- the exact times
can be found.
page
95
Figure 5-13: Time needed for each task (with and without contextualization)
Task 1
Task 2
Task 3
Variant A
57
33
51
Variant B
87
97
39
Table 5-9: Time needed in average to complete a task (with and without contextualization) in seconds
For task 1 and task 2, the contextualized visualization is much faster than the non-contextualized
version. But for task 3, the non-contextualized version is faster. For task 1 and for task 2 visualization
leads to a time reduction as the test persons did not need to ask someone else for help. The system
provided the needed information. Task 3 consisted of finding a file in a file structure. For this task the
non contextualized variant was faster. The reason is that the file structure was clear and therefore the
persons did not need to think of where the needed file might be. Thus contextualization did not offer a
benefit in this situation. As the situation in the test was not realistic, we decided to do a second test
with a realistic scenario. This test is discussed in the next section
Effectiveness
The effectiveness measures the difference between correct answers and answers in total. Most of the
tasks were answered correctly (only 8 from 180 answers were wrong). From these 8 wrong answers 6
were given by persons working with variant B (without contextualization). Figure 5-14 shows the
distribution of correct / incorrect answers between the tasks and variants.
page
96
Figure 5-14: Effectiveness for all tasks
Most of the tasks were done correctly. Thus it can be said, that the test persons were able to perform
the tasks. Only in task 3 in variant A (A3), and in task 2 in variant B (B2) errors occurred. The reason
for the errors in A3 might be found in the bad visibility of the associations in the relationship tool. This
fact was mentioned in the questionnaires and observed in the experiments. The reasons for the errors
in B2 were overhasty decisions based on insufficient information. This lack of information was caused
by not asking colleagues in order to save time. A similar behaviour might be found in reality, when
knowledge workers do not have the time to ask their colleagues due to hard time restrictions. In this
case a contextualized visualization helps to find the needed information.
User satisfaction
A questionnaire allowed the participants of the user study to give personal feedback. In order to find
out how participants experienced the contextualized visualization, the questionnaire contained eleven
questions with multiple-choice answers and two open questions. The questions were based on the
ISO-Norm for ergonomic requirements on human-system-interaction.
For the open questions, most answers were related to the representation of the relationship tool.
There were suggestions like making the topic that is in the actual focus more visible, or make the
associations between topics more visible. (The grey lines between two topics are sometimes
insufficiently visible.)
Aim of the multiple-choice questions was to find out if the system is self explaining, non intrusive,
controllable, fault tolerant, and customizable. For all questions more than 50 % of the participants
gave a positive answer (see Figure 5-15). Thus it can be said, that the relationship tool meets all the
requirements of the ISO norm.
Figure 5-15: Answers in the study
page
97
All participants of the evaluation stated that the system is very helpful or at least helpful to fulfil the
given tasks. In addition they found that contextualized information offers good support. Thus it can be
said that contextualized visualization offers a useful possibility for a task support for knowledge
workers. In addition, 97 % of the participants stated that they can imagine to use tools supporting
contextualized visualization in the future.
5.1.9.3
Second Evaluation
As stated before, the third task in the evaluation was too far removed from reality as it was very
contrived. This can be seen in the time measurement as well as in the answers of the users in the
survey. Thus a second, more realistic scenario was evaluated.
Setup of the second evaluation
The setup of the second evaluation was quite similar to the original setup in order to ensure the
comparability of the data. Fourteen persons participated in the second evaluation. They had to search
for two resources, once with a contextualized visualization and once without. Again a cross evaluation
was done. But in contrast to the first evaluation, a real file structure was provided. The median depth
of this structure had three levels.
In addition, it was taken into account if a participant also took part in the first evaluation or already
knew the visualization tool. If this was not the case, the persons got some time to get used to the
visualization tool. It was also taken into account if the participant already knew the file structure.
Results of the second evaluation
With a realistic file structure, the contextualized visualization (variant A) leads to a much faster result
than variant B without contextualized visualization (compare Table 5-10). Only one person in the
second evaluation was faster without contextualized visualization. But this person stated to be very
familiar with the used file structure.
Time
Variant A
27
Variant B
52
Table 5-10: Average time needed to complete a task (with and without contextualization) in seconds
Thus it can be said that in a realistic scenario, all three tasks can be done much faster and with a
higher user satisfaction if a contextualized visualization is used.
5.1.9.4
Summary
With several user studies we examined if a contextualized visualization supports knowledge workers
in their daily work. We compared a contextualized system with a non contextualized system. The
qualitative as well as the quantitative results of the user studies have shown that a contextualized
visualization can offer very good support for knowledge workers. This can be seen in the shorter time
needed to complete the tasks as well as in the better user satisfaction.
To summarize, it can be said that this kind of support has a clear potential to offer useful support for
knowledge workers. As the results of the user studies have shown, the contextualized visualization
can also lead to faster and better decisions. Furthermore, the evaluation has shown that knowledge
workers stated that contextualized visualization can be useful in their daily work. They would like to
use similar systems in the future.
5.1.10 Component evaluation: user profile and MyExperiences
The APOSDLE User Profile builds the basis for the system‟s adaptation, that is, for the ranking of
Learning Goals and, hence, the recommendation of information within APOSDLE Suggests. In
page
98
addition, the information in the User Profile is necessary for identifying knowledgeable Persons for a
Topic or Learning Goal at hand.
The APOSDLE User Profile is designed as an overlay of the Topics in APOSDLE. In other words, for
each of the Topics, it is decided whether or not the user has knowledge or skills related to it. The
APOSDLE User Profile is automatically maintained applying the approach of knowledge indicating
events (KIE). In a nutshell, different types of naturally occurring actions of users are observed and
inferences are made on the user‟s underlying knowledge level for a certain Topic (see also Section
4.3.3). For each Topic, three levels of expertise are distinguished in APOSDLE‟s third prototype:
Learner, Worker and Supporter (Note, these labels differ from the labels used in previous versions).
For instance, “carrying out a task” is a KIE for the “Worker” level. Thus, whenever a user executes a
task (for example, prepares a creativity workshop) within the APOSDLE environment, the counter of all
Topics related to that Task within his or her User Profile for the “Worker”-level is incremented. At any
point in time, a weighted algorithm in the APOSDLE User Profile Service decides at which level of
expertise a user is located with respect to every Topic in the domain. Users can access the
information in their user profiles through an open learner model called MyExperiences (see Section
5.1.6 and Figure 5-8).
With this component evaluation, the following question should be answered:
Is the information in the APOSDLE User Profile valid? Do the levels of expertise of a user,
as represented in APOSDLE, correspond to the user‟s “true” levels of expertise in each
topic?
5.1.10.1 Method
The participants in our study were 5 employees of ISN who had used MyExperiences for a period of
approximately 2 months (end of September 2009 to mid December 2009) in the course of the
workplace evaluation.
Quantitative and qualitative techniques were applied for answering the research question. For
quantitative estimates of the agreement of the information in the user profile, external values of the
expertise level of a user in each topic were needed. Therefore, each of the participants was provided
with a set of 145 cards, one for each topic in the ISN domain. In a first (self-assessment) trial, the
participant was asked to sort each of the cards (topics) into one of five categories: (i) “I am rather
inexperienced in this topic”, (ii) “I am neither very inexperienced nor very experienced in this topic”, (iii)
“I am very experienced in this topic”, (iv) “This is not my area of work”, and (v) “I do not understand the
topic”. In a second trial (peer assessment), each participant was asked to sort the same cards in those
categories for one of their colleagues. In addition to the five categories in the self-assessment, a sixth
category, (vi) “I cannot decide on the level of expertise of my colleague in this topic”, was available for
peer-assessment. That way, for each participant, we obtained one complete self-assessment and one
complete peer-assessment for each topic in the ISN domain. In addition, qualitative data was
collected. For this purpose, we presented to each participant his or her own visualization of domain
knowledge in MyExperiences. Then, we asked him or her to comment on salient aspects of their own
User Profile, for instance, topics for which MyExperiences represented the right knowledge level, and
topics for which MyExperiences was wrong in their opinion. Further comments which occurred in the
course of this interview were also recorded.
5.1.10.2 Results
Only the contingency coefficient between the knowledge level of each of the five participants as
represented within MyExperiences (Learner, Worker, Supporter) and the self-assessment and the
peer-assessment for the three categories (i) “I am rather inexperienced in this topic”, (ii) “I am neither
very inexperienced nor very experienced in this topic”, and (iii) “I am very experienced in this topic”
shall be reported here. In the following, these three categories of self-and peer-assessment will be
abbreviated as (i) beginner, (ii) advanced, and (iii) expert. In case of agreement between self- and
peer-assessment and assessment by theAPOSDLE User Profile , whenever the self-or peerassessment says beginner, the assessment of the system should be Learner. Whenever the self-or
page
99
peer-assessment says advanced, the system should detect Worker and whenever the self-or peerassessment is expert, the knowledge level in MyExperiences should be Supporter.
P1
For the first participant P1, the contingency of self-assessment and assessment by MyExperiences
was c=.323 (n.s.). While the user, in her self-assessment had mostly used the category “expert”, the
system had detected the category “Worker” in most cases. In 6 out of 12 cases where the user
classifies herself as “expert”, APOSDLE believes she is a “Learner”. Only in one of these topics where
the user sees herself as “beginner”, also the system classifies her as “Learner”. In five out of six cases
where the user assessed herself as “beginner”, APOSDLE detected “Worker” level, the other topic
was classified in accordance with the self-assessment into “Learner”. Interestingly, in all five cases
where APOSDLE classified the user as “Supporter”, the user assessed herself as “expert”.
The situation looks very similar in the peer-assessment of this user; the contingency coefficient of
peer-assessment with the assessment in MyExperiences is c=.28 (n.s.).
When asked about salient aspects of her User Profile as represented in MyExperiences, P1 remarked
that expert topics were detected generally well. For some of the “Supporter” topics, she did not
understand why APOSDLE had classified her as “Supporter”. In general, she found that APOSDLE
“underestimated” her abilities.
P2
For the second participant P2, the contingency coefficient of self-assessment and assessment by
MyExperiences was very low with c=.090 (n.s.). Indeed, the category of “advanced” and “Worker”
respectively were the most frequent categories both in self-assessment and assessment by the
system, but there was no systematic co-occurrence of these categories. When the system said
“Supporter”, all three self assessments, “beginner”, “advanced” and “expert” occurred approximately
equally frequently. Of the 20 topics where the participant assessed herself as “beginner”, the system
classified her as “Worker” in 9 cases. Only for the case of a “Worker”, the user also assessed herself
as “advanced” in 29 out of 68 cases.
Even though the contingency coefficient is higher for peer-assessment (c=.206, n.s), the contingency
table shows that the agreement between peer-assessment and the system‟s decision is even lower
than the agreement with self-assessment: In only 6 out of 23 cases where the system classifies P2 as
a “Supporter”, the peer-assessment also says “expert”. In only 21 out of 62 cases where the system
says “Worker”, the peer-assessment also says “advanced” and in only 6 out of 22 cases where the
system says “Learner”, the user is also seen as a “beginner” in the peer assessment.
The main source of error, both in the comparison of the knowledge level detected by APOSDLE with
self-assessment and peer assessment, is the fact that the most frequent knowledge level in
APOSDLE cases is “Worker”.
P2 explicitly mentioned two topics for which she found it very positive and also surprising that
APOSDLE correctly detected that she was “Supporter”. On the other hand, P2 listed five topics where
APOSDLE had detected the wrong knowledge level according to P2.
P3
Also for the third participant P3, the contingency coefficient of self-assessment and assessment by
MyExperiences was very low with c=.095 (n.s.). Here, the reason for the low contingency is that the
participant assessed himself as “expert” for 54 out of 57 topics and as “advanced” in the other three
cases. However, APOSDLE detected “Worker” level in 49 cases and “Learner” level in 8 cases.
Consequently, in 46 cases when P3 said he was “expert”, APOSDLE classified him as “Worker” and in
page
100
8 cases as “Learner”. Agreement between self-assessment and APOSDLE could be observed in all 3
cases where P3 classified himself as “advanced”; there, APOSDLE also detected “Worker” level.
The situation was very similar for peer-assessment which resulted in a contingency coefficient of
c=.136 (n.s.), with the difference that in the peer-assessment P3 was considered to be “advanced” in
14 topics where APOSDLE also classified P3 as “Worker”.
Looking at MyExperiences, P3 found it salient that some topics in “Worker” level were correct.
P4
The contingency coefficient of P4‟s self-assessment with the assessment in MyExperiences was
c=.338 (n.s.). In 12 out of 14 cases where APOSDLE classified P4 as “Supporter”, also P4 regarded
himself as “expert”. For knowledge levels in the other two categories, the agreement between
APOSDLE and P4 was rather low.
The contingency coefficient for the peer-assessment of P4 was lower (c=.201, n.s.). While APOSDLE
had detected “Worker” level for 38 out of 59 topics, the peer considered P4 as an “expert” in 52 of the
topics. In addition, P4 was not a “beginner” in any of the topics according to the peer-assessment but
APOSDLE classified P4 as a “Learner” in 7 topics.
P4 stated that he had corrected some of the “wrong“ entries in MyExperiences. One thing which he
found salient was that APOSDLE considered him as “Learner” in one topic where he considered
himself as “expert”.
P5
The comparison of P5‟s self-assessment with assessment by APOSDLE resulted in a low contingency
coefficient of c=.125 (n.s.). Again, the category “Worker” was overrepresented in the assessment by
APOSDLE: the system classified P5 as “Worker” in 47 out of 58 topics while P5 considered herself to
be “advanced” only in 14 cases. In 30 out of 47 cases where the system had detected “Worker” level,
P5 considered herself as “expert”. The second aspect, which is salient in the contingency table, is the
fact that APOSDLE classified P5 as “Learner” in 6 topics where P5 said she was an “expert”, and in 2
topics where P5 considered herself as “advanced”. Only in 1 topic both APOSDLE and P5 agreed that
she was “beginner/Learner”.
The agreement of the information in P5‟s User Profile is slightly higher for peer-assessment (c=.254,
n.s.). In 2 out of 3 cases where the system classifies P5 as “Supporter”, also the peer-assessment
says “expert” and in 4 out of 9 cases where APOSDLE says “Learner”, also the peer-assessment says
“beginner”. However, due to the overrepresentation of the “Worker” level in the system‟s assessment
of P5‟s knowledge levels, low contingency results for these topics where APOSDLE detected “Worker”
level: Out of these 47 topics, the peer-assessment classifies P5 as “expert” in 13 cases and as
“beginner” in 18 cases.
Looking at MyExperiences, P5 found that “Learner” levels were classified quite well but that
APOSDLE was not able to correctly classify “Supporter” levels for her.
5.1.10.3 Conclusion
The aim of our component evaluation was to investigate validity of the information in the APOSDLE
User Profile as presented to the users in MyExperiences. Even though we carried out a simulation
study, see section 4.3.3, the component evaluation of the User Profile and MyExperiences was our
first study where the underlying algorithms were tested in a practical setting. The outcomes of our
study will be used to further improve MyExperiences and the APOSDLE User Profile.
Our evaluation study with five users showed that there is a low contingency between the detected
knowledge levels in the user profile with self- and peer-assessment. What were the reasons for this
low contingency? First of all, it became obvious that the category “Worker” is generally
page
101
overrepresented in the automatic assessment of knowledge levels. This is a characteristic of the
underlying inference mechanism: Whenever the algorithm cannot decide between two knowledge
levels (for example, between “Worker” and “Supporter”, or between “Learner” and “Supporter”), it
decides for “Worker”. This may be re-considered. Another reason for the overrepresentation of the
“Worker” level is due to the fact that “Performing a task” is a KIE for “Worker”-level for all topics related
to that task. In other words, if a user clicks on a task, the counter of KIE voting for “Worker” for each
topic related to the task will be incremented. This may be problematic, if a task has a large number of
learning goals (and topics respectively).
One finding which seems contradictory at first glance is that some users found that “expert” topics
were very well classified into “Supporter” level, while others said that “expert” topics were not identified
at all. The reason for this finding is that some of the “experts” were involved in the process of
annotating documents, which is a “Supporter”-KIE. Some others, like P3, did not create annotations at
all for several reasons. As “editing an annotation”, besides “being contacted as an expert” (which did
not work correctly at ISN), is the only “Supporter”-KIE, it is difficult for a person to be classified as a
“Supporter” in a topic, without creating one or several annotations for this topic. Additional ways will be
needed to detect “Supporter” level.
In all, for APOSDLE it is easier to detect “Learner” and “Worker” levels because more KIE exist for
these levels than for “Supporter” level. However, one may argue that a conservative approach is
preferable anyway: in the context of a learning system, it may be better to underestimate a person‟s
knowledge than overestimating it.
5.2 Supporting cooperation processes
The utilisation of knowledge for daily work purposes is one of the main characteristics of knowledge
workers. They collect, apply and create knowledge in terms of information or data sources. In the
same context they also transfer knowledge implicitly or explicitly in cooperative processes. APOSDLE
addresses this by focusing not only on the individual work and learning situation, but also by covering
cooperative ways of working and learning.
Cooperation support within APOSDLE focuses specifically on communication and cooperation support
in learning situations. That is, it is not intended to support the entire palette of possible user to user
interactions, but to investigate which interactions are especially important during a learner-expert
exchange.
This section uses an activity model to illustrate the different possible cooperative activities within an
APOSDLE community. Depending on the type of interaction desired, relevant channel selection needs
to take place. The APOSDLE approach is to first help the user choose the right person to cooperate
with, and then to provide the user with a choice of different cooperation channels depending on
several aspects: availability of the person to be contacted, preferences of the person to be contacted,
properties of the learning situation to be supported, etc.
Another important APOSDLE approach to cooperation is the contextualization of cooperation
channels. Instead of simply offering the channels, APOSDLE aims to initialize them by conveying the
context of the learner to the person being contacted (peer or knowledgeable person). This includes
providing information about the current work task being executed by the learner, the relevant
resources visited by the learner, relevant documents accessed and open, etc.
In order to enable other users to learn also from an interaction between one specific learner and
knowledgeable person, the documentation and publishing of cooperation outcomes is part of the
overall conceptual approach.
5.2.1
Challenges
Historically, the most effective way to collaborate is in face-to-face situations when all participating
individuals are able to use all verbal, mimic and gestural communication channels. This observation
page
102
was also made by the APOSDLE workplace learning study (de Hoog et al., 2006) in which workers
from different backgrounds were asked to provide information about their organisational work and
learn situation. Replacing face-to-face communication by electronic forms may lead to a loss or
misinterpretation of information because these media are not as rich (see for example, Daft & Lengel,
1986). The major challenge we face is providing sufficiently rich channels for communication that
enable the sharing of work contexts between communicators in order to improve learning at work.
The second challenge derives from the APOSDLE specific approach towards learning. Learning is not
only considered as self directed and task driven, but also by the utilisation of interpersonal information
exchange through cooperation. Here the APOSDLE workplace learning study (de Hoog et al., 2006)
has revealed that 70% percent of 146 interviewed knowledge workers prefer an interpersonal
information exchange. Thus, the APOSDLE learning strategy can be summarised as “learning by
doing” and “learning by social interaction”. This means that it should also be possible to share learning
materials to enable other people to give meaningful feedback to the learner and to stimulate reflective
thought.
rd
The third challenge which is specifically addressed with the 3 APOSDLE prototype, is addressing the
requirements of APOSDLE as a socio-technical system in which people interact with material
resources or with other people. They require up-to-the-minute knowledge about their virtual workplace
environment and its information (resources) as well as about their colleagues (Bardram & Hansen,
2004) in order to allow:
 the alignment of own activities with activities of others in cooperative processes,
 the reduction of communication overhead needed to communicate individual cooperation
contexts and situations, and
 the reduction of unplanned interruptions and context switching in cooperative working.
APOSDLE should support knowledge workers here to keep them aware of their workplace and
individuals around.
While most information which is used and produced in cooperation processes is bound to individuals,
dealing with personal privacy will become a further challenge in APOSDLE, especially balancing
between using personal data for cooperation purposes and hiding it for privacy reasons (this will not
be elaborated here, but in a separate deliverable).
The deliverable DII.8& DIII.11 APOSDLE Approach to self-directed work-integrated learning describes
the major concepts of contextualized cooperation. The following section focus on implementations
specific aspects.
5.2.2
Cooperative Activities in APOSDLE
The APOSDLE approach regarding the support of cooperation processes focuses on communication
and cooperation as major activities in collaborative knowledge worker communities. In addition to
socialising effects, the sharing of knowledge for work or learning purposes plays a crucial role for the
building and performance of such cooperative communities.
page
103
Figure 5-16: APOSDLE activity model for cooperative communities
Figure 5-16 shows an activity model describing the interaction between members of cooperative
communities – the knowledge workers - as well as knowledge artefacts involved:

Knowledge workers create knowledge artefacts by making explicit their tacit knowledge about
facts, concepts and processes related to work situations, for example, using documentation
techniques.

Knowledge workers use knowledge artefacts for work or learning purposes by applying the
obtained information.

Knowledge workers rate the subjective and objective quality of knowledge artefacts with
respect to their individual experiences.

Knowledge workers comment on knowledge artefacts to add extra information, they discuss or
reflect on their content or simply give feedback.

Knowledge workers annotate knowledge artefacts to classify it as a piece of information with
relations to other pieces.

Knowledge workers discuss about topics which are relevant to the community‟s work and
learning situations.

Knowledge workers communicate with other knowledge workers about work or learning
related topics as well as about completely unrelated topics.

Knowledge workers collaborate in a certain context to share, apply and create knowledge
together.

Knowledge workers rate the subjective and objective competence of other knowledge workers
with respect to their individual experiences.

Knowledge workers annotate other knowledge workers to classify their competences.
rd
For the 3 APOSDLE prototype, the complete activity model will not be taken into consideration. The
focus will be on supporting communication, cooperation, and commenting as well as rating activities.
page
104
Additionally the creation of knowledge in cooperative processes is a major goal of the 3
prototype.
5.2.3
rd
APOSDLE
Contextualized Cooperation Framework
The APOSDLE contextualized cooperation framework is a technical concept transforming the
contextualized cooperation model into a software implementation. It is generic, based on existing
software applications and services and may be extended with new functionality at any time.
The following sections give an overview of the contextualized cooperation framework and especially
on how key concepts of the contextualized cooperation model can be transferred into a technical
solution.
The APOSDLE contextualized cooperation framework is based on a generic software framework
which can be extended to integrate more or different tools for communication, cooperation or
coordination purposes as well as for cooperative authoring functionality.
The contextualized cooperation framework contains components which are responsible for:

Providing cooperation services as well as authoring services depending on the cooperation
context and cooperation objects by plugging in appropriate software tools,

Performing, logging and documenting cooperation activities as well as authoring/re-authoring
activities,

Processing cooperation transcripts, summarizing the outcome of cooperation activities and
publishing them into the organizational cooperation space,

Producing and ensuring awareness for cooperation and cooperative authoring/re-authoring
activities.
The APOSDLE contextualized cooperation framework aims at adapting or enhancing existing tools to
provide cooperation as well as authoring/re-authoring functionality. These tools are selected based on
application partner requirements on one side and suitability of existing tools on the other side.
5.2.3.1
Cooperation components
The APOSDLE contextualized cooperation framework consists of three main architectural parts:

Awareness Service
This component implements the backend functionalities to support the broadcasting and
monitoring of awareness knowledge within the APOSDLE workspace.

Awareness Client
This component provides visual awareness elements which display contextual information
about items of the APOSDLE workspace and co-workers.

Cooperation Service
This component implements all backend functionalities needed to provide contextualized
cooperation features, cooperation spaces and standard storage, logging as well as retrieval
functionalities.

Cooperation Client
This component provides the graphical user interface to the knowledge worker and allows for
providing cooperation awareness and context information, requesting and performing
cooperation as well as cooperation guidance.
page
105

External Services and Features
This component bundles a set of externally available software applications and technologies
which are utilized by the APOSDLE contextualized cooperation framework. They provide
organizational as well as technical access points to work integrated cooperation and learning.
A detailed explanation of these three architectural main parts of the APOSDLE contextualized
cooperation framework can be found in Deliverable DIV.5 Software Architecture Version 3.
Furthermore, this explains how the contextualized cooperation framework integrates with other
components of the APOSDLE systems implementation.
5.2.3.2
Cooperation features
The APOSDLE contextualized cooperation framework provides functionality to the individual
knowledge worker based on existing cooperation features available at the workplace and on
APOSDLE specific additions. As far as possible, we try to integrate and re-use already present
functionality to increase the knowledge worker‟s acceptance and familiarity as well as work process
integration. Nevertheless, it has to be acknowledged that not any technology can be integrated and
adopted by the APOSDLE contextualized cooperation framework due to technical barriers and effort
limitations.
We categorize cooperation features depending on their relation to the APOSDLE contextualized
cooperation model. The following sections describe how the contextualized cooperation framework
rd
provides cooperation features which are planned to be delivered with the APOSDLE 3 Prototype.
5.2.3.2.1
Cooperation spaces
In 5.2.2 we pointed out the importance of personal as well as organizational spaces in which
cooperation is managed and performed. These spaces are needed to support the management of tacit
as well as explicit knowledge which is related to or created through cooperation on personal or
organizational level.
In APOSDLE three different types of cooperation spaces are provided:

The organizational cooperation space is represented by the APOSDLE Main Application
and the APOSDLE Resource Viewer which provide explicit information about cooperation
entities (knowledge artefacts, people, topics, etc.) that can be internalized by knowledge
workers and used for cooperation. Additionally, the organizational authoring space provides
the functionality for externalizing cooperation knowledge through cooperative authoring
processes.

The personal cooperation space is represented by the My APOSDLE component which
offers functionality to externalize and internalize cooperation knowledge to the individual
knowledge worker.

The temporary cooperation space can be seen as intermediate between both other spaces.
It is available to cooperating knowledge workers and provides the functionality to personally
externalize and internalize knowledge during cooperation.
The APOSDLE contextualized cooperation framework implements or integrates these three
nd
rd
cooperation space types. Below their primary functions and implementation in the 2 and 3
APOSDLE Prototype are presented.
APOSDLE Main Application and APOSDLE Resource Viewer
The APOSDLE main application represents the organizationally shared workspace of knowledge
nd
workers and thus it can represent the entities of shared cooperation spaces. In the 2 APOSDLE
Prototype it was used to:
page
106

Display a recommended set of resources (documents, learning events and knowledgeable
persons) depending on the automatically detected or manually adjusted work task of a
knowledge worker, and to

Search for resources (documents, learning events and knowledgeable persons) without
dependence to the current context.
rd
In the 3 Prototype this component is enhanced with a functionality to browse for resources (see
Figure 5-17).
Figure 5-17: Mock-up of APOSDLE Main Application with browse functionality in the 3rd Prototype
Figure 5-17 shows a diagrammatic overview of the work-learn domain with topics and tasks (see also
Figure 5-3). This provides another way of viewing and accessing organisational resources.
Another part of the organizational cooperation space is represented by the APOSDLE Reader (see
Figure 5-5) which displays detailed information about certain resources as well as enables their
annotation by knowledge workers. This functionality is also used for cooperation purposes.
The APOSDLE contextualized cooperation framework integrates with both essential components of
the organizational cooperation space.
The organizational cooperation space and the included cooperative authoring space
This component is part of a virtual cooperation space of the organization. Here the functionality is
provided to manage externalization of cooperation outcomes through cooperative information
authoring. Authoring refers to cooperative reflection and reviewing activities which are carried out in
the post-cooperation phase by knowledge seeker and knowledgeable persons.
The aim is to provide a persistent space for maintaining cooperation outcomes which are made
available to all knowledge workers and free for further reviewing and authoring activities.
nd
In the 2 APOSDLE Prototype there was already a Cooperative Authoring Space available for
authoring reflections and viewing transcripts. With the post-cooperation phase a reflection link was
created in the Cooperative Authoring Space which then could be accessed by knowledge seekers and
knowledgeable persons to reflect on adequate solutions for cooperation requests (see Figure 5-18). In
parallel the transcript information (as far as available) was linked with the reflection stub in order to
page
107
ease the re-use of information pieces from cooperation. Once a reflection was published to the
Cooperative Authoring Space, this information was accessible by all other knowledge workers and
thus available for further peer reviewing and authoring.
Figure 5-18: Newly created reflection part in the APOSDLE Cooperative Authoring Space
The Cooperative Authoring Space was implemented using existing software tools namely a
20
MediaWiki . It was adapted to the special needs of APOSDLE in order to reduce extra workload
through publication activities. One adaptation was the re-design of its graphical user interface to
address the overall APOSDLE look and feel for a common usage experience. In addition, we reduced
the available set of functions to page authoring, retrieval, page history and page discussions.
rd
Since the 3 Prototype Content of the Cooperative Authoring Space will be automatically fed back into
the APOSDLE knowledge artefact storages to enable easy access to its information via APOSDLE
Suggests as well as browse and search functionalities of the APOSDLE client application.
The personal (cooperation) space: My APOSDLE
The My APOSDLE component instantiates a virtual cooperation space for the individual knowledge
worker. Here functionality can be found for configuring personal cooperation settings and preferences,
accessing information about previous, current and upcoming cooperation activities, and explicit
authoring of mental models which can be used in cooperation to work on a common understanding of
terms and relations.
The aim is to provide a persistent space for maintaining all information needed for and produced by
cooperation from the knowledge worker‟s perspective.
nd
Essential parts of the personal cooperation space have already been available in the 2 Prototype,
namely the cooperation settings and history. They were part of the APOSDLE User Profile
Management Tool and were used to control individual cooperation tool settings as well as to access a
history on individual cooperation activities.
rd
In the 3 Prototype a separate space was made available for managing, externalizing and creating
personal knowledge in APOSDLE. This space also inherits the role of a personal cooperation space.
The My APOSDLE Space will include following functionality:

A visualization of the current experience level in relationship to the topics of the domain
model. The visualization will provide overview and detailed information which can be retrieved
20
MediaWiki is an open source software which is also used for hosting the Wikipedia.org network. The software is made
available at http://www.mediawiki.org.
page
108
through interaction. This feature also allows for adjusting expertise levels manually where
appropriate (see Figure 5-19 which is the same as Figure 5-8).
Figure 5-19: Mock-up of My APOSDLE Space with competence levels visualization

A history of the knowledge worker’s activities in APOSDLE including cooperation
activities. This feature allows for accessing cooperation contexts and outcomes after the end
of the contextualized cooperation process.

A schedule of upcoming and planned activities in APOSDLE. Here the knowledge worker
can access information about scheduled cooperation activities as well as resuming paused
cooperation.

A place to collect information pieces (see Figure 5-20) which are available within or outside
APOSDLE. The knowledge worker can create domain model independent collections of items
(resources, people, activities, notes, etc.). These collections can be structured manually by
defining relations between items to reproduce the knowledge worker‟s mental model. Thus
collections can be used in cooperation to communicate and work on a common mental model
of things.
page
109
Figure 5-20: My APOSDLE with collection authoring functionality

A place to manage personal learning paths. Learning paths are used to create a
personalized learning activity within APOSDLE which can be created and accessed from
within the My APOSDLE Space (see also Section 5.1.4.2).
Any functionality provided by the My APOSDLE Space intends to support the management and
externalization of tacit knowledge from the knowledge worker‟s perspective. Once knowledge was
made explicit it can be shared with others using the My APOSDLE Space as starting point.
Temporary cooperation spaces
For any cooperation a temporary cooperation space is created to maintain all required and shared
information which is used during a cooperation‟s lifetime as well as to provide a place for externalizing
and internalizing knowledge during cooperation. After the cooperation is finished, the information is
transferred from temporary cooperation spaces into the Cooperative Authoring Space or My
APOSDLE according to its primary nature and further purpose.
nd
In the 2 Prototype temporary cooperation spaces provided contextual information for cooperation
(see Figure 5-21). The contextual information included resources, knowledgeable persons, task and
learning goal of the knowledge seeker. It was explicitly made available to knowledgeable persons
during cooperation and could be used to access background information about the cooperation
request and its context. After cooperation was finished the information from temporary cooperation
spaces were still present but not further accessible from the Cooperative Authoring Space or
somewhere else.
page
110
Figure 5-21: Temporary cooperation space in APOSDLE 3rd Prototype
rd
In the 3 Prototype we provided a bi-directionally accessible temporary cooperation space, and added
new functionality to:

Enable the addition of extra information to a temporary cooperation space by both
knowledge seeker and knowledgeable persons.

Embed the knowledge seekers virtual workplace into the temporary cooperation space by
using screen sharing technologies.
5.2.3.3
Cooperation contextualization
Contextual information about cooperation helps to improve the anticipation of cooperation situations
as well as contexts as it was worked out in the APOSDLE contextualized cooperation model.
Furthermore, it helps to decide about cooperation features which can be made available for a certain
context. The APOSDLE contextualized cooperation framework addresses contextualization in four
different ways:

By providing contextual awareness information for each cooperation phase from within the
graphical user interface,

By a context dependent recommendation of knowledgeable persons,

By a context dependent recommendation of cooperation media, and

By making cooperation contexts explicit and accessible to all cooperation partners.
The next sections describe more in detail how the APOSDLE contextualized cooperation framework
deals with contextualization in the APOSDLE Prototypes.
page
111
Contextual awareness information
In the literature examples can be found of techniques which present awareness information to the
knowledge worker (Bardram & Hansen, 2004; Gutwin & Greenberg, 2002) from and within the work
space. APOSDLE adopts similar techniques and integrates them with the APOSDLE contextualized
cooperation framework to provide awareness information:

Knowledge worker lists provide information about the presence and identity of members of
the organizational cooperation space. They can be searched, browsed or found in relation to
resources. Knowledge worker lists further can be used to display the availability, action,
location or situation of other knowledge workers. Thus they provide a rich set of information to
maintain knowledge about knowledgeable persons who are related to one‟s own cooperation
context.

Knowledge worker smart annotations provide information about the presence, identity and
authorship of knowledge workers within the organizational cooperation space. They are used
to display the context-dependent relation of knowledge workers to other entities within the
cooperation space, for example, knowledge workers who have contributed to a recommended
resource. Further, knowledge worker smart annotations provide a common access point for
interactions with the knowledge worker. They display actions that can be used to cooperate
with the knowledge worker in addition to her availability, action, location or situation.

Context widgets provide information about resources (knowledge artefacts, notes,
collections, people, etc.) that relate to a knowledge worker‟s current context. This information
includes the identity, location of and possible interactions with resources.
nd
In the 2 Prototype context widgets were used to provide contextualized awareness information. The
system used automatic task detection (Lokaiczyk, Godehardt, Faatz, Goertz, Kienle, & Wessner et al.,
2007; see also Section 4.1) to estimate the current context of a knowledge worker and to display
knowledgeable persons as well as resources (documents, learning events) which are related to the
detected context (see Figure 5-22).
Figure 5-22: Knowledgeable person context widget in the 2nd Prototype
rd
In the 3 Prototype the context widgets are replaced by more general context containers which
present the same information but provide a different display mechanism. We use knowledge worker
lists and smart annotations to provide cooperation awareness information depending on the current
cooperation context of the knowledge worker and his location in the cooperation space.
page
112
Recommendation of knowledgeable persons
Before cooperation actually takes place, knowledge workers may ask themselves a question like “Who
of my colleagues could help me with the work I‟m currently doing?” To support knowledge workers in
answering questions like this, recommendation of cooperation partners is the first step of our sociotechnical support. In APOSDLE recommendations are based on the work context learning model (Ley
et al., 2008b; see also Section 3.3) that integrates roles with each of the spaces of the 3spaces model.
It thus comprises a knowledge worker‟s context within the APOSDLE system. The aim is now to
recommend for a knowledge worker (that is, a knowledge seeker) currently adopting the worker or
learner role a list of knowledgeable persons (that is, possible cooperation partner) adopting the expert
role. To find them, APOSDLE maintains user profiles (see Section 4.2) based on a multi-layered
overlay user model whereas three of these layers correspond to the three spaces contained in the
work context learning model.
The work layer of the user model is the most important one for recommendation of knowledgeable
persons. It is backed by the work space and the process package. It describes all tasks and work
processes a knowledge worker has done in the past or is currently doing. The work layer of the user
profile is continuously updated with usage data as the knowledge worker interacts with the system.
Figure 5-23: Overview of the APOSDLE Recommender System
Besides usage data about task executions, additional information about which competencies are
needed to perform a certain task are retrieved. This is based on the mapping between tasks and
competencies required to perform them. This mapping allows for running a learning need analysis
which is located at the core of the recommender system shown in Figure 5-23 and analyzes which of
the competencies a knowledge worker already has and which ones are missing to perform a task at
hand. The learning need analysis therefore consists of two main steps:

First step: calculating the current knowledge state by creating a competency profile based on
summing up competencies gathered by task executions collected in the user profile.

Second step: inference of the learning need based on the gap resulting from the difference
between the competencies needed for a certain task and the competency profile calculated in
the first step.
The recommender system uses this analysis as its foundation to calculate a knowledge seeker‟s
learning need as well as to find persons whose knowledge state could cover it. All found persons are
then ranked based on how often they have gained competencies covering the learning need. At the
end of this process a ranked list of knowledgeable persons has been generated which then will be
presented to the knowledge seeker.
page
113
Recommendation of cooperation media
In the first version of this deliverable, we gave a brief overview on different media selection theories
and approaches. All these theories try to describe the user behaviour regarding media selection. It
was shown that all these theories are not proven empirically.
For the APOSDLE contextualized cooperation framework we designed a media selection model that
meets our requirements and combines the strengths of different media selection theories like the
technology acceptance model, the channel expansion theory, the critical mass theory, the social
influence theory or social interaction perspective. In Figure 5-24 an overview of this model is
presented including selected factors. Our media selection model provides an approach towards
recommending and pre-selecting cooperation media for a certain cooperation context. We have two
types of knowledge workers who influence the media selection, the knowledge seeker as the
cooperation initiator and at least one knowledgeable person who acts as the cooperation partner.
While most approaches consider only the perspective of a knowledge seeker, we want to focus also
on knowledgeable persons who should have an equal say in media selection. Therefore our model
takes into account three elementary influencing factors: knowledge seeker, knowledgeable person and
communities of practice or a collaborative community.
Figure 5-24 Influencing factors of media selection model for intra-organizational cooperation
On the knowledge seeker‟s side we discern the following aspects:

The factor working task contains complexity and information spreading.

The factor cooperation context contains time pressure, time to answer and location.

The factor cooperation experiences reflects the personal perspective and experiences of the
knowledge seeker from earlier cooperation with a knowledgeable person.
The knowledgeable person also has an own cooperation context. In addition, both knowledge seeker
and knowledgeable persons are influenced by their individual media experience. This factor reflects
the personal experiences with cooperation media and tools of each cooperation partner. W e also need
to consider further environmental influences which affect media selection at both ends:
page
114

The social environment media experiences contain the media experiences of the social
context of the community of practice. This could be the media experiences of a team,
department or of other communities.

The company media demand contain media experiences of the company or organization.
Most of the time companies develop rules for media or communication tools from their own
media experiences (for instance in many companies public messaging networks are not
allowed to be used for cooperation).
The objective media properties (richness, feedback time, synchronicity, re-use etc.) have an indirect
influence in the model. They influence the individual media experience.
Our model describes the influence factors which affect recommending and pre-selecting a certain
medium for cooperation. In practice we have to select not only a medium, we have to select also a
cooperation tool which provides the functionality to cooperate through a certain medium. We solved
this problem in APOSDLE by a tool medium categorization. Thus a multi-channel messaging tool like
Skype21 goes into different media categories: text messaging, audio conferencing (phone) and video
conferencing.
We described the factors and the dependencies of our model. However, to implement this model we
need to calculate the influence of each factor f on a certain medium m. This will be done by weighting
the normalized factor‟s influence w and setting it in relation to all other factors. The following formula
(1) describes the general calculation of an appropriateness ranking value r(m) for a certain medium m
out of a range of media M:
(
(1)
The experience-based factors f are derived from the knowledge workers cooperation history, whereas
the non-experience-based factors f are derived from objective media properties. For example, the
factor time pressure is related to the media synchronicity property of a medium. One major problem is
the automatic determination of the influence factor values. In APOSDLE we use a complex model
which models these values.
Beside other factors, our approach focuses in particular on the medium experiences and preferences
of the individual and their social context – for example their community of practice. Thus it requires an
in depth knowledge about these experiences and preferences to allow a technical meaningful
implementation.
The media experience of knowledge workers can be derived from their cooperation history. However,
without having initial history information our approach would not be able to produce meaningful
recommendations. Therefore we combined this method with a straightforward cooperation profile
based media recommendation taking the cooperation profiles of knowledge seeker as well as
knowledgeable persons into account. In APOSDLE we use such profiles to model the medium specific
preferences of knowledge workers. A cooperation profile contains:

A list of preferred cooperation tools and their preferred usage period together with tool related
data (for example, account information), and

A list of preference-value-pairs that formally describe the personal occurrence of a certain
medium preference (for example, synchronicity weight).
The cooperation profile based media recommendation uses a linear ranking algorithm and calculates
the intersection of preferred and available media22 from the profiles of cooperating knowledge workers
Skype is a widely used software application which provides functionality for internet mediated calls, instant messaging, file transfer,
short message service and video conferencing.
21
22
Available media are derived from existing or provided cooperation tools at the cooperation partners‟ workplaces.
page
115
and of their community of practice. An essential pre-condition is that the cooperation partners have
adapted their own profile to their specific requirements and preferences. Then usage periods and
general preferences (for example, preferred usage of synchronous cooperation or of channels with
high social presence) are part of the calculation process and result in a ranking of available
cooperation tools.
With this combination of cooperation profile and experience based recommendation of cooperation
media, we support the technical aspect of social interactions within a community of practice as well as
increase the efficiency in working with knowledge.
Cooperation contexts
Information about a cooperation context needs to be accessible for all cooperation partners. Therefore
the context elements have to be made explicit and provided to the knowledge workers in each phase
of cooperation. The APOSDLE contextualized cooperation framework uses the temporary cooperation
space to provide access this kind of information.
nd
In the 2 Prototype the temporary cooperation space were used to display information about learning
rd
goal, task and resources (documents, knowledgeable persons) of the knowledge seeker. In the 3
Prototype this approach will be enhanced for each of the resource types: knowledge artefact, topic,
note, collection, learning path, etc. Additionally, the temporary cooperation space is used for
interaction with resources and the knowledge seekers virtual workplace in cooperative manner.
5.2.3.4
Cooperation guidance
The APOSDLE contextualized cooperation model includes an approach for guiding cooperation
partners through the process of cooperation. Therefore scripting cooperation on different levels and
granularities is proposed. The APOSDLE contextualized cooperation framework transfers cooperation
scripts into a technical part of the Cooperation Client – the Wizard. This component provides scripted
support on a macro as well as on a micro level. On the macro level the cooperation partners are
visually guided through the three cooperation phases. On the micro level for each phase of
cooperation, guiding hints as well as scripted interface elements are provided to the knowledge
worker.
Figure 5-25: Cooperation Wizard in pre-cooperation phase with script support,
fading level 1 (left) and 2 (right)
The cooperation request script (see Figure 5-25) is a social script that divides the request into three
prompts that have to be filled by the knowledge seeker in particular order. Furthermore, the knowledge
page
116
seeker receives hints how and with which content to fill the input fields. This sequenced prompts can
support the knowledge seeker in asking a well elaborated, comprehendible and comprehensive
question that the knowledgeable person understands
This script with step by step instructions to remind the knowledge seeker of the steps to formulate a
comprehendible and comprehensive request is either internalized by an experienced user or will be
internalized with increasing expertise. Therefore an approach has been developed to fade out the
script. This can be done automatically by the system when it detects the increasing expertise or by the
knowledge seeker herself during the request phase via the slider. If the knowledge seeker feels more
experienced in asking questions than by the system detected she can reduce the level of script
support.
There are three levels of script support. The first level is using sequenced prompts and is described in
the previous paragraph. The second level of fading the script is non-sequenced and using also four
types of prompts within the hints but in one combined instruction and is providing question starters
integrated into the input field. The knowledge seeker is completely free to use this script support.
Whether the user makes use of the script support when she formulates her request cannot be
checked during this phase. This is a social script and thus describes mainly the “how” of formulating
the request and therefore the system can hardly match the script with the input of the knowledge
seeker.
At the third level of fading the script has nearly no scripts support. There is only one input field that can
be freely filled by the knowledge seeker and he can use the simple hints on the right hand side in
Figure 5-25.
The possibility of the knowledge seeker to adjust the level of script support during the pre-cooperation
phase is also available in the default settings menu of the APOSDLE contextualized cooperation
framework. When the knowledge seeker has adjusted the level, she can immediately start the
cooperation request with her favoured script support.
5.2.3.5
Cooperation media
Computer mediated cooperation needs a medium that transfers and stores information until it is
accessed by the cooperation partners especially when cooperation takes place asynchronously. The
APOSDLE contextualized cooperation framework provides cooperation based on available
cooperation services and tools. Below an overview of relevant cooperation services and tools which
can be plugged into the APOSDLE contextualized cooperation framework is given.
Tools and services for asynchronous cooperation
Tools and services for asynchronous cooperation may support work-integrated cooperative learning
by people co-located at different times or being remote at different times. The following tools and
services can be integrated into the APOSDLE contextualized cooperation framework:

E-mail
E-mail is a store-and-forward method of cooperation via connected computers. It allows for
multilateral cooperation whereas text and images are used as content formats and any other
document format may be used as attachments. Cooperation transcripts (generated using the
forward option) may be stored as text files or in proprietary formats.
Tools and services: Outlook, Thunderbird, Google Mail, Yahoo!, Windows Live Hotmail etc.

Electronic mailing list
An electronic mailing list is a specialized use of e-mail for widespread distribution of information. It allows for multilateral cooperation whereas text and images may be used as content
formats and any other document format may be used as attachment. Cooperation transcripts
(generated using the forward option) may be stored as text files or in proprietary formats.
page
117
Tools and services: Yahoo! Groups, MSN Groups, Innercircle.cc, Google Groups,
Freelists.org, Jiglu.com etc.

Internet forum
An internet forum is a web application for holding discussions. It allows for multilateral cooperation whereas text and images may be used as content formats. Cooperation transcripts may
be stored as text files or database entries.
Tools and services: MetaForum, bbPress, phpBB, vBulletin, wtcBB, InvisionFree, ZetaBoards,
HomepageModules etc.

Wiki
A Wiki is a web application facilitating easy creation, editing and linking of web pages. It allows
for multilateral cooperation whereas text and images as well as documents may be used as
content formats. Cooperation transcripts are available as a revision history.
Tools and services: Pbwiki, MediaWiki, Twiki, ScrewTurn, Clearspace, MindTouch, TikiWiki,
Xwiki as well as the personal wikis Zim, ZuluPad, Luminotes, wikiPad etc.
The specific collection of tools and services for asynchronous cooperation integrated into an
organization tailored version of the APOSDLE system has to be configured depending on this
organization and its social-technical parameters.
Tools and services for synchronous cooperation
Tools and services for synchronous cooperation may support work-integrated cooperative learning
remotely and at the same time. The following tools and services can be integrated into the APOSDLE
contextualized cooperation framework.

Instant messaging
Instant messaging is a peer-to-peer form of synchronous cooperation based on typed text. It
allows for multilateral cooperation. It is possible to save cooperation transcripts as text.
Tools and services (most popular): AOL Instant Messenger, ICQ, Windows Live Messenger,
Yahoo! Messenger, Google Talk, climm, Lisq, aMSN, emesene, Psi, Mcabber, Coccinella,
Exodus, Kopete, Miranda, eBuddy, SamePlace etc. etc.

Chat
A chat room is a client/server application for synchronous mass cooperation. The range of
available technology goes from text-based, real-time online chat to fully immersive graphical
social environments. Chat allows for multilateral cooperation whereas text, audio and/or video
may be used as content formats. It is possible to log cooperation transcripts on the client side
as well as on the server side.
Tools and services: Bersirc, ChatZilla, jIRC, KVIrc, leafChat, mIRC, Visual IRC, AbsoluteChat.Com, AKIVA Software, ChatSpace Community, ParaChat, Visual Chat, ByteCam etc.

Social networking
Virtual networking software is a web application for synchronous (and asynchronous) mass
cooperation. The range of available technology goes from text-based to immersive graphical
environments. Social networking allows for multilateral cooperation whereas text, audio and/or
video may be used as content formats.
Tools and services: Facebook, XING, LinkedIn, Twitter, Jaiku, Passado, Orkut, Active worlds,
Second life, Uni-Verse
page
118

Webcast, Podcast
A webcast or podcast is a media file distributed via Internet. It allows for multilateral cooperation whereas text, audio and/or video may be used as content formats. The availability of transcripts depends on the content format.
Tools and services: HelloWorld.com, YouTube, SlideShare, TalkShoe

Web conferencing
Web conferencing is a client/server or peer-to-peer application used to conduct live meetings
or live presentations via Internet. It allows for multilateral cooperation whereas text, audio
and/or video may be used as content formats. The availability of transcripts depends on the
content format.
Tools and services: WebTrain Web Conferencing, Genesys Meeting Center, Adobe Acrobat
Connect, IBM Lotus Sametime, Microsoft NetMeeting, Windows Meeting Space, Microsoft
Office Live Meeting, WebEx, Croquet Project, WebHuddle, Fronter, dimdim as well as
Redianet, Virtual Whiteboard, ACE, Shared Page, MoonEdit, SubEthaEdit, Gobby, CoWord,
Mindmeister etc.

Voice Chat, VoIP
Voice chat is a peer-to-peer application to facilitate audio cooperation via Internet. It allows for
multilateral cooperation whereas audio is used as content format. It is possible to record transcripts by specialized streaming audio recording applications.
Tools and Services: Skype, sipgate, Jabbin, Gizmo Project, minisip, Kcall, WengoPhone,
SJPhone

Data conferencing
Data conferencing is a client/server or peer-to-peer application used to exchange computer
data in real time. It allows for bilateral cooperation with a variety of different content formats.
Transcripts may be generated through copying.
Tools and services: NX technology, RealVNC, rdesktop, Terminal Services, Apple Remote
Desktop, Sun Secure Global Desktop, pcAnywhere, Citrix XenApp
The specific collection of tools and services for asynchronous cooperation integrated into an
organization tailored version of the APOSDLE system has to be configured depending on this
organization and its social-technical parameters.
Cooperation media in the APOSDLE Prototypes
nd
rd
In the 2 and 3 Prototypes a series of asynchronous and synchronous cooperation media was made
available within the APOSDLE contextualized cooperation framework:

Desk Phone
APOSDLE application partner representatives stated the physical telephone to be the most
important cooperation tool after face-to-face cooperation (see the two Workplace learning
studies). As in most organisations a TAPI (Telephony Application Programming Interface) or
CAPI (Common ISDN Application Programming Interface) is not available, there has to be at
least a low level integration of telephones with the APOSDLE contextualized cooperation
framework. Thus the callable phone numbers are also provided through the automatic
calculation and recommendation of external tools.

Email
Cooperation via email is widely used in intra-organizational as well as extra-organizational
cooperation. The APOSDLE contextualized cooperation framework provides integration with
standard email applications like Mozilla Thunderbird, Microsoft Outlook Express and Microsoft
page
119
Outlook. In that way, a cooperation request can be sent by using the knowledge workers‟
native email client application.

Instant Messaging/Chat
Instant messaging is an increasingly popular way to cooperate, especially in organisation with
a mid to higher staff level. The APOSDLE contextualized cooperation framework integrates
Microsoft MSN Messenger and Microsoft Live Messenger, Novell Groupwise Messenger,
Skype as well as instant messaging based on XMPP protocols in general.

Voice/Video Chat
Instant messaging via audio or video based cooperation channels is supported by means of
the Microsoft MSN Messenger and Microsoft Live Messenger as well as Skype.

Cooperative Authoring
The cooperative authoring of information is supported by integrating and supporting access to
a MediaWiki.
23
In addition to the external cooperation tools listed above, the prototype version of ConcertChat was
nd
used in the 2 Prototype for text based messaging, white board and context sharing and is still
integrated into the APOSDLE contextualized cooperation framework.
rd
For the 3 Prototype cooperation tools and services were made more transparent to the user. The
recommendation engine and graphical user interface was re-worked in order to allow for the
recommendation and selection of cooperation features based on media instead of tools and services.
5.2.3.6
Cooperation lifecycle
In the APOSDLE contextualized cooperation model the cooperation process was defined as
terminating activity that runs through three phases: pre-cooperation phase, cooperation phase and
post-cooperation phase. Each phase is divided into process steps. The APOSDLE contextualized
cooperation framework implements the cooperation process by defining states and transitions which
describe the lifecycle of a cooperation session (see Figure 5-26). Each session is created by a
knowledge seeker and terminates to a final state after stopping or cancelling cooperation.
23
ConcertChat was developed in the first project phase to support contextualized collaboration in a rich-client solution. See here
also DIII.2: 1st Prototype APOSDLE – Collaboration Tools.
page
120
PRE-COOPERATION PHASE
REQUESTING
schedule
initiate
cancel
schedule
SCHEDULED
PENDING
initiate
cancel
join
COOPERATION PHASE
reject
pause
PAUSED
RUNNING
cancel
resume
stop
stop
POSTCOOPERATION
PHASE
STOPPED
CANCELED
Figure 5-26: Diagram of cooperation states and transitions in APOSDLE contextualized cooperation
framework
The cooperation lifecycle contains the following states:

REQUESTING
In this state a cooperation request is being prepared by the knowledge seeker. This includes
all activities carried out to define a cooperation request statement and to prepare the content
of its context.

SCHEDULED
In this state cooperation is scheduled either by a knowledge seeker or by a knowledgeable
person to a starting point sometime the future. Until this moment is reached, the cooperation
remains in this state.

PENDING
In this state the cooperation request was already sent to the cooperation partners and is
waiting for their response.

RUNNING
In this state the cooperation is running by performing cooperation with a provided medium.
page
121

PAUSED
In this state the cooperation is paused by one of the cooperation partners and will be resumed
sometime in the future. Until this moment is reached, cooperation remains in this state.

STOPPED
In this state the cooperation phase is finished and the reflection can be carried out by the
cooperation partners. The cooperation lifecycle terminates with this state.

CANCELED
In this state the cooperation is cancelled by all knowledgeable persons or by the knowledge
seeker. The cooperation lifecycle terminates with this state.
Furthermore, the cooperation lifecycle contains the following transitions which cause a change of
states:

initiate
This transition includes the submission of the cooperation request from a knowledge seeker to
knowledgeable persons.

schedule
This transition includes the scheduling of the cooperation request to a time in the future.

cancel
This transition includes the cancelling of the cooperation by all knowledgeable persons or by
the knowledge seeker.

join
This transition includes the commitment of at least one knowledgeable person to answer the
cooperation request.

reject
This transition includes the rejection of knowledgeable persons not wanting to answer the
cooperation request.

pause
This transition includes the interruption of a running cooperation by at least one cooperation
partner for a certain time period.

resume
This transition includes the resumption of a cooperation session after it was paused before.

stop
This transition includes the stopping of a running or paused cooperation by all knowledgeable
persons or by the knowledge seeker.
The cooperation lifecycle above is described from the general systems perspective. Components from
within the system, for example the Cooperation Client or Cooperation Services participate only in parts
of the lifecycle and thus have to be synchronized with state transitions that are caused in other
components. Therefore the APOSDLE system uses a notification mechanism to synchronize
cooperation sessions between Cooperation Clients and Cooperation Services.
5.2.4
Component evaluation: scripted communication
In order to test the theoretical basis of the scripts and their practical value, a component evaluation is
performed. The main research question here focuses on the intended “goals” of the script as
page
122
mentioned in D2.8 & D3.5 “The APOSDLE Approach to Self-directed Work-integrated Learning”
(Lindstaedt et al., 2009) and the work plan. These goals were:
1) that the cooperation script on a macro level guides knowledge seekers and knowledgeable
persons through the overall contextualized cooperation process
2) and that the cooperation scripts on a micro level helps knowledge seekers and knowledgeable
persons to individually use each process phase as efficient as possible for problem solving
and learning.
5.2.4.1
Introduction: main research questions
Based on these goals, the focus of the study is to collect data about 1) the guidance of the
cooperation script to knowledge seekers and knowledgeable persons through the overall
contextualized cooperation process (macro level) and 2) support of knowledge seekers and
knowledgeable persons to individually use each process phase as efficient as possible for problem
solving and learning (micro level). The activities problem solving and learning are reformulated in
acquiring and sharing knowledge, as these terms are used in other deliverables, such as in D2.5
“Workplace Learning study 2”. In addition, these terms are easier to operationalise given the context of
the evaluation.
Point 2 refers to efficiency of process phase usage. However, as measuring efficiency implies that
there is comparison between two options, measuring efficiency is not possible, as users will only use
APOSDLE P3 and no other tool. Therefore, measuring efficiency is not part of the research question.
Instead, we look at the effectiveness of using each process phase for knowledge acquiring and
knowledge sharing.
The main research questions are:
“To what extent does using the script of the APOSDLE Contextualized Cooperation Wizard
contributes to effective guidance of knowledge seekers and knowledgeable persons through the
process phases of cooperation?”
“To what extent does using the script of the APOSDLE Contextualized Cooperation Wizard
contributes to helping knowledge seekers and knowledgeable persons to individually use each
process phase effectively for knowledge acquisition and knowledge sharing.”
5.2.4.2
Measurements
For answering the research questions goals are used to determine if the objectives of the cooperation
scripts are met. In this component evaluation, the goals are derived from the operationalization of the
goals of the cooperation script as mentioned in Lindstaedt et al., 2009. These goals are made more
specific by using indicators at the second operationalization level. The goals and accompanying
indicators are used to answer the research questions. These are the four goals with their
accompanying indicators:
Goal I – Knowledge seekers micro level
The script supports knowledge seekers to individually use each process phase for acquiring and
sharing knowledge.
Indicators


The script is understandable for knowledge seekers.
The script is applicable for knowledge seekers.
page
123




The script supports a knowledge seeker in communicating about his relevant knowledge with
the knowledgeable person.
The script supports a knowledge seeker in communicating about knowledge with the
knowledgeable person in an understandable way.
The script supports a knowledge seeker in achieving his cooperation goal as complete as
possible.
The script supports reflection on the cooperation by the knowledge seeker.
Goal II – Knowledgeable person micro level
The script supports knowledgeable persons to individually use each process phase for acquiring and
sharing knowledge.
Indicators





The script is understandable for knowledgeable persons.
The script is applicable for knowledgeable persons.
The script supports a knowledgeable person in communicating about his knowledge in a
descriptive manner.
The script supports a knowledgeable person in communicating about his knowledge in a for
the knowledge seeker understandable manner.
The script supports a knowledgeable person in communicating about relevant knowledge with
a knowledge seeker.
Goal III –Knowledge seeker macro level
The script guides knowledge seekers through the overall contextualized cooperation process.
Indicators



The different levels of scripted support are sufficient to satisfy knowledge seekers‟ need for
support.
The script can be used by knowledge seekers to guide their overall cooperation process.
The script guides knowledge seekers through the cooperation process in a self-evident way.
Goal IV Knowledgeable person macro level
The script guides knowledgeable persons through the overall contextualized cooperation process.
Indicators



5.2.4.3
The different levels of scripted support are sufficient to satisfy knowledgeable persons‟ need
for support.
The script can be used by knowledgeable persons to guide their overall cooperation process.
The script guides knowledgeable persons through the cooperation process in a self-evident
way.
Method
Two methods were used. Both will be elaborated below.
5.2.4.4
Study 1: evaluation study with students and experts
In this study, data is collected using students in a lab setting. Although they are not employees they
can be considered as knowledge seekers who have to perform a task. Experts can be considered as
knowledgeable persons in the knowledge domain of the task. Two experts participated in the study.
page
124
Eleven students participated; one man and ten women. They were aged between 18 and 22 years old.
They were all in their first year of the bachelor program of Psychology (4 students) or Communication
Science (7 students).
Students and experts participated in pairs (11 pairs); one expert worked with six students and the
other with five. One expert worked from a workplace at home, the other form his workplace at the
office. The students worked in a lab setting where besides the student only the researcher was
present. All participants involved worked with the APOSDLE P3 version of the Contextualized
Cooperation Wizard, see 1.2.3. Each expert had chosen and formulated his own subject area and
student task. The student started by reading an introduction and instruction. The expert also had to
read an introduction and instruction before the first session. Thereafter, the students filled in a short
questionnaire which measured their knowledge of the topic of the task. In this questionnaire, they were
presented with some terms. The students were asked, within the context of the topic of the task, to
what extent they were familiar with these terms. In addition, in the questionnaire afterwards two
questions were asked: if their educational program paid attention to the subject and how much they
learned. Also the experts were asked in the questionnaire afterwards if they had the impression that
the students had learned something. This way it is possible to see if the students had little or much
knowledge of the topic of the task.
After this short questionnaire, the students were introduced to and instructed about the usage of the
Contextualized Cooperation Wizard. The experts also had to perform this introductory task before the
first session. After completing an introductory task, where they discovered the main functions of the
Contextualized Cooperation Wizard, the student performed the task. Students were motivated to
perform well, as the best three students would get a present: a book token. They got 25 minutes to
work on the task. Each task contained two questions. The task of students who had to work with
expert one was formulated as follows: “How does the positive impact of computer-supported
collaborative learning manifests itself? Which problems could occur during computer-supported
collaborative learning?” The task for the students of expert number two was formulates as follows:
“Explain, by making a comparison between the present information dissemination situation in
emergency response and the net centric approach, which problems the net centric approach is a
solution for, and, secondly what are the downsides of the net centric approach to information
distribution in the domain of emergency response.”
The task was thus related to the knowledge domains of the experts: computer supported collaborative
learning and information distribution in emergency response situations. In this setting, the student was
the knowledge seeker who lacks the knowledge to complete the task on his own and can only acquire
knowledge needed to complete the task by contacting an expert by using the Contextualized
Cooperation Wizard. The expert was instructed to share his knowledge via the Contextualized
Cooperation Wizard when the student approached him. The expert was not obliged to use the script,
although the student was instructed that he should use the Cooperation Wizard. When the task was
completed, the student and expert ended the cooperation via the Contextualized Cooperation Wizard.
The student could work out the task on paper or in Word. After completion of the task, a questionnaire
was handed out to the student which asked the student questions about the guidance and support the
scripts offered through the overall contextualized cooperation process. The indicators described in
1.1.1, were translated into questions and statements. The expert only had to fill in this questionnaire
with questions and statements after completing the last session. The experts were also asked some
questions about usage of the tool in case they needed knowledge. The questionnaire of the students
and experts also contained a question that measured the overall attitude towards the wizard. This
question asked if they would want to use the wizard at their (future) workplace.
A short interview with each student was conducted after each session; the experts were only
interviewed after their last session. During the session the researcher sat next to the students and
noted down relevant behavior that showed use of the wizard during the three phases of the
cooperation process. Also, technical problems were noted down.
page
125
5.2.4.5
Study 2: Expert walkthrough
Expert walkthrough is an evaluation method mainly used and grounded in usability research. In this
study however, the focus is on evaluating the concept of scripted communication as used in the
Contextualised Cooperation Wizard, see subsection 1.2.4. This concept of scripted communication
was not implemented in P3 and was only available on paper. This concept of scripted communication
relates to the script as described in Lindstaedt et al. (2009).
The expert evaluation focussed on the aspects mentioned in the research question, that is on his
opinion about the effectiveness for and the effect on the users of the script, and the quality and the
usage of the cooperation scripts for acquiring and sharing knowledge. The expert evaluation was
based on the expert‟s knowledge and experience in the area of technology enhanced learning
environments, CSCW and scripted communication. The data gathered from the expert evaluation is
supplemental to the data gathered in the first study, as it evaluates an alternative script which could
have been implemented.
The expert was introduced to and instructed about the usage of the Contextualized Cooperation
Wizard and its role within APOSDLE. Next the two roles that are addressed in the script, that is
knowledge seeker and knowledgeable person, were explained briefly. The expert was asked to
critically review the scripted communication from first the knowledge seeker‟s and then the
knowledgeable person‟s point of view. During the walkthrough, the expert could take notes about his
experiences. The researcher also observed the expert and took down relevant notes. After the expert
walkthrough, the expert had to judge statements about the Contextualized Cooperation Wizard which
were based on the indicators described in 1.1.1. In addition, the expert was questioned about his
experiences so he could share his thoughts.
5.2.4.6
APOSDLE P3 version of the Contextualized Cooperation Wizard used in Study 1
In this subsection, using shots and explanations, the appearance and functioning of the prototype 3
version of the Contextualised Cooperation Wizard will be illustrated. The knowledge seeker refers to
the student and knowledgeable person refers to the expert.
The knowledge seeker can find the knowledgeable person in the APOSDLE Suggest tab and contact
him via the channel he prefers, in this case Skype chat (see Figure 5-27). The knowledgeable person
is called „Eval Expert” and is available as his icon is green.
page
126
Figure 5-27: Contacting the knowledgeable person via APOSDLE P3 Suggest tab
Figure 5-28 shows the Prepare: stating the request. This is the screen the knowledge seeker sees
when he wants to collaborate with a knowledgeable person. The scripts are embedded in the
Contextualised Cooperation Wizard. One can modify the script level by adjusting it at the bottom line
of the wizard (high, medium, low). There one can find the scripts as hint texts (orange texts) and as
pre-selectable texts left from the input fields on each wizard tab. The pre-selectable texts of the first
two tabs are placed in the text boxes; only the last tab shows how it normally works.
The level of support is high here. With medium support fewer scripts are shown and with low support
no scripts are shown.
page
127
Figure 5-28: Prepare phase: stating the request
The window in Figure 5-29 shows the request as the knowledgeable person can see it in this phase.
Figure 5-29: Recall request tab in Cooperate phase
page
128
Figure 5-30: Cooperate phase (right hand pane is knowledgeable person, left hand pane is student)
Once the request is accepted, the Cooperate phase starts (see Figure 5-30). At the left hand one can
see the screen of the knowledge seeker, at the right hand one can see the screen of the
knowledgeable person. The self-evaluation scripts are shown in the right hand screen. These scripts
are meant to let the user look into his objectives and evaluate this information. The strategy and selfmonitoring scripts are shown in Figure 5-31. They refer to respectively the strategy for information
collection and the monitoring of the achievement of objectives. Again, with medium support fewer
scripts are shown and with low support no scripts are shown. All scripts are the same for the
knowledge seeker and knowledgeable person.
Figure 5-31: Cooperate phase: the strategy scripts (left hand) and self-monitoring scripts (right hand)
page
129
Figure 5-32: Cooperate phase 2: the valuation scripts
Figure 5-32 shows the valuation scripts of the Cooperate phase; here the user can judge the value of
the shared information. In Figure 5-33 the third phase, the Reflect phase, is shown. Here the
knowledge seeker and the knowledgeable person can reflect on their cooperation.
Figure 5-33: Reflect, phase 3
5.2.4.7
Concept of scripted communication used in Study 2
The paper concept of the scripted communication, used in Study 2, was presented in Lindstaedt et al.,
2009. Cooperation begins with the request of a knowledge seeker which has to be stated in a way the
knowledgeable person understands, thus we developed a cooperation request script, see Figure 5-34.
In the request phase knowledge seeker and knowledgeable persons work on collaboratively satisfying
the stated cooperation request. Here the support of the proposed cooperation script varies, depending
on the learning goal type which is connected with the cooperation request. For the learning goal type
page
130
“what” the focus is on remembering and understanding factual knowledge. For the learning goal type
“how” the focus is on applying procedural knowledge. And for learning goal type “why” the focus is on
conceptual learning, about its relations. In Figure 5-35 and Figure 5-36 more details of the scripts are
represented. For each learning goal the learning activities are specified. Learning activities describe
actions useful for attaining a specific learning goal. Of course, these actions are kept general. as they
have to be applicable in different situations. Furthermore, scripted support related to learning actions
is specified.
In the post-cooperation phase, see Figure 5-37, we also provide a script for guiding knowledge seeker
and knowledgeable person through the last steps of reflecting and reviewing the cooperation outcome
from both perspectives.
Activities
Scripted Support
Request
Present the request
Fading Level:
LOW
Guiding Hints
Formulate a question.
Summarize the important constraints.
Describe your problem and not a solution.
Fading Level:
MEDIUM
Guiding Hints
State your question.
Summarize the actual state of your problem. How far have you
already come?
Describe the barriers that prevent you from proceeding.
Formulate your desired goal.
Possible question starters, if you are seeking:
Fading Level:
HIGH

Facts: Can you give me information about ...

Ideas: Do you have an idea of ...

Opinion: Do you think ...

Verification: Do you also think ...
Guiding Hints
State an open and clear question.
Use one of the following question starters to categorize your
question - if you are seeking:

Facts: Can you give me information about ...

Ideas: Do you have an idea of ...

Opinion: Do you think ...
 Verification: Do you also think ...
Adaptation of User Interface
Standard input elements for stating the request are split up into
single input elements for question, present cognition, barriers
and desired goal
Figure 5-34: Cooperation request script for knowledge seeker in pre-cooperation phase with fading levels
page
131
Role
Scripted Support
Knowledgeable person
Retrieve the relevant definition/introduction
and keep in mind the existing knowledge of
your colleague: try to relate the given
definition/introduction to that.
Knowledge seeker
Give an example that fits the definition.
Knowledge seeker
Does your point of interest belong to the
category given in the definition?
Knowledge seeker
Compare the given definition/introduction to
your point of interest and try to identify
Knowledge seeker
Compare the given definition to a definition of
a related topic and try to identify differences
and similarities.
Knowledge seeker
Compare the given example to another
example of the same topic and try to identify
Knowledge seeker
Check whether the elements of the definition
are clear, consistent and unambiguous.
Knowledge seeker
Check whether the information given
introduction is reason to get to know more and
look for more information.
Evaluate the given example/illustration: does it
fit your definition/introduction?
Role
Scripted Support
Knowledgeable person Retrieve the relevant procedure/checklist and keep
in mind the existing knowledge of your colleague:
try to relate the given procedure/checklist to that.
Knowledge seeker
Try to rephrase the presented procedure/checklist
in your own words.
Knowledge seeker
Compare the given procedure to your task and try
to identify differences and similarities.
Knowledge seeker
Compare the given procedure to a procedure of a
related topic and try to identify differences and
similarities.
Knowledge seeker
Compare the given example to another example of
the same topic and try to identify differences and
similarities
Knowledgeable person Demonstrate and carry out (part of) the task.
Knowledge seeker
Give an example how you could do it by carrying
out (part of) the task
Knowledgeable person Evaluate (part of) the task that was carried out:
does it fit with your procedure?
Figure 5-35: Cooperation script sequence for knowledge seeker and knowledgeable person in
cooperation phase and learning goal type what (left) and how (right)
Role
Scripted Support
Knowledgeable
person
Recall (the) relevant
guidelines/constraints/template/checklist and
keep in mind the existing knowledge of your
colleague: try to relate the given
guidelines/constraints/template/checklist to
that.
Knowledge seeker
Give an example of how you will use the
guidelines/constraints/template/checklist into
practice.
Knowledge seeker
Derive guidelines /constraints/ a checklist/a
template for your task from the given
information.
Knowledge seeker
Compare the given
guidelines/constraints/template/checklist to
your task and try to identify differences and
similarities.
Role
Scripted Support
Recall (the) relevant explanation/information
and keep in mind the existing knowledge of
your colleague: try to relate the given
explanation/information to that.
Knowledge seeker
Try to rephrase the presented explanation in
your own words and, if something is unclear,
ask for clarification.
Knowledge seeker
Summarize the main points of the information
presented.
Knowledge seeker
Compare the given explanation/information to
your task and try to identify differences and
Knowledge seeker
similarities.
Explain why, this means, explain the reasons
Knowledge seeker
why or explain more about the logic behind it.
Knowledge seeker
Check if there are any contradictions between
Knowledge seeker
the given information and what you already
knew about the topic.
Knowledgeable
person
Select the most important guideline(s) from the
given guidelines/constraints/template/checklist.
Can your task be structured according to the
presented template?
Check if there are any contradictions in the
given guidelines/constraints/template/checklist.
Evaluate the approach that is presented; is it
appropriate?
Figure 5-36: Cooperation script sequence for knowledge seeker and knowledgeable person during
cooperation and learning goal type why (left) and how to produce (right)
page
132
Role
Activities
Scripted Support (Guiding Hints)
Knowledge
seeker
Reflect
What was the most important information given? Give a
short summary.
Knowledgeable
person
Present a summary
Evaluate
Do you need to know more?
Evaluate own knowledge;
Diagnose if there is still a
lack of knowledge, faults.
Can the found information be used for other tasks too?
Review
Review the provided summary.
Review the summary
Does the summary contain all necessary information? Is
there something to add or comment?
Please rate how happy you are about the learning
outcomes
Figure 5-37: Cooperation script sequence for knowledge seeker and knowledgeable persons in the postcooperation phase
5.2.4.8
Results
Study 1
The first goal is related to script supports knowledge seekers and knowledgeable persons to
individually use each process phase for acquiring and sharing knowledge. The second goal focuses
on the guidance of the script for both knowledge seekers and knowledgeable persons through the
overall contextualized cooperation process. These two goals and their indicators are investigated in
the first study.
Technical problems
Technical problems that occurred during the evaluation sessions were kept track of. The following
problems occurred:

APOSDLE wouldn‟t start. This problem could be solved, but the student had less time to work
on the assignment. Therefore he only answered one of the two questions of the task.

The student had to log in for a second time; the expert seemed to be offline in the APOSDLE
start screen (APOSDLE Suggests) although the expert was online. Several invitations were
sent but none of them seem to arrive. Also the system message that the expert had declined
the invitation was shown a few times. The expert explained to have received some invitations
and that he only got the option to decline the invitations. After these start-up problems things
went better. At the right hand bottom of the screen however, system messages with „expert
available‟ and „expert unavailable kept appearing alternately. The chat kept working however.
This occurred quite a few times.

Skype didn‟t start properly: although the expert was online it showed that he was offline. The
expert however, could see the question and sent an answer. From his side no problems
seemed to occur. The sent answer didn‟t arrive at first: it arrived after 10 minutes and then
Skype showed the expert as being offline.

In several cases (seven) the Contextualized Cooperation Wizard didn't move into the Reflect
phase. There was no warning or system message, so the reason for this failure is unknown.

The system was sometimes slow; windows kept „hanging‟ on the screen.

Also the Contextualized Cooperation Wizard once didn‟t pop-up when the expert was invited.
The Skype tool does pop-up. Fortunately, a second try works so the student can start working
on the assignment.

The start screen of APOSDLE changed; the expert was not listed anymore. After searching,
logging out and logging in and searching again the expert was found. However, the location
was different; he was not visible as a search result for the topic used before.
Even though the above technical problems occurred, the evaluation could be performed but
sometimes not as completely as planned.
page
133
Due to the technical problem, seven students could not see the Reflect phase and two students could
not perform the tasks. However, all students performed the try-out assignment and did see all aspects
of the wizard. Their opinions were therefore also taken into account for the data collection.
Questionnaire and interviews of students
First the results of the questionnaire will be presented and subsequently the results of the interviews.
The results of the questionnaire show that the students had no to little knowledge of the subject of the
task and that their educational program did not pay attention to the subject. The majority of the
students (72.7%) stated to have learned at least something during the evaluation. So there was a
genuine need for support.
The questionnaire started with some questions about the Prepare phase. Some students answered
that they used the selection tabs in this phase; some only for sentence starters (18.2%) and some for
sentence starters and as an inspiration for their request formulation (18.2%). However, most students
did not use the selection tabs (63.6%). If students used the selection tabs, most of these students
(75%) found them helpful for improving the request.
The hint that was given in the Prepare phase, was found useful for guiding their actions by 36.4% of
the students. Hence, most students (63.3%) did not found the hints useful for guiding their actions.
In the Cooperate phase, no students used the selection tabs. Some students (27.3%) found the hint in
the Cooperate phase helpful for guiding their actions, but most students (72.7%) did not agree with
this and found the hint not useful. Two students (18.2%) stated to have made notes in the Reflect
phase. However, this was never observed and it is assumed that these students were confused with
taking notes in the Cooperate phase. The Reflect phase was seen by only four students, due to
technical problems. All four students (100%) however found the hint in this phase useful.
The support given in the Cooperation Wizard is realized a script. This was explained in the
questionnaire to the students. Nine statements about the script had to be judged by the students.
These statements were translations of the indicators of Goal I and Goal III. Table 5-9 shows the mean
judgments, which shows that the script is reasonable understandable with a mean judgment of 6.27.
The mean judgments show that the students did not have strong opinions about all other statements,
as most mean scores vary between 3.6 and 5.2, what refers to disagree till no opinion. However, the
minimum and maximum scores show that the opinions were not univocal and that some students
agreed more with the statements than others.
page
134
Minimum
Maximum
Mean
(scale 1 10) *
Std.
Deviation
1. The script is understandable for me.
2
10
6.27
2.57
2. The script is applicable for me.
2
10
5.27
2.65
3. The script supports me in communicating about
my relevant knowledge with the expert.
1
8
4.09
2.07
4. The script supports me in communicating about
knowledge with the expert in an understandable
way.
1
9
4.00
2.32
5. The script supports me in achieving my
cooperation goal as complete as possible.
1
7
3.64
1.43
6. The script supports reflection on the cooperation
by me.
1
7
4.18
1.60
7. The different levels of scripted support are
sufficient to satisfy my need for support.
1
6
4.45
1.64
8. The script is used by me to guide the overall
cooperation process.
1
7
2.91
1.87
9. The script guides me through the cooperation
process in a self-evident way.
1
7
3.45
2.16
Statements
Table 5-9: The statements for the students (knowledge seekers)
* 1 refers to strongly disagree, 10 refers to strongly agree
The students were also asked if they would use the wizard in a future workplace. Some students
answered that they would use it for contacting someone else for gaining information (18.2%) and
some students indicated that they would use it for both gaining and sharing information (18.2%). Most
students (63.6%) however, would not like to use the tool.
The observation of the students shows that, if a request was stated in the first phase, it contained not
more than a few sentences in one of the text fields. Students that used the sentence starters said
these sentence starters were useful as they reflected their questions or because they need support
with the English language. One student explicitly stated that it was not clear that he had to use the
wizard to describe his request. Other students explained that they could not see the relation between
the task and the usage of the wizard; they did not understand what the added value of its usage would
be. They stated that the hint in the Prepare phase helped them to understand the wizard and how they
should use it. One student stated that the hints only made obvious what was already obvious.
Although none of the students used the selection tabs, it was seen that some students used the text
field in the Cooperate phase for typing down notes or pasting chat texts. One student sees the note
taking option as the big advantage of the wizard. However, only one student wrote down his entire
answer in this text field, the other students eventually ended up with using Word. Most students
started to work out their answer in Word and as soon as they opened Word they ignored the
Contextualised Cooperation Wizard and used only Skype Chat and Word.
Most students explained their low marks on the statements by the fact that they hardly used the wizard
because they were working on answering the assignment. They did contact the expert via the wizard
by sending a request, but after this the communication took place via Skype chat. One student
mentioned that the tool is patronizing and that it should me more focused on guiding via, for example,
page
135
flow charts. One student stated that the script helps in making knowledge sharing more structured,
another student said to understand the idea of the three phases. However, also some students explain
that they find the other tools (Word, Skype chat) faster and that these tools cover their need for
support.
Questionnaire and interviews of experts
The experts hardly used the Contextualised Cooperation Wizard. Results from the questionnaire show
that they were not satisfied with the received request. One expert answered he had a neutral opinion
about the received requests and one expert stated he was not very satisfied.
The hints given in the three phases were found useful by one expert. The selection tabs in the
Cooperate phase were not used by the experts, just like taking notes in the Reflect phase. If the
experts had to contact someone, one expert would use the selection tabs in the Request phase to
formulate his request. The other expert would not use them. Both experts, thus also the one who
would not use the sentence starters in the Request phase, do find them helpful for improving the
request. The hint given in the request phase is found useful for guiding their actions in this phase.
Table 5-10 shows the experts‟ judgement of the eight statements they received about the script. From
Table 5-10 it becomes clear that the experts were positive about the comprehensibility of the script.
They also agree on the support it gives for communicating about their knowledge (statements 3, 4 and
5): they do not find the script supportive. Although one expert finds the script rather applicable, the
other disagrees. They also disagree on three other topics; one expert used the script for guidance
through the overall cooperation process, finds the different levels of scripted support sufficient to
satisfy his need for support and finds the guidance of the script self-evident. The other expert
disagrees with the last two statements and did not use the script for guidance during the overall
Minimum
Maximum
1. The script is understandable for me.
8
10
9.00
2. The script is applicable for me.
3
7
5.00
3. The script supports me in communicating about my knowledge in a
descriptive manner.
2
4
3.00
4. The script supports me in communicating about knowledge in a for the
student understandable manner.
3
4
3.50
5. The script supports me in communicating about relevant knowledge with
a student.
3
3
3.00
6. The different levels of scripted support are sufficient to satisfy my need
for support.
4
8
6.00
7. The script is used by me to guide the overall cooperation process.
4
8
6.00
8. The script guides me through the cooperation process in a self-evident
way.
2
6
4.00
Statements
Mean
(scale 1 10) *
Table 5-10: The statements for the experts (knowledgeable persons)
One expert explained in the interview that he did not use the tool, which explains his low rating of the
statements. One expert mentioned that when using these kinds of tools, interaction between expert
and student is needed. He stressed that you need to have interaction. If a person asks a specific wellformulated question, than he can expect a precise well-formulated answer. Therefore, if the student
formulates no or a vague request, it is difficult to make use of the tool.
page
136
After acceptation of the request one expert couldn‟t find the request. So it was not obvious that this
was also available in the Contextualised Cooperation Wizard window. This expert therefore mentioned
that it would be nice if the request would be static and continuously visible or if it would be visible in
Skype. Also, if the text field of the request was not filled, you saw an empty pop-up window at the right
hand bottom of the screen saying: “Problem:…”.
One expert experienced that almost all students asked a lot of open questions. Most students wanted
to know everything with one question. One expert knows, because of his own background in CSCL,
how important it is to support stating of a good request. One expert stated that he experienced the
wizard as a detour when using Skype chat. The added value of the Contextualized Cooperation
Wizard is not clear to him. One expert said that the support given in the tool, was not designed from
his perspective but more from the students‟ perspective. Despite this, one expert said that the scripts
were easy to understand. One expert that found the hints useful, explained this referred only to the
first time he used the tool. Using it more frequently, makes it annoying. According to one expert the
communication principles are good, but the chat system is too long-winded; it goes against the natural
habitual way of chatting. One expert also said that the script could also be on paper during the session
or in a manual; there was no added value of having it on screen.
One expert stated that during the cooperation it stays unclear if someone is working and what
someone does at a certain time. The experts were doing their daily tasks when they participated in the
study. One expert said that the time aspect is not easy; the students expected that he reacted
immediately but he was also busy with other tasks. In a normal organization this is also the case. So
sometimes he didn‟t notice immediately that a request or chat message was sent. The other expert
noted that answering questions is normally not the major activity, so that is why, in this form, it is
keeping you from your daily task.
Both experts stated that working with two windows is difficult; if the windows would be integrated it
would make life easier. One expert added that the window of the Contextualized Cooperation Wizard
was also large; this was experienced as intrusive for your other activities on the pc.
Summary
There were several technical problems with the Cooperation Wizard, especially at the students‟ side.
Nevertheless, the study could be performed. The results of this first study show that the students did
not actively used the wizard during the cooperation process. Some students found the hints useful for
guiding their actions in the Prepare, Cooperate and Reflect phase. Furthermore, the script was found
to be reasonably understandable. The students did not have strong opinions about all other
statements. However, the opinions were not univocal as some students were more positive than
others. Most students however, would not like to use the tool in their future workplace.
The experts also did not actively used the wizard during the cooperation. The experts were positive
about the comprehensibility of the script. They do not find the script supportive for sharing their
knowledge. On the other statements which are based on the indictors, they do not agree, and so their
opinions differ.
Study 2
General walkthrough of the script
First the statements and opinions of the expert, given during the walkthrough are described.
Having fading levels in the request phase is good. The fading level low is good for people who have
their own strategy. In the request phase there is not much difference between the support given in the
medium fading level and high fading level. Also, fading should be reformulated in fading in or support.
Specifying steps is something that is not present now. With this it is meant to specify for the user: “now
do this”. Now the users are free to use the scripts or not. Making it more visible, saying to them “This is
the process, you should follow these steps”, could be useful.
Scripts can change expectations of someone, what he expects to get from a collaboration partner.
That is a powerful way to foster knowledge sharing and creation.
page
137
Currently the knowledgeable person sees only the scripts that are meant for him and the knowledge
seeker only sees the scripts that are meant for him. The knowledgeable person could also want to
gain more knowledge. The advice would be to make him more aware of what the knowledge seeker
wants to know, so he can anticipate on that. So he can check for himself if he has answered all the
questions, if he has told all the relevant things. Vice versa this is also true; the knowledge seeker
could also benefit from seeing what the knowledgeable person is asked to do.
In addition, this could also benefit the reflect phase. A knowledgeable person could there see if he has
given the information that answers each „hint‟ correctly. He can see if he has met the end goals. This
way there is a better match between question and answer. It also prevents disappointment from the
knowledgeable person‟s side that he cannot adequately help the knowledge seeker. So specifying the
steps reduces uncertainty and tension.
There is also a content dimension. A script is sometimes adapted to a certain domain to make it more
helpful (e.g. what, how, why and how to do). I really like this. However, it would be more helpful if the
knowledge seeker and knowledgeable person were aware of each others‟ mini assignments (the
scripts). This way you create shared knowledge about what to expect from each other, it clarifies and
raises expectations.
The scripts are shown in an un clickable manner. Scripts should be integrated in the chat tool and
should be clickable. Then, if you click on them they appear in the chat tool. That way you could also
divide the two roles: you can only click on the scripts relevant for your role.
An idea is to let the knowledgeable person see what the knowledge seeker writes down (his notes)
based on the scripts. This is a kind of in between mini evaluation in addition to the reflect phase at the
end. This way the knowledgeable person can see if the knowledge seeker has understood it correctly.
Maybe literature about expert – lay person communication could be an interesting area to investigate
more.
What is also remarkable is that there are quite a lot of prompts. There is the risk that they don‟t want to
use it, because they reject the extra work. The idea of a script is to make it clearer so that it can be
done with less effort.
Scripts relate to procedural knowledge, they tell what to do. An interesting question to investigate is
“To what extent do users internalise these scripts”. To what extent do learners learn how to
communicate about knowledge?
Statements
The expert was asked to indicate to what extent he agreed or disagreed with 17 statements about the
script. As can be seen, the expert was reasonably positive about the script (see Table 5-11).
He made some additional comments for a few statements. The script can be made easier to
understand with a graphical overview (statement 1). Also, there is uncertainty about what to cover to
meet lay person understanding (statement 4). The support for knowledgeable persons could be
improved by making knowledge seeker prompts visible for the knowledgeable person (statement 9).
Also, reflection could be improved by the knowledgeable person, checking whether the knowledge
seeker got it (”protocol”) (statement 11).
page
138
Statements
Judgement
(scale 1- 10) *
1. The script is understandable for knowledge seekers.
7
2. The script is applicable for knowledge seekers.
9
3. The script is understandable for knowledgeable persons.
7
4. The script is applicable for knowledgeable persons.
8
5. The script supports a knowledge seeker in communicating about his
relevant knowledge with the knowledgeable person.
9
6. The script supports a knowledge seeker in communicating about
knowledge with the knowledgeable person in an understandable way.
9
7. The script supports a knowledge seeker in achieving his cooperation goal
as complete as possible.
9
8. The script supports a knowledgeable person in communicating about his
knowledge in a descriptive manner.
8
9. The script supports a knowledgeable person in communicating about his
knowledge in a for the knowledge seeker understandable manner.
7
10. The script supports a knowledgeable person in communicating about
relevant knowledge with a knowledge seeker.
7
11. The script supports reflection on the cooperation by the knowledge seeker.
8
12. The different levels of scripted support are sufficient to satisfy knowledge
seekers‟ need for support.
8
13. The different levels of scripted support are sufficient to satisfy
knowledgeable persons‟ need for support.
7
14. The script can be used by knowledge seekers to guide their overall
8
15. The script can be used by knowledgeable persons to guide their overall
6
16. The script guides knowledge seekers through the cooperation process in a
self-evident way.
8
17. The script guides knowledgeable persons through the cooperation process
in a self-evident way.
7
Table 5-11: Expert ratings
* 1 refers to strongly disagree, 10 refers to strongly agree
Summary
The results of the expert walkthrough show that the expert is rather positive about the concept script.
The expert is positive about several aspects like the fading levels in the Request phase. The expert
also makes some suggestions for improvement like making the knowledgeable person more aware of
what the knowledge seeker wants to know. A remarks is made about the large number of prompt, that
may inhibit usage. The judgement of the statements, which were based on the indicators, indicate that
the expert is reasonably positive about the script.
5.2.4.9
Conclusions
The results of the two studies show that the expert is reasonably positive about the script and that the
goals, based on his opinion, are met. Nevertheless the expert sees room for improvement. For
page
139
example, the knowledge seeker could benefit from seeing what the knowledgeable person is asked to
do or to let the knowledgeable person see what the knowledge seeker writes down.
The first study however, results in a totally different outcome, namely that the two goals are not met.
The students and experts of involved in this study are not so positive and feel that there is no real
benefit from using the Contextualised Cooperation Wizard.
The main research question cannot be answered in an univocal way, because the results of the two
studies are so dissimilar. The effectiveness of the script for acquiring and sharing knowledge is not
proven, as students hardly used the tool. Nevertheless, the expert thinks that users could benefit from
the script when acquiring and sharing knowledge. The participants of study 1 have furthermore not
really formed an opinion about the quality of the scripts due to the absence of active usage during the
study. The expert however is positive about the quality of the script for acquiring and sharing
knowledge.
page
140
6 Knowledge Ar tefact Lifecycle
This chapter describes the lifecycle of knowledge artefacts. First, a document enters the APOSDLE
system as a repository object (for example, during system instantiation, during nightly updates, after
communication events). Next, the entire document or parts thereof are turned into knowledge artefacts
by manually or automatically attaching two types of metadata (annotations): domain concept(s) and
material use(s). Finally, the knowledge artefacts are related to each other and retrieved via an
associative network. First ideas on how user feedback mechanisms on the different steps can be
utilized are included. The individual steps are described below.
Concerning knowledge artefacts, two component evaluations were carried out:
Multi media: does multimedia content improves APOSDLE and in which way can multimedia content
best made accessible?
Retrieval of knowledge artefacts: which semantic similarity measure used for retrieval is the best one
for a specific application domain and can one measure be seen as the default measure across
domains?
6.1 What is an APOSDLE knowledge artefact?
In APOSDLE a knowledge artefact is defined as a document, or part thereof, together with two types
of metadata: the learning domain concept (see Section 3.1) addressed/described within the document
(piece) and the material use type (see Section 5.1.4.1) of the document (piece). Optionally a
knowledge artefact can also have a free text comment associated with it.
6.2 Access to knowledge artefacts
Explicit knowledge in a company, incorporated in documents (reports, articles, manuals, video and so
forth) is often stored in various back-end systems (repositories) of this company. As APOSDLE aims
at taking into account the entire intellectual capital of a company and make it available to the
knowledge workers, this explicit knowledge has to be accessed somehow. Documents retrieved will
further serve as a basis for creating knowledge artefacts, which themselves will be used within
APOSDLE for learning or cooperation.
6.2.1
Challenges
Concerning the access to back-end systems some challenges can be identified that have to do with
unified access.
IT-Systems of modern companies are based on a highly networked infrastructure using different
operating systems with various interconnection and communication protocols. Due to this variety of
back-end systems, a first major challenge is to provide a single point of access for all modules
APOSDLE consists of. Whenever an APOSDLE component wants to retrieve documents, it should not
matter where this documents originally resides. For this purpose, a general interface may be used
which provides homogeneous access to all documents, independent of the underlying infrastructure
the documents are stored in.
Especially in companies with many employees, privacy issues play an important role. There are
various access restrictions based on user or group rights. In APOSDLE another challenge is that
privacy issues concerning document access rights need to be taken into account. The APOSDLE
System may not be allowed to access all documents stored in the back-end systems. This privacy
aspect has to be taken into account, but will be discussed in a separate deliverable.
page
141
6.2.2
Theoretical Background
The theoretical background of our work is based on technologies like GRID computing. Hyatt and
Vrabik (2004) define Grid computing as distributed computing over a network of heterogeneous
resources, enabled by open standards. The special “Information Grid” tries to provide secure access
to any information source (heterogeneous files, databases, and storage systems) regardless of where
it resides. To achieve this, so-called virtualization of data across various back-end systems is used.
With virtualization, the data appear to come from one source even though the data is stored in a
distributed way.
APOSDLE´s implementation of a homogeneous access to various back-end systems was inspired by
the Java Content Repository (JCR). The idea behind the JCR is that with the growing popularity of
content management applications, the need for a common, standardized API for content repositories
has become apparent. JCR tries to provide such an API by means of having an interface that can be
layered on top of a wide variety of existing content repositories. The JCR architecture consists of a socalled repository model holding different workspaces. Each workspace consists of nodes and
properties. The homogeneous access approach used in APOSDLE uses somewhat the same
architecture, although it is not exclusively related to content management systems. There is one
interface used by other APOSDLE components (the Data Object Repository) which connects to
various back-end systems. The back-end systems store the documents.
6.2.3
APOSDLE Approach
To guarantee homogeneous access to all documents stored in different repositories, a so-called Data
Object Repository is used. The Data Object Repository provides a general interface and therefore a
single point of access for all other modules and tools within the APOSDLE System (see Figure 6-1).
Documents can be retrieved on the basis of a unique identifier created for each document. Within the
Data Object Repository the document itself is represented by a so-called Repository Object (RO).
Repository 1
(File System)
Repository 2
(Samba)
Repository 3
(Wiki)
Figure 6-1: Data Object Repository within the APOSDLE Platform
Repository Objects can be understood as a document representation enriched with automatically
obtained information in terms of metadata (like mime type or file size). After creating a Repository
page
142
Object based on a document, this RO is stored and made available to the APOSDLE System. The
following issues concerning a Repository Object must be addressed:

Each Repository Object is identified by a unique identifier within the APOSDLE System. One
possible solution for this is to use a Global Unique Identifier (GUID) as proposed in RFC 4122
(RFC4122, 2005).

Based on this unique identifier the corresponding document may be retrieved. The content is
further used, for example, for indexing and classification by the APOSDLE´s Knowledge
Discovery and Extraction Service.

Furthermore, a Repository Object may provide some system-managed, read-only additional
information (metadata) based on some standardization of digital objects like the Dublin Core
Metadata Element Set (Dublin Core). Metadata may consist of Document-Title, the GUID,
mime type, file size and so forth.

Repository Objects underlie an access mechanism, which, in our case, is primarily based on
the availability of the corresponding document within the appropriate repository integrated into
the APOSDLE platform. Documents can be deleted or revoked from the underlying backend
system. The Data Object Repository has to cope with this. Furthermore, access rights for the
APOSDLE System to backend systems have to be taken into account.
We choose this approach of using a Repository Object because the Repository Object represents a
general interface to deal with documents and corresponding metadata within the APOSDLE System.
The Data Object Repository, as one module of the APOSDLE System, handles both the Repository
Objects and the connected repositories. Repository Objects are stored in a database for further
retrieval by other modules. A so-called Repository Manager deals with the repositories used in the
APOSDLE System. Repositories may be located in or outside the APOSDLE Platform. Each
repository is defined by having one out of several repository types and additional information about
how to access the appropriate back-end system. A repository type represents a local or remote File
system, a connection via Samba, an Email Server etc. Technologically, the Product Factory design
pattern (Metsker & Wake, 2006) is applied within the Repository Manager. Using the Repository
Manager, repositories may be managed (activated/deactivated) on-the-fly.
One issue concerning the Data Object Repository is to detect changes in the connected back-end
systems (updated, moved or deleted files), so that the APOSDLE´s Knowledge Discovery and
Extraction Service can re-classify the data and the APOSDLE System is able to present the most
accurate results to the knowledge workers. Determining the changes in the back-end systems is not
done in real-time but on a periodical (daily) basis.
Concerning repository types, the Data Object Repository is build up quite modular. Access to the
rd
different repository types is implemented using 3 Party libraries and APIs using the same overall
interface.
We choose this approach because it has to be flexible in adding additional repository types: each
Company‟s IT-Infrastructure APOSDLE will be used in, will be (very) different. Using database storage
for the Repository Objects is necessary to determine changes. As mentioned, detecting those
changes is performed on a periodical basis. On the one hand because hardly any back-end system
provides a proper and non proprietary notification mechanism, on the other hand document
conversion and re-classification may be a very time-consuming process.
6.2.4
Where else has something like this been used and proven to work
A similar approach of gathering data from back-end systems has been shown in (Aperture 2007).
Aperture is a Java framework including an open source library for extracting full-text content and
metadata from various information systems (for example, file systems, web sites, mail boxes) and the
file formats (for example, documents, images) present in these systems. The Aperture framework is
created in cooperation between the German DFKI institute (http://www.dfki.de) and the Dutch software
page
143
firm Aduna (http://aduna.biz). APOSDLE´s Data Object Repository will partly benefit from Apertures
functionality. Compared to APOSDLE, the framework uses a crawler-based architecture and is not
intended to provide access to single documents. Furthermore it lacks privacy management.
6.2.5
New challenges and ideas to solve them
During the implementation of Prototype 2 and the planning phase of Prototype 3 several new
challenges emerged.
A key idea behind the APOSDLE System is that documents retrieved using the Data Object
Repository are presented to the knowledge workers, based on their actual work context. Knowledge
workers are then able to annotate those documents (respectively parts of them), for example, with
domain concepts. This is the process of creating knowledge artefacts on the basis of the documents
(see also the next Section). Due to the huge amount of different file formats found in the back-end
systems, it is almost impossible to provide such an annotation facility for all those file formats. So the
idea to solve this, presenting a new challenge, is to convert the different files into a format that can be
easily displayed and annotated. Plans for Prototype 3 of APOSDLE are to use PDF as a target format.
For the Data Object Repository, as the underlying infrastructure, this means using different concepts
in delivering the converted documents. Additionally, document formats and standards are subject to
change (see for example, Microsoft Office document formats), so a mechanism must be found that
new document formats upcoming in future can also be easily integrated into APOSDLE.
Furthermore, not only textual documents play an important role to transfer knowledge but also audio
rd
and especially video files are becoming more and more important. These are now accessible in the 3
Prototype, but improvements in terms of annotations and retrieval are still possible.
Finally, in the Application Partner‟s IT-Infrastructures we have to cope with highly secured intranets
containing more or less confidential information. For this reason it will be quite difficult to access this
information for a prototype application like the APOSDLE System, which is not fully integrated into the
Partner‟s IT-Security concepts. Storing access passwords within APOSDLE will not be allowed. For
this purpose we will on one hand deal only with non confidential public information extracted from the
productive systems. On the other hand we try to find solutions to gain access to the back-end systems
without compromising the security architecture. This could for example be done by using some sort of
single-sign-on mechanisms as used in various network architectures (see, for example, Microsoft
2007).
6.2.6
Component evaluation: multi-media
Goal of the study reported in this section was to investigate if a knowledge artefact can also consist of
multimedia like video. The following questions were leading in this study:

Does multimedia content improve a system like APOSDLE?

What are the circumstances to make it successful?

Which type of multimedia fits best?
6.2.6.1
Materials and participants
The main APOSDLE components dealing with multimedia are the Add-Resource Tab in the Main
application and the APOSDLE Reader which can display videos and video lectures in the same way
as PDF documents, which means that Snippets can be created and manipulated using the same GUI
and functionality.
The interesting question is if the users understand the concept and if they like the idea to have one
GUI for textual and multimedia material. Therefore the test persons had to play with example videos
and video lectures and to create a screen cast on their own.
The evaluation contained:
page
144

the functionality of the video player in the APOSDLE Reader

the screen cast – record and ingest functionality of the APOSDLE Add-Resource Tab
Four persons of JRS in Graz that are not involved in the APOSDLE project participated in the study.

Person 1: male, 28 years old, developer / researcher

Person 2: male, 36 years old, IT project manager / researcher

Person 3: female, 30 years old, IT project manager / researcher

Person 4: male, 33 years old, developer / researcher
Materials used in the study consisted of

Current client (not the open source version)

Current Platform with ISN domain and documents

Some additional documents to work on (videos, PowerPoint Presentations for screen casting)

Script for Screen cast (oral introduction for producing and entering)

Survey questions adapted from Shneiderman (2002) Questionnaire for User Interaction
Satisfaction.
6.2.6.2
Results
Below is a summary of the data for each of the survey questions:
1. Characters and signs on the screen: easy to read / difficult to read

4x easy

The feedback for searching is a blurred APOSDLE windows and confuses me
2. Integrated Help System used / not used

4x not used
3. Number of simultaneous displayed information on the screen: adequate / inadequate

4x adequate

Rating of Snippets should be also shown in the theme-river (colour-coded)
4. Order of information (elements) on the screen: logical / illogical

3x logical, 1x illogical

Video overview should be there. Something like a key frame overview.
5. Is the terminology understandable: yes / no

2x yes, 2x no
6. Reaction time of the system: acceptable / unacceptable

4x unacceptable

Feedback during the annotation is not given. Which start and endpoint was chosen?

System reaction when clicking on different snippets in the list is too slow – no
feedback!
page
145

Click on snippet list -> new position of the video takes too long -> feedback required

Video player start / stop with space key or a mouse click in the timeline

Selecting a section of the video via dragging a rectangle in the timeline
7. Can you imagine to create / use video lectures? Yes / no

3x yes, 1x no
8. Is the implementation of one viewer for textual and multimedia content an advantage? yes / no

4x yes

Very good idea to handle videos in that way
Does multimedia content improve a system like APOSDLE?
Each of the four test persons liked the way APOSDLE deals with multimedia and were able to work
with the multimedia tools within a few minutes. As all of them give talks about work related topics from
time to time, they all are happy to have the ability to record their PowerPoint slides from the screen
and their speech and entering it into APOSDLE later on.
The main advantage they see is that within APOSDLE they are able to annotate the content in a very
useful manner that makes them confident that other users will benefit of it.
What are the circumstances to make it successful?

Fast system reaction

Easy way of browsing through the video content

Easy way of selection a section of the video

Not depending on the type of content

Not depending on the extra amount of work for annotating
Which type of multimedia fits best?
Three of the test users preferred the screen cast and enter functionality for their talks. One person
wanted to have many video lectures. Finally, it appeared that all kind of video material fits into the
tested scenarios.
The answers to the prepared questions were not as useful as expected. They all disliked the slow
system reaction of the client / server application. Due to the fact that the system is a prototype, this
problem would be solved by creating a “real world product” of APOSDLE.
Even more useful and relevant was the 15 minutes long face to face talk afterwards.
6.3 Creation of knowledge artefacts
One essential step in the APOSLDE knowledge artefact lifecycle is how knowledge artefacts are
created. One approach in creating knowledge artefacts is to do this in a collaborative fashion, where
users create knowledge artefacts by annotating documents with a domain concept and/or a material
use as described below. Due to the huge amount of knowledge items targeted by APOSDLE, one
single user or a small user group will not be capable to annotate all knowledge artefacts. Therefore,
tools must support a collaborative annotation, so that users can share their knowledge about
knowledge artefacts with each other.
However, manual annotation is a labour intensive task and we do not necessarily expect manual
annotations to provide training data covering all necessary concepts for classification. To overcome
page
146
this natural limitation and to bootstrap an annotation free repository, APOSDLE makes use of linguistic
and machine learning approaches to automatically assign domain concepts and material uses to
knowledge artefacts.
6.3.1
Challenges
As defined, knowledge artefacts are documents or parts thereof enriched with metadata. Following the
APOSDLE concept of the 3spaces concerning knowledge worker, learner and expert they are mainly
used in learning activities and are displayed as usable resources for the knowledge worker (see
Section 5.1). So the basic challenge is to extract parts of data objects, enrich them with specific
metadata and present them to the knowledge worker. This includes providing different possibilities for
adding metadata to integrated data objects.
6.3.2
APOSDLE Approach
In APOSDLE knowledge artefacts are created in two ways. On the one hand knowledge artefacts can
be created manually by annotating documents or media files. On the other hand they can be created
automatically, based on learning from given manually annotated test cases.
6.3.2.1
Manual creation of Knowledge Artefacts
Among others, manually created artefacts serve as bases for further automatic creation of knowledge
artefacts. In APOSDLE the manual creation of artefacts is supported by the usage of annotation
facility in the APOSDLE Reader (see Figure 5-7, this facility is available through the “Snippet” tab).
This facility not only helps the user to enrich parts of integrated documents or media files with specific
metadata (material-use, semantic concept, free notes), but it is also used for presenting knowledge
artefacts and their assigned annotation. Therefore, from a user‟s point of view the annotation facility is
more a support facility for her daily work than a facility for annotating documents. This approach
follows successful Web 2.0 approaches which have shown that annotation has to be embedded in the
knowledge task rather than being a separate tasks. Similar findings were reported by (Cutrell et al.,
2006). Their experiments, conducted at Microsoft Research, revealed, that while users are willing to
annotate documents, the annotation process must be included in tools they use for their work, as done
in the APOSDLE approach.
Concerning documents firstly the Annotation Tool can deal with Portable Document Format (PDF).
Before a user can annotate documents they have to be converted into PDF. The conversion is
performed by the Data Object Repository whenever a new document is integrated into the APOSDLE
system. Secondly, videos encoded in a common format can be used.
The first step in manually annotating documents (files) is to select one specific document by using the
Repository Object Browsing Tool (ROBT). The ROBT is a graphical tool, which lists all available data
objects in the APOSDLE System, independent of the backend system where the data objects are
actually stored.
By selecting the data object in the graphical user interface (GUI) of the ROBT the Annotation Tool is
started. In case of a text document the Annotation Tool retrieves the content of the converted PDF via
a Web Service Request to the APOSDLE Platform. In case of a media file the media file content is
retrieved.
6.3.2.2
Automatic creation of knowledge artefacts
Automatic creation of knowledge artefacts distinguishes between the annotation of material use and
the annotation of domain concepts to document parts. Both approaches utilize supervised learning
techniques which require learning a model from previously annotated documents (for example,
training data). The model is afterwards used for annotating new, previously unseen, artefacts.
From an architectural point of view the functionality is embedded in the APOSDLE architecture via the
Annotation Service. The service interacts with the structure and document repository to load the
page
147
training data and to store the newly created annotations. Functions are triggered via the coordinator
and results are obtained from the Annotation Tool by directly accessing the structure repository. Due
to its loose coupling with other components, the Annotation Service may be of further use in future
scenarios, like annotating web pages on the fly.
The AnnotationService has basically two modes of operation:

training of a classification model for the annotation of documents (material use, domain
concepts) based on manual annotations from knowledge workers using the annotation facility.

automatic classification of documents within the document repository
Both activities are triggered by the “Coordinator” from within the APOSDLE framework. Based on the
mode of operation the AnnotationService performs the following, simplified workflow:

Training: Get all manually annotated knowledge artefacts from the Structure Repository and
train a classification model for each view (material use, domain concept)

Classification: Iterate over all unprocessed documents from the Document Repository. Find
and classify knowledge artefacts within each document and store the results in the Structure
Repository as “automatic annotation”
For both, the Common Access Tool is needed for a normalized view on the documents (PDF), so that
each component (Annotation Tool, Annotation Service) operates on the same input. In the learning
phase, the annotation service fetches a document from the document repository together with
annotated concepts and material uses.
From an algorithmic perspective, the document is pre-processed using standard text mining
approaches like tokenisation, sentence splitting and part of speech tagging. Since parts of documents
should be annotated, the document is split into parts following two strategies. The top down approach
analyses the document structure (for example, headings, number of line breaks etc.) and tries to
identify meaningful blocks. The blocks are annotated afterwards. While exploiting a documents
structure is preferable, structural elements are not recognizable for some document formats. Therefore
a bottom up strategy is applied in addition. This approach starts at the sentence level and merges
succeeding sentences, which are annotated with the same material use or domain concepts. Future
activities also plan to incorporate more sophisticated counting strategies like majority votes or
weighted majority votes.
The annotation of material use(s) in a document will be performed by utilizing inherent semantic and
structural information. For example, a material use of the type “Question” will consist mainly of
sentences which close with a question mark. Such information will be used to train a statistical model
based on the maximum entropy principle, which is currently state-of-the-art for statistical language
processing. Maximum Entropy modelling has been proven to be a valuable tool for a broad range of
language processing tasks (see (Ratnaparkhi, 1998), and therefore has been chosen as the approach
for knowledge artefact labelling.
Assignment of domain concepts can be seen as classical text classification problem of assigning
documents to a set of predefined classes (Sebastiani, 2002). From the accurate and fastest
algorithms, Support Vector Machines (SVM‟s) and k-Nearest-Neighbour algorithms seem to perform
well in most test cases. Since k-Nearest-Neighbour algorithms do not require excessive computing
time and can be update in an iterative manner with lesser effort than Support Vector Machines, we
choose to use k-Nearest-Neighbour algorithms for the assignment of domain concepts. Especially in
the case of dynamic repositories, iterative update is of great importance. Since the given domain
structure is hierarchical in nature, the classifier has been modified to consider hierarchical
relationships to increase accuracy (Granitzer & Auer, 2005 ).
page
148
6.3.2.3
Displaying a knowledge artefact
Of special importance is the visual representation of annotations, which is not only used during the
annotation process itself, but also for displaying relevant document parts with respect to a users
information need. Providing such a visual overview has been demonstrated to greatly enhance the
perception of relevant information while reducing the cognitive burden and information overload
induced by only textual content (Kienreich et. al., 2005).
In general the visualisation is based on the theme river paradigm (Havre et al., 2002): Given a set of
annotations, each occurrence of an annotation is visualised in a side bar where the vertical axis
denotes the position of the occurrence of the annotation in the document and the horizontal axis
denotes it‟s confidence or relevance. Thereby, the beginning of the vertical axis denotes the beginning
of the document and the end of the vertical axis the end of the document respectively. A marker line is
used to show the current position in the document.
The relevance of an annotation thereby comes from two possible sources: one source is the automatic
annotation. In this case the relevance shows the confidence of the automatic annotation tool in
assigning an annotation to a specific document position. A second source for the confidence is the
query a document was opened from. In this case, the relevance shows the importance of an
annotation with respect to the query a document has been opened from. While the latter source is of
higher importance for the knowledge worker, since it immediately shows relevant document parts, the
first sources is helpful in validating the automatic annotation process. In both cases, it is expected that
navigation within large documents will be greatly enhanced by using the adapted theme river
visualisation.
6.3.3
Why did we choose this approach?
In APOSDLE we opted for the integration of manual annotation with automatic annotation because the
difficulty is that manual annotations are in general better than automated ones, but also more
expensive to obtain. Our approach therefore is to motivate users to manually seed-annotate selected
documents and do the rest automatically. Over time, both more manually created knowledge artefacts
will exist and the classifier will be better trained.
Due to the huge amount of available text document formats it is nearly impossible to develop an
Annotation Tool, which supports every single format (concerning, for example, text selection). For this
purpose the conversion to PDF is performed, so that the Annotation Tool has only to deal with PDF.
We use a video player capable of viewing common video formats and that has the ability to be set to a
predefined position to jump directly to the beginning of a knowledge artefact.
6.3.4
6.3.4.1
Defining metadata for annotation
Concerning environments of the different application partners, one challenge is to define the set of
metadata, which fits best to the partner‟s workflow. This is done be analyzing the core processes the
corresponding application partner deals with.
6.3.4.2
Automatic creation of knowledge artefacts
One obvious problem in using machine learning methods for automatic annotation is the number of
training examples required for learning the accurate classification model. Especially if the documents
differ in style and content, a large number of training examples is required. The following solutions
have been foreseen to tackle this challenge:
1. Feature selection and development of different heuristics for annotating material uses. Since
there are only a few classes of material use, in contrast to the larger number of domain
concepts, different models are learned for each material use based on different, handcrafted
heuristics. Furthermore, each material use is based on different, manually selected feature
page
149
selection strategies, reflecting special properties of a material use. By manually tuning the
material use models in this way, human a-priori knowledge can be incorporated thereby
reducing the amount of necessary training examples. A combination of learning and rule
based approaches would be a natural extension to this.
2. Bootstrapping training examples using unsupervised approaches. For annotating domain
concepts it is not possible to tune every domain model due to the number of concepts and
changes from domain to domain. Therefore, we rely on bootstrapping training examples using
statistical patterns and unsupervised learning approaches. In particular, all documents are
analysed in an unsupervised manner by extracting associations between terms of knowledge
artefact. Those associations are used to extend the classification hypothesis and to consider
also terms not contained in the training examples. Another approach for annotating domain
concepts would be the use of transductive learning techniques like, for example, transductive
support vector machines (Zien et al., 2007), which is planned if performance estimates of our
current approach fall below expectations.
3. Accuracy expectations. One important aspect in applying machine learning techniques is the
achieved accuracy by automatic means. Before applying such techniques, only rough
guesses can be made on how a system will perform and how the algorithm meets the
expectation of the user. Depending on the results of our evaluations, means for increasing
the quality of feedback may be foreseen. So for example users not only assign annotations to
a paragraph or not but may also state the important reason(s) why an annotation is assigned
or not. While this process is more labour intensive, it may yield more accurate results in
future.
6.4 Retrieval of knowledge artefacts
6.4.1
Challenges
The environment the knowledge worker operates in can be represented by formal domain models (see
Section 3.1 and Section 3.2) which conceptualize those entities of the worker‟s domain relevant for
work-integrated learning. For example, a model for the domain of requirements engineering defines all
of the methods for the elicitation of engineering and how they are related. Formal models define the
entities of a domain and the semantic dependencies between them by using a formalism. APOSDLE
uses a set of models to formalize the context of the knowledge worker employed by a company where
APOSDLE is deployed. This raises challenge 1: the retrieval of resources for learning based on
entities within the models.
To use the information present in the models for the retrieval of learning material from the document
base of a company, the models and the document base have to be related in some form. Per se those
24
two tiers are not connected , as the domain models are created by experts for a special purpose,
while the resources in the organisational memory are created by the workers in the company during
25
their daily work. This leads to challenge 2 : the relation between the models and the resources
present in the document base.
To interconnect the two tiers and thus make the documents available for retrieval and the creation of
learning material from them possible, the entities present in the models are used to annotate the
resources in the document base of the company. In this process, concepts stemming from the models
become semantic metadata for the resources present in the document base.
24
A similar problem is present in establishing the semantic web: many resources already exist in the current web, but most of
them are not annotated semantically. Together with the large number of resources already present on the web (or in a
company) this is one of the major obstacles in establishing a broad use of semantic technologies. Therefore this research
project will precisely follow current and future developments in fields such as ontology learning (Buitelaar et al., 2005) and the
(semi-) automatic creation of semantic metadata (Handschuh & Staab, 2003).
25
This challenge is addressed with the annotation facility of the Domain Modeling Tool (see Section 3.1).
page
150
Two options are described in Section 6.3 for assigning semantic metadata to a certain resource:
(1) Manual assignment of semantic metadata – which is a time consuming task.
(2) Automatic assignment of semantic metadata – which does not equal the precision of metadata
assigned by human experts.
The difficulties associated with assigning semantic metadata can easily lead to sparse annotation of
resources in the document base: in this case, only a few concepts from the models are used for
annotation, and only a few resources are annotated.
Sparse annotation of resources with semantic metadata, however, handicaps the search process for
resources based on this metadata: a search based on a set of concepts will not be able to retrieve
results relevant to the corresponding query if the resources in the document base are not semantically
annotated. This leads to challenge 3: retrieval of resources despite a sparse annotation of resources
with semantic metadata.
6.4.2
APOSDLE approach
In this section we will address the first and third challenge presented above, the second one was
already taken up in Section 6.3.
6.4.2.1
Challenge 1: Retrieval based on semantic metadata
APOSDLE uses an associative network as foundation for the context-based retrieval of knowledge
artefacts. In this network, every concept included in the domain model and every knowledge artefact is
represented by a node. Semantic annotations of knowledge artefacts created with the Domain
Modelling Tool (see Section 3.1) are modelled as edges in the network, connecting nodes that name
concepts with nodes that stand for knowledge artefacts. Furthermore, concept nodes are connected to
other concept nodes in matters of their semantic similarity; just as well document nodes to other
document nodes regarding their textual similarity.
The network is queried by the User Profile Service (see Section 4.3) to retrieve knowledge artefacts
corresponding to the current context of the user. These knowledge artefacts are then presented to the
learner (see Section 5.1).
Queries to the associative network are currently expressed by sets of concepts, for which the retrieval
process returns the most suitable knowledge artefacts. For processing the information present in the
associative network, a technique called spreading activation is employed. Initially, a set of nodes
representing the concepts of the search query is activated. Starting with this set of nodes, activation
spreads over the edges of the network to the nodes in the neighbourhood. Those nodes with the
highest level of activation are seen to be the most similar ones to the set of nodes activated initially;
and are returned as retrieval results. At the moment, only nodes representing knowledge artefacts are
taken into account as result to a search.
We see APOSDLE as a retrieval approach for the Semantic Desktop (see Sauermann et al., 2006), as
we use technology originally developed for the Semantic Web to build a desktop application for
supporting a knowledge worker. In our approach, we aim at combining database-like queries to a
knowledge representation as found in Semantic Web technology with classical information retrieval
approaches, that is, the statistical analysis of document content.
6.4.2.2
Challenge 3: Sparse annotation of resources with semantic metadata
The difficulties associated with assigning semantic metadata (Section 6.3) can lead to a sparse
annotation of resources in the organizational document base. This sparse annotation can make the
search for resources based on semantic metadata less successful as the retrieval process is not able
to identify resources relevant to a query due to a lack of documents annotated with semantic
metadata.
page
151
The APOSDLE approach addresses this by employing a technique called associative retrieval in order
to find knowledge artefacts not directly annotated with certain concepts, but indirectly associated with
them.
We perform the associative retrieval approach based both on the semantic metadata and the content
of the documents to increase the amount of relevant knowledge artefacts which can be provided. For
this purpose, we benefit from the associations between concepts in the domain model (based on
semantic similarity provided by the Semantic Service) and associations between documents in the
document base (based on textual similarity provided by the Classification Service) - see Figure 6-2 for
a graphical representation of this approach.
Semantic similarity is
provided by Semantic
Service
Concept layer
Concepts are
semantic metadata
to Knowledge
Artefacts
Document layer
Content-based
similarity is
provided by
Classification
Service
###
######
####
###
####
####
####
###
#####
Term layer
Figure 6-2: The associative network uses two different forms of similarity, which are semantic similarity
and content based similarity. Semantic annotations act as “glue” between these two types of similarity.
6.4.3
Why did we choose this approach?
The APOSDLE solution relies on semantic metadata assigned to resources (knowledge artefacts)
within the backend systems of the companies where APOSDLE is deployed. This semantic metadata
is used to identify resources suitable for the knowledge worker to learn about a certain topic. At
present, assigning of semantic metadata is either done manually or semi-automatically, with both
approaches introducing difficulties that can lead to a situation in which not the entire document base is
annotated with semantic metadata.
This sparse annotation of resources with semantic metadata makes search for resources simply
based on this metadata difficult. A query based on concepts will not be able to identify resources
which are lacking semantic metadata. We use associative retrieval to find resources that are not a
direct result of a query but are – looking at associations with relevant material – promising of being
relevant.
The use of the associative network as an underlying structure for the retrieval process allows the
search for resources based on semantic metadata (challenge 1) and at the same addresses the
problem of low coverage of the resources in the document base with semantic metadata (challenge 3).
As associative retrieval is quite important, we will briefly provide some theoretical background for this
approach in the next section, also including references to data retrieval relying on semantic metadata.
page
152
6.4.4
6.4.4.1
Theoretical Background
Associative Retrieval
Crestani (1997) characterizes associative retrieval as a form of information retrieval which tries to find
appropriate information by retrieving information that is by some means associated with information
that is already known to be relevant. Associated information items can be documents, parts of
documents, extracted terms, concepts, etc. The idea of associative retrieval goes back to the 1960s,
when investigations (Salton, 1963; Salton, 1968) in the field of information retrieval tried to increase
retrieval performance using associations between documents or index terms, which were determined
in advance. Crestani (1997) presents a detailed survey of existing approaches regarding associative
retrieval.
6.4.4.2
Associative Networks
In associative retrieval, association of information is frequently modelled as a graph, which is referred
to as an associative network (Crestani, 1997). Nodes in this network represent information items such
as documents, terms or concepts, edges stand for associations between information items. Edges can
be weighed and / or labelled, expressing the degree (and type) of associations between two
information items.
6.4.4.3
Spreading Activation
Spreading activation originates from cognitive psychology (see Anderson, 1983) where it serves as
mechanism for explaining how knowledge is represented and processed in the human brain. The
human mind is modelled as a network of nodes, which represent concepts and are connected by
edges. Starting from a set of initially activated nodes, the activation spreads over the network
(Sharifian & Samani, 1997). During search, activation flows from a set of initially activated information
items over the edges to their neighbours. The information items with the highest level of activation are
seen to be the most similar to the set of nodes activated initially.
Spreading activation found its way into applications in both neural and semantic networks for
information retrieval (Crestani, 1997). It is comparable to other retrieval techniques regarding its
performance (Mandl, 2001). A detailed introduction to spreading activation in information retrieval can
be found in (Crestani, 1997); for a description of our studies refer to (Scheir & Lindstaedt, 2006).
6.4.4.4
Retrieval based on semantic metadata
While retrieval based on metadata is a well known paradigm in information retrieval research,
semantic metadata introduces new aspects regarding the retrieval process. In earlier metadata
schemas, such as taxonomies or gazetteers, elements from the knowledge organization scheme have
been related by a distinct set of relation types. In the case of semantic metadata the elements in a
knowledge representation are related by arbitrary semantic relations, which can be taken into account
for retrieval purposes (Lux et al., 2006).
6.4.4.5
Semantic Desktop
The Semantic Web (Berners-Lee et al., 2001) movement led in recent years to the development of
new, standardized forms of knowledge representation and technologies for coping with them, such as
ontology editors, triple stores or query languages. The Semantic Desktop paradigm (Sauermann et al.,
2005; Decker & Frank a,b, 2004) stems from the Semantic Web movement and aims at applying
technologies developed for the Semantic Web to desktop computing to finally provide a closer
integration between (Semantic) Web and (Semantic) Desktop.
page
153
6.4.5
6.4.5.1
Where else has something like this been used and proven to work
Information retrieval in the Semantic Web and on the Semantic Desktop
There is a steady growing amount of proposed models and implemented systems which make use of
semantic metadata for information retrieval. The work of Heflin and Hendler (2000), Shah et al. (2002)
or Guha et al. (2003) can be viewed as pioneering work in this domain. Kiryakov et al. (2004) and
Castells et al. (2007) present logical follow-ups of this early work with fine tuned, evolved systems.
Applications of semantic metadata range from indexing it together with textual data (Shah et al., 2002;
Chirita et al., 2006) over extensions of the vector space model (Kiryakov et al., 2004; Castells et al.,
2007) to modelling documents as part of knowledge bases (Zhang et al., 2005). Beagle++ (Chirita et
al., 2006) is an example of a search engine for the Semantic Desktop; and SEAL is an approach for
ranking search results in semantic portals (Stojanovic et al., 2001; Stojanovic et al., 2003). A detailed
overview can be found in Esmaili and Abolhassani (2006) and Scheir et al. (2007).
6.4.5.2
Network models, spreading activation and associative retrieval
A detailed survey of classical approaches to associative retrieval and spreading activation can be
found in Crestani (1997). More recently, (Mandl, 2001) states that - considering the good results of
spreading activation based text retrieval systems PIRCS (Kwok et al., 1996) and Mercure
26
(Boughanem et al., 1999) at the TREC - spreading activation based retrieval models are comparable
to other information retrieval approaches regarding their performance.
Besides the systems mentioned afore that operate on textual data, there also exist approaches that
make use of spreading activation to search in a knowledge based system. Ontocopi (Alani et al.,
2003) identifies communities of practice in an ontology using spreading activation based clustering.
Rocha et al. (2004) present a hybrid approach for searching the (Semantic) Web that combines
keyword based search with spreading activation search in an ontology for search on web sites. Berger
et al. (2004) present a tourist information system whose underlying knowledge base is searched using
27
spreading activation. Finally, Huang et al. (2004) address the scarcity problem in recommender
systems using a network-based associative retrieval approach.
6.4.6
6.4.6.1
User Feedback
At present, the users of the APOSDLE system have no option to influence the results of the retrieval
process by delivering their opinion about the relevance of the retrieved knowledge artefacts back into
the system. However, giving the knowledge worker the possibility to submit feedback could improve
both ranking and quality of retrieved documents significantly. If a resulting knowledge artefact is very
bad, it should not be included in the result set any more (or at least ranked below), if it is very good, it
should get a higher rank.
We are evaluating a user feedback system using a connectionist learning approach which aims at
updating the associative network using the feedback provided by the users.
Besides the basic technical implementation, there are also some conceptual challenges to meet:
There has to be a critical mass of users giving feedback to derive an average value, so that single,
erroneous, or irrelevant valuations do not affect the results. Nevertheless, the user should be able to
notice the effect of the feedback given to encourage further participation. Limits for the minimal
amount of valuations and quantitative threshold values have to be defined.
26
Text REtrieval Conference
27
In the field recommender systems sparse associations between users of the systems and recommended items, leads to a
similar problem as discussed by challenge 3: items can not be recommended adequately if there is not a decent set of
associations between users and items.
page
154
A problem of rating relevance is how to define the relevance of a piece of information; does a rating
simply characterize good quality of the document, or it is a measure how suitable it is as a result of the
given query. Furthermore, it is not really useful to provide functionality for rating a knowledge artefact
before the user actually had a chance to inspect it accurately, just as well if there are only pieces
provided instead of the whole document.
We are right now evaluating a connectionist learning approach to tweak the weights of the edges
between concepts and knowledge artefacts in the associative network. If a delivered knowledge
artefact is very good, it should be ranked higher, so the relevance of this piece of information should
be boosted. Since the relevance is derived from the weights of the edges between the starting concept
and the knowledge artefact itself, those weights should be elevated. If the result is bad, it should be
ranked below and therefore the weights should be decreased.
These adjustments called judgements are derived by using a technique called back-propagation; it is
identical to the process of spreading activation but is performed in reverse: stating from the knowledge
artefact a user submitted feedback for, the system traverses the path back to the source node(s) of a
query. Those paths that are traversed from a result node to the original source node(s) of a query
receive higher (for positive feedback) or lower (for negative feedback) weights.
Technically, in addition to the associative network including concepts, knowledge artefacts and their
connections modelled as graph with weights we use a second network simply to store the adjustments
to the edge weights of the associative network. This way we can always switch between the original
associative network and the one changed by user feedback.
While evaluating the user feedback system, we are also engaged in finding best practice solutions for
the challenges presented above.
6.4.6.2
Combination of concept and term based search
To increase the options of the knowledge worker in the context of search and further enhance the
possibilities provided by retrieval in APOSDLE, we added the possibility to query the network structure
using concepts and terms (words) at the same time (see, for example, Figure 5-1 the Search facility).
Hence, the search can also be conducted with an additional set of keywords used to find knowledge
artefacts not annotated with concepts, but still relevant in matters of these keywords.
Additionally, a user can start a search for knowledge artefacts that deal with topics that are not
modelled in the domain model; or narrow a search based on concepts from the domain model by
using terms (words). Like the connectionist learning approach, this functionality is being evaluated
right now and not yet available in the user interface.
6.4.7
Component evaluation: similarity measures
The Associative Network (AN) in APOSDLE is responsible for delivering a suitable list of snippets
(fragments of documents) to a given input concept. In delivering the snippets, the AN has the
(optional) possibility to make use of its built in Concept To Concept Layer, which connects all the
concepts available by means of their semantic similarity. Turning this layer on, the most relevant
concepts in the neighbourhood of the input concept are added to the input of the AN.
This Concept to Concept layer is directly built from the domain ontology, where the edge weights
connecting pairs of concepts of the domain ontology are derived using a semantic similarity function
which measures how much two concepts are alike. Several semantic similarity functions have been
proposed in literature, which correspond to different criteria of „being alike‟, and the evaluation of these
functions (hereafter, measures) has shown that no single measure always performs better, but
different measures can perform differently depending on the criteria of „being alike‟ required. As a
consequence of this, APOSDLE offers the possibility to use several semantic similarity measures: (i)
Conceptual Similarity (Wu & Palmer, 1994), (ii) Shortest Path, (iii) Scaled Shortest Path (Leacock &
Chodorow, 1998), (iv) Resnik (Resnik, 1995), (v) Lin (Lin, 1998), (vi) Jiang Conrath (Jiang &
Conrath1997), and (vii) the Trento Vector (Scheir, Lindstaedt & Ghidini, 2008).
page
155
Measures (i)-(iii) belong to the so-called path based measures, as they measure similarity using the
distance (or path length) between concepts in the is-a hierarchy. Measures (iv)-(vi) belong to the socalled information content based measures, as they measure similarity using the closeness of the
information content of different concepts. Measure (vii), instead, measures similarity by looking at how
many relations concepts have in common.
In this evaluation, the following research questions are covered:
(i) Which semantic similarity measure is the best one for a specific application domain?
(ii) Is there a specific similarity measure which is performing well on every domain tested and
could therefore be proposed as a default measure?
(iii) How much do the covered measures actually differ?
To examine the behaviour in different application domains, two individual domains are evaluated: ISN
and CCI.
6.4.7.1
Method
The data required for this evaluation is gathered using a Web application called Web Tool (for
Ontology) Evaluation (WTE) (see Figure 6-3)
Figure 6-3: Screenshot of the WTE
Two domain experts of every domain are rating the similarity between concepts (as shown in Figure 62) in at least two individual sessions, yielding a test set of at least four sessions. To reduce memory
effects, the experts are asked to wait at least three days before they do the second session. For a
certain number iterations, two concepts are presented and the domain expert has to grade the degree
of interconnection between those two concepts. Using these ratings, an “ideal” model of the Concept
to Concept layer can be created. To ensure a correct understanding of the concepts shown, the parent
concepts (if available) for both concepts are shown.
According to (Charles, 2000) we have chosen a five point similarity scale, containing the following
values:
page
156
Scale Description
Value
identical in meaning
1
similar in meaning
0.75
vaguely similar in meaning
0.55
different in meaning
0.27
opposite in meaning
0
Table 6-1: Similarity Scale used in WTE
For the domain to be evaluated, the APOSDLE Concept similarity component provides the derived
measures for each edge connecting two concepts. A domain containing 120 concepts is therefore
yielding a maximum of (120*119) / 2 = 7140 Concept Pairs to be rated.
Recursive edges (an edge from one concept back to the same one) are not evaluated, also there is
always only one edge between two concepts - concept1 to concept2 is the same as concept2 to
concept1 and, therefore, has the same similarity values assigned. It is impossible to let the experst
evaluate all the 7140 Concept Pairs, so there is the need for some pre selection.
The number of Concept Pairs to be rated in one session can be defined individually for each
application domain: (i) fixed concepts defines Concept Pairs which are queried in every session –
allowing us to compare ratings from different experts and over time; (ii) variable concepts defines the
number of totally randomly selected Concept Pairs to be rated in a session. Fixed concepts are
preselected manually by inspecting the similarity matrix of the domain including all the measures with
the aim of being able to discriminate between measures. For example, Concept Pairs scoring very
different values for the different measures are helping us to see which measures are more similar to
the human evaluation.
Having more individual rating results for significant Concept Pairs tends to be more useful than having
a larger number of totally randomly selected Concept Pairs. The number of total Concept Pairs to be
rated is set to 100, taking approximately 15 to 20 minutes per session. In the ISN domain we
preselected 32 Concept Pairs, in the CCI domain 28 ones.
Only completed sessions are stored into the database, ensuring that stored sessions are always
available. All tasks (import of measures, import of concepts, import of Concept Pairs etc.), except the
pre selection of Concept Pairs, can be carried out automatically, which makes it very easy to add other
domains or work with different measure settings.
The results of the expert ratings then are evaluated against the computed measures. All measures are
normalized (values between 0 and 1) and the average deviations of each measure compared with the
expert ratings are derived.
6.4.7.2
Results
In the ISN domain two experts rated in six sessions, therefore yielding 600 individual ratings, rating
435 different Concept Pairs. The 32 fixed Concept Pairs have been rated in every session (so they
have 6 ratings), 5 Concept Pairs have been rated twice (meaning that they have been randomly
selected twice). The six individual ratings of the fixed Concept Pairs are fairly consistent, they differ in
only one point at most; the average value is taken to erase some possible erroneous entries.
The best three measures compared to the expert ratings in ISN are:
1. Conceptual Similarity:
Average Deviation: 0.1899
2. Jiang Conrath:
page
157
3. Shortest Path:
In the CCI domain two experts rated in four sessions, yielding 400 individual sessions, rating 306
different Concept Pairs. The 28 fixed Concept Pairs have been rated in every session (so they have 4
ratings), 10 Concept Pairs have been rated twice (meaning that they have been randomly selected
twice).
The best three measures compared to the expert ratings in CCI are:
1. Scaled Shortest Path:
2. Shortest Path:
3. Conceptual Similarity:
Answers to research questions
(i) Which semantic similarity measure is the best one for a specific application domain?
In the ISN domain, the Conceptual Similarity measure is the best one when compared with the
expert ratings. Jiang Conrath and Shortest Path are second and third, but they do already
have a deviation larger than 0.20.
In the CCI domain, Scaled Shortest Path delivers the best results, Shortest Path and
Conceptual Similarity do follow with a quite significant distance.
These results show that the perfect similarity measure has to be chosen for every domain
individually.
(ii) Is there a specific similarity measure which is performing well on every domain tested and
could therefore be proposed as default measure?
Because we are testing only two of the APOSDLE domains, we only can speak about those
two. There is no similarity measure which performs perfectly on both domains, so there is now
clear winner.
However taking both domains into account, the path based measures seem to be the most
appropriate for the kind of ontologies which constitute the domain models. In fact both the
Conceptual Similarity and the Scaled Shortest Path belong to this family of measures.
Focusing on a single measure the Shortest Path measure seems to be the best one suitable
for a default setting. Conceptual Similarity performs very well on ISN, but is far behind in CCI,
while the distance between winner and Shortest Path in ISN is not that big.
(iii) How much do the covered measures actual differ?
The difference between the different layers can be seen clearly in the top 3 ranking of the
results section. While in ISN the differences are not that large, Scaled Shortest Path is the
clear winner in CCI. The worst measures in both domains deliver very high average deviations
up to 0.35, meaning that choosing the wrong can measure significantly degrade the results.
6.4.7.3
Conclusion
Overall, path based measures seem to be the most appropriate measures for the APOSDLE models
and the very simple shortest path measure performs well on both tested domains. Nevertheless, the
best measure has to be evaluated on a case-by-case manner for each domain. In fact, the evaluation
page
158
has emphasised the existence of large differences between the top scorer and the other measures,
making the selection of the correct similarity function a crucial task for a good performance of the
Associative Network.
Since WTE can derive those measure evaluations on the fly via the Web interface, it is also possible
to have a look at the results while the evaluation is still ongoing, for example after each individual
session or user.
page
159
7 References
Abowd, G.; Dey, A.; Orr, R.; Brotherton, J. (1997). Context-awareness in wearable and ubiquitous
computing. Technical report, Georgia Institute of Technology, 1997. cited on pp. 4.
Adomavicius, G. & Tuzhilin, A. (2005). Toward the Next Generation of Recommender Systems: A
Survey of the State-of-the-Art and Possible Extensions, in: IEEE Transactions on Knowledge
and Data Engineering, 17 (6), 734-749.
Adomavicius, G. Sankaranarayanan, R., Sen, S., & Tuzhilin, A.. (2005). Incorporating contextual
information in recommender systems using a multidimensional approach, in: ACM
Transactions on Information Systems, 23, 1, 103-145.
Agrawal, R.,Faloutsos, C. & Swami, A. (1993). Efficient similarity search in sequence databases,
Lecture Notes in Computer Science, 730, 69-84.
Alani, H.; Dasmahapatra, S.; O'Hara, K. & Shadbolt, N. (2003). Identifying Communities of Practice
through Ontology Network Analysis. IEEE Intelligent Systems 18(2), 18-25.
Albert, D. & Mori, T. (2001). Contributions of Cognitive Psychology to the Future of e-Learning. Bulletin
of the Graduate School of Education Hiroshima University, Part I (Learning and Curriculum
Development) (Vol.50, pp. 25-34).
Albert, D., Hockemeyer, C., Mayer, B., Steiner, C. M (2007). Cognitive Structure Modelling of Skills for
Technology-Enhanced Learning. Paper presented at the 7th IEEE International Conference on
Advanced Learning Technologies (ICALT 2007), July 18-20, 2007, Niigata, Japan.
Albert, D., Hockemeyer, C., Wesiak, G., 2002. Current Trends in eLearning based on Knowledge
Space Theory and Cognitive Psychology. Psychologische Beiträge, 4, 44, 478-494.
Anderson, J. R. (1983). A spreading activation theory of memory. Journal of Verbal Leaning and
Verbal Behaviour, 22, 261-295.
Anderson, L. W., & Krathwohl, D. A. (2001). A taxonomy for learning, teaching, and assessing: A
revision of bloom's taxonomy of educational objectives. New York: Longman.
Aperture 2007. A Java framework for getting data and metadata. http://aperture.sourceforge.net/
Arvey, R. D. & Murphy, K. R. (1998). Performance Evaluation in Work Settings. Annual Review of
Psychology, 49, 141-168.
Baader, F., Calvanese, D., McGuinness, D.L., Nardi, D. & Patel-Schneider, P.F. (2003). The
Description Logic Handbook: Theory, Implementation, and Applications. Cambridge University
Press,.
Baldauf, M.; Dustdar, S.; & Rosenberg, F. (2007). A survey on context aware systems. International
Journal of Ad Hoc and Ubiquitous Computing, 2, 4, 263 – 277.
Bardram, J.E. (1997). Plans as situated action: an Activity Theory approach to workflow systems. In
Proceedings of the Fifth European Conference on Computer Supported Cooperative Work
(ECSCW‟97), pages 17–32, Lancaster, United Kingdom, 1997. Kluwer. 2, 4, 58
Bardram, J. E., & Hansen, T. R. (2004). The AWARE architecture: supporting context-mediated social
awareness in mobile cooperation. In Proceedings of the 2004 ACM conference on Computer
supported cooperative work (pp. 192–201). Chicago, Illinois, USA: ACM.
Benyon, D. R., & Murray, D. M (1993). Adaptive systems; from intelligent tutoring to autonomous
agents. Knowledge-Based Systems, 6, 4, 197-219.
Berger, H.; Dittenbach, M. & Merkl, D. (2004),An Adaptive Information Retrieval System based on
Associative Networks, in S. Hartmann & J. Roddick, ed.,'Proceedings of the 1st Asia-Pacific
page
160
Conference on Conceptual Modelling (APCCM 2004)', Australian Computer Society Inc.,
Dunedin, New Zealand, pp. 27--36.
Berners-Lee, T.; Hendler, J. & Lassila, O. (2001). The Semantic Web. Scientific American.
Billett, S. (1996a). Accessing and engaging vocational knowledge: Instructional media versus
everyday practice. Education and Training, (38), pp.18–25
Billett, S. (1996b) Situated learning: Bridging sociocultural and cognitive theorising. Learning and
Instruction, (6), pp.263–80.
Bloom, B. (Ed.) (1956). Taxonomy of educational objectives. Handbook 1: The cognitive domain. New
York: David McKay.
Bonestroo, W. J., & De Jong, T. (2009). Planning in technology enhanced learning environments.
Manuscript submitted for publication.
Boughanem, M.; Dkaki, T.; Mothe, J. & Soule-Dupuy, C. (1999). Mercure at trec7, in Proceedings of
The Seventh Text REtrieval Conference (TREC-7)'.
Boyatzis, R. E. (1982). The Competent Manager: A Model for Effective Performance. New York: John
Wiley & Sons.
Brown, P. J. (1996). The stick-e document: a framework for creating context-aware applications. In
Proceedings of EP’96, Palo Alto, pages 259–272, January 1996.
Brusilovsky, P. (2004). KnowledgeTree: A Distributed Architecture for Adaptive E-Learning. In:
Proceedings of the WWW 2004, May 17–22, 2004, New York, USA, pp 104-113.
Brusilovsky, P., Schwarz, E. & Weber, G. (1996). ELM-ART: An Intelligent Tutoring System on World
Wide Web, in C. Frasson, G. Gauthier & A. & Lesgold (Ed.): Intelligent Tutoring Systems
(Lecture Notes in Computer Science, Vol. 1086), pp. 261-269, Berlin: Springer Verlag.
Budzik, J. & Hammond, K. J. (2000) User Interactions with Everyday Applications as Context for Justin-time Information Access, in: Proceedings of the 2000 International Conference on Intelligent
User Interfaces, pp. 44-51, New Orleans: ACM Press.
Budzik, J., Hammond, K. J. & Birnbaum, L. (2001) Information Access in Context, in: KnowledgeBased Systems, 14 (1-2), 37-53.
Buitelaar, P.; Cimiano, P. & Magnini, B., ed. (2005). Ontology Learning from Text: Methods,
Evaluation and Applications, Vol. 123. Amsterdam: IOS Press..
Carroll, J. M., & Rosson, M. B. (1987). The Paradox of the Active User. In: Carroll, J. M. (eds.):
Interfacing Thought: Cognitive Aspects of Human-Computer Interaction. Cambridge: MIT
Press, pp. 80-111.
Cassell, J. (2000). Embodied conversational interface agents. Communications of the ACM, 43, 4, 70–
78.
Castells, P.; Fernández, M. & Vallet, D. (2007). An Adaptation of the Vector-Space Model for
Ontology-Based Information Retrieval. IEEE Transactions on Knowledge and Data
Engineering, 19, 2, 261-272.
Charles, W.G. (2000). Contextual correlates of meaning. Applied Psycholinguistics, 21, 505–524.
Chirita, P.; Costache, S.; Nejdl, W. & Paiu, R. (2006). Beagle++: Semantically Enhanced Searching
and Ranking on the Desktop. In The Semantic Web: Research and Applications. Berlin /
Heidelberg: Springer, pp. 348-362.
Choo, C.W. (1998). The knowing organisation: How Organizations Use Information to Construct
Meaning, Create Knowledge, and Make Decisions. New York: Oxford University Press.
Christl, C., Ghidini, C., Guss, J., Pammer, V., Rospocher, M., Lindstaedt, S., Scheir P. & Serafini, L.
(2008). Deploying semantic web technologies for work integrated learning in industry. A
page
161
comparison: SME vs. large sized company. In Proceedings of the 7th International Semantic
Web Conference (ISWC 2008), Semantic Web In Use Track. Karlsruhe, Germany. 28-30
October 2008. Volume 318/-1/2008 of LNCS. Pages 709-722. Springer Berlin/Heidelberg
Collin, K. (2006). Connecting work and learning: design engineers' learning at work. Journal of
Workplace Learning, 18 (7/8), 403-413.
Crestani, F. (1997). Application of Spreading Activation Techniques in Information Retrieval. Artificial
Intelligence Review, 11, 6, 453-482.
Cristani, M. & Cuel, R. (2005). A survey on ontology creation methodologies. International Journal
Semantic Web Information Systems, 1, 2, 49–69.
Cutrell, E., Robbins, D.C., Dumais, S.T. & Sarin, R. (2006). Fast, flexible filtering with Phlat - Personal
search and organization made easy. In Proceedings of CHI'06, Human Factors in Computing
Systems, (Montréal, April 2006), ACM press, 261-270.
Czerwinski, M. & Horvitz, E. (2002). An investigation of memory for daily computing events. In
Proceedings of HCI ’02.
de Hoog, R., Kabel, S., Barnard, Y., Boy, G., DeLuca, P., Desmoulins, C., Riemersma, J. & Verstegen,
D. (2002b). Re-using technical manuals for instruction: creating instructional material with the
tools of the IMAT project. In: Workshop proceedings Integrating technical and training
documentation, 6th International Intelligent Tutoring Systems conference (ITS 2002), San
Sebástian, Spain, 28-39.
de Hoog, R., Kooken, J., Ley, T. & Maiden, N. (2006). Workplace Learning Study. Deliverable D2.1,
EU Project 027023 APOSDLE (92 p.), Know-center, Graz, Austria.
delMas, R., Garfield, J., Ooms, A., & Chance, B. (2007). Assessing students' conceptual
understanding after a first course in statistics. Statistics Education Research Journal, 6, 28-58.
Decker, S. & Frank, M. (2004) The Social Semantic Desktop. DERI Technical Report 2004-05-02,
Galway, Innsbruck.
Decker, S. & Frank, M. R. (2004). The Networked Semantic Desktop. In WWW Workshop on
Application Design, Development and Implementation Issues in the Semantic Web.
Dehn, D. & van Mulken, S. (2000). The impact of animated interface agents: a review of empirical
research. International Journal of Human-Computer. Studies, 52, 1,1–22,
Dewey, J. (1933). How We Think. New York: D. C. Heath.
Dey, A.K., & Abowd, G.D. (1999). Towards a better understanding of context and context-awareness.
GVU Technical Report GIT- GVU-99-22, College of Computing, Georgia Institute of
Technology.
Dey, A.K. & Abowd, G.D. (2000). Cybreminder: A context-aware system for supporting reminders. In
Proceedings of the 2nd International Symposium On Handheld and Ubiquitous Computing
(HUC), pages 172–186, Bristol, UK, September 2000.
Dey, A. K.; Abowd, G. D. & Salber, D. (2001). A Conceptual Framework and a Toolkit for Supporting
the Rapid Prototyping of Context-Aware Applications, in: Human-Computer Interaction (HCI)
Journal, 16, 2-4, 97-166.
Doignon, J. & Falmagne, J. (1985). Spaces for the assessment of knowledge. International Journal of
Man-Machine Studies, 23, 175-196.
Doignon, J. & Falmagne, J. (1999). Knowledge Spaces. Heidelberg: Springer.
Dreyfus, H. L. & Dreyfus, S. (1986). Mind over Machine. The Power of Human Intuition and Expertise
in the Era of the Computer. New York: Basil Blackwell.
Dublin Core (n.D.). Dublin Core Metadata Initiative. http://dublincore.org/
page
162
Dumais, S.,Platt, J.,Heckerman, D. & Sahami, M. (1998). Inductive learning algorithms and
representations for text categorization. In Proceedings of CIKM-98, 7th ACM International
Conference on Information and Knowledge Management (Bethesda, MD, 1998), 148–155.
Engelmore, R. & Morgan, T. (1988). Blackboard Systems. Addison-Wesley,
Eraut, M.R. (1994). Developing Professional Knowledge and Competence. London: Falmer Press.
Eraut, M., & Hirsh, W. (2007). The Significance of Workplace Learning for Individuals, Groups and
Organisations. SKOPE, Oxford & Cardiff Universities.
Esmaili, K. S. & Abolhassani, H. (2006). A Categorization Scheme for Semantic Web Search Engines.
In '4th ACS/IEEE International Conference on Computer Systems and Applications (AICCSA06)'.
Field, A. (2009). Discovering statistics using spss. London: SAGE Publications Ltd.
Fink, J. (2004). User Modelling Servers: Requirements, Design, and Evaluation, Amsterdam: IOS
Press.
Fink, J. & Kobsa, A. (2002). User Modelling for Personalized City Tours, in: Artificial Intelligence
Review, 18, pp. 33-74.
Furnkranz, J., & Widmer, G. (1994). Incremental Reduced Error Pruning. In Proceedings of the
Eleventh International Conference on Machine Learning, 70–77. San Francisco, Calif.: Morgan
Kaufmann.
Garavan, T. N., Morley, M., Gunnigle, P., & McGuire, D. (2002). Human resource development and
workplace learning: emerging theoretical perspectives and organisational practices. Journal of
European Industrial Training, 26, 2, 3, 4, 60-71.
Garcia-Barrios, V.M.; Beham, G. & Kump, B. (2008). Scrutinising Competencies: Retraceable Clouds
of Learning Goals in the APOSDLE System. In Proceedings of 6th International Conference
on Community Based Environments: OpenACS and .LRN Conference 2008, February 2008,
Guatemala, ISBN 978-99922-2-434-2, pp. 34-45
Gemmel, J.; Bell, G.; Lueder, R.; Drucker, S.; Wong, C. (2002). MyLifeBits: Fulfilling the Memex
Vision. In Proceedings of the ACM Multimedia, Sophia-Antipolis, France, December 2002,
pages 235–238
Gery, G.J. (1991). Electronic Performance Support Systems. How and Why to remake the Workplace
Through the Strategic Application of Technology. Boston: Weingarten Publications.
Ghidini, C., Rospocher, M., Serafini, L., Kump, B., Pammer, V., Faatz, A., Guss, J. (2007). Integrated
Modelling Methodology Version 1. Deliverable D1.3, APOSDLE project.
Gigerenzer, G., Todd, P. M.; the ABC Research Group (1999). Simple Heuristics That Make Us Smart.
Oxford University Press.
Goecks J, & Shavlik J. (2000). Learning users' interests by unobtrusively observing their normal
behavior. In: IUI 2000: International Conference on Intelligent User Interfaces. New York:
ACM, pp 129-132.
Granitzer M., Auer P. Experiments with Hierarchical Text Classification, Proceedings of 9th IASTED
International Conference on Artifical Intelligence, ACTA Press, Benidorm, Spain, 2005,
IASTED
Green, P. C. (1999). Building Robust Competencies: Linking Human Resource Systems to
Organizational Strategies. San Francisco: Jossey-Bass.
Gruber, T. R. (2007). Collective Knowledge Systems: Where the Social Web meets the Semantic
Web. To appear in: Journal of Web Semantics.
page
163
Guha, R.; McCool, R. & Miller, E. (2003),Semantic search, in 'WWW '03: Proceedings of the 12th
international conference on World Wide Web', ACM Press, New York, NY, USA, pp. 700--709.
Gutwin, C., & Greenberg, S. (2002). A Descriptive Framework of Workspace Awareness for Real-Time
Groupware. Computer Supported Cooperative Work (CSCW), 11(3), 411–446.
Guzmán, R., & Conejo, R. (2004). A Model for Student Knowledge Diagnosis Through Adaptive
Testing. In: J.C. Lester et al. (eds.): ITS 2004, LNCS 3220. pp 12-21.
Handschuh, S. & Staab, S., ed. (2003). Annotation for the Semantic Web, Vol. 96, IOS Press.
Hart, S. G., & Staveland, L. E. (1988). Development of nasa-tlx (task load index): Results of empirical
and theoretical research. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (pp.
139-183). Amsterdam, The Netherlands: North Holland Press.
Harter, Andy; Hopper, Andy; Steggles, Pete; Ward, Andy; Webster, Paul (1999). The anatomy of a
context-aware application. In Mobile Computing and Networking, pages 59–68, 1999.
Havre S., Hetzler E., Whitney P., Nowell L., (2002). ThemeRiver: Visualizing Thematic Changes in
Large Document Collections, IEEE Transactions on Visualization and Computer Graphics,
8(1), pp. 9-20.
Heckmann, D. (2005). Ubiquitous User Modeling. Saarbrücken: Dissertation Universität des
Saarlands.
Heckmann, D.et al. (2005). GUMO – the General User Model Ontology, in: Proceedings of the 10th
International Conference on User Modeling, pp. 428-432, Berlin, Heidelberg: Springer.
Heflin, J. & Hendler, J. (2000). Searching the Web with SHOE. In Artificial Intelligence for Web Search.
Papers from the AAAI Workshop. WS-00-01., AAAI Press, Menlo Park, CA, pp. 35-40.
Henze, N. & Nejdl, W. (2001). Adaptation in open corpus hypermedia, in: International Journal of
Artifical Intelligence in Education (JAIED) Special Issue on Adaptive and Intelligent WebBased Systems, 12, 325-350.
Hiermann, W., & Höfferer, M. (2003). A Practical Knowledge-based Approach to Skill Management
and Personal Development. Journal of Universal Computer Science, 9, 12, 1398-1409.
Hockemeyer, C., Conlan, O., Wade, V. & Albert, D. (2003). Applying Competence Prerequisite
Structures for eLearning and Skill Management. Journal of Universal Computer Science,
9(12), 1428-1436.
Hockemeyer, C., Held, T. & Albert, D.(1998). RATH - A Relational Adaptive Tutoring Hypertext WWWEnvironment Based on Knowledge Space Theory. In: C. Alvegard (eds.) Computer Aided
Learning and Instruction in Science and Engineering. Proceedings of the Fourth International
Conference (CALISCE'98), Göteborg, Sweden.
Hoog, R. de, Kooken, J., Ley, T., Kump, B., & Lindstaedt, S. (2008) Workplace Learning Study 2.
APOSDLE-D02.5-UT-WorkPlaceLearningStudy2, APOSDLE consortium
Horvitz, E.; Apacible, J.; Subramani, M.; Sarin, R.; Koch, P.; Cadi, J.J.; Narin, A.; Rui, Y. (2003).
Experiences with the design, fielding, and evaluation of a real-time communications agent.
Technical Report MSR-TR_2003-98, Microsoft Research, 2003.
Houle, C. O. (1961). The inquiring mind: A study of the adult who continues to learn. Madison,
University of Wisconsin Press.
Howes, T. A. & Smith, M. (2006). Lightweight Directory Access Protocol (LDAP): Uniform Resource
Locator. RFC 4516: The Internet Society. Accessed at 2007-06-29:
http://rfc.sunsite.dk/rfc/rfc4516.html
Huang, Z.; Chen, H. & Zeng, D. (2004). Applying associative retrieval techniques to alleviate the
scarcity problem in collaborative filtering. ACM Transactions on Information Systems, 22, 1,
116-142.
page
164
Hyatt M, & Vrabik R. (2004). The information grid - Secure access to any information, anywhere, over
any network. Accessed at http://www.ibm.com/developerworks/library/gr-infogrid.html
IEEE (2002). Draft Standard for Learning Object. IEEE 1484.12.1-2002
Jameson, A. (2003). Adaptive interfaces and agents. In: J. A. Jacko, &. A. S. (eds.): Human-computer
interaction handbook. Mahwah, NJ: Erlbaum, pp 305-330.
JCR (n.d.). Introducing the Java Content Repository API,
http://www.ibm.com/developerworks/java/library/j-jcr/
Jiang, J.J. & Conrath, D.W. (1997). Semantic similarity based on corpus statistics and lexical
taxonomy. In Proceedings of the 10th International Conference on Research on
Computational Linguistics, Taiwan, 19–33.
Kass, R., & Finin, T. (1988). Modeling the User in Natural Language Systems. Computational
Linguistics, 14, 3, pp 5-22.
Kay, J. (1995). The UM Toolkit for Cooperative User Modeling, in: User Modeling and User-Adapted
Interaction, 4, 3, pp. 149-196, Springer Netherlands.
Kay, J. & Kummerfeld, B. (2006). Scrutability, User Control and Privacy for Distributed Personalization,
in: CHI 2006. Workshop on Privacy-Enhanced Personalization.
Khuwaja, R., Desmarais, M. & Cheng, R. (1996). Intelligent Guide - Combining user knowledge
assessment with pedagogical guidance. Intelligent Tutoring Systems. Lecture Notes in
Computer Science Vol.1084, Springer Verlag.
Kienreich, W.; Sabol, V.; Granitzer, M.; Klieber, W.; Lux, M. & Sarka, W. (2005), A visual query
interface for a very large newspaper article repository; Database and Expert Systems
Applications, 2005. Proceedings. Sixteenth International Workshop on, 22-26 Aug. 2005
Page(s):415 - 419
Kiryakov, A.; Popov, B.; Terziev, I.; Manov, D. & Ognyanoff, D. (2004), 'Semantic annotation, indexing,
and retrieval'. Journal of Web Semantics: Science, Services and Agents on the World Wide
Web, 2, 1, 49-79.
Kobsa, A., Koenemann, J. & Pohl, W. (2001). Personalised hypermedia presentation techniques for
improving online customer relationships, in: The Knowledge Engineering Review, 16, 2, 111155.
Koda, T. & Maes, P. (1996). Agents with faces: the effect of personification. In Proceedings of the 5th
IEEE International Workshop on Robot and Human Communication, pages 189–194, 1996.
Kooken, J., de Hoog, R., Ley, T., Kump, B. & Lindstaedt, S. (2008). Workplace Learning Study 2.
Deliverable D2.5, EU Project 027023 APOSDLE (92 p.), Know-center, Graz, Austria.
Korossy, K. (1997). Extending the theory of knowledge spaces: A competence-performance approach.
Zeitschrift für Psychologie, 205, pp 53-82.
Kotis, K., Vouros, G. & Stergiou, K. (2006). Towards automatic merging of domain ontologies: The
hcone-merge approach. Journal of Web Semantics, 4, 1, 60–79.
Kwok, K. & Grunfeld, L. (1996). TREC-4 Ad-Hoc, Routing Retrieval and Filtering Experiments using
PIRCS, in D. K. Harman, ed., The Fourth Text REtrieval Conference (TREC-4)'.
Lave, J., & Wenger, E. (1991): Situated Learning - Legitimate Peripheral Participation, Cambridge
University Press
Leacock, C. & Chodorow, M. (1998). Combining local context and wordnet similarity for word sense
identification. In C. Fellbaum (Ed.), WordNet: A Lexical Reference System and its Application.
Cambridge, MA: MIT Press.
Lei, Y. & Singh, M.P. (1997). A comparison of workflow metamodels. In Proceedings of the ER-97
Workshop on Behavioral Modeling and Design Transformations: Issues and Opportunities in
page
165
Conceptual Modeling, Los Angeles, California, USA, November 1997. 2, 5, 34
Ley, T., Kump, B., Ulbrich, A., Scheir, P., & Lindstaedt, S. (2008). A Compentence-based Approach
for Formalizing Learning Goals in Work-Integrated Learning. In Proceedings of the ED-MEDIA
2008 - World Conference on Educational Multimedia, Hypermedia & Telecommunications.
Vienna.
Ley, T., Lindstaedt, S. N., & Albert, D. (2005). Supporting Competency Development in Informal
Workplace Learning. In K. Althoff, A. Dengel, R. Bergmann, M. Nick, & T. Roth-Berghofer
(Ed.), Professional Knowledge Management (pp. 189-202). Berlin: Springer.
Ley, T., Ulbrich, A., Scheir, P., Lindstaedt, S. N., Kump, B., & Albert, D. (2008). Modelling
Competencies for supporting Work-integrated Learning in Knowledge Work. Journal of
Knowledge Management, 12, 6 pp 31-47.
Leymann, F. (2006). Workflow-based coordination and cooperation in a service world. In R. Meersman
and Z. Tari, (eds). Proceedings of the 14th International Conference on Cooperative
Information Systems (CoopIS’06), volume 4275 of Lecture Notes in Computer Science, pages
2–16, Montpellier, France, November 2006. Springer-Verlag. 2
Liebermann, H. & Selker, T. (2000). Out of context: Computer systems that adapt to, and learn from,
context. IBM Systems Journal, 39, 3.
Lievens, F., Sanchez, J. I. & De Corte, W. (2004). Easing the Inferential Leap in Competency
Modeling: The effects of Task-related Information and Subject Matter Expertise. Personnel
Lin, D. (1998) An information-theoretic definition of similarity. In Proc. 15th International Conf. on
Machine Learning, 296–304. San Francisco, CA: Morgan Kaufmann.
Lindstaedt, S. Beham, G., Aehnelt, M., Hambach, S., Musielak, M. Kooken, J.P., Hoog, R. de &
Leemkuil, H.H. (2009). The APOSDLE approach to self directed work integrated learning.
APOSDLE deliverables D2.8 and 3.5. Graz: Know Center
Lindstaedt, S. & Mayer H. (2006), A Storyboard of the APOSDLE Vision. In W. Nejdl and K.
Tochtermann (Eds.), Innovative Approaches for Learning and Knowledge Sharing (LNCS
4227), 628-633, Springer, Berlin.
Lokaiczyk, R., Godehardt, E., Faatz, A., Goertz, M., Kienle, A., & Wessner, M., et al. (2007). Exploiting
Context Information for Identification of Relevant Experts in Collaborative WorkplaceEmbedded E-Learning Environments. In E. Duval, R. Klamma, & M. Wolpers (Eds.), Springer11645 /Dig. Serial]: Vol. 4753. Creating new learning experiences on a global scale. Second
European Conference on Technoloy Enhanced Learning, EC-TEL 2007, Crete, Greece,
September 2007 ; proceedings (pp. 217–231). Berlin: Springer.
Lucia, A. D. & Lepsinger, R. (1999). Competency Models: Pinpointing critical success factors in
organizations. San Francisco: Jossey-Bass.
Lux, M.; Tochtermann, K. & Granitzer, M. (2006),Retrieval basierend auf Semantischen Metadaten, in
Franz Lehner; Holger Noesekabel & Peter Kleinschmidt, ed., Multikonferenz
Wirtschaftsinformatik 2006, Gito-Verlag, Passau, Germany.
Maes, Pattie (1994). Agents that reduce work and information overload. Communications of the ACM,
37, 7, 31–40.
Mandl, T. (2001), Tolerantes Information Retrieval. Neuronale Netze zur Erhцhung der Adaptivität und
Flexibilität bei der Informationssuche, PhD thesis, University Of Hildesheim.
Martin, D., Burstein, M., Hobbs, J., Lassila, O., McDermott, D., McIlraith, S., Narayanan, S., Paolucci,
M., Parsia, B., Payne, T. , Sirin E., Srinivasan, N., & Sycara, K. (2004). Owl-s: Semantic
markup for web services, w3c member submission 22 November 2004.
http://www.w3.org/Submission/OWL-S/.
page
166
Maurer, H. & Tochtermann, K. (2002). On a New Powerful Model for Knowledge Management and its
Applications, in: Journal of Universal Computer Science JUCS, pp. 85-96.
Maus, H. (2001). Workflow context as a means for intelligent information support, in V. Akman; P.
Bouquet; R. Thomason; R.A. Young (Ed.): Modeling and Using Context. 3rd Int. and
Interdisciplinary Conference, Springer LNAI 2116.
Merriam, S. B. and Caffarella, R. S. (1991) Learning in Adulthood. A comprehensive guide, San
Francisco: Jossey-Bass
Merrill, M. D. (2001). Components of instruction: toward a theoretical tool for instructional design.
Instructional Science, 29, 4-5, 291-310.
Merrill, M. D. (2002). First principles of instruction. Educational technology research and development,
50, 43-59.
Metsker S.J., & Wake W.C., (2006). Design Patterns in Java, 2nd edition. Boston: Pearson.
Microsoft (2007). Single Sign-On in Windows 2000 Networks, http://technet.microsoft.com/enus/library/Bb742456.aspx
Middleton, S. E., Shadbolt, N. R. & De Roure, D. C. (2004) Ontological User Profiling in Recommender
Systems, in: ACM Transactions on Information Systems, 22 (1), 54-88.
Montaner, M., López, B. & De la Rosa, J. L. (2003). A Taxonomy of Recommender Agents on the
Internet, in: Artificial Intelligence Review, 19, 4, 285-330.
Moore, D. S., & McCabe, G. P. (1997). Introduction to the practice of statistics (2nd ed.). New York:
Freeman.
Murray, T., 1999. Authoring Intelligent Tutoring Systems: An Analysis of the State of the Art.
International Journal of Artificial Intelligence in Education, 10, 98-129.
National O*NET Consortium (2005). O*NET OnLine.
New, G. E. (1996). A three-tier model of organizational competencies. Journal of Managerial
Psychology, 11, 8, 44-51.
Nordsieck, Fritz (1972): Betriebsorganisation. 4. edition. Stuttgart : Poeschel.
Norman, D.A. (1988). The design of everyday things. New York: Doubleday.
O'Donnell, A. M., Dansereau, D. F., & Hall, R. H. (2002). Knowledge maps as scaffolds for cognitive
processing. Educational Psychology Review, 14, 71-86.
Oberweis, A. (2005). Person-to-Application Processes: Workflow Management. In Dumas et al. (Eds.).
Process aware information systems., chapter 2, pages 21–36. John Wiley & Sons, Inc.
Oppermann, R. (2005). From User-adaptive to Context-adaptive Information Systems, in: i-com
Zeitschrift für interaktive und kooperative Medien, 9 (3), 4-14.
Pernici, B., Locatelli, P., & Marinoni, C. (2006). The eCCO System: An eCompetence Management
Tool Based on Semantic Networks. In R. Meersman, Z. Tari, & P. Herrero (Eds.): On the Move
to Meaningful Internet Systems 2006: OTM 2006 Workshops (pp. 1088-1099). Heidelberg:
Springer.
Perugini, S., Goncalves, M. A. & Fox, E. A. (2004). Recommender Systems Research: A ConnectionCentric Survey, in: Journal of Intelligent Information Systems, 23, 2, 107-143.
Piaget, J. & Inhelder, B. (1973). Memory and intelligence. NY: Basic Books.
Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1993). Reliability and predictive validity
of the motivated strategies for learning questionnaire (mslq). Educational and Psychological
Measurement, 53, 801-813.
page
167
Platt, J.C. (1998). Sequential minimal optimization: A fast algorithm for training support vector
machines, Microsoft Research Technical Report MSR-TR-98-14.
http://research.microsoft.com/en-us/um/people/jplatt/smoTR.pdf
Prahalad, C. & Hamel, G. (1990). The Core Competence of the Corporation. Harvard Business
Review, 68, 3, 79-91.
Puustinen, M., & Pulkkinnen, L. (2001). Models of self-regulated learning: A review. Scandinavian
Journal of Educational Research, 45, 269-286.
Quinlan, J.R. (1986). Induction of Decision Trees, Machine Learning, 1(1), 81-106.
Rath, A., Kröll, M., Lindstaedt, S. N. & Granitzer, M. (2007) Low-Level Event Relationship Discovery
for Knowledge Work Support, in N. Gronau (Ed.): Proccedings of the 4th Conference on
Professional Knowledge Management WM2007, pp. 227-234, Berlin: GITO-Verlag.
Ratnaparkhi A., Maximum Entropy Models for Natural Language Ambiguity Resolution, Ph.D. thesis,
University of Pennsylvania, Philadelphia, PA, 1998.
Resnik, P. (1995). Using information content to evaluate semantic similarity in a taxonomy. In
Proceedings of the 14th International Joint Conference on Artificial Intelligence, Vol. 1, 448453, Montreal, August 1995.
RFC4122 2005. A Universally Unique IDentifier (UUID) URN Namespace,
http://www.ietf.org/rfc/rfc4122.txt
Rhodes, B.J. (2000). Margin notes: Building a contextually aware associative memory. In Proceedings
of the International Conference on Intelligent User Interfaces (IUI ’00), pages 9–12, 2000.
Rhodes, B.J.; & Starner. Th., (1996). The remembrance agent. In AAAI Symposium on Acquisition,
Learning and Demonstrations, 1996.
Rist, T.; Andre, E. & Mugller, J. (1997). Adding animated presentation agents to the interface. In
Proceedings of the International Conference on Intelligent User Interfaces (IUI ’97), pages 79–
86, 1997.
Robinson, D.G. (2003): Skill and Performance: They are not equal. Apartment Professional Magazine
Rocha, C.; Schwabe, D. & Aragao, M. P. (2004). A hybrid approach for searching in the semantic web,
in 'WWW '04: Proceedings of the 13th international conference on World Wide Web', ACM
Press, New York, NY, USA, pp. 374--383.
Rospocher, M., Ghidini, C., Serafini, L., Kump, B., Pammer, V., Lindstaedt, S.N., Faatz, A. & Ley, T.
(2008). Collaborative Enterprise Integrated Modelling. In: Gangemi, A., Keizer, J., Presutti, V.
& Stoermer, H. (Eds.): Proceedings of the 5th Workshop on Semantic Web Applications and
Perspectives (SWAP2008), Rome, Italy, December 15-17, 2008. CEUR Workshop
Proceedings 426 CEUR-WS.org 2008
Rumsey, D.(1997). The Changing Workplace of the 21st Century - Implications for Educators and
Trainers. AARE Conference, Researching Education in New Times, Brisbane
Salton, G. (1963), Associative Document Retrieval Techniques Using Bibliographic Information.
Journal of the ACM, 10, 4, 440--457.
Salton, G. (1968). Automatic Information Organization and Retrieval. McGraw Hill.
Sánchez-Alonso, S. & Frosch-Wilke, D. (2007). An Ontological Representation of Competencies as
Codified Knowledge. In M.-A. Sicilia(Ed.), Competencies in Organizational E-Learning.
Concepts and Tools. (pp. 169-184). Hershey, London, Melbourne, Singapore: Information
Science Publishing.
Sauermann, L. et al. (2006). Semantic Desktop 2.0: The Gnowsis Experience, in: Proceedings of the
International Semantic Web Conference (ISWC), 887-900.
page
168
Sauermann, L.; Bernardi, A. & Dengel, A. (2005). Overview and Outlook on the Semantic Desktop. In
Proceedings of the 1st Workshop on The Semantic Desktop at the ISWC 2005 Conference.
Scheir, P. & Lindstaedt, S. N. (2006). A network model approach to document retrieval taking into
account domain knowledge. In: Martin Schaaf & Klaus-Dieter Althoff, (ed.). LWA 2006. Lernen
- Wissensentdeckung - Adaptivitдt, 9.-11.10.2006 in Hildesheim. Universitдt Hildesheim, pp.
154-158.
Scheir, P., Lindstaedt, S. & Ghidini, C. (2008). A network model approach to retrieval in the Semantic
Web. International Journal on Semantic Web and Information Systems (IJSWIS). 4(4), 56-84.
Scheir, P., Ghidini, C. & Lindstaedt, S. (2007). Improving Search on the Semantic Desktop using
Associative Retrieval Techniques Proceedings of I-MEDIA 2007 and I-SEMANTICS 2007,
Graz, Austria, September 5-7, 2007, 221-228.
Scheir, P.; Pammer, V. & Lindstaedt, S. N. (2007). Information Retrieval on the Semantic Web - Does
it exist? In 'LWA 2007, Lernen - Wissensentdeckung - Adaptivität, 24.-26.9. 2007 in
Halle/Saale (accepted for publication)'.
Schilit, B. (1995). A System Architecture for Context-Aware Mobile Computing. PhD thesis, Columbia
University, 1995.
Schilit, B. Adams, N. & Want, R. (1994). Context-aware computing applications. In IEEE Workshop on
Mobile Computing Systems and Applications, Santa Cruz, CA, US, 1994.
Schippman, J. S., Ash, R. A., Battista, M., Carr, L., Eyde, L. D., Hesketh, B., Kehoe, J., Pearlman, K.,
Prien, E. P. & Sanchez, J. I. (2000). The practice of competency modeling. Personnel
Schmidt, A. & Kunzmann, C. (2006). Towards a Human Resource Development Ontology for
Combining Competence Management and Technology-Enhanced Workplace Learning. On
the Move Conferences 2006, OnToContent Workshop, Montpellier,
LNCSSpringerHeidelberg/Berlin:.
Schmidt, A. (2005). Bridging the Gap between Knowledge Management and E-Learning with ContextAware Corporate Learning. In: K.-D. Althoff; A. Dengel; R. Bergmann; T. Roth-Berghofer
(eds.) Professional Knowledge Management. Third Biennal Conference, WM 2005,
Kaiserslautern, Germany, April 2005. Revised Selected Papers, Lecture Notes in Artificial
Intelligence (LNAI)Springer, pp. 203-213.
Schmidt, A.; Aidoo, K.A.; Takaluoma, A.; Tuomela, U.; Van Laerhoven, K.; Van de Velde, W. (1999).
Advanced interaction in context. In HUC, pages 89–101.
Schreiber, A.Th., Akkermans, J.M., Anjewierden, A.A., de Hoog, R., Shadbolt, N.R., Van de Velde, W.,
and Wielinga, B.J. (2000). Knowledge engineering and management: The CommonKADS
methodology. Cambridge Massachusetts, MIT Press
Schwab I, & Kobsa A. (2002). Adaptivity through Unobstrusive Learning. KI Special Issue on
Adaptivity and User Modeling, 3, pp 5-9.
Sebastiani, F. (2002), 'Machine learning in automated text categorization', ACM Computing Surveys
(CSUR) 34(1), 1--47.
Selker, Ted (1994). Coach: A teaching agent that learns. Communications of the ACM, 37, 7, 92–99
Shah, U.; Finin, T. & Joshi, A. (2002). Information retrieval on the semantic web. In 'CIKM '02:
Proceedings of the eleventh international conference on Information and knowledge
management', ACM Press, New York, NY, USA, pp. 461-468.
Sharifian, F. & Samani, R. (1997),Hierarchical spreading of activation, in Farzad Sharifian, ed.,'Proc.
of the Conference on Language, Cognition, and Interpretation', IAU Press, , pp. 1--10.
Shneiderman, B. (2002). User Interface Design. Reading (MA), Addison-Wesley.
page
169
Sicilia, M. (2005a). Ontology-Based Competency Management: Infrastructures for the Knowledgeintensive Learning Organization. In M. Lytras & A. Naeve (Eds.), Intelligent Learning
Infrastructures in Knowledge Intensive Organizations: A Semantic Web perspective (pp. 302324). Hershey: Idea Group.
Sicilia, M. (2005b). On the representation of change according to different ontologies of learning.
International Journal of Learning and Change, 1(1), 66-79.
Sicilia, M. A. (Eds.) (2007). Competencies in Organizational E-Learning. Hershey: Information Science
Publishing.
Simons, PRJ. (2000). Towards a constructivistic theory of self-directed learning. In Straka, G.A. (Eds.).
Conceptions of self-directed learning: Theoretical and conceptional considerations, (pp. 155169). Münster, Germany: Waxmann
Smith, H. & Fingar, P. (2002). Business Process Management: The Third Wave. Tampa, Florida :
Meghan-Kiffer Press.
Spear, G.E. & Mocker, D.W. (1984). The organizing circumstance: Environmental determinants in selfdirected learning. Adult Education Quarterly, 43, 1-10.
Stojanovic, N.; Maedche, A.; Staab, S.; Studer, R. & Sure, Y. (2001),SEAL: a framework for
developing SEmantic PortALs. In 'Proceedings of the First International Conference on
Knowledge Capture (K-CAP 2001), October 21-23, 2001, Victoria, BC, Canada', pp. 155-162.
Stojanovic, N.; Studer, R. & Stojanovic, L. (2003). An Approach for the Ranking of Query Results. In
the Semantic Web., in 'International Semantic Web Conference', pp. 500-516.
Teltzrow, M. & Kobsa, A. (2004). Impacts of User Privacy Preferences on Personalized Systems: a
Comparative Study, in C.-M. Karat; J. Blom; J. Karat (Ed.): Designing Personalized User
Experiences for eCommerce, pp. 315-332, Dordrecht: Kluwer Academic Publishers.
U.S. Department of Commerce (2000). Safe Harbour Privacy Principles, Washington, DC.
Ulbrich, A., Scheir, P., Lindstaedt, S. N. & Görtz, M. (2006) A Context-Model for Supporting WorkIntegrated Learning, in W. Nejdl & K. Tochtermann (Ed.): Innovative Approaches for Learning
and Knowledge Sharing (LNCS 4227), pp. 525--530, Berlin: Springer.
Van Assche, F. (2007). Linking Content to Curricula by using Competencies. Paper presented at the
First International Workshop on Learning Object Discovery & Exchange (LODE'07),
Crete/Greece, September 18, 2007. Available from: http://fire.eun.org/lode2007/lode11.pdf
Van den Berg, P. T. (1998). Competencies for Work Domains in Business Computer Science.
European Journal of Work and Organisational Psychology, 7(4), 517-531.
Van der Aalst, W.M.P. & ter Hofstede, A.H.M. (2002). YAWL: Yet Another Workflow Language.
Technical Report. Brisbane: Queensland University of Technology.
Vrandecic, D., Sofia Pinto, H., Sure, Y., & Tempich, C. (2005). The diligent knowledge processes.
Journal of Knowledge Management, 9, 5, 85–96.
Want, R.; Hopper, A.; Falcao, V.; Gibbons, J. (1992). The active badge location system. ACM
Transactions on Information Systems, 10, 1, 91–102.
Want, R.; Schilit, B.N.; Adams, N.I.; Gold, R.; Petersen, K.; Goldberg, D.; Ellis, J.R.; & Weiser, M.
(1995). An overview of the parctab ubquitous computing environment. IEEE Personal
Communications, 2, 6, 28–43.
Weber, G. & Brusilovsky, P. (2001). ELM-ART: An Adaptive Versatile System for Web-based
Instruction. International Journal of Artificial Intelligence in Education (JAIED), 12, 4, 351-384.
Weinert, F. E. (1999). Concepts of competence. DeSeCo Expert Report. Neuchatel: Swiss Federal
Statistical.
Weiser, Mark (1991). The Computer for the 21st Century. Scientific American, 265, 3, 94–104.
page
170
Wessner, M. (2005). Kontextuelle Kooperation in virtuellen Lernumgebungen, Lohmar, Germany:
Josef Eul Verlag.
Winograd, T. (2001). Architectures for context. Human Computer Interaction, 16, 2/4, 401–419
Wolpers, M., Martin, G., Najjar, J., & Duval, E. (2006). Attention Metadata in Knowledge and Learning
Management. In: Proceedings of the I-Know 2006, Graz, Austria.
Wöls, K., Kirchpal, S., & Ley, T. (2003). Skills Management - An 'all-purpose' Tool? Proceedings of IKnow '03 - 3rd International Conference on Knowledge Management (pp. 138-143). Graz:
Know-Center.
Won, M. & Pipek, V. (2003). Sharing Knowledge on Knowledge - The eXact Peripheral Expertise
Awareness System. Journal of Universal Computer Science , 9, 12, 1388-1397.
Wu, Z. & Palmer, M. (1994). Verb semantics and lexical selection. 32nd Annual Meeting of the
Association for Computational Linguistics, pages 133–138, Las Cruces, New Mexico.
Yimam-Seid, D. & Kobsa, A. (2003). Expert-Finding Systems for Organizations: Problem and Domain
analysis and the DEMOIR approach. Journal of Organizational Computing and Electronic
Commerce, 13(1), 1-24.
Zhang, L.; Yu, Y.; Zhou, J.; Lin, C. & Yang, Y. (2005). An enhanced model for searching in semantic
portals. In WWW '05: Proceedings of the 14th international conference on World Wide Web,
ACM Press, New York, NY, USA, pp. 453--462.
Zien, A., Brefeld, U., and Scheffer, T. 2007. Transductive support vector machines for structured
variables. In Proceedings of the 24th international Conference on Machine Learning (Corvalis,
Oregon, June 20 - 24, 2007). Z. Ghahramani, Ed. ICML '07, vol. 227. ACM, New York, NY,
1183-1190. DOI= http://doi.acm.org/10.1145/1273496.1273645
Zimmerman, B. J., & Schunk, D. H. (Eds.). (2001). Self-regulated learning and academic achievement.
London: Lawrence Erlbaum Associates.
page
171
8 Glossar y
Associative network: An associative network is a generic network of information items represented
by nodes. Relations between information items are expressed by edges interconnecting the
nodes. Weights on the edges can be used to represent the strength of the relation between
the information items.
Associative retrieval: Associative Retrieval is a form of information retrieval which tries to find
relevant information by retrieving information that is by some means associated with
information that is already known to be relevant.
Cooperation Profile: A cooperation profile is part of the user profile and contains information on
personal preferences and habits regarding collaboration.
Collaborative Community: A collaborative community is a social group of knowledge workers within
the context of a larger organisational entity which share knowledge in work or learning
situations.
Communication Channel: A communication channel is a medium which transfers the communication
message from sending to receiving communication partners. Communication channels can be
divided in synchronous and asynchronous channels. Synchronous communication channels
connect two or more partners at the same time. The communication message will be delivered
directly with a short medium based delay. Asynchronous communication channels do not
connect communication partners immediately. The communication message will be saved by
the medium until it is fetched by the receiver.
Data Object Repository: ought to be part of APOSDLE‟s underlying knowledge infrastructure and
deals with managing Repositories and Repository Objects.
Domain model: A formal model of one domain of knowledge which is relevant to the knowledge
worker. “One” domain of knowledge stresses the fact that the knowledge worker might move
from one domain of knowledge to another. “Formal model” refers to the fact that the model is
machine-readable. Human-readable definitions of model entities may be provided for
convenience. The domain model consists of definitions of basic domain concepts, relations
between them and restrictions on both concepts and relations (axioms)
Knowledge Artefact: a digital entity which has been created in a knowledge-based activity. Examples
for Knowledge Artefacts is a (part of a) requirements engineering document or a scene in a
video. Every Knowledge Artefact in APOSDLE is addressable via a unique identifier.
Knowledge Artefacts can be described by a set of concepts from a formal model and related
to other Knowledge Artefacts. They can be parts of documents, as long as they are
addressable via a unique identifier.
Knowledge Indicating Event (KIE): these denote user activities which indicate that the user has
knowledge about a certain topic
Knowledge State: A knowledge state is given by a set of tasks and the associated set of knowledge
and skills needed to perform these tasks. A learner is said to be in one knowledge state if
he/she has been sufficiently engaged with this set of tasks and the associated skills and
knowledge either through performing the tasks or through learning. This state is then called
the competence of the learner. Because of the prerequisite relation on learning goals, not all
knowledge states are permissible.
Knowledge worker: an employee of an organisation whose essential operational and value creating
tasks consist in the production and distribution of knowledge.
Knowledge seeker: a person who encounters a learning need and intends to contact other people
concerning this learning need.
page
172
Knowledgeable person: a person who possess above average knowledge about a topic or a set of
topics in the domain.
Learning Goal Interpretation: The learning goal interpretation of a task is the set of knowledge and
skills which are required at the minimum for performing the task
Learning Goal Type: The type of knowledge or skill which the learner wants to acquire. This is
expressed as a cognitive activity which a learner can perform after acquiring the knowledge or
skill (i.e. remember, understand, apply, create) with regard to a certain element of the domain
ontology.
Learning Goal: One skill or element of knowledge which a learner wants to acquire at a certain point
in time. A learning goal is expressed as a tuple of a domain model element and a learning
goal type.
Learning hint: A suggestion for how a learner could process learning material. It consist in several
cases of an engagement activity with a what? and a Why? part. In possible, references are
provided to support for learning that are available elsewhere in the APOSDLE system.
Learning History: The collection of all learning goals which the learner has been engaged with in the
past. This may have happened either by learning (i.e. by engaging in learning events) or by
applying certain knowledge and skills (i.e. by performing a task).
Learning Need: A set of learning goals which the learner needs to tackle for performing a certain task.
A learning need arises from considering the knowledge and skills required for a certain task,
and the knowledge and skills the learner has available. In APOSDLE, the latter is
approximated by the learning goals the learner has engaged with in the past.
Learning Path: The learning path is a sequence of topics for which knowledge can be acquired.
Prerequisite Learning Goal: We assume that learning goals are (partially) ordered by a prerequisite
relation. The prerequisite relation is interpreted as follows: before tackling a certain learning
goal, a learner needs to have tackled all learning goals that are regarded as its prerequisites.
Repository Object: a representation of one document retrieved from a back-end system and stored in
the APOSDLE System.
Repository: a back-end system accessible by the APOSDLE System by using existing
communication protocols and interfaces.
Semantic desktop: The Semantic Desktop paradigm (Sauermann et al., 2005; Decker and Frank,
2004) aims at applying technologies developed for the Semantic Web to desktop computing to
finally provide for a closer integration between (semantic) web and (semantic) desktop.
Semantic metadata: Semantic metadata is metadata that stems from knowledge representations.
The unique characteristic of semantic metadata opposed to classical metadata is that
elements of knowledge representation that are used as metadata can be related by arbitrary
semantic relations (Lux et al., 2006).
Task Learning Goal Mapping: A mapping which assigns to each task all skills and elements of
knowledge (learning goals) which are required for a successful performance of this task and
which are in the learning domain.
Task: In the context of work-integrated learning, a task is an activity which has to be accomplished
within a certain period of time, and which requires specific knowledge and skills to be
performed.
Task-Learning Goal Structure: The collection of all knowledge states and their associated task
representations.
page
173
User Context: In our understanding context is the substrate in which events occur and which allows a
meaningful interpretation of data. Context is characterized by a relevant subset of all
surrounding potentially dynamic (e.g. temporal, environmental) information and external and
internal conditions. To assess the relevance of information a goal must be considered, which
is bound to a knowledge worker. The context is characterized by a potentially dynamic subset
of all environmental features. Within APOSDLE, we restrict the information considered for the
User Context to those aspects where the acquisition of this information in an IT-based model
28.
is feasible. The user context is implemented in the user profile
User Profile Service (User Profile Manager): The user profile service (UPS) is the software
component, which stores and maintains user profiles. The UPS offers user-focused services
to the other parts of the APOSDLE Platform. The UPS makes four different types of services
available to the platform: logging services, inference services, production services and control
29
services .
User Profile: A user profile is a data structure, which represents an individual users‟ personal history
and current context with respect to the user's work-, learning and collaboration-related
experiences. While the user context is a conceptual formal model, the user profile is the data
structure that implements parts of the user context. A user profile of a given user consists of
30.
four different types of data: user data, environment data, usage data and inferred data
28
https://aposdle.itc.it/glossary/index.php/User_Context
29
https://aposdle.itc.it/glossary/index.php/User_profile_service
30
https://aposdle.itc.it/glossary/index.php/User_Profile
page
174

Conceptual Architecture including Component Evaluation

Transcription

Similar documents

advantage - Join RE/MAX

Fayetteville Observer - Program in the Humanities

Draft brochure.-update

January 2016 Dreams Come True at Participant

Latent dwelling structures in the Final Palaeolithic: Niederbieber IV

Using Semantic Technologies for Resource Allocation in Computing

Semantic annotation of satellite images

Preserving our

here - Remember the Triangle Fire Coalition