UNIVERSIT`A DEGLI STUDI DI TRENTO “TOUCHING DANTE”: A

UNIVERSITÀ DEGLI STUDI DI TRENTO
FACOLTÀ DI SCIENZE MATEMATICHE, FISICHE E NATURALI
Master of Science in Computer Science
Thesis
“TOUCHING DANTE”: A PROXIMITY-BASED
PARADIGM FOR TABLETOP BROWSING
prof. Antonella de Angeli
Co-advisor: prof. Massimo Zancanaro
Advisor:
Candidate:
Silvia Bordin
ACADEMIC YEAR
2010 - 2011
First of all, I would like to thank the I3 unit at Fondazione Bruno Kessler for giving
me the opportunity to have a rich, hands-on experience in a stimulating, collaborative
and friendly environment, the team of Practix for always being available for
suggestions and technical support, and the users who participated in the prototype
evaluation.
Then, I would like to thank my family: in spite of all the hard times, they were always
there for welcoming me back home and pushing me to do my best.
I would like to thank Andrea, for always hugging me when I most needed it.
I would like to thank the “pious women”, my sisters during these years at university: I
will never forget our happy, insane moments together.
Finally, I would like to thank all the people I met during Erasmus for showing me
how vast and variegated the world is, and all the people I met during university for
sharing this path of growth.
ii
Contents
1
Introduction
1
2
Related work
2.1 Natural User Interfaces . . . . . . . . . . . . . . . . . . . . .
2.2 Interactive tabletops . . . . . . . . . . . . . . . . . . . . . . .
2.2.1 Collaborative information exploration in public spaces
2.2.2 Collaborative learning . . . . . . . . . . . . . . . . .
2.3 Information visualization and visual analytics . . . . . . . . .
2.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
2
2
3
4
5
5
6
3
Towards a proximity-based paradigm
3.1 A case study: Dante’s Inferno . . . . . . . . . . . . . . . . . . . . . .
3.2 The design procedure . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Implemented principles of interaction design . . . . . . . . . . . . . .
10
10
11
12
4
The prototype
4.1 Tabletop devices . . . . . . . . . . . . . . . . . . . . .
4.1.1 Microsoft Surface . . . . . . . . . . . . . . . .
4.2 The platform . . . . . . . . . . . . . . . . . . . . . .
4.3 Technicalities . . . . . . . . . . . . . . . . . . . . . .
4.3.1 Inheritance of controls . . . . . . . . . . . . .
4.3.2 Card snapping to the text page . . . . . . . . .
4.3.3 Data serialization . . . . . . . . . . . . . . . .
Extracting dialogs . . . . . . . . . . . . . . .
4.4 Problems encountered . . . . . . . . . . . . . . . . . .
4.4.1 Dynamic dimensions . . . . . . . . . . . . . .
4.4.2 Ambiguities: coordinates and dimension setting
4.4.3 Incomplete customizability of components . .
4.4.4 Special effects . . . . . . . . . . . . . . . . .
4.4.5 Data cleanup . . . . . . . . . . . . . . . . . .
19
19
19
21
21
23
25
29
29
33
33
33
34
35
35
iii
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
5
Evaluation
5.1 Co-discovery learning . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Semi-structured interview . . . . . . . . . . . . . . . . . . . . . . . .
5.3 Attrak-Diff2 questionnaire . . . . . . . . . . . . . . . . . . . . . . .
36
37
39
42
6
Conclusions and future work
44
Bibliography
47
iv
Chapter 1
Introduction
This thesis stems from what learned and experimented during my internship by the I3
unit of the Fondazione Bruno Kessler in Trento, Italy. The main contribution is the
design of a proximity-based interaction paradigm for browsing complex information.
The paradigm builds on the intrinsic peculiarities of interactive tabletops, namely their
physical spatial dimension, direct-touch input modality and affordance for multi-user
interaction. In order to assess the validity of such proposal, I sought a topic that could
be addressed by a prototypical application aimed at a realistic public setting. After
some research, I chose Dante’s Inferno, due to the complexity, vastity and universality
of such writing; furthermore, previous research had already made available partiallystructured data about the relationships between characters.
The prototype application was evaluated by four couples of users, so as to qualitatively
assess whether the interaction paradigm would help users in browsing a collection of
data collaboratively. Preliminary results are encouraging: user studies confirmed that
the proposed paradigm can be easily understood, that it is perceived as simple while
still engaging, that it allows for collaborative learning while not forcing it, and that it
helps highlighting and remembering connections among data.
Future directions for this work include the integration of users’ suggestions in the
prototype (especially in light of the considerations about the need to bridge the current
adaptation phase to touch-enabled devices), the application of the paradigm to different
settings (both public and business) and the extension of the concept of proximity also
to tag-recognizable objects.
1
Chapter 2
Related work
This chapter presents the related work which provides the grounding for my contribution. I will first of all introduce Natural User Interfaces, which can in particular be
applied on interactive tabletops. I will then go more in detail about how interactive
tabletops placed in public spaces can facilitate collaborative information exploration
and learning. Furthermore, I will provide a brief introduction to information visualization and visual analytics. In conclusion, I will present some examples of software that
are close to what developed for the present thesis.
2.1
Natural User Interfaces
This thesis can be contextualized within the realm of Natural User Interfaces, one of
the most recent and interesting evolutions in computing and human-computer interaction. A Natural User Interface provides “a clear and enjoyable path to unreflective
expertise in its use. It makes skilled behavior seem natural in both learning and expert
practice” [35]. In other words, Natural User Interfaces allow users to feel right from the
start that “their performance is ideal and that it seems effortless and graceful” [35], i.e.
they are immediately and continuously successful and rewarded. Furthermore, Natural User Interfaces “promise to reduce the barriers to computing still further, while
simultaneously increasing the power of the user, and enabling computing to access still
further niches of use” [35].
In this context, the word natural refers to “the user’s behavior and feeling during the
experience” of interaction; direct, multi-touch devices such as the tabletop can be considered as natural input interfaces since, for instance, they remove the intermediate
layer of control devices such as mouse and keyboard and exploit several types of human memory. In fact, tabletops provide a physical space for the user to organize information and require physical gestures to be operated: such combination has been widely
2
acknowledged to reinforce the persistence of learned information. The overall goal of
Natural User Interfaces is indeed to leverage “the potential of modern technologies to
better mirror human capabilities” [35].
It is paramount to realize that, in order to take full advantage of the capabilities provided by this new paradigm, it does not suffice to convert or wrap designs targeted at
traditional GUIs, nor does mimicking some other experience, even if it is familiar to
the user. Rather, it is better to understand what the unique features of Natural User Interfaces are and in which niches they best fit in, by integrating technologies and design
that rely on natural gestures that can be quickly discovered [5]. To this end, this thesis
proposes a new interaction paradigm to be used at a tabletop interface.
2.2
Interactive tabletops
Multi-user multi-touch tabletop displays are becoming more and more affordable and
widely available: therefore, interest has been growing towards their “potential to support natural simultaneous interaction of multiple individuals with digital materials”
[28]. Tabletop displays combine a large, shareable display area with a physical arrangement that leverages both on the well-established experience humans have in collaborating around tables and on the fluid, dynamic interaction that takes place in such
context [33]. Furthermore, such a configuration promotes discussion and interaction
among peers by providing equal access and shared ownership over the activity at hand.
As stated in [23], such social benefits combined with the productivity advantages of
computing can be applied to a variety of settings: for instance, Microsoft Surface,
the tabletop platform addressed by the present thesis, is admittedly targeted at entertainment in public places or in commercial settings; interactive tabletops have started
appearing in restaurants or bars; some studies, such as [19], have investigated their
impact in a business setting, and so forth.
As claimed in [30], “where it is considered desirable for a group to participate in
collaborative activities that involve using and creating an assortment of representations
[...], then horizontal interactive surfaces that closely couple the resources used and
created during the various activities are most effective”. In fact, the study described
in the article highlights that cohesive collaboration is enticed by tabletop displays, as
they provide the following benefits:
• a shared representation of data the group can refer to and which facilitates contribution of all members to problem solving;
• the support for more fluid interaction, given by the tight coupling of the information to be accessed to their graphical representation;
3
• a physical surface affording easy role swapping, which encourages users to put
forward alternatives for discussion.
Furthermore, the technology of interactive tabletops can enable even users with little if
no computer experience (for instance, the elderly) to begin interaction with computing
technology quickly and in a satisfying way [12].
2.2.1
Collaborative information exploration in public spaces
The capability of tabletops to augment information exploration and analysis, and their
increasing affordability suggests that they are very likely to become commonplace in
several settings in the near future. For instance, they might help domain experts in their
workplace. As an example, let us think of business analysts looking at the trends in
financial data, or at biologists analysing the outcomes of laboratory experiments. They
are also increasingly available in public spaces, such as libraries or museums, thus
conveying information in an interactive and potentially engaging way while leveraging
on their ease of use [20].
In this thesis I focus on interaction in public spaces, where the goal is to “attract
people’s attention, draw them closer to the installation and promote lightweight information exploration that leads to serendipitous discoveries, some reflection on the
presented data and/or active discussion of this data with peers” [20]. This situation
is aimed at an audience whose socio-cultural background, interests, and level of acquaintance with technology can be highly diverse and unpredictable. Moreover, a key
point is in the serendipity of the interaction. Such audience typically does not have any
clearly defined questions to pose to the system, nor has it specific goals with respect to
it: rather, it is willing to explore information mostly based on spontaneous interest.
These considerations clearly point out that “interaction techniques need to be designed
with a walk-up-and-use scenario in mind” [20]: users are not likely to read lengthy
or complex instructions to operate the installation, but will rather try to figure out by
themselves how to do it in an occasional and possibly single interaction experience,
which should thus provide immediate and rewarding feedback.
Another fundamental feature of the public spaces scenario is group interaction and, in
particular, its orientation towards social, collaborative exploration of data: users are
likely to explore visualizations in parallel (especially in the case of groups of people
not knowing each other), possibly sharing their insights also verbally from time to
time. As a result, a large amount of research has been devoted to investigating how
interactive tabletops could be integrated into a didactical setting, i.e. in collaborative
learning.
4
2.2.2
Collaborative learning
As defined in [13], the goal of computer-supported collaborative learning (CSCL) is to
“design software tools and collaborative environments that facilitate social knowledge
construction via a valuable assortment of methodologies, theoretical and operational
definitions, and multiple structures”.
Several studies proved the effectiveness of shared tabletop interfaces in supporting collaborative learning. According to [10], such benefit is clearer if we consider not only
the outcome of the learning per se, but also how tabletop-facilitated collaboration affects the learning process: it is widely agreed that such technology “is enjoyable to use,
promotes playfulness, can encourage equity of participation and can lead to learning”
[10] and that the coupling of verbal and physical interaction both with the machine and
with other users can boost remembrance and favors exploration and discussion.
One way to foster around-the-table collaboration is the implementation of cooperative
gestures. These are defined in [24] as “interactions where the system interprets the
gestures of more than one user as contributing to a single, combined command”. They
can be used in multi-user interaction techniques to “enhance users’ sense of teamwork,
increase awareness of important system events, facilitate reachability and access control” and so forth.
2.3
Information visualization and visual analytics
Another area that contributed to the development of the present thesis is visual analytics, which, as stated in [21], can be defined as “an iterative process that involves
information gathering, data preprocessing, knowledge representation, interaction and
decision making” and which aims at gaining insight into a problem described by a
vast amount of heterogeneous and possibly contradictory or incomplete data by trying
to combine automatic analysis methods with human intuition and background knowledge.
Visual analytics is a branch of the more general field of information visualization,
which concerns the communication of abstract, relevant data through the use of interactive interfaces; its goals are mainly the execution of either confirmatory or exploratory analysis and the efficient and effective presentation of obtained results. One
of the first examples of a system implementing collaborative visual analytics is Cambiera [19], developed for a tabletop display and supporting the analysis of large text
documents collections; by providing a shared space, it combines both the benefits of
face-to-face interactions and of artefact-centered information sharing.
5
2.4
Examples
An example of interactive tabletop application allowing to explore a network of information is mæve [26]. It is in many ways similar to the prototype developed for this
thesis: in fact, mæve is an interactive system which aimed at enhancing the discovery
of architectural projects presented at the Entryville student competition in the 2008
Venice Biennale.
As described in [2], each exhibit is described in a paper card marked with a tag; such
cards can then be placed on an interactive tabletop, which can recognize their tags
and present a concept network consisting of related projects, metadata and mutually
shared concepts as can be seen in figure 2.2. This representation is also proposed on a
supplemental wall projection, enriched with further media and text displays: the whole
configuration can be seen in figure 2.1. As a result, this tangible system allows users
to easily explore both the unifying themes of the exhibition and the individual features
of the various architectural projects: this promotes an informal, playful learning that
reinforces remembrance and provides an overview of the exposition, while at the same
time affording the development of social interaction.
An evaluation study has demonstrated that mæve and its innovative spatial setup provide a new browsing experience to users, “by combining tangible interaction, complex
visualization and collaborative data examination”, and invite them to interact with the
system without posing barriers or demanding too much concentration.
6
Figure 2.1: The setting of mæve.
Figure 2.2: A screenshot of the relations among architectural projects shown by mæve.
7
Similar objectives were addressed by the ImpulsBauhaus installation [1], which aimed
at the discovery and exploration of background information in the context of an exhibition about the artists belonging to the Bauhaus movement. The overall setting,
shown in figure 2.3, is similar to that of mæve. In particular, the installation shows
biographical information about such artists, linking them in a sort of social network;
furthermore, it exploits the potentialities of augmenting objects through mobile technology and tag recognition as shown in figure 2.4.
Figure 2.3: The setting of ImpulsBauhaus.
8
Figure 2.4: A screenshot of the relations among artists shown by ImpulsBauhaus.
However, more research is needed to investigate whether new interaction paradigms
would be more suitable to ease and improve user experience on Natural User Interfaces
in the public spaces scenario. This is the niche this thesis aims at, by proposing a new
interaction paradigm that will be described in detail in chapter 3.
9
Chapter 3
Towards a proximity-based paradigm
This chapter addresses the main contribution of the present thesis, that is the design
of an innovative paradigm of interaction. This piece of work was sketched during
my internship by the I3 unit at Fondazione Bruno Kessler in Trento, Italy. The main
idea of this paradigm relates to activation of interactive elements induced by physical
proximity: while users browse the interface, new information is revealed as they move
items near to each other. Moreover, the semantics of proximity and thus the effect
produced differ according to the nature of the items involved in the interaction.
3.1
A case study: Dante’s Inferno
The context in which the interaction paradigm has been instantiated has been suggested
by the availability of an ontology describing the characters of Dante Alighieri’s Inferno
and the relationships linking them. The KDD-Lab of the National Center for Research
(CNR), directed by prof. Amedeo Cappelli, aimed at making explicit and implicit concept relations emerge from a manually annotated XML version of the whole text of the
Commedia. Here, the dialog, topological, political and even linguistic relations between characters were highlighted by domain experts and composed in a sort of social
network. Such database identifies 66 characters and the dialogs that occur between
pairs of them; furthermore, the ontology provides biographical and literature information not only about them, but also about characters that are just mentioned in the text.
The knowledge mining process is described in [8], while the subsequent “application
of network analytics and visualization” is presented in [9].
10
Figure 3.1: The social networks built in [9]
As shown in [9], the KDD-Lab adopted a visual analytics methodology to represent the
multidimensional social network extracted: namely, a directed graph. Their goal was
in fact to provide organized access to the data to scholars and in general specialized
users, allowing them to have a simultaneous overview of Dante’s text, knowledge and
time. However, the amount of information and the effort required for gathering it
are enormous and it would be worthwhile to share these also with casual users, for
instance in a public library or in an exhibition about Dante and his Commedia: a niche
for taking advantage of the low barrier of Natural User Interfaces therefore emerges.
3.2
The design procedure
The design is the result of an iterative process based on prototype refinement, which
combined what was learned from a literature review, an overview of similarly-targeted
applications and the available data about Dante’s Inferno. The design was immediately
oriented to the realization of a multi-user interface, so as to exploit the affordances of
tabletops; however, since the targeted device (referenced in chapter 4) was not capable
of detecting the author of each touch, cooperative gestures were not integrated in the
design.
At first, the visualization approach outlined in [9] was considered, drafting an initial
design and interface based on graphs. However, it soon became clear that it would be
better to exploit the peculiarities of the target interactive tabletop more in depth: therefore, some low-fidelity paper prototypes were produced trying to take this into account.
11
Following this path, a different design emerged, more oriented towards interaction with
user-recognizable objects and built on top of the spatial capabilities provided by the
tabletop.
With subsequent refinements of the design, also informed by further literature examination and a deeper investigation on tabletop-based applications, higher-fidelity prototypes were produced in Adobe Illustrator and Microsoft Expression Design. Finally, as
the design converged to a stable version, a proper application began to be developed:
it will be described more in detail in chapter 4.
3.3
Implemented principles of interaction design
The main concept around which the whole design revolves is that of proximity: the
reactions of the system are triggered by the user justaposing the elements she can
interact with. Following is a description of the design which highlights how Norman’s
principles were implemented [27]. The corresponding medium-fidelity prototypes will
also be presented.
In the standby state (figure 3.2), a page reading the first canto of the Inferno is shown
together with some objects that suggest how it is possible to interact with it, i.e. cards
representing the most famous characters of the writing and a handle for the opening
of an additional panel. This choice has been inspired by the principle of visibility, i.e.
making evident to the user what the state of the system is and what action it is possible
to perform.
Feedback to user actions is given, as mentioned, in response to proximity, namely by
applying a sort of magnetism metaphor to elements. For instance, a card drawn near the
central page snaps to it (figure 3.3), highlighting the first dialog in which the character
represented in the card speaks. Each snapped card is associated with a different color
(3.4), which is used to both highlight the lines of the related character in the text and
frame the card itself: this enforces the principle of consistency.
12
Figure 3.2: The initial screen.
Figure 3.3: One card snapped to the page.
13
Figure 3.4: Two cards snapped to the page.
When a new card is snapped to the text, any previously snapped card is repelled in
case the character shown in it does not speak with the former character within the
whole writing; this suggests to the user that the two figures are somehow mutually
exclusive. Such behavior enforces the constraints principle: constraints prevent the
users from doing what they are not allowed to. Other examples of the implementation
of this principle are that elements not related to each other are not activated by mutual
proximity nor provide any feedback to the user and so do non-manipulation-enabled
elements as well.
On top of the screen there is a handle bar which affords dragging (figure 3.5). Affordances are about what is active or can be manipulated in the interface; more precisely,
they are attributes of an object that allow people to know how to use it [31]. In the case
of screen-based interfaces, we talk about ”perceived affordances, which are essentially
learned conventions”. When dragged downwards, the bar opens a panel containing selectors (figure 3.6), i.e. elements representing a category of characters. Such selectors
(which can be clearly distinguished visually from the usual cards) can be dragged in
and out of the container panel.
14
Figure 3.5: The handle bar.
Figure 3.6: The selector panel opened.
Furthermore, when a selector is drawn near a card, the magnetism metaphor is applied once again: if the character has some relation with the category represented, an
additional panel slides out of the card (figure 3.7) presenting information on such relation (for example, the political career of the character, or the poem it occurs in, or the
family it belonged to, and so forth).
Figure 3.7: A card interacting with a selector.
15
In all of these actions, the lights provide an important feedback that an active element
has a lit border that allows to tell active and inactive elements apart.
A special widget is structured as a contacts book (figure 3.8) in order to provide access
to characters directly through their name, rather than through the various categories.
The user can then select the name of a character and the related card pops out at the
bottom of the screen.
Figure 3.8: The contacts book.
Cards and selectors are the two main kinds of manipulation-enabled controls available
in the system, each with a homogeneous, recognizable look-and-feel. Furthermore,
the content of the central page can be scrolled by touching. The background is not
manipulation-enabled instead and is not responsive to the interaction with other elements, except for a special area at the bottom which is sensitive to card dropping
(figure 3.9). In fact, cards about characters suggested by the system pop out from this
area and float around here: the user can either choose to drag them upwards, in the
main interaction area, or to ignore them, in which case the cards disappear beyond the
table border after a while. The user can also return to this area the cards she wishes to
eliminate from the table.
16
Figure 3.9: The dropping area.
On top of the verses displayed, there is a header indicating which canto is currently being shown (figure 3.10). By touching such header, a popup list of all cantos will open;
each of them can then be selected with a new touch and the page will automatically
scroll up to the chosen canto.
Figure 3.10: The canto selector header.
17
Finally, for what concerns the overall layout of the application, we have that since
the central element is the page containing the text of the Commedia, the system has
been designed so as to provide a fixed orientation. In a way, this disrupts one of the
advantages of using a tabletop, namely accessibility from all sides, but at the same time
it favors readability. Furthermore, layout and spacing of items have been informed by
reviewed literature about visual analytics, even though the typical scenarios in which
the latter is applied usually involve much larger amounts of data than those at hand.
It is worth noticing that the design is not intrinsically collaborative: for instance, there
is no action which necessarily requires at least two users to collaborate in order to
perform it. This choice was made to take advantage of a larger variety of use case
scenarios: a user might not be willing to collaborate with others, or there might not
be anyone else to collaborate with; nonetheless, such a user could use the application
all the same. In contrast, it is conjectured that the commonly accepted affordance for
collaboration of tabletops will naturally foster spontaneous cooperation in the case of
two or more users that are willing to engage in a shared interaction.
18
Chapter 4
The prototype
In this chapter I will present the prototype software that was developed so as to allow
a user evaluation of the proposed design.
4.1
Tabletop devices
Direct multi-touch tables are “surfaces on which input sensing and output displays are
superimposed, and on which multiple touches can be detected simultaneously” [32].
Some of them use capacitive touch sensing: for example, Mitsubishi’s Diamond Touch
[4] can also identify which user is touching which particular location on the surface;
others rely on computer vision techniques. Their unifying trait, however, is the support
of direct-touch and gesture-based interactions.
4.1.1
Microsoft Surface
The prototype developed for this thesis targets Microsoft Surface 2.0 [3], a surface
computing platform provided by Microsoft which combines both a touch-enabled operating system and a dedicated hardware by Samsung. The current version, on which
the prototype has also been tested, consists of a 40” LED backlit LCD display with an
integrated PC running Windows 7. The PixelSense technology [6] relies on infrared
sensors and on computer vision to recognize touches and objects placed on top of the
display. With respect to its previous version, which was equipped with cameras, Microsoft Surface 2.0 is thus a much thinner table; furthermore, the interactive display
can even be mounted vertically, thus broadening the number of scenarios in which it
can be used.
19
Microsoft Surface has been explicitly designed for use in public, possibly commercial
settings by several users at once, providing the capability of sharing information and
without requiring specific previous knowledge to operate it. It was chosen as target
platform because it was available at Practix, one of FBK’s spin-offs with which the I3
unit has tight relations, and it was more advanced, reliable and vividly-colored than
the camera-equipped unit possessed by the I3 unit.
Figure 4.1: Microsoft Surface 2.0.
Figure 4.2: Interaction around the tabletop.
20
4.2
The platform
The prototype has been developed under the .NET 4.0 framework, exploiting the capabilities of the graphical library Windows Presentation Foundation (WPF). Furthermore, the Microsoft Surface SDK 2.0 has been included in the Visual Studio 2010 IDE
to handle touches and manipulations in order to take advantage of the capabilities of
Microsoft Surface. The code has been written in C# for the part concerning the logic
and in the XAML markup language for the graphical part, with some exceptions that
will be described in next section.
Microsoft Surface SDK includes an Input Simulator which converts mouse clicks into
touches: this allowed developing the prototype on an ordinary non-touch enabled laptop, without requiring the constant availability of the proper tabletop (kindly provided
by Practix, http://www.practix.net).
4.3
Technicalities
Here follow some screenshots of how the mockups introduced in chapter 3 evolved
into the prototype application.
Figure 4.3: The main screen.
21
Figure 4.4: The main screen on the tabletop.
Figure 4.5: Some details: the address book, the canto selector and the open selector panel.
22
Figure 4.6: Card snapping.
In this section I am going to present some of the most relevant solutions found to
implementation issues.
4.3.1
Inheritance of controls
In WPF, controls typically consist of two complementary parts: the graphical part is
written in XAML, a dedicated markup language, whereas the business logic is written
in C# in the so-called code-behind. However, a control created in this way cannot be
extended by further controls. My idea was that of creating different types of cards:
some plain ones, some with an image of the character, and some allowing an extension
showing additional information about the character. Since however their base behavior
and appearance was to be the same, I wanted to be able to reuse code.
23
Figure 4.7: The different types of card implemented.
Therefore, I created a base control called ICarta, entirely written in C#: this seems to
be the only way to allow inheritance of controls. ICarta is extended by CartaBase,
which represents a plain card with no image attached, and by CartaExtended,
where an image of the character (usually one of the most famous ones) is shown. The
former control is written as a regular one, partly in XAML and partly in C#; the latter is entirely written in C# instead, as it is further extended by CartaExpandable:
such control features both an image and a resizing capability so as to show information
about the related character once the card is snapped to a selector.
24
4.3.2
Card snapping to the text page
ICarta mainly contains the code that manages the manipulation of a card. In WPF,
a manipulation fires the events presented in figure 4.8:
Figure 4.8: Events fired by a manipulation in WPF.
The ManipulationStarting event occurs as the user places the finger on the object. At this point, it is possible to set the ManipulationContainer: the position
of the manipulation will be considered as relative to this element from then on. In
ICarta, the corresponding handler method has been implemented as follows:
Listing 4.1: ManipulationStarting method
1
2
3
4
5
6
7
8
9
10
11
12
v o i d Carta_ManipulationStarting ( o b j e c t sender , ManipulationStartingEventArgs e )
{
/ / t h e c a n v a s l y i n g a t t h e b a s e o f t h e whole a p p l i c a t i o n
e . ManipulationContainer = t h i s . CartaParent ;
/ / t h e b o r d e r of t h e c a rd i s h i g h l i g h t e d so as t o p r o v i d e
/ / t h e u s e r w i t h f e e d b a c k a b o u t t h e f a c t t h a t an a c t i o n r e c o g n i z e d
/ / by t h e s y s t e m i s a c t u a l l y t a k i n g p l a c e
t h i s . CardBorder . BorderBrush = C o l o r M a n a g e r . Instance . BASE_GLOW ;
e . Handled = t r u e ;
}
25
The ManipulationStarted event occurs next and reports the origin of the manipulation. Then, as long as the user moves the touched object on the screen, the
ManipulationDelta event is fired multiple times, recording the kind of transformation applied to the object. In ICarta, the corresponding handler method has been
implemented as follows:
Listing 4.2: ManipulationDelta method
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
v o i d Carta_ManipulationDelta ( o b j e c t sender , ManipulationDeltaEventArgs e )
{
/ / rotation is applied
t h i s . rotation . CenterX = t h i s . center . X ;
t h i s . rotation . CenterY = t h i s . center . Y ;
t h i s . rotation . Angle += e . DeltaManipulation . Rotation ;
/ / translation is applied
translate ( e . DeltaManipulation . Translation . X , e . DeltaManipulation . Translation . Y ) ;
/ / t h e p o s i t i o n o f t h e c e n t e r o f t h e c a r d i s computed w i t h r e s p e c t t o
/ / t h e c o o r d i n a t e s system of t h e c e n t r a l page
Point relativePos = t h i s . TranslatePoint ( t h i s . center , Pagina ) ;
/ / i f t h e c a r d i s above t h e dr opping a r e a
i f ( relativePos . Y > 0 && relativePos . Y < t h i s . referenceHeight )
/ / i f t h e c a r d i s on t h e l e f t o f t h e p a g e
i f ( relativePos . X <= 0 )
{
i f ( relativePos . X <= − t h i s . wider_proximity )
{
/ / i f t h e c a r d i s w i t h i n a ” s e n s i t i v e ” a r e a around t h e page
i f ( t h i s . isAttached )
{
/ / t h e c a r d i s b e i n g d e t a c h e d from t h e p a g e ;
/ / t h e h i g h l i g h t i n g o f t h e c a r d i t s e l f and o f t h e
/ / l i n e s p r o n o u n c e d by t h e c h a r a c t e r i n t h e t e x t i s t u r n e d o f f
t h i s . Pagina . theContent . HighlightOff ( t h i s ) ;
t h i s . isAttached = f a l s e ;
}
}
/ / as the card approaches the
/ / t h u s p r o v i d i n g a cue about
/ / p o s s i b i l i t y of snapping i t
e l s e i f ( t h i s . rotation . Angle
page , i t i s f o r c e d t o be h o r i z o n t a l ,
the
t o t h e page
! = 0 ) t h i s . rotation . Angle = 0 ;
}
// if
else if
{
//
if
{
t h e c a r d i s on t h e r i g h t o f t h e p a g e
( relativePos . X >= t h i s . referenceWidth )
i f t h e c a r d i s w i t h i n a ” s e n s i t i v e ” a r e a around t h e page
( relativePos . X >= t h i s . referenceWidth + t h i s . wider_proximity )
i f ( t h i s . isAttached )
{
/ / t h e c a r d i s b e i n g d e t a c h e d from t h e p a g e ;
/ / t h e h i g h l i g h t i n g o f t h e c a r d i t s e l f and o f t h e
26
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
/ / l i n e s p r o n o u n c e d by t h e c h a r a c t e r i n t h e t e x t i s t u r n e d o f f
t h i s . Pagina . theContent . HighlightOff ( t h i s ) ;
t h i s . isAttached = f a l s e ;
}
}
/ / a s t h e c a r d a p p r o a c h e s t h e page , i t i s f o r c e d t o be h o r i z o n t a l ,
/ / t h u s p r o v i d i n g a cue about t h e
/ / p o s s i b i l i t y of sn apping i t t o t h e page
e l s e i f ( t h i s . rotation . Angle ! = 0 ) t h i s . rotation . Angle = 0 ;
}
e . Handled = t r u e ;
}
/ / t h i s method s e t s t h e c o o r d i n a t e s o f t h e c a r d
i n t e r n a l v o i d translate ( d o u b l e x , d o u b l e y )
{
Canvas . SetTop ( t h i s , Canvas . GetTop ( t h i s ) + y ) ;
Canvas . SetLeft ( t h i s , Canvas . GetLeft ( t h i s ) + x ) ;
}
No inertia is applied on the cards. Finally, the ManipulationCompleted event
signals that the manipulation is completed and that the object is still. In ICarta, the
corresponding handler method has been implemented as follows:
Listing 4.3: ManipulationCompleted method
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
v o i d Carta_ManipulationCompleted ( o b j e c t sender , ManipulationCompletedEventArgs e )
{
/ / the border takes a neutral color
i f ( ! t h i s . isAttached ) t h i s . CardBorder . BorderBrush = cm . TEXT_COLOR ;
Point relativePos = t h i s . TranslatePoint ( t h i s . center , Pagina ) ;
/ / p r o x i m i t y t o t h e page i s checked so as t o d e t e r m i n e whether t o snap t h e c a r d
t h i s . CheckProximityToPage ( relativePos ) ;
/ / p r o x i m i t y t o a s e l e c t o r i s checked so as t o d e t e r m i n e whether t o snap t h e c a r d
ProximityManager . Instance ( ) . handleProximity ( t h i s ) ;
e . Handled = t r u e ;
}
p r i v a t e v o i d CheckProximityToPage ( Point relativePos )
{
/ / i f the card i s not in the dropping area
i f ( relativePos . Y > 0 )
/ / i f t h e c a r d i s w i t h i n t h e h e i g h t of t h e page
i f ( relativePos . Y < t h i s . referenceHeight )
{
/ / i f t h e c a r d i s on t h e l e f t o f t h e p a g e
i f ( relativePos . X <= 0 )
{
/ / i f t h e c a r d i s c l o s e enough t o t h e p a g e
i f ( relativePos . X >= − t h i s . snapping_proximity )
{
27
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
/ / s n a p t h e c a r d and h i g h l i g h t t e x t p r o n o u n c e d by t h e c h a r a c t e r , i f any
translate ( Math . Abs ( relativePos . X − ( t h i s . Width / 2 ) ) − Width , 0 ) ;
t h i s . Pagina . theContent . HandleSnapping ( t h i s ) ;
}
}
/ / i f t h e c a r d on t h e r i g h t o f t h e p a g e
e l s e i f ( relativePos . X >= t h i s . referenceWidth )
{
/ / i f t h e c a r d i s c l o s e enough t o t h e p a g e
i f ( relativePos . X <= t h i s . referenceWidth + t h i s . snapping_proximity )
{
/ / s n a p t h e c a r d and h i g h l i g h t t e x t p r o n o u n c e d by t h e c h a r a c t e r , i f any
translate(−relativePos . X + t h i s . referenceWidth + ( t h i s . Width / 2 ) ,
0) ;
t h i s . Pagina . theContent . HandleSnapping ( t h i s ) ;
}
}
}
e l s e / / drop the card
t h i s . BeginAnimation ( Canvas . TopProperty , disappearingAnimation ) ;
}
ProximityManager is a class dedicated to detecting the proximity of a card to a
selector: since both elements can be freely rotated and moved by the user, this has
required writing a sort of customized hit testing method, which can account for the
closest selector to a given card (or vice versa), can give precedence to the snapping
of a card to the central page and, when snapping a card to the relevant selector, can
position it in a suitable way to let the card expand and show its information about
the character without messy overlappings. The code of the class is rather lengthy and
complex, therefore it will not be reported here.
28
4.3.3
Data serialization
One of the most laborious parts of the development consisted in the reading, loading
and management of the XML data provided by the KDD-Lab. A preliminar cleaning
phase was first of all required so as to standardize the structure of such files: it will be
described more in detail in section 4.4.5. Once clean data was available, it was first
loaded by class XmlManager; here follows a sample code for the reading and loading
of one of the XML files used.
Listing 4.4: Xml loading
1
2
3
4
5
XmlDocument inferno_text = new XmlDocument ( ) ;
X m l R e a d e r S e t t i n g s xrs = new X m l R e a d e r S e t t i n g s ( ) ;
xrs . D t d P r o c e s s i n g = D t d P r o c e s s i n g . Parse ;
XmlReader reader = XmlReader . Create ( ” f i l e s / i n f e r n o . xml ” , xrs ) ;
inferno_text . Load ( reader ) ;
XML files provided input data to associate characters with categories, to extract dialogs
from the text and to obtain contextual information about the characters. Let us see how
each of these actions was performed in turn.
Extracting dialogs
Once KDD-Lab’s document about dialogs was loaded, it had to be traversed so as
to read in memory the extremes of each dialog. To this end, a jagged array called
dialogo nodes was created: it has 34 rows, one for each canto in the Inferno,
and each row contains as many XmlNodeList items as the dialogs occurring in the
corresponding canto, defined by the numbers of their initial and ending verses.
Listing 4.5: Dialog reading
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
XmlNodeList dialoghi_canti = cnr_document . SelectNodes ( ” / T e s t o 1 / T a b l e / D i a l o g a t o /
Dialoghi ” ) ;
f o r e a c h ( XmlNode canto_xml i n dialoghi_canti )
{
XmlNode num_canto_roman = canto_xml . SelectSingleNode ( ” C a n t o ” ) ;
s t r i n g roman = num_canto_roman . InnerText ;
i n t index = generic_utils . RomanToInt ( roman ) ;
XmlNodeList dialogs = canto_xml . SelectNodes ( ” D i a l o g o ” ) ;
i n t i = 0;
dialogo_nodes [ index − 1 ] = new XmlNodeList [ dialogs . Count ] ;
f o r e a c h ( XmlNode dialogo i n dialogs )
{
dialogo_nodes [ index − 1 ] [ i ] =
dialogo . SelectNodes ( ” S e g m e n t i D i a l o g o / Segmento ” ) ;
i++;
}
}
29
Such information now has to be converted into a suitable format to be displayed in the
scrolling content of the central text page. Each fragment of text (corresponding either
to the lines of a character or of the narrator) has to be first of all reconstructed as proper
text from its bounding verses obtained at the previous step and then annotated with the
name of the character that possibly pronounces it. Meanwhile, names of characters are
saved into an auxiliary set called personaggi. All such information, including the
text of each fragment, are saved in objects called HighlightTextBlocks: they
extend regular TextBlocks by adding the possibility of dynamically coloring (i.e.
highlighting) the background of parts of the text.
Listing 4.6: Dialog serialization
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
/ / t h i s method i s c a l l e d o n c e f o r e a c h c a n t o , whose i n d e x i s i
i n t e r n a l L i s t <H i g h l i g h t T e x t B l o c k > GetCantoHTB ( i n t i )
{
L i s t <H i g h l i g h t T e x t B l o c k > result = new L i s t <H i g h l i g h t T e x t B l o c k >() ;
XmlNodeList [ ] segments = dialogo_nodes [ i ] ; / / s e g m e n t s i n c a n t o i
/ / i n d e x o f t h e v e r s e where t h e n e x t s e g m e n t i s s u p p o s e d t o b e g i n
i n t current_start = 1 ;
s t r i n g text = ” ” ;
C o l o r M a n a g e r cm = C o l o r M a n a g e r . Instance ;
f o r e a c h ( XmlNodeList segmentList i n segments )
f o r e a c h ( XmlNode segment i n segmentList )
{
H i g h l i g h t T e x t B l o c k tmp ;
/ / i n i t i a l v e r s e of t h i s segment
i n t verso_start = i n t . Parse ( segment . SelectSingleNode ( ” T e s t o / I n i z i o /
V e r s o ” ) . InnerText ) ;
/ / ending v e r s e of t h i s segment
i n t verso_end = i n t . Parse ( segment . SelectSingleNode ( ” T e s t o / F i n e / V e r s o ”
) . InnerText ) ;
/ / i n i t i a l v e r s e of next segment
i n t next_start = ( segment . NextSibling ! = n u l l ) ? i n t . Parse ( segment .
NextSibling . SelectSingleNode ( ” T e s t o / I n i z i o / V e r s o ” ) . InnerText ) :
verso_end ;
/ / ending v e r s e of next segment
i n t next_end = ( segment . NextSibling ! = n u l l ) ? i n t . Parse ( segment .
NextSibling . SelectSingleNode ( ” T e s t o / F i n e / V e r s o ” ) . InnerText ) :
verso_end ;
i f ( current_start ! = verso_start )
{
/ / the n a r r a t o r says something before another c h a r a c t e r speaks :
/ / s u c h p i e c e o f t e x t i s c o n v e r t e d i n t o an a u t o n o m o u s
/ / HighlightTextBlock
tmp = new H i g h l i g h t T e x t B l o c k ( ) ;
tmp . CantoIndex = i ;
tmp . NomePersonaggio = n u l l ;
tmp . NomeInterlocutore = n u l l ;
tmp . FilterText = n u l l ; / / t e x t t o be h i g h l i g h t e d
/ / whole t e x t o f t h e H i g h l i g h t T e x t B l o c k
tmp . RawText = t h i s . GetFullText ( i , current_start , verso_start ) ;
tmp . FilterTextBackground = Brushes . Transparent ;
tmp . FilterTextForeground = cm . TEXT_COLOR ;
30
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
result . Add ( tmp ) ;
current_start = verso_start ;
}
/ / a c h a r a c t e r speaks i n t h e c u r r e n t segment
tmp = new H i g h l i g h t T e x t B l o c k ( ) ;
tmp . CantoIndex = i ;
s t r i n g nome = segment . SelectSingleNode ( ” C h i P a r l a ” ) . InnerText ;
/ / t h e name o f t h e c h a r a c t e r i s a d d e d t o t h e g l o b a l H a s h S e t o f names
i f ( ! personaggi . Contains ( nome ) ) personaggi . Add ( nome ) ;
tmp . NomePersonaggio = nome ;
s t r i n g interlocutore = segment . SelectSingleNode ( ” AChi ” ) . InnerText ;
/ / t h e name o f t h e i n t e r l o c u t o r i s a l s o s a v e d
tmp . NomeInterlocutore = interlocutore ;
text = t h i s . GetFullText ( i , verso_start , verso_end + 1 ) ;
tmp . FilterText = text ;
tmp . RawText = ( ( next_start == verso_end + 1 | | segment . NextSibling ==
n u l l ) ) ? text : text + t h i s . GetFullText ( i , verso_end + 1 ,
next_start ) ;
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
tmp . FilterTextBackground = Brushes . Transparent ;
tmp . FilterTextForeground = cm . TEXT_COLOR ;
/ / s a v e an a s s o c i a t i o n b e t w e e n t h e name o f e a c h c h a r a c t e r and t h e
/ / H i g h l i g h t T e x t B l o c k s i n which i t s p e a k s
i f ( ! page . htb_dictionary . ContainsKey ( nome ) )
{
HTBListWrapper list = new HTBListWrapper ( ) ;
list . Add ( tmp ) ;
page . htb_dictionary . Add ( nome , list ) ;
}
else
{
HTBListWrapper list ;
i f ( page . htb_dictionary . TryGetValue ( nome , o u t list ) ) list . Add ( tmp
);
page . htb_dictionary [ nome ] = list ;
}
result . Add ( tmp ) ;
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
i f ( segment . NextSibling == n u l l )
{
/ / t h e r e a r e no o t h e r d i a l o g s i n t h i s segment ,
/ / so t h e remainder of t h e t e x t i s saved as t h e n a r r a t o r s p a r t
tmp = new H i g h l i g h t T e x t B l o c k ( ) ;
tmp . CantoIndex = i ;
tmp . NomePersonaggio = n u l l ;
tmp . NomeInterlocutore = n u l l ;
tmp . FilterText = n u l l ;
tmp . RawText = t h i s . GetFullText ( i , current_start , t h i s . CountVerses
(i) + 1) ;
tmp . FilterTextBackground = Brushes . Transparent ;
tmp . FilterTextForeground = cm . TEXT_COLOR ;
result . Add ( tmp ) ;
}
else
current_start = next_start ;
}
r e t u r n result ;
}
31
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
/ / t h i s method c o u n t s t h e number o f v e r s e s i n a g i v e n c a n t o
p r i v a t e i n t CountVerses ( i n t canto )
{
XmlNode node = inferno_text . SelectSingleNode ( ” i n f e r n o / c a n t o [ @numero =
canto + 1 ) + ” ] ” ) ;
XmlNode ultima_terzina = node . LastChild ;
XmlNode ultimo_verso = ultima_terzina . LastChild ;
r e t u r n i n t . Parse ( ultimo_verso . Attributes [ ” numero ” ] . Value ) ;
}
” + (
//
t h i s method g e t s t h e t e x t b e t w e e n two g i v e n i n d e x e s o f v e r s e s w i t h i n a
specified canto
p r i v a t e s t r i n g GetFullText ( i n t canto , i n t index_start , i n t index_next_start )
{
s t r i n g result = ” ” ;
s t r i n g query = ” i n f e r n o / c a n t o [ \ @numero = ” + ( canto + 1 ) + ” ] ” ;
XmlNode terzina_corrente = inferno_text . SelectSingleNode ( query ) ;
query = ” ∗ / v e r s o [ \ @numero = ” + index_start + ” ] ” ;
XmlNode verso_corrente = terzina_corrente . SelectSingleNode ( query ) ;
terzina_corrente = verso_corrente . ParentNode ;
i n t tmp = index_start ;
w h i l e ( tmp < index_next_start )
{
result += verso_corrente . InnerText + ” \n ” ;
i f ( tmp % 3 == 0 )
{
result += ” \n ” ;
terzina_corrente = terzina_corrente . NextSibling ;
verso_corrente = terzina_corrente . FirstChild ;
}
e l s e verso_corrente = verso_corrente . NextSibling ;
tmp++;
}
r e t u r n result ;
}
As soon as a card is snapped to the page, all the HighlightTextBlocks associated
to the related character is retrieved in the whole text and the background of the lines
that character pronounces is highlighted.
32
4.4
Problems encountered
In this closing section I am going to point out the most relevant issues that arose during development. The implementation phase turned out to be very time-consuming
for at least two major reasons: first of all, the prototype needed to have a variety of
features and to be stable enough so as to ensure a realistic user evaluation; secondly,
the technology framework was totally new to me and it required a rather long training
phase. In fact, it seems that the framework was targeted to professional developers
who can afford spending a long time training or who have a substantial previous experience with Microsoft’s technologies. This is probably due to Microsoft Surface
being evidently not (yet?) a consumer product because of its price and dimensions.
Nonetheless, having released a public SDK could be interpreted as a way of lowering
such barriers: rather, it turns out that the framework is often hard to understand, when
not puzzling, as I will describe here.
4.4.1
Dynamic dimensions
At the beginning of the implementation, it was not clear what the target device would
be, i.e. whether a sort of home made interactive tabletop available at the I3 unit or a
more recent full-HD Microsoft Surface; moreover, the resolution of the screen of the
laptop I was working on was even different. Therefore, the whole software had to be
parametrized so that the size of all elements drawn would be dynamically computed
base on the device where it was running. The framework however does not easily
provide this capability: as a result, a significant amount of code had to be written so as
to allow such device independence.
4.4.2
Ambiguities: coordinates and dimension setting
The Microsoft Surface SDK provides two different ways of setting the coordinates
of an element on the basic layout. One way is to act on the manipulation applied to
the object, i.e. by registering a TransformGroup on the object and then setting its
translation, rotation and scaling in the ManipulationDelta handler method. Alternatively, one can set the value of the attached properties Bottom, Left, Top or Right
of the Canvas object, as it happens in the translate method presented in listing
4.2. One of the two methods has to be chosen at the beginning and to be maintained
consistently throughout the development.
33
Moreover, apparently complementary methods, such as Canvas.SetBottom and
Canvas.SetTop for instance, cannot actually be mixed in the code: rather, only
one of them has to be used consistently. In fact, let us consider the following snippet
drawn from the code-behind of an hypothetical user control:
Listing 4.7: Coordinates setting
1
2
Canvas . SetTop ( t h i s , 1 0 0 . 0 ) ;
C o n s o l e . WriteLine ( Canvas . GetBottom ( t h i s ) ) ;
The first instruction places the control at a distance of 100 pixels from the top edge
of the window. Intuitively, the second instruction should print out a value that corresponds to the height of the window minus 100. Instead, it outputs a value of NaN.
For what concerns dimensions, it happens that every control has several properties
that relate to its width or height: for example, MinWidth, MaxWidth, Width,
ActualWidth. In particular, Width is usually set in the constructor, but it can be
overwritten by an automatically computed value of ActualWidth after the loading is completed. Hence, in order to set dynamic dimensions, one has to specify
ActualWidth and ActualHeight of both controls and their children in the Load
event handler instead of taking advantage of the XAML capabilities.
It is possible that these design choices are due to the fact that Microsoft Surface is
a specific device with a given resolution and that one should only use absolute dimensions when specifying controls, or let the system take care of the arrangement of
items. Nonetheless, this behavior is in my opinion rather confusing, overly restrictive
and often complicates the code to be written.
4.4.3
Incomplete customizability of components
Not all of the components can be freely customized. For example, the LibraryBar
component (used to implement the scrollable list of selectors in the categories panel)
caused me a lot of trouble as its width would stay fixed to 300 pixels no matter what
my specifications were in the code or in the XAML, while I wanted it to span over
the whole width of the screen. It turned out that, in order to apply such a simple
change, I would need to rewrite the whole template of the control (which, by the way,
cannot be done on a LibraryBar). In the end, I discovered that it sufficed to set
the MinWidth of the component to obtain the desired behavior. As said before, this
redundancy of similar properties and these poorly documented restrictions often get
in the way of the developer, who has to take care of a considerable amount of details
instead of focusing on the core flow of the application.
34
4.4.4
Special effects
Another unclear point concerns the capability of the software platform to actually
support the special effects it provides to the developer. Initially, I had designed the
feedback related to a snapped card as a colored halo; furthermore, each object had a
shadow drawn by the system. Performance rapidly degraded as the number of objects
on the scene grew, so I had to replace the colored halo with a solid colored border and
dynamically-created shadows with images.
I have some experience in game development and have often used dedicated game
engines: in such context, the system takes care of most of the necessary optimizations
in a way that is transparent to the developer, so as to obtain smooth executions even
in presence of shadows or large numbers of polygons and on ordinary hardware. It is
possible that this led me to overlook the performance issue at first. However, I found
it rather disappointing that the framework promises several capabilities which then
cannot be properly managed at runtime.
4.4.5
Data cleanup
Another issue that arose but that is not related to the technology in use was that of
data cleanup. As described in section 3.1, XML data was provided by the KDD-Lab
of the National Centre for Research. Upon a closer examination, however, it turned
out that there were several flaws in the files. For example, the documents were not
always well-formed, with open or incorrectly nested tags. Moreover, categories were
not homogeneous but often referred to only one character and thus had to be redefined
and assigned manually. Then, the ontology did not always provide information about
the life of actually speaking characters, but rather on cited ones that did not occur in
the text; this required me to gather some biographical and historical information (on
the most famous characters, at this prototypical stage) and insert it manually in the
system. This is the reason why the interaction with selectors was implemented only
for few cards.
Most importantly, only the initial and ending verses of dialogs were provided, but not
the proper text which had to be integrated by external sources. This process proved to
be time-consuming indeed, but was essential to the implementation of a functioning
prototype.
35
Chapter 5
Evaluation
Here follows the description of how a qualitative evaluation has been performed by
means of observational studies. Four pairs of users were asked to test the system in
turn according to the following procedure:
• users were first asked to freely explore the system for five minutes (co-discovery
learning): the aim of this part was to gain insights on their mental model of the
interaction and of the interface using the think aloud technique.
• then, three tasks were posed: users had to find the dialog between Dante and
Ulisse, then the prophecy that Farinata degli Uberti makes about Dante’s upcoming exile, and finally some information about Dante’s political career as a
Guelph.
Users have been videorecorded while interacting with the table. Such phase was
followed by the administration of an Attrak-Diff2 questionnaire [15] and by a semistructured interview with both the members of each pair.
Users came from different backgrounds. Two pairs of them consisted of master students, with little or no previous experience with touch-screen devices and definitely
no experience with interactive tabletops. Other two pairs of users consisted of professionals aged between 26 and 40, with some experience in the field of interface design,
all well acquainted with touch-screen devices and with at least a basic experience with
interactive tabletops. This allowed to have different perspectives on the design, which
were reflected in the evaluation outcomes.
In the remainder of this section I will present the findings resulting from the codiscovery phase, from the semi-structured interview and from the Attrak-Diff2 questionnaire.
36
5.1
Co-discovery learning
In general, it appears that most functionalities were discovered by users within the first
two or three minutes and that the behavior of the system was well understood. Users
were all able to complete given tasks within a very short time and, from what revealed
by the think aloud technique, built an overall correct mental model of how the application works and of what information it offers. It also seems that the user experience,
after some initial moments of frustration that will be clarified below, developed in a
satisfactory way: one pair of users even kept interacting with the application for as
long as 45 minutes after the study terminated.
In the remainder of this section I will discuss the so-called critical incidents that were
noted during the co-discovery phase. At first, users often tried to put two cards one on
top of the other, or a card on top of the text page in order to obtain further information
(figure 5.1).
Figure 5.1: Users trying to place a card on the text page.
37
Moreover, they tried (especially in the case of users not acquainted with touch devices)
to “double click” on the items expecting some reaction to occur. However, once users
realized that these inputs were not recognized by the system, they started trying to
drag items around. This suggests that there is some discrepancy between the users’
apparent mental model and the model proposed by the presented design: such point
will be addressed more in detail in next section.
Users often interpreted the sliding panel containing the selectors as a generic “drawer”,
in which they tried to put cards they wanted to dispose (figure 5.2).
Figure 5.2: Users trying to dispose the cards in the sliding panel.
In contrast, they were frequently puzzled by cards disappearing when drawn too close
to the bottom edge of the screen: often, this gesture was not intended to dispose the
cards, but rather to put them back into their initial position. It also happened that at
some point there were too many cards scattered around the tabletop, thus cluttering
the interface (figure 5.3), and users asked for a way of clearing the working space
altogether.
The most frequent difficulties concerned understanding how to use selectors (i.e. that
they could be attached to cards). Often, additional information obtained by snapping
a card to a selector were only discovered by chance and not intentionally. However,
this might be due to the fact that such behavior was implemented only for some of the
cards.
38
Figure 5.3: One example of interface cluttering.
Interestingly, some users interpreted the central text literally as a book and tried to
flip its pages to move through the text. All of them immediately grasped the meaning
of color codes used to differentiate characters’ lines. However, someone was puzzled
when dialogs spanned beyond the currently visible section of text and possibly only
the lines of one snapped characters appeared, especially if this happened the first time
users snapped cards to the page.
5.2
Semi-structured interview
The most interesting and rich findings resulted from semi-structured interviews, that
were carried out immediately after the user studies with the tabletop and involved both
participants at the same time. Several dimensions of the interaction paradigm and of
the interface were assessed during these interviews: first of all, whether the paradigm
was understandable and self-revealing; then, whether it helped users in discovering
and managing complex information, and whether the interface actually succeeded in
exploiting tabletop capabilities; finally, whether the overall design fostered collaboration.
In general, users agree on the fact that the interaction paradigm is pretty simple (one
of them even stated that it was “much simpler than expected”) and that most of the
functionalities can be discovered and understood after a few attempts. Particularly in
the case of users not acquainted with touch-screen devices, however, it was evident
39
that the GUI mental model is very strong and often conditions the way in which users
perceive new interactive environments, up to the point of hampering the interaction
itself. For instance, the very first action tried by some users to interact with the system
was double-tapping on a card, as mentioned above; some users also asked for explicit
“trashcans” in which cards could be disposed, or for more explicit cues about object
affordances (such as “visible scrollbars” or “little arrows”).
It is possible that as familiarity with similar devices increases, such mental model will
get more flexible; however, it seems that this is an issue which should be addressed
when designing for public spaces, at least in this phase in which interactive tabletops
are still not so common. Some of the more technically expert users even suggest that
some explicit instructions about usage should be placed near the tabletop so as to explain how interaction can take place (“they do not need to be long, as users are not
likely to read them, and I reckon it’s a ugly solution, but it might help unacquainted
users to overcome their initial fears”). I would like to add that, from the observations
drawn from these user studies, I got the impression that users can have two different
approaches to tabletops: some of them are familiar if not fluent with technologies and
thus not intimidated by them, and so they are not afraid of getting their hands on the
tabletop (“at last, you enter a museum and you can touch something! I’m fed up of all
those signs saying that you can’t touch this, you can’t touch that, and the glass display
cases. . . ”); some others perceive technology as something likely to crash or to make
them feel inadequate or dumb, and are thus rather reluctant to experiment. Therefore,
I considered it as a positive result when one of the “non-technical” users stated that
touch-mediated interaction with the text (“and with the characters”!) is more direct
and engaging as it “removes the barriers” created by the traditional mouse and keyboard. The same user also stated that such a system can contribute to “renovating the
image of the Commedia and making it closer to new generations”.
Users widely appreciated that functionalities are revealed with discovery, rather than
being all laid out at once. They say that this engages users, makes them curious and
pushes them to keep interacting with the system in order to find out more about its capabilities (“you get curious and want to know more about what it can do”). Moreover,
some users believe that one of the strengths of such a system is that it allows browsing
the text in unconventional ways, for instance by following the dialogs of a character
or the occurrences of a specific group of them, rather than through the traditional, sequential way of reading (“it allows for a stratification of information and for multiple
levels of detail”).
40
The majority of users states that the interface favors the organization of knowledge:
characters can be clustered according to several features, or they can be contextualized easily and with fast access to collateral information. Therefore, it is frequently
pointed out that another one of the main strengths of the system is its capability of
helping users not getting lost within the complexity of both the text and its context,
but rather focusing on the details they deem relevant. In particular, one of the users (a
law student) thinks that a similar system which allowed to browse the civil code would
be very useful in her field, as the latter is another vast and complex text where crosstopic connections are difficult to maintain (“I always have to group articles according
to their topic by hand. It is easy to forget something, and at the least producing the
schemes and searching them is very time-consuming. If I could browse the code with
a similar application, it would be much clearer”).
Users also appreciated the possibility of discovering less-known characters and of getting information about their background; they find this feature to be engaging and
attractive of users’ curiosity (“I find this Tesifone and then ask myself, who is Tesifone?
And I want to know more about it”; “I might realize that some character, or some
group of characters, speaks in more than one canto. I had not considered this aspect
before”). Interestingly, a few users state that the interface makes it easier to approach
a traditionally difficult text such as the Commedia because the text itself almost fades
in the background (“it becomes of secondary importance”), while the characters gain
a prominent position.
Technical users stated that the look-and-feel of the interface is “thoughtfully laid out”:
spatial sections serving different purposes can be clearly identified (with the possible
exception of the dropping area, as seen in previous section). Non-technical users stated
that the overall layout is unobtrusive and allows to focus on relevant details; some of
them believe that this is another element that facilitates the approach to the text, as
the interface looks “less intimidating” even though the tabletop might be unfamiliar.
Some users state that the overall color scheme attracts attention while not distracting
from the content of the application, and that it fits with the Commedia being an antique writing; some others argue that the colors are too soft, dark and unattractive. In
general, however, users agree that the use of images and colors, together with physical
gestures, helps persistence of learned information, eases recall and facilitates making associations and focusing on the parts that are most relevant to users at a certain
moment (“usually you have the text, which is already difficult in itself, and then it is
overwhelmed by all those notes, and you don’t want to read all of them, you are only
interested in some”). Furthermore, some users propose that a similar color code should
be used also to help identifying which characters belong to each category (“you might
put colored shapes both on the folder [the selector] and on the cards that are in it”).
41
In conclusion, let us consider the collaborative aspect of the design. One of the users
correctly points out that “there is no intrinsic need in the system for more than one
user”: he also adds that the only possible social aspect is due to the attempt of understanding how the system works. On the contrary, his partner argues that collaboration
is strongly afforded by the tabletop device and that collaborative discovery is more satisfying and “rewarding” than the individual one. Such latter observation is confirmed
by all of the “non-technical” users, who say that the tabletop itself strongly promotes
sharing and collaborative learning even though the software does not force them: users
are in fact “all at the same level”, without any of them being dominant as it would happen in a traditional personal computer setting. Moreover, one of them states that being
the Commedia a “neutral topic”, it does not inhibit communication even among casual
users who do not know each other, as it would often happen in a public setting, but
rather provides some hints about starting a conversation.
5.3
Attrak-Diff2 questionnaire
A more quantitative evaluation resulted from the administration of the Attrak-Diff2
questionnaire to all the participants to the user study. As explained in [15], this questionnaire tries to evaluate and compare the hedonic and pragmatic qualities of interactive products, thus trying to assess independently beauty and practical usability.
The questionnaire consists of several pairs of words representing opposite extremes;
the user can choose from seven possible gradations between such extremes. Pairs of
words are grouped into those concerning hedonic qualities (separating the two aspects
of stimulation and identity) and those concerning pragmatic qualities.
The Attrak-Diff2 questionnaire was administered as an Italian translation obtained
from its English version, as all participants in the study were Italian speakers and their
knowledge of English was in some cases not sufficient to ensure their understanding
of the questions. As suggested in [14], HQI, HQS and PQ scores were computed by
averaging the respective item values per participant; table 5.1 presents such results. In
table 5.2 below, further aggregated data obtained by averaging the scores over all the
participants are also shown to provide a more synthetical representation of results.
42
Table 5.1: HQI, HQS and PQ scores averaged per participant.
Participant
1
2
3
4
5
6
7
8
HQI
5.86
5.14
7.00
6.14
5.00
5.23
5.43
5.14
HQS
6.29
4.86
6.71
6.00
6.29
4.57
6.43
4.43
PQ
5.71
4.43
5.86
6.14
5.14
5.71
5.71
4.86
Beauty Goodness
7
6
6
6
7
7
7
7
5
6
5
6
6
7
6
6
Table 5.2: Aggregate values for HQI, HQS and PQ scores.
HQI
5.62
HQS PQ
5.70 5.45
Beauty Goodness
6.12
6.38
Following the interpretation suggested in [14], it appears that aggregate scores are
overall satisfactory from all points of view (recall that the higher the score, which can
range from 1 to 7, the more positive the evaluation for that quality). Users perceive a
good “capability of communicating identity to others” (HQI score) and a good degree
of “perceived novelty, stimulation and challenge” (HQS score); moreover, they also
perceive a good degree of usability (PQ score). This appears to be an encouraging
result. It might be worth investigating whether proposing a more polished application
prototype could affect the scores, in particular that one concerning pragmatic qualities,
and how.
43
Chapter 6
Conclusions and future work
This thesis has addressed the design of a proximity-based interaction paradigm for
browsing complex information which could build on the intrinsic peculiarities of interactive tabletops; this design has subsequently been evaluated with a user study carried
out on a prototypical application aimed at a public setting and presenting the relations
between characters in Dante’s Inferno. As seen in the previous chapter, user evaluations reported encouraging results which are summarized in the following.
User studies confirm that, after a brief initial phase of discovery, the interaction paradigm
is well understood and accepted. It is also confirmed that exploiting the intrinsic features of tabletops (spatiality, collaboration affordance and direct-touch manipulation)
favors the organization of notions, especially in case of a vast and complex knowledge
base. Categorization becomes in fact not only mental, but also spatial: furthermore,
involving gestures reinforces previously gained knowledge, as once again structuring
is not only mental but also involves physical movement.
This prerogative of spatiality, together with the low barriers for use of tabletops, may
favor the integration of such devices in several settings, both public and business, as
suggested by users themselves in section 5.2. Schools, or didactical settings in a broad
sense, might exploit such capabilities to reinforce learning, or to facilitate the connection of notions. Libraries and museums might profit from interactive tabletops by developing applications (such as the prototype presented here or as those cited in section
2.4) that highlight contextual or relational information about the exhibits. Spatiality
might very well be exploited also to analyze medical or business data, where relations
can easily be lost in the overwhelming amount of information.
44
User studies have also confirmed that the designed interface captures attention and
engages users, enticing them to continue interacting with the system so as to discover
all of its capabilities; they have also underlined the importance of carefully designing
such interface so as to further lower barriers for the use of tabletops.
Moreover, it appears that the social aspect of interaction emerges even though it is
not enforced by the design itself: this provides flexibility in the use of the system and
broadens the number of scenarios in which it can be applied. On the one hand in fact,
users can browse information also on their own, without forcing them to collaborate
if they are not willing to or forbidding them to use the application in case they do not
have companions; on the other hand, social interaction afforded by tabletops has been
demonstrated to take place all the same.
These findings sum up to a positive and encouraging evaluation of the proposed interaction paradigm for information browsing. Here follow some suggestions for further
improvement and extension:
• following users’ suggestions and considerations made in section 5.2 about this
transitory phase in which touch-enabled device are not yet familiar to everybody,
some additional cues about how to operate the system could be added, for example as a demonstrative video to be displayed on standby mode or as balloons
to be proposed in response to the first touches issued by some user.
• the prototype itself could be refined, mainly by having an expert adding more
background information about characters: this would allow to investigate further
how selectors are perceived and thus to perform a more thorough user study.
• a different target platform could influence the design. For instance, Mitsubishi’s
Diamond Touch [4], with its capability of detecting who issued each touch, could
allow the introduction of collaborative gestures, making the design more collaborative in nature.
• as hinted above, the capability of the interaction paradigm to highlight associations among data could be applied also to other contexts, i.e. to complex and
rather large knowledge bases. The required structuring of data can be described
in the common XML format: therefore, it can be realized also by non-IT people,
as it happened in the presented case study about Dante.
45
• the interface could be enhanced with audio, opening even further scenarios of
application: let us think, for instance, of the exploration of the musical production of classical composers.
• finally, in the proposed interaction paradigm space is physical, but manipulated
objects are still virtual. Implementing tag recognition could bring proximitybased interaction to a further level, thus integrating virtual and “real” space.
46
Bibliography
[1] Exhibition dedicated to the examination of the social networks of the Bauhaus
movement. http://vimeo.com/5333614.
[2] mæve installation website.
http://portal.mace-project.eu/
maeve/installation.php.
[3] Microsoft Surface website.
en/us/default.aspx.
[4] Mitsubishi’s diamond
DiamondTouch/.
http://www.microsoft.com/surface/
touch.
http://www.merl.com/areas/
[5] The Natural User Interface group. http://nuigroup.com/go/lite/.
[6] PixelSense technology. http://www.microsoft.com/surface/en/
us/pixelsense.aspx.
[7] A. Cappelletti, G. Gelmini, F. Pianesi, F. Rossi, and M. Zancanaro. Enforcing
cooperative storytelling: First studies. In Proceedings of the IEEE International
Conference on Advanced Learning Technologies, pages 281–285, Washington,
DC, USA, 2004. IEEE Computer Society.
[8] A. Cappelli, M. N. Catarsi, P. Michelassi, L. Moretti, M. Baglioni, F. Turini, and
M. Tavoni. Knowledge mining and discovery for searching in literary texts. In
Proceedings of the Third International Conference on Language Resources and
Evaluation, 2002.
[9] A. Cappelli, M. Coscia, F. Giannotti, D. Pedreschi, and S. Rinzivillo. The social
network of dante’s inferno. Leonardo, 44:246–247, June 2011.
[10] R. Fleck, Y. Rogers, N. Yuill, P. Marshall, A. Carr, J. Rick, and V. Bonnett.
Actions speak loudly with words: unpacking collaboration around the table. In
Proceedings of the ACM International Conference on Interactive Tabletops and
Surfaces, ITS ’09, pages 189–196, New York, NY, USA, 2009. ACM.
47
[11] D. Furniss, A. Blandford, and P. Curzon. Confessions from a grounded theory
phd: experiences and lessons learnt. In Proceedings of the 2011 annual conference on Human factors in computing systems, CHI ’11, pages 113–122, New
York, NY, USA, 2011. ACM.
[12] S. Gabrielli, S. Bellutti, A. Jameson, C. Leonardi, and M. Zancanaro. A singleuser tabletop card game system for older persons: General lessons learned from
an in-situ study, pages 85 –88. 2008.
[13] C. L. Z. Gress, M. Fior, A. F. Hadwin, and P. H. Winne. Measurement and
assessment in computer-supported collaborative learning. Comput. Hum. Behav.,
26:806–814, September 2010.
[14] M. Hassenzahl. The interplay of beauty, goodness, and usability in interactive
products. Hum.-Comput. Interact., 19(4):319–349, Dec. 2008.
[15] M. Hassenzahl, M. Burmester, and F. Koller. AttrakDiff: Ein Fragebogen zur
Messung wahrgenommener hedonischer und pragmatischer Qualität. In Mensch
& Computer 2003: Interaktion in Bewegung, pages 187–196, Stuttgart, Germany,
2003. B. G. Teubner.
[16] M. Hassenzahl and A. Monk. The Inference of Perceived Usability From Beauty.
Human-Computer Interaction, 25(3):235–260, 2010.
[17] K. Höök. User-centred design and evaluation of affective interfaces, pages 127–
160. Kluwer Academic Publishers, Norwell, MA, USA, 2004.
[18] S. Houde and C. Hill. What do prototypes prototype ? Current, 2:367381, 1997.
[19] P. Isenberg, D. Fisher, M. Ringel, M. Kori, and I. Mary. An exploratory study
of co-located collaborative visual analytics around a tabletop display. October,
pages 179–186, 2010.
[20] P. Isenberg, U. Hinrichs, M. Hancock, M. Tobiasz, and S. Carpendale. Information visualization on interactive tabletops in work vs. public settings. In Proc.
CoVIS 2009, 2009. Appeared in the Technical Reports series of the Department
of Media Informatics of the Ludwig-Maximilians-University of Munich, Germany in 2010.
[21] D. A. Keim, F. Mansmann, J. Schneidewind, J. Thomas, and H. Ziegler. Visual
data mining. chapter Visual Analytics: Scope and Challenges, pages 76–90.
Springer-Verlag, Berlin, Heidelberg, 2008.
48
[22] C. Leonardi, A. Albertini, F. Pianesi, and M. Zancanaro. An exploratory study
of a touch-based gestural interface for elderly. In Proceedings of the 6th Nordic
Conference on Human-Computer Interaction: Extending Boundaries, NordiCHI
’10, pages 845–850, New York, NY, USA, 2010. ACM.
[23] M. R. Morris. Designing tabletop groupware. In Adjunct Proceedings of the 18th
annual ACM Symposium on User Interface Software and Technology, 2005.
[24] M. R. Morris, A. Huang, A. Paepcke, and T. Winograd. Cooperative gestures:
multi-user gestural interactions for co-located groupware. In Proceedings of the
SIGCHI conference on Human Factors in computing systems, CHI ’06, pages
1201–1210, New York, NY, USA, 2006. ACM.
[25] M. R. Morris, J. O. Wobbrock, and A. D. Wilson. Understanding users’ preferences for surface gestures. In Proceedings of Graphics Interface 2010, GI
’10, pages 261–268, Toronto, Ont., Canada, Canada, 2010. Canadian Information Processing Society.
[26] T. Nagel, L. Pschetz, M. Stefaner, M. Halkia, and B. Müller. m&#230;ve — an
interactive tabletop installation for exploring background information in exhibitions. In Proceedings of the 13th International Conference on Human-Computer
Interaction. Part III: Ubiquitous and Intelligent Interaction, pages 483–491,
Berlin, Heidelberg, 2009. Springer-Verlag.
[27] D. Norman. The design of everyday things. Basic Books, 1988.
[28] A. M. Piper and J. D. Hollan. Tabletop displays for small group study: affordances of paper and digital materials. In Proceedings of the 27th international
conference on Human factors in computing systems, CHI ’09, pages 1227–1236,
New York, NY, USA, 2009. ACM.
[29] J. Ramey, T. Boren, E. Cuddihy, J. Dumas, Z. Guan, M. J. van den Haak, and
M. D. T. De Jong. Does think aloud work?: how do we know? In CHI ’06
extended abstracts on Human factors in computing systems, CHI EA ’06, pages
45–48, New York, NY, USA, 2006. ACM.
[30] Y. Rogers and S. Lindley. Collaborating around vertical and horizontal large
interactive displays: which way is best? Interacting with Computers, 16(6):1133
– 1152, 2004.
[31] Y. Rogers, H. Sharp, and J. Preece. Interaction Design: Beyond HumanComputer Interaction. John Wiley and Sons Ltd, 2002.
49
[32] K. Ryall, C. Forlines, C. Shen, M. R. Morris, and K. Everitt. Experiences with
and observations of direct-touch tabletops. In Proceedings of the First IEEE
International Workshop on Horizontal Interactive Human-Computer Systems,
pages 89–96, Washington, DC, USA, 2006. IEEE Computer Society.
[33] S. D. Scott, K. D. Grant, and R. L. Mandryk. System guidelines for co-located,
collaborative work on a tabletop display. In Proceedings of the eighth conference
on European Conference on Computer Supported Cooperative Work, pages 159–
178, Norwell, MA, USA, 2003. Kluwer Academic Publishers.
[34] O. Stock, M. Zancanaro, C. Koren, C. Rocchi, Z. Eisikovits, D. Goren-bar,
D. Tomasini, and P. T. Weiss. A co-located interface for narration to support
reconciliation in a conflict: initial results from jewish and palestinian youth. In
Proceedings of the twenty-sixth annual SIGCHI conference on Human factors
in computing systems, CHI ’08, pages 1583–1592, New York, NY, USA, 2008.
ACM.
[35] D. Wigdor and D. Wixon. Brave NUI World: Designing Natural User Interfaces
for Touch and Gesture. Morgan Kaufmann, 2011.
50