Effective temporal graph layout: A comparative

Transcription

Effective temporal graph layout: A
comparative study of animation versus
static display methods
Michael Farrugiaa, ∗ and
Aaron Quigleyb
a
University College Dublin, 8 Talbot Down,
Dublin15, Ireland.
E-mail: [email protected]
b
Chair of Human Computer Interaction,
School of Computer Science, The University
of St. Andrews, UK.
E-mail: [email protected]
∗
Corresponding author.
Abstract Graph drawing algorithms have classically addressed the layout
of static graphs. However, the need to draw evolving or dynamic graphs
has brought into question many of the assumptions, conventions and layout
methods designed to date. For example, social scientists studying evolving
social networks have created a demand for visual representations of graphs
changing over time. Two common approaches to represent temporal information in graphs include animation of the network and use of static snapshots
of the network at different points in time. Here, we report on two experiments, one in a laboratory environment and another using an asynchronous
remote web-based platform, Mechanical Turk, to compare the efficiency of
animated displays versus static displays. Four tasks are studied with each visual
representation, where two characterise overview level information presentation, and two characterise micro level analytical tasks. For the tasks studied in
these experiments and within the limits of the experimental system, the results
of this study indicate that static representations are generally more effective
particularly in terms of time performance, when compared to fully animated
movie representations of dynamic networks.
Information Visualization (2011) 00, 1 -- 18. doi:10.1057/ivs.2010.10
Keywords: graph drawing; dynamic social networks; information visualisation;
evaluation; animation
Introduction
Received: 27 August 2009
Revised: 23 August 2010
Accepted: 13 September 2010
Real-world networks are not static, they are in a state of constant evolution. Networks, such as social science networks, biological networks and
computer networks, are constantly changing with new nodes added and
new relations forming, while old relations may either persist or end. In
this article, we focus on dynamic social networks and their effective visualisation. Despite the lack of widespread research publications on dynamic
social networks, Doreian and Stokman1 report that research on dynamic
networks is increasing. Recently, Volume 325 (July 2009) of the Science
Journal featured a special edition on Complex Systems and Networks. In
this feature, written by the most prominent authors in network science,
the study of dynamic networks was identified as one of the current main
challenges in network theory.
It is recognised that collecting longitudinal data is a time consuming and
challenging task. This explains why network data is typically only collected
once for analysis. Owing to this, the field of social network analysis has
often concentrated on single snapshot instances of a network. Thanks in
part to the proliferation of online technologies and social media, the data
collection bottleneck has been partially overcome and dynamic network
data sources have become more accessible.
© SAGE Publications, 2011. 1473-8716
Information Visualization
Vol. 00, 0, 1 – 18
Farrugia and Quigley
Dynamic social network analysis on longitudinal
data can be useful for investigating long-term human
behaviour. Christakis and Fowler2 studied the dynamics
of smoking behaviour over a period of 32 years. Smoking
decreased throughout the data collection period and interestingly connected groups of smokers stopped smoking
together suggesting a network effect. In another study,
van Duijn et al,3 studied the evolution of friendships
in University freshmen. Their results show that physical proximity of actors and the individual similarities
between actors, are only contributors to the beginning
of friendship formation. Dynamic network analysis can
also be important in criminal investigations. Xu et al4
describe the processes and measures applicable when
using dynamic network analysis in a criminal context
and also make effective use of visualisation, including
animation, to present the evolution process of criminal
networks.
From the earliest research in social network analysis,
visualisation has been a central tool. Moreno, the inventor
of sociometry,5 devised the first graphical representation
of a relationship between two entities and called it the
sociogram. Since Moreno’s days there has been a significant increase in complexity and improvement in the
methods for how social networks are drawn.6 However
in network visualisation, the temporal aspects of the
data within the network have often been ‘flattened out’
or ignored. The realisation that the study of different
versions of a network over time can lead to insights not
possible through static display alone has given rise to an
increasing interest in dynamic graphs within the visualisation community. A testament to this, is the fact that
at least three major conference competitions, GD’98,7
InfoVis’048 and VAST’089 contained tasks requiring the
visualisation and characterisation of dynamic networks.
Beyond the problem of developing either a graph
layout technique or a new visual design for dynamic
networks lies the presentation of the visualisation. Classically, there are three ways in which dynamic networks,
visualised as node-link diagrams, can be presented. The
most common presentation of dynamic graphs is to
animate the different layouts for each time point, and
produce a movie of the dynamic network.6,10–13 Alternatively, the individual layouts can be displayed adjacent
to each other in one screen akin to a photo album with
thumbnail images.14,15 In a variant of this presentation,
the node link diagrams can be displayed as static images
in sequence, similar to a slideshow.16,17
When it comes to evaluating the mode of presentation of dynamic networks, studies usually either rely on
the general intuition that animation is beneficial, or else
ask for experts to judge the usefulness of animation.4 In
reality, no empirical studies have been conducted on the
actual benefits of animation when visualising dynamic
social networks. In this study, we attempt to empirically
evaluate the benefits of animation and understand the
tasks and cases for which animation is beneficial over
static images.
2
As part of our evaluation, we conduct two experiments
using different experimentation platforms. The first experiment is a laboratory user study with a group of computer
science students. In the second experiment, we attempt
to generalise the results obtained in the laboratory by
extending the study to anonymous users, who might not
have experience with social network visualisations, in a
remote experiment using Mechanical Turk (MTurk). As
part of our results, and discussion we detail our experiences using a remote experimentation platform and the
implications it has on visualisation experiments.
The rest of the article is organised as follows. In
Section ‘Related work’, we review related work in the
area of dynamic social network visualisation, animation
and visual perception, including approaches to visualisation evaluation with an emphasis on remote experiments for visualisation. In Section ‘Experiment setup and
materials’, we describe our experiments’ design, materials
and methodology used to compare two visual representations. In Sections ‘Results’ and ‘Discussion’, we report
on the results and then discuss them. Finally, we derive
conclusions based on the results and discuss areas for
further work, both in the evaluation and implementation
of such visual systems.
Related Work
This work draws on three principal areas of visualisation
research; graph drawing, visual perception and visualisation evaluation methods. Dynamic graph drawing
provides the technical background to visualise networks
following desirable criteria of graph layouts with a
temporal component. As the aim of the study is to
compare animated and static modes of presentation,
research on the efficiency of both mediums as a perceptual aid in different visualisation applications, is reviewed.
Part of this research includes a user study, where a relatively new platform that takes advantage of a web-based
micro-market to engage a user study population. Early
studies this platform, report promising results if the
necessary care is take to interpret the results, in the appropriate context. Recent work on the applicability of this
platform for the purpose of resourcing participants and
conducting online experiments is also reviewed.
Dynamic graph drawing
In its most basic form a graph consists of a number of
nodes, representing elements of information and the
interrelationships between these elements, or edges.
Graph drawing is the challenge of determining a
geometric position for every node and a route for every
edge, so that a picture of the graph can be rendered.
Dynamic graph drawing is an area within graph drawing
that studies the layout of graphs with a temporal component. This dynamic component can come about from new
Effective temporal graph layout
nodes or edges being added to the network or existing
ones being removed or updated.
For the purposes of this research, we focus specifically on social networks. Graph theory provides a solid
framework onto which the concepts and ideas from
such networks can be mapped. This brings a vocabulary
which can be employed to denote structural properties and a set of primitive concepts and mathematical
operators to measure these structural properties that are
representing social structures.18 Given the natural representation of a social network as a graph, graph drawing
itself is the natural approach to the visualisation of social
networks. Graph drawing is a well-researched area in
computer science and several good references exist on the
subject.19–22 In addition, Freeman6 provides a detailed
reference on which of these algorithmic approaches are
best suited to social network visualisation.
In traditional graph drawing, a single static version of
the network is drawn. However, when drawing ‘dynamic
networks’ several related versions of the same basic
network need to be drawn. Here, the time dimension
adds an additional constraint beyond the traditional optimisation of aesthetic criteria20,23,24 with static graphs.
When drawing temporal dynamic graphs it is generally
considered beneficial to keep node movement between
time periods at a minimum. This is to preserve the mental
map between different related layouts.25,26
Where dynamic social network visualisation has been
studied, the focus has been predominantly on the formation of network structures. The most common approach
to the dynamic graph drawing problem builds upon the
flexible nature of force directed layout algorithm27 to
control node movement. These layout algorithms28,29
often include a stiffness parameter to dampen node movement between successive time-steps to help maintain a
smooth change to the layout.
When the sequence of graph versions are known before
the drawing, the complete layout information can be
exploited to improve the node placement. These class of
graph drawing techniques are called offline layout algorithms. In one of these techniques, Diehl and Görg30
use the supergraph of all graph iterations to calculate the
node positions. These positions are then used to inform
the individual node locations at the drawing of each time
step.
Lee et al31 extend layout heuristics first applied to static
graphs, that of simulated annealing by Davidson and
Harel.32 They formulate the dynamic graph drawing question as an optimisation problem, whereby they attempt to
minimise a cost function of graph changes between iterations. Although encoding different graph criteria separately gives the user more control over the layout, these
algorithms are typically quite slow to execute. Dwyer and
Gallagher33 employ a two and a half-dimensional layout
where time is mapped to the third dimension providing
a 3D layered view of the changes in the data over time.
In this study, we fix the parameters related to graph
layout. We employ the Social Network Image Animator
(SoNIA)34 software framework whose focus is on the optimisation of dynamic graph layouts and use it to generate
the graph images and videos for our studies. Our approach
is to vary the mode of presentation to study the impact
on graph understanding.
When evaluating dynamic graph drawing algorithms
one needs to compare the difference between layouts
across different time steps and measure the change to
the layout over time.35 Friedrich and Eades36 provide a
comprehensible list of criteria and measures of a good
animation. Frishman and Tal29 use the distance between
nodes in successive layouts to calculate the average
displacement of the nodes. They complement this with
measures of the energy (stress) of the graph at successive
time points as a measure of the ‘niceness’ of the layout.
While several good attempts have been made to understand the impact of graph layouts and presentation on the
graph comprehension,23,24,37 work on this is still in its
formative stages. It is difficult, if not impossible, to claim
that a group of measures can be used as an absolute indicator of optimal layout or presentation. Instead, quantitative claims from a new algorithm provide an indication
that it improves upon certain criteria, that are currently
believed to impact upon perception.
Mode of presentation
Animation is essentially a sequence of images displayed
in rapid succession to give the illusion of movement.
Using animation, one can embed time within the
network structure itself without changing any additional
attributes. Animation is a natural way of presenting
temporal information,38 therefore it is hardly surprising
that animation is the most popular manner of presenting
dynamic networks.
One of the most concrete applications of dynamic
network visualisation is the study by Moody et al17 that
uses dynamic network visualisation to characterise two
real and one synthetic dynamic network. Both animation and static images are used to present the results.
The animation presentation is used to create movies of
the networks, whereas static images are presented using
flipbooks (slideshows). In the animations, nodes are free
and can change position between different time periods,
however in flipbooks all the nodes are fixed between time
periods. In flipbooks only one common graph layout is
used, an approach that the authors claim is appropriate
for relatively sparse networks.
The authors also distinguish between continuous and
discrete time in networks. Continuous network data is
made up of streaming data with known start and end times
for each relationship, for example, a network extracted
from a log of phone calls. Discrete networks, on the other
hand, collect data at specific time intervals. Two classroom
interactions from the McFarland classroom network39 are
used as an example of a continuous time network. This
data consists of a stream of interactions of conversion
turns in classrooms.
3
In their article, the patterns identified in the three
networks are presented convincingly using animation.
It is clear that animation was the preferred presentation method for analysis and presentation. Perhaps, one
reason for this is the fact that animation provided less
interactive burden owing to the number of time periods
in the networks (for instance, 200 iterations for the
syntactic network). Animation has the ability to condense
many images in a short space of time and to be executed
with one play action. To visualise this data with flipbooks
one has to manually browse through all the images in
sequence. In addition, the fact that nodes were fixed in
flipbooks, does not provide a comparable alternative to
animation were the nodes are moving.
Animated visualisations of node link diagrams are
also popular and successful in visualisation competitions where dynamic network data needs to be analysed.
The best two submissions in the Graph Drawing ’98
competition40 were both presented as recorded animations. In Info Vis ’04 the winning submissions analysed
the temporal data set of visualisation publications using
mainly static network images41 or alternative visualisations to node link diagrams.42 In VAST 2008, there was
no single winner but a number of different rewards for
significant contributions were awarded. Among these
was an award for the best animation which clearly highlighted the structural change occurring in the evolving
network.43 The submission explains how the animated
visualisations were used to support the analysis and
assisted in finding the solution to the problem scenario.
Most of the submissions in VAST 2008 used multiple
static images of node-link diagrams to show the visualisation. Few of the more advanced systems such as44 added
interactive features to allow image scrolling or viewing
multiple images in a single window. In addition, some
systems extracted the time element from the graph data,
and used the time-dimension in a traditional time-based
line chart. A notable visualisation in this case was the
contribution by the University of Maryland45 that used
the Social Action platform46 to visualise the activity of
each node throughout the time period as a stacked line
chart.
Animation studies in node link diagrams
Although animation is an intuitive and almost natural
way of presenting temporal information, there were very
few efforts to evaluate animation as a way of presenting
node link diagrams. Most of the evaluations of dynamic
networks visualised as node link diagrams study the
importance of retaining the mental map in for the graph
layout.26 When animation is used for presentation, it is
often assumed that animation is the logical and hence
only way to represent such information.
One of the few studies in this area was conducted by
McGrath and Blythe47 who study the impact of motion
on viewers’ perception of graph structure. The authors
conduct a user study based on an analytical scenario,
4
rather than a purely syntactical task-based experiment,
to understand the impact of both motion and layout in
understanding the network. Motion is used to facilitate
the tracking of the nodes’ between two time periods in
the network. The results show that motion has a positive
effect when used to understand change in the status of
a node. A change to the network layout alone does not
have a statistically significant impact unless motion is
added to the visualisation.
Ware and Bobrow48 investigate motion in node link
diagrams, however in this study motion is only used
as a highlighting mechanism. In this context, the preattentive property of motion is exploited and used as a
visual coding attribute of nodes, such as colour and shape.
The three different experiments conducted all suggest
that motion is effective as a means of highlighting in
node link diagrams.
Animation for perception
Animation as a mode of presentation, has been studied
in different fields with mixed results. Most of this work
studies animation in a learning environment.38 In
computer science, this was reflected in using animation to
explain the internal workings of complex algorithms.49
Similar to the case with dynamic networks, the first developers of algorithm animations believed that animations
are beneficial as a learning tool, without empirically evaluating the perceived positive effect. When scientists started
to validate this claim by running perception experiments
to understand the benefits of animation, animation was
not found to perform significantly better or worse than
static images.50 Interestingly, followup studies in this
area51 attempted to find instances where animation is
helpful. In these studies, the benefits of animation were
seen when students were allowed to use it in a less rigid
environment supplemented with other material. Animation was also found to be useful when the users were
allowed to actively interact with the animation by using
their own data to be visualised with the animation.52
Inspired by the favourable audience response to Hans
Rosling’s presentations at TED 200653 and TED 2007,54
Robertson et al55 addressed the effectiveness of animation
in trend visualisation. The authors differentiate animation
as used for analysis (when the results are not known), as
opposed to when animation as used for the presentation
of results (when the results are known). In their study,
animation is compared to two static image representations, small multiples and a slideshow view with traces
of movement shown. In this study, on multivariate data
animation some positive effects of animation are identified in the presentation tasks, yet even for presentation
the accuracy in the animated visualisations was lower
than the static representations. For the analytical tasks,
the two static image representations had better performance in both accuracy and time measures. Interestingly,
in the informal survey at the end of the experiment, the
participants thought that animation was easier and more
enjoyable than their static counterparts. The appeal of
using animation in competition submission40,43 and in
explaining network patterns in research publications2,17
can be partially explained by the positive effect animation
has when presenting data as described by Robertson.
When describing animation as a mode of presentation (in Section ‘Mode of presentation’), we distinguish between ‘a presentation mode’ as opposed to
‘for presentation’. In the context of this article, mode
for presentation refers to the way the visualisation is
presented, whether it is presented using static images or
animation. The graph layout and the node link diagram
are the visualisation, whereas animation and static images
are two ways that visualisation can be presented.
Other work36 has investigated aspects of animation
especially the interpolation between keyframes in animations. Yee et al56 explore the case of interpolating between
radial-based images and from the informal usability
studies find that interpolating the polar coordinates of the
nodes radial layout, makes the transition between screens
easier to follow. Animation can also be used to assist
with transitioning between different visualisations.56–58
When animation is used as a transition the animation is
not the main component of the visualisation but animation is used to assist with a shift of viewpoints to make
the retention of context or the ‘mental map’ easier. Both
the formal evaluation by Heer and Robertson and the
informal study by Yee et al report positive results in favour
of animation. In this case, animation is used as a tool to
assist with transitions rather than as the mode of presentation itself. This application of animation however is
quite distinct from animation as a means of presentation, which is the way that animation is being studied
in this article. In these experiments, we are investigating
animation as a tool in it’s own right.
Current systems support for dynamic network
visualisation
The number of software packages that can be easily used
by social scientists to visualise dynamic social networks
is limited when compared to the number of packages
available for studying static networks. Social scientists
who study dynamic networks tend to employ strictly
mathematical tools such as the Simulation Investigation
for Empirical Network Analysis (SIENA) package,59 which
do not support visualisation. If we look at the benefits
of information visualisation,60 we can see that dynamic
Social Network Analysis can clearly benefit from visualisation. However, a simplistic approach to this problem may
further obscure the data and reduce the utility of the visual
display with clutter and gratuitous glyph movement.
Few visualisation packages support a time element to
allow for the network to change over time. For example,
Netdraw, which comes packaged with the popular
UCINET61 package does not support dynamic networks.
Pajek62 supports time-based networks and can generate
images of the network at different points in time. Pajek
presents the different networks snapshots as single images
in a slideshow requiring the user to manually click on a
sequence of images.
The biggest contribution to date, to the visualisation of
dynamic network data, has been from Skye Bender-deMoll
and Daniel McFarland,34 who developed an open source
software framework called SoNIA to visualise dynamic
networks. The main focus of SoNIA is to provide a platform for testing and comparing different graph layout
techniques, using dynamic instead of static network data.
The network animation in SoNIA is created by joining
together a sequence of images of the network, and interpolating the position of nodes between keyframes.
The authors define a framework for representing timebased network events. They categorise different time
sequences in typical network data sources, and suggest
ways how time can be represented in the source data
for use in the network visualisations. They introduce the
concept of ‘slicing’ event data, as a metaphor to describe
a network at a point in time. The slice can be either a thin
slice or a thick slice. In a thin slice, the network is extracted
for an exact point in time. Thin slices are good to query
network data that contains a duration element in the
network events. In a thick slice, all the events in particular
time window are considered. This is good when network
events don’t have duration, or have a small duration, and
a time window is used to group multiple events in the
network. The framework used for defining time-based
events in network data is scalable to various data sources.
Recently, Loet Leydesdorff et al12 have developed a new
layout technique based on multidimensional scaling for
dynamic network layouts. Unlike previous approaches,
such as SoNIA, they do not use linear interpolation
between independent static frames. In the new layout
algorithm proposed, the authors specifically add a parameter to control the change and movement of nodes over
time. This parameter in turn controls how much the
layout changes between subsequent frames and can be
tuned to retain the mental model of the layout between
different frames. This method is implemented in Visone63
a publicly available software system for visualising and
analysing social networks.
Despite the interest and advances from the visualisation community in dynamic social networks, the number
of available systems used by social network scientists that
handle dynamic networks is still lagging behind the developments reported from visualisation. Perhaps, the main
reason for this is that these advances are developed in
prototype systems and not extended into comprehensive
general purpose systems that are widely available.
Crowdsourcing experiments
As part of our experiment, we employ a service provided
by Amazon called Mechanical Turk (Mturk).64 MTurk
5
allows requestors (the employers) to outsource, or better
crowdsource, small, stand alone tasks to a pool of workers.
The tasks are typically small enough to be completed in a
few seconds or minutes, and each task is usually paid in
US cents. In the site’s vernacular, the tasks are called HITs
– Human Intelligence Tasks. On the basis of sample usage
examples in the service’s documentation, the site was
envisaged to help with tasks such as image categorisation,
survey responses and website filtering.
The site also provides a facility to filter the workers that
can participate on tasks either by measures of past performance or general demographics such as the worker’s location. Users can also be requested to qualify for certain HITs
by first completing a qualification task designed to test
the suitability of the worker to complete those tasks. This
provides a good way to filter different workers based on
their skill set and knowledge.
Studies on the demographics of Mechanical Turk
workers65,66 report an evolving trend in the worker population. Currently the highest concentration of workers is
around 56 per cent US based, about 30 per cent Indian
and a minority from other countries. The gender has
become more balanced at almost 48 per cent male, 52
per cent female, as opposed to a 60 per cent female population in the early years of the service. The average age
of workers is 31.6 and the education level is high with
59 per cent of workers having a Bachelor degree or higher.
The reasons survey respondents give for working on
the site are predominantly for making money (especially among the Indian population), for fun and to pass
time.
Such a platform is attractive to researchers as it provides
a set of participants that are easily available, willing to
conduct small tasks such as those involved in experiments, and usually involves a lower payment and time
overhead in the process of conducting the experiments.
The platform has already been used with promising
success in user studies67 and to validate the quality of
machine translation68 where Mechanical Turk workers
were asked to judge the quality of a translation.
Recently, Heer and Bostock69 studied the suitability
of MTurk as an experimentation platform for visual
perception, and found that the platform has potential for
conducting viable visualisation experiments at a lower
cost, faster completion time and with more participants
than typical laboratory studies. The seminal study on
graphical perception by Cleveland and McGill70 was
replicated on MTurk with identical results to the original study. In the replicated study on contrast,71 the
authors find that web-based experiments might, in fact,
be more representative of the general usage in day to day
life, rather than results in a laboratory based on a single
display. Among their findings, Heer and Bostock claim
that long tasks are not suited to MTurk as they are more
prone to ‘gaming’. On the basis of this insight, we split
the whole experiment in smaller chunks, yet promote
the completion of the entire set of experimental tasks by
offering a bonus for a complete set of answers.
6
Experiment Setup and Materials
In order to study the impact of the mode of presentation
on understanding dynamic networks, we conducted two
distinct yet related experiments. Before the actual experiments, the experiment design and material was tested
with a pilot group of social scientists, as distinct from
our subsequent computer science students and online
test subjects. Following the observations from the pilot
study, our first experiment is a group laboratory experiment with computer science students familiar with node
link diagrams. In the second experiment, we expand
the audience of the first experiment to unspecialised
participants by conducting the same experiment on the
MTurk platform as introduced in Section ‘Crowdsourcing
experiments’.
A within subjects factorial design was employed in both
experiments. The independent variables tested were 2×
visualisation Type (Animation versus Static Images), 2 × 2
Tasks (Global Network overview versus Local Individual
node versus Specified time period versus Unspecified time
period) 2× Density (Low density versus High density) and
2× questions. The total number of questions for each
participant were 2 × 4 × 2 × 2 = 32. In addition to this, four
questions were presented to each participant in the beginning of the experiment to help them familiarise themselves with the system and question types. This data was
subsequently discarded from the analysis.
Tasks
When studying dynamic networks, scientists are interested in change that occurs at different time periods in
the network. Change in the network can occur at two
levels; at the global network overview level and the local
individual node level. Overview level tasks, are tasks that
require the analysis of the entire network, whereas individual actor tasks focus an individual actor or a small
group of actors. On the basis of prior experience with
analysing dynamic networks,13 an analyst typically starts
by identifying the overall change in the network before
drilling down to investigate the actors that are responsible
for that change. This task distinction is also applicable
to static social network analysis and is synonymous with
overview and detailed tasks when analysing attribute data.
The temporal search space of a dynamic network is
another important variable when studying dynamic
networks. In a similar manner in which a network can be
studied from an overview perspective and a local node
perspective, the temporal search space can either be global
across the entire time period or else localised to focus on a
specified time period. Entire time period tasks question the
complete data collection period of the network whereas
specified time period tasks, question the network (or
individual actor) at a specific point in time, for example
in weeks 2, 3 and 4. Each of the first two task variables
can be combined with tasks from the second category to
Table 1: Examples of experiment questions for each task
Network overview – Specified time period (Connection)
Network overview – Entire time period (Density)
Actor detail – Specified time period (Degree)
Actor detail – Unspecified time period (Connection)
Find a node that has only one connection on weeks 3, 4 and 5?
Which week has the least number of connections?
How do the number of connections of Node 1 change in weeks 2, 3 and 4?
For how many weeks does the connection between Node 1 and Node 6 last?
obtain network level tasks within a specified time period,
network tasks throughout the entire time period, individual tasks with a specified time period and individual
tasks throughout the entire time period.
To formulate the user questions, we selected a number of
tasks related to graph visualisation72 and formulated them
in a dynamic context. For the individual actor tasks we
developed questions on actor degree, path length between
actors and general questions about the creation of new
links, and the removal of old links. As a purely dynamic
task for individual actors, we employed tasks on transitive
relations between actors.
For the overall network tasks we asked questions about
network density and the number of groups observed
throughout the entire network. An important network
wide observation in dynamic networks characterises how
actors form new ties, or change their linking preferences
over time.15 As a pure dynamic concept question in this
case, we included questions on homophilious patterns of
connection between actors in the network. The principle
of homophily states that new ties are more likely to form
with other actors that resemble oneself.73 An example of a
question on homophily is ‘Throughout the 6-week period,
green nodes tend to form more connections with blue
nodes, than with other green nodes?’. Other examples of
questions for each task type are described in Table 1.
The pilot questions asked to the social science students
and subsequently the computer science students were
questions that assumed a prior knowledge in interpreting
social network or graph drawings. As MTurk workers
might not necessarily be familiar with node link diagrams
and graph concepts, questions about paths were removed
and reformulated into other forms of questions. Also,
based on the results of the pilot study and the first laboratory user study, questions on transitivity proved to be
quite challenging, therefore they were removed from the
set of question types for the MTurk. The four different
types of question types used for MTurk included questions
about connections, density, degree and homophily. The
questions did not use any technical terms and were all
phrased in terms of the connection patterns of the nodes.
Hypotheses
In this work, we ground our hypotheses on previous work
on animation and the effect of motion on perception and
apply it to social network visualisation.
Hypothesis 1: Network overview tasks and no specified time
period– Based on the work on the benefits of motion in
overall tasks by McGrath and Blythe,47 we hypothesise
that animation will be beneficial, therefore (a) faster
and (b) more accurate, in analysing network overview
task without a constraint on the time period.
Hypothesis 2:Individual Actors and no specified time period
– The positive results of animation used for
transitions57 suggests that animation can be useful to
facilitate the retention of context between different
visualisations. If this notion is extrapolated a step
further, one can hypothesise that animation may be
beneficial, therefore (a) faster and (b) more accurate, for
tasks that require the user to follow a node throughout
the network.
Hypothesis 3: Specific time period tasks
– For localised tasks constrained by a time period, the
overhead of interaction with the video for searching,
stopping and pausing might result in animated presentations constrained by time being (a) slower than static
versions. The interaction overhead however should not
impact the (b) accuracy of the results.
Hypothesis 4: Network density – Following Robertson et
al,55 we hypothesise that the average time to respond
will be (a) faster and (b) more accurate in lower density
networks than in higher density networks. We are
unsure about the effect of density and representation
on each other, to this effect, we would like to investigate if the performance in the different representations
is effected by the density of the network.
Data sets
Two data sets with different densities were used in this
study. The first network (shown in Figure 1) is a low
density, manually created network containing nine nodes
across six different time periods. In this network, each
node has one attribute encoded by shape and colour. The
second network (shown in Figure 2) has a higher density
and consists of 32 nodes, 2 node attributes, 2 types of
edges and 6 time periods. In the visualisations, one of the
node attributes is represented by colour and the other by
shape. The second network is adapted from van Duijn
et al74 who studied the evolution of friendship among a
class of freshmen. We extracted part of this network that
included only relationships of strength 1, 2 and 5 and
removed edge direction. The reason for this simplification
is to make the network more visually legible by removing
edges of low semantic significance. Throughout, the
experiment no reference was made to the source of the
data in order to avoid any bias by participants who might
have been familiar with the data set.
7
coordinates and each subsequent frame uses the node
coordinates of the previous frame as a starting point. This
technique was used as it has been found to be the most
effective to generate these type of networks in previous
studies.17 The layouts generated by the algorithm were
improved by manually positioning some of the nodes
to increase readability and retain context between time
frames. SoNIA34 was used to generate all the visualisations and movies.
Web-based experiment system
Figure 1:
Low density network week 4.
Figure 2:
High density network week 4.
Network images and animation
The two data sets in the study are presented to the participants as static network images and as animated movies.
The two representations have an identical graph layout,
colours and labels. In fact, the static images are taken from
the six keyframes in the movie, one at each time point
in the network. The movies were created by interpolating
the positions of the nodes between the six keyframes
used as static images. The movies for each network were
generated at three speeds, normal (182 frames), slow
(242 frames) and slower (302 frames), according to the
feedback received during the pilot study.
The algorithm used for generating the layouts is the
Kamada-Kawai algorithm.75 To minimise the displacement of node position, chain-based anchoring is used
between the successive layouts. In this approach, the first
frame of each network is generated using random node
8
An online web-based system was specifically designed to
run the experiment. The system is compatible with all
major browsers with Flash and Javascript enabled to watch
the movies and collect time measurements, respectively.
The visual representations were displayed on the left,
while the question was displayed on the same page to the
right. The static images were laid out in a 3 × 2 grid, with
each of the static images having the same size as the video
(see Figure 3). Participants answered questions either by
inputting a number in a text box, or using radio buttons
for Likert scale questions. An option to skip a question
was available if the participant was unable to answer. For
any skipped questions participants were asked to supply
a reason for skipping the question. The questions of the
experiment were ordered using a Latin square design.
The interaction techniques of the video animation were
limited to play, pause and search for specific time position on the time bar. Pausing the video and clicking on
the timeline enabled participants to move directly to a
specific point in time. Participants were made aware of
this functionality but no effort was made to promote this
mode of interaction. Participants could also select their
speed preference for the movie playback by clicking on
the speed mode under the movie (see Figure 4).
At the end of the experiment, each participant was
presented with a list of adjectives and asked to select the
top five words applicable to each representation. The list
contained 100 adjectives and was equally split between
positive adjectives and negative adjectives. The lists of
words were randomised for each participant but kept the
same for each representation.
This qualitative data was gathered in an effort to understand the sentiments and impressions the participants
experienced while conducting the experiment. Although
a representation mode might be more effective in terms
of measurable quantities such as accuracy and timeliness,
there are other less easily quantifiable aspects that might
provide a reason for using alternative presentation modes.
Furthermore, a checklist of terms tends to encourage
feedback more than simple open-ended questions on
user preferences and this was reflected in the number of
answers received for the adjective list as opposed to the
open-ended question on the reason for representation
preference. The list of adjectives also attempts to understand more subtle reasons for a preference towards a
Figure 3:
Web-based experiment system showing static image questions.
representation by prompting the users with a list of
emotions that they might not have necessarily thought
about expressing but know that they have experienced.
Adaptation of the lab System to MTurk
The original system that was developed for the laboratory group study was adapted for MTurk. All the primary
materials used in this experiment, images, videos, data set
and experimentation system were kept the same. A minor
addition was made to the experiment system to allow the
users to pause the experiment. In a laboratory setting,
the surrounding environment is designed to minimise
external distractions and users are visually monitored.
When an experiment is being run over the internet in a
remote location, the experimenter loses all control of the
surrounding environment and distractions. To minimise
the effect of distractions on the timing of the results in
performing the experiment, a pause button was added to
the experimental system to allow the user to pause the
timer. When the pause button is pressed the experiment
materials are not displayed until the user presses the
resume button to continue the experiment. The experiment still timed out after 30 min of inactivity, therefore if
someone abandoned the experiment for a long time the
data was discarded.
Participants
In the experiments different sets of participants were used.
In the first experiment, the participants were seven males
(average age: 29, min: 23) postgraduate research students
in computer science, all of whom were familiar with node
link diagrams. The student background was known and
all the participants were previously screened before the
experiment in terms of suitability for the tasks.
In the second experiment, the participants were anonymous workers on Mechanical Turk. From the collected
gender and age information, there were 26 females
(average age: 29, min: 19, max: 57) and 33 males (average
age: 30, min: 17, max: 58). In this case, there was no
control over the participants and participants did not
necessarily have knowledge of node link diagrams, social
network or computer graphs. As reported in other studies
a large proportion of MTurk workers have a high level of
eduction,66 therefore understanding the basic principles
behind node link diagrams was probably not difficult.
The HITs submitted on MTurk were entered in four
batches of 24 questions each, over a period of 11 days.
Out of the 96 requests issued, 12 were rejected outright
and reissued for a total of 108 submissions. From the 108
submissions, 48 submissions were discarded because they
were completed by the same person more than once.
These multiple submissions, identified by the unique
worker id, were completed by seven workers. In this
9
Figure 4:
Web-based experiment system showing animated image questions.
case, only the first submission was taken into consideration and the others were discarded. From the number
of submissions only three people did not complete the
experiment in full. In order to have standardised results
and retain the question ordering and within subjects
design, these three submissions were discarded. All the
submissions were screened for obvious signs of gaming of
the system such as answering quickly and inaccurately.
After cleaning the data, the total number of participants
for the second experiment were 57.
Laboratory group experimental method
The experiments for the group of computer scientists
were held in a classroom environment with each of the
seven participants conducting the experiment simultaneously. Before each experiment, a brief tutorial was
presented to explain the use of the system. A set of four
test questions, each with a different representation and
10
task type were given to make the participants familiar
with the system. Participants were encouraged to experiment with the system controls during these four example
questions.
MTurk experimental method
The actual experiment platform was hosted on our
servers to ensure maximum control on the presentation
of content, collecting accurate timing data per question
and control the question order to conform to the original Latin Square design. The MTurk HIT consisted of a
description of the task with a link pointing to the experiment system and a passcode to uniquely identify each
HIT. Upon finishing the experiment each participant was
given a code to input in MTurk to finish the HIT.
In order to keep in spirit with the small discernible
tasks forming HITs on MTurk, we provided the option
for the participants to finish the experiment early after
completing each batch of four tasks. Each batch of four
questions was worth 0.10c, and there was a total of nine
batches. After each batch of four tasks was completed by
the participant, a screen showing the monetary reward
up to that point and asking if the user wants to continue
was displayed. In order to incentivise the participant
to complete the entire experiment, the participant was
notified of a US $1 bonus if the entire experiment was
completed. If the participant opted out of the experiment
a question asking the reason for opting out was presented
followed by the final qualitative questions. After the
participant finished the experiment (or decided to stop
prematurely), a passcode reflecting the level of completion was provided. We did this to prevent publishing the
entire experiment as one long task with a high reward
possibly attracting workers that want to ‘game’ the system
and gain a ‘large’ some of money for a little reward.69
The MTurk platform does not provide a way to bundle a
group of HITs together as a HIT can only be either rejected
or approved. For this reason, all the HITs were published
with the minimum amount of money for each HIT (0.10c)
and not with the total amount, as otherwise people who
do not complete the entire experiment will still get the
maximum amount of money. The monetary balance based
on the completion of the experiment was transferred to
the participant in the form of a bonus in MTurk.
When conducting an experiment remotely the experimenter has no control over the selection of participants,
the surrounding environment and the computer setup of
the participants. In order to ensure that the participants
had a computer setup that conforms to the minimum
requirements of the experiment, each remote participant
had to pass a screening task before being able to participate. The screening task text provided the basic information necessary to interpret node link diagrams and
answer the experiment questions. Each participant then
had to pass a simple test that tested both the basic understanding of node link diagrams and also ensured that each
participant had the minimum requirements to conduct
the experiment (that is, watch a movie using Flash).
because of the small number of participants, these results
are not statistically significant. In the second experiment
the same trends can be observed, but in this case most of
the results are statistically significant. The performance
of both groups of different users suggests that throughout
most of the experiments, questions answered using static
presentation were faster and sometimes more accurate
than questions answered using animation. In general,
this difference is stronger in measures related to time
than measures of accuracy.
In the laboratory experiment, questions with a static
mode of presentation were answered faster (M = 56.45 s)
than those with an animated presentation (M = 62.22 s).
This difference however is not significant. In the second
experiment, the t-test did reveal a significant difference
in time to complete tasks between static and animated
displays, t(904) = 9.154, (P < 0.001). This indicates that
the average time to complete tasks using static representations (M = 35.16 s) was significantly lower than
the average time to complete tasks using animation
(M=47.55 s). The average time to complete is lower in the
second experiment because the questions were simplified
and in general were easier to answer. The experiment was
designed this way due to the remote experimentation
platform use and the anonymous possibly non specialised
participants, taking part in the second experiment.
The total accuracy in the first experiment was 85 per
cent, whereas the accuracy of the second experiment was
86 per cent. The accuracy is sufficiently high in both cases
to indicate that the participants were generally able to
answer correctly and the questions were not too difficult.
Participants in the second experiment who might have
had no previous experience with node link diagrams were
still able to answer most of the questions correctly. The
similar accuracy rating of the participants suggests that
the simplification of the questions for the second experiment using a non-specialised audience was appropriate
and adequate. The number of errors made when animated
presentation was used were higher (59 per cent of the total
in the first experiment and 56 per cent of the total in the
second), and the difference was significant t(904) = 2.660,
(P < 0.01).
Results
In this section, we report on error rates, time to complete
task, and descriptive preferences of the participants. The
paired samples t-test was used to analyse both measures.
General observations
In the two experiments with the two different user groups
the main results obtained are generally consistent. In
terms of results, the first laboratory experiment with the
smaller computer science students user group acted as
a prequel to the larger experiment conducted with an
anonymous audience. Most of the results of the first
experiment are indicative of the general trend, however
Task differences
The 2 × 2 tasks for each group were cross tabulated and
analysed in terms of average number of errors and average
completion time. Once again both experiments reported
similar results, in that static images were more effective in
both time and accuracy. Figures 5 and 6 give an overview
of the results in graphical format and Tables 2–5 give the
results in tabular format.
Network overview tasks with no specified time period
Throughout the conditions tested in both experiments,
the only measure that reported a positive difference for
11
Figure 5:
Experiment 1 result overview.
Figure 6:
Experiment 2 result overview.
Table 2: Average time to complete task – experiment 1
Table 4: Average time to complete task – experiment 2
Average time to
complete
Time
Representation
Animated
Static
Average time to complete
Time
Representation
Animated Static
Actor detail
No specified time
Specified time
41.10
56.28
36.31
53.15
Actor detail
No specified time
Specified time
62.83*
37.99*
44.81*
32.37*
Network overview
No specified time
Specified time
71.01
77.42
70.66
67.66
Network overview
No specified time
Specified time
48.14*
42.88*
35.39*
29.65*
∗ Significant differences between averages.
* Significant differences between averages.
Table 5: Average error per task – experiment 2
Table 3: Average error per task – experiment 1
Average no of errors
Time
Representation
Animated
Static
Actor detail
No specified time
Specified time
0.04
0.13
0.03
0.04
Network overview
No specified time
Specified time
0.21
0.31
0.28
0.13
Average no of errors
Time
Representation
Animated
Static
Actor detail
No specified time
Specified time
0.25
0.14
0.22
0.11
Network overview
No Specified Time
Specified time
0.16*
0.06
0.11*
0.03
∗ Significant differences between averages.
animation was the accuracy measure for network overview
tasks with no specified time period (Hypothesis 1b) in the
first experiment. In the first experiment, animated images
M = 0.21 reported a lower error rate than static images
12
M = 0.28, yet this difference is not significant. For the
time to respond, the means are almost equal (M = 71.01 s)
for animated and (M = 70.66 s) for static images and the
difference is also not statistically significant.
The only time the results between the experiments show
different trends is in this task. Network wide questions
without a specified time period were answered faster using
static images (M = 35.39) than using animated movies
(M = 48.14), t(226) = 3.815, P < 0.001. Here, questions
answered using animated movies had a higher error rate
M=0.16 than those answered with static images M=0.11.
In this case, the difference is also significant t(226) = 1.95,
P < 0.05. The results obtained from the second experiment
contradict our first Hypothesis (1a and 1b), where animation is believed to be faster and more accurate in tasks on
network overviews and no specified time.
statistically significant. In the second experiment, the
average time to reply in the dense networks is M = 47.57 s
whereas in the lesser dense network it is M = 35.70 s,
t(903) = −8.559 , P < 0.001. In case of errors, the difference is 0.13 in the lesser dense network and 0.14 in the
dense network.
We also conduct an RM-ANOVA analysis on the time
measure of density data of the second experiment, to
study the relationship between density and representation. From this analysis, we observe an effect of both
density F(1448) = 70.48, P < 0.001 and representation
F(1448) = 64.49, P < 0.001, but there was no significant
interaction between density and representation.
Individual actors and no specified time period
In the first experiment, the results show that static images
were both faster (M = 36.31 s for static, M = 41.10 s for
animated) and had a smaller error rate (0.03 for static,
0.04 for animated) than animated movies, however both
differences are not statistically significant.
In the second experiment, a significant difference
confirming that static images (M = 44.81s) are faster than
animated movies (M = 62.83) is reported t(226) = 6.017,
P < 0.001. The difference in terms of error rate is not
statistically significant P < 0.286. This result also contradicts our initial hypothesis related to speed (2a), but is
inconclusive on the second part of the hypothesis related
to accuracy (2b).
For tasks querying individual actors where the time
condition is specified, the difference is significant for the
timing measurement t(226) = 3.493, P < 0.01) but not for
the error measurement.
Participants’ comments and preferences
Specific time period tasks
In the first experiment, both the response time (M=60.41s
for static, M = 66.85s for animated) and the error rate of
static images (M = 0.08 for static, M = 0.21 for animated)
have better performance than the animated representation, for tasks with a specific time period specified. Again,
the results in this experiment are not significant.
In the second experiment, the result mirror the results
in the first experiment but have different significance
values. The tests revealed a lower time to respond in the
case of static visualisation (M = 30.19 s) when compared
to animated visualisations (M = 40.41 s), t(451) = 6.264,
P < 0.001. For the error rate measurement, the differences between static (M = 0.07) and animated (M = 0.10)
representations is statistically significant t(451) = 1.636,
P < 0.103. This result confirms our third Hypothesis 3a
and 3b.
Network density
In Hypothesis 4, we postulate that denser networks
lead to more errors and take a longer time to complete.
This hypothesis is supported in the second experiment
for the time to respond but not for the number of
errors. In the first experiment, both differences are not
At the end of the exercise, participants were asked to
comment on both visual representations and indicate
their preference between the static and animated displays.
From the participants who answered that question, six
preferred animations and 22 preferred static images.
Perhaps, the most informative comment on this can
be expressed by one of the participants who wrote ‘I
really liked the videos but they were difficult to use if
you are being timed, I felt a little rushed with them and
turned them to the slowest speed’. One other participant
remarked that he preferred ‘Static mostly, for accuracy.
But the video was useful for following a single node
around more easily’. This was in line with our original
intuition, yet the quantitative results did not confirm it.
The main reasons in favour of static displays included;
‘easier to access information selectively’, ‘easier comparisons over time periods’, ‘easy to verify results’. Some
participants reported that they had to stop the animation
and manually browse through the static frames to register
the change. This behaviour was also noticed during the
running of the laboratory experiment. In the words of
one participant, ‘when trying to compare the transition
between two weeks, it was very useful to be able to flip
back and forth between two frames in the timeline and
have them presented one on top of the other as in the
animation.’
Some MTurk workers also commented positively on
the nature of the HIT and expressed encouragement
towards publishing other similar HITs. One participant
also remarked on the novel structure of the HIT; ‘Hit was
fairly enjoyable to complete. I like how it was broken up
into four parts each, and I was able to stop and collect
the money I earned without having to do all the questions if I didn’t want to. More hits broken up like that
would be great!’. Some workers were even interested in
the experiment and asked questions on the purpose of
the experiment and the outcome (‘I wonder what you
were really testing. It was interesting stuff anyway.’).
It would probably be a good idea for future experiments to
provide more information on the publications resulting
from the experiment to create a stronger bond with the
participants.
13
site words to the ones used for describing animation
can be observed. The visualisation also clearly illustrates that participants preferred static images for their
efficiency.
Discussion
Figure 7: Word cloud for adjectives selected for static representations.
Figure 8: Word cloud for adjectives selected for static representations.
In order to get a general impression of the sentiments that the visualisations elicited from the participants, each participant was asked to choose the top
five words from a list of 100 words equally divided
between positive or negative sentiments. Figures 7 and
8 show the list of static and animated words chosen by
the participants respectively. In the figures, the size of
the words corresponds with the number of times the
word was chosen, whereas the locations of the words are
random.
The sentiments expressed for the respective representations are evident from the two diagrams. Most
participants felt that animated images were harder and
more difficult to use than the static alternatives. Participants also remarked on the time consuming nature
of animation, a measure which was clearly reflected in
the quantitative results. Time is of essence to workers
on Mechanical Turk, since the longer a task takes the
less money per HIT they can make, therefore this sentiment is well understood. Conversely, when one looks
at the sentiments expressed for static images, oppo-
14
Similarly to other experimenters who were the first to test
animation in a field that considered animation to be the
natural way of representing temporal information,50 we
were surprised at the overwhelming evidence in favour
of static images. From the results, it is clear that for these
experiments the static representations allowed participants to perform the tasks faster, and in some cases more
accurately than animated representations. While for some
tasks this was expected from the original hypotheses,
most of the hypotheses on the benefits of animation
were contradicted. In this discussion, we shall attempt to
outline and understand some possible reasons for this.
One possible reason is that interaction with animated
displays was very limited, therefore the full potential
of using animation was not exploited. Previous studies
particularly in algorithm animations52 and our previous
experiences43 have observed that interaction with
animated displays is crucial during analysis. In the experiments, animation interaction was limited to pause, seek
and play operations on the movie, rather than the interactions typically available in a complete interactive system.
Time constraints and a desire to limit confounding factors
in our experiments limited the use of such a complex
system in the experiments.
When animating one is effectively interpolating
between many different frames creating many in-between
or artificial graph layouts which in reality do not exist. If
a user is interacting with these animations by starting and
stopping the animations the search space is much larger
because of the added interpolated frames. In this case,
when a user is searching a video it takes much longer
than searching a static image that quickly provides access
to the final solution of the graph. A possible suggestion
in this case, considering that some participants admitted
that they used the animation like a static image by
browsing the video, would be to let the user flip between
keyframes and only animate when the play button is
on. This means adding an additional interaction parameter to allow the user to browse between timeframes,
slideshow style. Alternatively, keyframe locations can be
highlighted on the video time line.
Another possible reason is that most of the tasks in the
experiment were low level topological tasks that require
careful analysis of the network instead of a general overall
perspective. In our first hypothesis, we believed that
animation will be beneficial for general overview tasks,
however this hypothesis was contradicted in both time
and accuracy measures. While the network tasks were
real overview tasks the questions might have been too
specific, requiring the participants to look carefully for
answers instead of trying to identify a general trend.
General overview questions might also benefit from a
layout scheme that emphasises the trend with respect to
the position of the nodes.
One of the participants commented that questions
answered using animation felt like trying to hit a moving
target. This comment might give some insight why our
second hypothesis, where we thought that animation will
be beneficial in tracking a single node, was contradicted.
In our animation all the nodes were moving throughout
the whole animation, thus possibly distracting the attention of the participants. One needs to be very careful
when using animation, and it is probably better if animation is used sparingly rather than throughout a whole
visualisation. Selectively animating parts of the network
to exploit the pre-attentive nature of motion can prove
to be more effective then simply animating the whole
network.
Interestingly, in our tests on density, the number of
errors in the denser network was not significantly higher
than the number of errors in the lower density network.
One aspect of density that was not tested in this experiment was time density. Both networks spanned a period
of six weeks, which resulted in 6 individual images and
six key frames of video. It will be interesting to test animation on a higher time density network, say for example
spanning a period of 20–200 time points. In such a case,
looking at 20–200 different images of the network might
be impractical as static images consume plenty of screen
space, whereas animation or a flipbook may be easier
to control as they can be contained in one screen. One
problem with this sort of experiment would be the time
required to run the experiment and the necessary precaution not to cognitively overload participants with a task
that is too difficult. In this respect, we believe that a more
ethnological type of study is probably a more realistic
approach to study the analyst’s toolbox.
The quantitative results from the measurements support
the comments written by the participants in the final
survey. This result is noteworthy because it informally
validates the positive benefits of static images. The major
complaint for animation, as can be clearly seen from the
graphic in Figure 8, is that animation is hard to use. In
our laboratory-based studies, some of the participants felt
that animation took more time to become familiar with.
From our word analysis, we found that participants in the
laboratory tended to be more careful in selecting negative
sentiments than anonymous participants, especially with
respect to animation.
Although for most tasks static representations were
generally more accurate, the results were for the most
part not statistically significant. We believe that there
are still possibilities for using animation as an analytical
tool, especially if animation is used selectively and with
care. For example, animation can be useful to illustrate
dynamics when presenting solutions, aided by additional voice over explanations of the network dynamics.
Participants used positive sentiments such as ‘exciting’,
‘engaging’ and ‘fun’ when describing animated representations.
Using remote web-based services for experimentation
Web-based experimentation and web-based workers give
the researcher the opportunity to reach a diverse participant group that extends beyond the typical sample of
students typically available within a university context.
This setup is a good stepping stone to start generalising
certain types of experiments in a possibly more authentic
setup than a laboratory space. Although we are still at
a very early stage of understanding the benefits, drawbacks and possibilities that these web-based platforms
can provide for research, we are encouraged by the results
obtained and the quality of work obtained from the
participants using the site. We caution other researchers
to structure their experiments carefully and to consider
any novelty effect in future uses of such a platform in
experimental design.
From our experience, we found that people who started
the experiment were willing to complete the whole experiment, even if they were given the option to opt out and
claim money at regular intervals. It will be interesting in
future studies to measure the effect of the size of the bonus
on the completion rate of the experiments. In some cases,
the same participants completed the experiment several
times in a row, giving the impression that workers on the
opposite side are working in a production line. Workers
on the site are accustomed to the repetitive nature of the
tasks typically found on the site, which makes running
longer experiments practical. In hindsight, the number of
tasks run in the experiment could have been extended for
the web-based experiment.
One area that we were sceptical about during the design
of the experiment with Mechanical Turk was in the collection of timing data from a remote experiment without
the ability to control distractions in the environment.
In reality, the timing data collected was well within the
expected range of task time completion and there were
no obvious instances where the time kept on running for
a long time that was disproportionate to the task length.
We can conjecture that the fact that the participants were
made aware of the importance of timing and the addition of the pause button helped to ensure more accurate
records.
Apart from a few users who completed the experiment
multiple times we did not find any instances of trying to
game the system by answering quickly and inaccurately.
We were careful to take into consideration the recommendations by other researchers who detected instances of
gaming, such as keeping the initial reward relatively low
not to attract undesirable participants and pre-screening
the workers. It could also be the case that workers are
learning that if they try to abuse the system in this way
their employees can easily detect their ploy and reject the
tasks accordingly.
15
We believe that there is promise in web-based experiments and that this experiment platform and traditional
laboratory experiments can complement each other. The
obvious benefits of participant resourcing, low cost and
relatively fast data collection times make remote web
systems attractive for the researcher. Naturally, the results
obtained from this platform need to be interpreted in
the context of the platform and care must be exercised
in ensuring that the data is clean before being used for
analysis.
Future Work and Conclusions
When interpreting networks it is impossible to be
completely independent of the graph layout used to
generate the images. As previous studies have shown23,37
the graph layouts have a significant effect on the interpretation and understanding of the graph structure. For
this experiment, we use a layout algorithm that was originally designed for static networks, yet there are still very
few graph layout algorithms designed specifically for
dynamic networks, that are available in social network
analysis toolkits. Furthermore, in these experiments the
dynamic network visualisation was restricted to node
link diagrams. There is further scope to look at different
visual representations for dynamic network data that
extends beyond the current paradigms of network visualisation, perhaps even being designed with an eye towards
exploiting the power of motion in subtler ways.
As we have discussed, there is scope for extending
this study to test other criteria such as time density and
perhaps investigate the difference between structural and
analytical question formulation for tasks. Furthermore,
more interactive features can be added to the animated
displays to study the effect of interaction on network
animation. Improvement in dynamic network layouts
will undoubtedly be of great help to dynamic network
visualisation irrespective of the representation medium
used to visualise the network.
The aim of this user study was to understand the
strength of static images and animated movies to analyse
dynamic social networks. We found that static images
provide better performance than their animated counterparts. The results obtained though do not justify giving
up further studies on animated representations. Nevertheless, improvements and further studies are required
to understand better the beneficial uses of animation in
applied social network analysis.
References
1 Doreian, P. and Stokman, F. (1997) Evolution of Social Networks.
Amsterdam: Gordon and Breach Publishers.
2 Christakis, N. and Fowler, J. (2008) The collective dynamics of
smoking in a large social network. New England Journal of Medicine
358(21): 2249.
16
3 van Duijn, M., Zeggelink, E., Huisman, M., Stokman, F. and
Wasseur, F. (2003) Evolution of sociology freshmen into a friendship network. Journal of Mathematical Sociology 27(2): 153–191.
4 Xu, J., Marshall, B., Kaza, S. and Chen, H. (2004) Analyzing and
visualizing criminal network dynamics: A case study. Proceedings
of the Second Symposium on Intelligence and Security Informatics
Springer.
5 Moreno, J.L. (1941) Foundations of sociometry: An introduction.
Sociometry 4(1): 15–35.
6 Freeman, L.C. (2000) Visualizing social networks. Journal of Social
Structure 1.
7 Eades, P., Marks, J., Mutzel, P. and North, S. (1997) Graph-drawing
contest report. In: G. Dibattista (ed.), Graph Drawing. Berlin,
Heidelberg: Springer, pp. 423–435.
8 Fekete, J., Grinstein, G. and Plaisant, C. (2004) InfoVis 2004
Contest: The History of InfoVis. IEEE, http://www.cs.umd.edu/
heil/iv04contest/.
9 Grinstein, G., Plaisant, C., Laskowski, S., O’ Connell, T., Scholtz,
T., and Whiting, M. (2008) Vast 2008 Challenge: Introducing
Mini-challenges. Visual Analytics Science and Technology, 2008.
VAST ’08. IEEE Symposium.
10 Erten, C., Harding, P., Kobourov, S., Wampler, K. and Yee,
G. (2003) GraphAEL: Graph animations with evolving layouts.
Lecture Notes in Computer Science 2912: 98–110.
11 Bender-deMoll, S. and McFarland, D. (2006) The art and science
of dynamic network visualization. Journal of Social Structure 7(2).
12 Leydesdorff, L., Schank, T., Scharnhorst, A. and De Nooy, W.
(2008) Animating the development of Social Networks over
time using a dynamic extension of multidimensional scaling. El
Profesional de Informacion 17(6): 611–626.
13 Farrugia, M. and Quigley, A. (2009) TGD: Visual data exploration
of Temporal Graph Data. In: K. Borner and J. Park (eds.)
Visualization and Data Analysis 2009, Vol. 7243, No.1, San Jose,
CA: SPIE, p.11.
14 Archambault, D. (2009) Structural differences between two graphs
through hierarchies. Proceedings of Graphics Interface 2009.
15 Powell, W., White, D., Koput, K. and Owen-Smith, J.
(2005) Network dynamics and field evolution: The growth of
interorganizational collaboration in the life sciences. American
Journal of Sociology 110(4): 1132–1205.
16 Pajek. http://pajek.imfm.si/doku.php.
17 Moody, J., McFarland, D. and Bender-deMoll, S. (2005) Dynamic
network visualization 1. American Journal of Sociology 110(4):
1206–1241.
18 Wasserman, S. and Faust, K. (1994) Social Network Analysis.
Cambridge UK.
19 Herman, I., Melançon, G. and Marshall, M. (2000) Graph
visualization and navigation in information visualization: A
survey. IEEE Transactions on Visualization and Computer Graphics
6(1): 24–43.
20 Di Battista, G., Eades, P., Tamassia, R. and Tollis, I. (1999) Graph
Drawing; Algorithms for the Visualization of Graphs. Upper Saddle
River, NJ: Prentice Hall.
21 Kaufmann, M. and Wagner, D. (2001) Drawing Graphs: Methods
and Models. Springer-Verlag: London, UK.
22 Di Battista, G., Eades, P., Tamassia, R. and Tollis, I. (1994)
Algorithms for drawing graphs: An annotated bibliography.
Computational Geometry: Theory and Applications 4(5): 235–282.
23 Purchase, H. (1997) Which aesthetic has the greatest effect on
human understanding? In: G. Di Battista (ed.) Graph Drawing.
Lecture notes in Computer Science, Vol. 1353. Berlin; Heidelberg:
Springer, pp. 248–261.
24 Ware, C., Purchase, H.C., Colpoys, L. and McGill, M.
(2002) Cognitive measurements of graph aesthetics. Information
Visualization 1(2): 103–110.
25 Misue, K., Eades, P., Lai, W. and Sugiyama, K. (1995) Layout
adjustment and the mental map. Journal of Visual Languages and
Computing 6(2): 183–210.
26 Purchase, H., Hoggan, E. and Görg, C. (2007). How important
is the ‘mental map’? – An empirical investigation of a dynamic
graph layout algorithm. Lecture Notes in Computer Science 4372:
184.
27 Brandes, U. (2001). Drawing on physical analogies. Lecture Notes
in Computer Science 2025: 71–86.
28 Brandes, U. and Wagner, D. (1997) A bayesian paradigm for
dynamic graph layout. Lecture Notes in Computer Science 1353:
236–247.
29 Frishman, Y. and Tal, A. (2008) Online dynamic graph drawing.
IEEE Transactions on Visualization and Computer Graphics 14:
727–740.
30 Diehl, S., Görg, C. and Kerren, A. (2001) Preserving the Mental
Map Using Foresighted Layout. Proceedings of the Joint EurographicsIEEE TCVG Symposium on Visualization VisSym ’01, Ascona,
Switzerland, Wien, Austria: Springer Verlag, pp. 175–184.
31 Lee, Y., Lin, C. and Yen, H. (2006) Mental Map Preserving Graph
Drawing Using Simulated Annealing. Proceedings of the 2006
Asia-Pacific Symposium on Information Visualisation-Volume 60,
Australian Computer Society, p. 188.
32 Davidson, R. and Harel, D. (1996) Drawing graphs nicely
using simulated annealing. ACM Transactions on Graphics 15(4):
301–331.
33 Dwyer, T. and Gallagher, D. (2004) Visualising changes in fund
manager holdings in two and a half-dimensions. Information
Visualization 3(4): 227–244.
34 Bender-deMoll, S. and McFarland, D. (2006) The art and science
of dynamic network visualization. Journal of Social Structure 7(2)
35 Bridgeman, S. and Tamassia, R. (2002) A user study in similarity
measures for graph drawing. Lecture Notes in Computer Science 6(3):
225–254.
36 Friedrich, C. and Eades, P. (2002) Graph drawing in motion.
Journal of Graph Algorithms and Applications 6(3): 353–370.
37 Blythe, J., McGrath, C. and Krackhardt, D. (1996) The effect
of graph layout on inference from social network data.
Proceedings of the Symposium on graph Drawing GD ’95. London:
Springer-Verlag, pp. 40–51.
38 Tversky, B., Morrison, J. and Betrancourt, M. (2002) Animation:
can it facilitate?: International Journal of Human-Computer Studies
57(4): 247–262.
39 McFarland, D. (2001) Student resistance: How the formal and
informal organization of classrooms facilitate everyday forms
of student defiance1. American Journal of Sociology 107(3):
612–678.
40 Brandes, U., Kääb, V., Löh, A., Wagner, D. and Willhalm, T. (2000)
Dynamic WWW Structures in 3D. Journal of Graph Algorithms and
Applications 4(3): 183–191.
41 Ke, W., Börner, K. and Viswanath, L. (2004) Major information
visualization authors, papers and topics in the ACM library.
Proceedings of the IEEE Symposium on Information Visualization.
Washington DC: IEEE Computer Society, p. 216.1.
42 Lee, B., Czerwinski, M., Robertson, G. and Bederson, B. (2004)
Understanding Eight Years of Info Vis Conferences Using PaperLens.
Washington DC: IEEE Computer Society, p. 216.3.
43 Farrugia, M. and Quigley, A. (2008) Cell phone mini challenge:
Node-link animation award animating multivariate dynamic
social networks. IEEE Symposium on Visual Analytics Science and
Technology, 2008. VAST ’08; October, IEEE, pp. 215–216.
44 Correa, C.D., et al. (2008) Cell phone mini challenge award:
intuitive social network graphs visual analytics of cell phone
data using Mobivis and Ontovis. IEEE Symposium on Visual
Analytics Science and Technology, 2008. VAST ’08; October. IEEE,
pp. 211–212.
45 Perer, A. (2008) Using socialaction to uncover structure in social
networks over time. IEEE Symposium on Visual Analytics Science
and Technology, 2008. VAST ’08; October. IEEE, pp. 213–214.
46 Perer, A. and Shneiderman, B. (2006) Balancing systematic and
flexible exploration of social networks. IEEE Transactions on
Visualization and Computer Graphics, 12: 693–700.
47 McGrath, C. and Blythe, J. (2004) Do you see what i want you
to see? the effects of motion and spatial layout on viewers’
perceptions of graph structure. Journal of Social Structure 5(2).
48 Ware, C. and Bobrow, R. (1985) Motion to support rapid
interactive queries on node-link diagrams. ACM Transactions on
Applied Perception (TAP) 1(1): 3–18.
49 Brown, M. and Sedgewick, R. (1985) Techniques for algorithm
animation. IEEE Software 2(1): 28–39.
50 Stasko, J., Badre, A. and Lewis, C. (1993) Do Algorithm
Animations Assist Learning?: An Empirical Study and Analysis.
Proceedings of the INTERACT’93 and CHI’93 Conference on Human
Factors in Computing Systems ACM CHI ’93; Amsterdam; The
Netherlands; New York: ACM, pp. 61–66.
51 Kehoe, C., Stasko, J. and Taylor, A. (2001) Rethinking the
evaluation of algorithm animations as learning aids: An
observational study. International Journal of Human Computer
Studies 54(2): 265.
52 Lawrence, A., Badre, A. and Stasko, J. (1994) Empirically
Evaluating the Use of Animations to Teach Algorithms. Proceedings
of the 1994 IEEE Symposium on Visual Languages, Citeseer,
pp. 48–54.
53 Rosling, H. (2006) Debunking myths about the ‘third world’,.
http://www . gapminder . org / videos / ted - talks / hans - rosling - ted2006-debunking-myths-about-the-third-world/.
54 Rosling, H. (2007) The seemingly impossible is possible,
http://www.gapminder.org/videos/ted-talks/hans-rosling-ted-talk2007-seemingly-impossible-is-possible/
55 Robertson, G., Fernandez, R., Fisher, D., Lee, B. and Stasko,
J. (2008) Effectiveness of animation in trend visualization.
IEEE Transactions on Visualization and Computer Graphics 14(6):
1325–1332.
56 Yee, K.-P., Fisher ,D., Dhamija, R. and Hearst, M. (2001) Animated
Exploration of Dynamic Graphs with Radial Layout. Proceedings
of the IEEE Symposium on Information Visualization. INFOVIS ’01.
Washington DC: IEEE Computer society, p. 43.
57 Heer, J. and Robertson, G. (2007) Animated transitions in
statistical data graphics. IEEE Transactions on Visualization and
Computer Graphics 13(6): 1240–1247.
58 Elmqvist, N., Dragicevic, P. and Fekete, J.-D. (2008) Rolling
the dice: Multidimensional visual exploration using scatterplot
matrix navigation. IEEE Transactions on Visualization and Computer
Graphics 14(6): 1148–1539.
59 Snijders, T., Steglich, C., Schweinberger, M. and Huisman,M.
(2007) Manual for SIENA, version 3.1.
60 Ware, C. (2004) Information Visualization: Perception for Design.
San Francisco, CA: Morgan Kaufmann.
61 UCINET. http://www.analytictech.com/ucinet/ucinet.htm.
62 Batagelj, V. and Mrvar, A. (1998) Pajek-program for large network
analysis. Connections 21(2): 47–57.
63 Brandes, U. and Wagner, D. (2003) Visone: Analysis and
visualization of social networks. Graph Drawing Software. Secaucus,
NJ: Springer-Verlag New York, pp. 321–340.
64 Turk, M. http://www.mturk.com.
65 Ipeirotis, P. (2010) Demographics of Mechanical Turk, http://behindthe-enemy-lines.blogspot.com/2010/03/new-demographics-ofmechanical-turk.html.
66 Ross, J., Irani, L., Silberman, M., Zaldivar, A. and Tomlinson ,
B. (2010) Who are the Crowd-workers?: Shifting Demographics
in Mechanical Turk. Proceedings of the 28th of the International
Conference Extended Abstracts on Human Factors in Computing
Systems ACM, CHI EA’10; Atlanta, GA. New York: ACM, pp.
2863–2872.
67 Kittur, A., Chi, E., and Suh, B. (2008) Crowdsourcing user studies
with Mechanical Turk. Proceeding of the Twenty-Sixth Annual
SIGCHI Conference on Human Factors in Computing Systems ACM,
CHI ’08; Florence, Italy. New York: ACM, pp. 453–456.
68 Callison-Burch, C. (2009) Fast, cheap, and creative: Evaluating
Translation quality using Amazon’s Mechanical Turk. Proceedings
of the 2009 Conference on Empirical Methods in Natural
Language Processing; Association for Computational Linguistics,
pp. 286–295.
69 Heer, J. and Bostock, M. (2010) Crowdsourcing graphical
perception: Using mechanical Turk to assess visualization design.
Proceedings of the 28th of the International Conference Extended
Abstracts on Human Factors in Computing Systems, CHI ’10; Atlanta,
GA; New York: ACK, pp. 203–212.
70 Cleveland, W. and McGill, R. (1984) Graphical perception:
Theory, experimentation, and application to the development of
graphical methods. Journal of the American Statistical Association
79(387): 531–554.
17
71 Stone, M., Bartram, L. and Consulting, S. (2009) Alpha, contrast
and the perception of visual Metadata. Sixteenth Color Imaging
Conference: Color Science and Engineering Systems, Technologies and
Applications, Vol. 16, November 2008, Portland, OR: The Society
for Imaging Science and Technology.
72 Plaisant, C., Lee, B., Parr, C., Fekete, J. and Henry, N. (2006) Task
taxonomy for graph visualization. Beyond Time and Errors: Novel
Evaluation Methods for Information Visualization (BELIV 06): 82–86.
18
73 McPherson, M., Smith-Lovin, L. and Cook, J. (2001) Birds of a
feather: Homophily in social networks. Annual Review of Sociology
27(1): 415–444.
74 van Duijn, M., Zeggelink, E., Huisman, M., Stokman, F. and
Wasseur, F. (2003) Evolution of sociology freshmen into a friendship network. Journal of Mathematical Sociology 27(2): 153–191.
75 Kamada, T. and Kawai, S. (1989) An algorithm for drawing general
undirected graphs. Information Processing Letters 31: 7–15.

Effective temporal graph layout: A comparative

Transcription

Similar documents

Toopy and Binoo`s Cubic Puzzles Set up the game… Ask a grown

resume - Dragonflyent.net

TV ART DESIGN

Lecture 13: Map animation

Computer Game and Animation Design

Fluid and Powerful Animation within Frame Restrictions