RemoteUI: A High-Performance Remote User Interface System for

RemoteUI: A High-Performance Remote User Interface System for

RemoteUI:
A High-Performance Remote User Interface System
for Mobile Consumer Electronic Devices
Daniel Thommes, Student Member, IEEE, Ansgar Gerlicher, Member, IEEE
Qi Wang, Member, IEEE, Christos Grecos, Senior Member, IEEE
Abstract — Modern consumer electronic devices can easily
integrate into distributed computing environments. By
applying network-based user interface systems to control these
devices, innovative applications become feasible. However,
several requirements like high responsiveness and low
bandwidth consumption need be satisfied. Particularly, to be
applicable in wireless and mobile local and wide area
networks, they must circumvent low available data rates and
high network latency. We propose a new solution called
RemoteUI, which works with abstract user interface (UI)
descriptions and their prioritized, incremental remote
replication. The system combines different approaches from
thin client computing and widget based UI systems and is
further optimized in the context of wireless networks and
mobile consumer devices. To demonstrate the advantages of
our solution, we experimentally compare the proposed
RemoteUI System with Virtual Network Computing (VNC)
implementations and an HTML-based solution1.
Index Terms — Remote User Interfaces, Mobile Consumer
Electronic Devices, VNC, Web4CE, HTML
I. INTRODUCTION
A growing number of consumer electronic devices can be
integrated into IP networks allowing them to use and offer
pervasive services. Particularly, the transfer of user interfaces
(UIs) to remote control devices is an essential technique to
facilitate such services in a user-friendly and locationindependent manner. Scheduling the next TV-recording on a
TV set or setting the temperature at home with the help of a
smart climate control device can be carried out from remote
locations. Versatile mobile consumer electronic devices like
smartphones or tablet computers are today's remote controls
"to go". They are connected to the Internet and capable of
presenting rich user interfaces for a comprehensive number of
applications. In this way they become a desirable amendment
1
Daniel Thommes is with the Institute of Applied Science, Stuttgart Media
University, 70569 Stuttgart, Germany (e-mail: [email protected]).
Ansgar Gerlicher is with the Institute of Applied Science, Stuttgart Media
University, 70569 Stuttgart, Germany (e-mail: [email protected]).
Qi Wang is with the Audio Visual Communications and Networks Group,
School of Computing, University of the West of Scotland, Paisley, PA1 2BE,
United Kingdom (e-mail: [email protected]).
Christos Grecos is with the Audio Visual Communications and Networks
Group, School of Computing, University of the West of Scotland, Paisley,
PA1 2BE, United Kingdom (e-mail: [email protected]).
Fig. 1. Using mobile devices to remotely control other consumer
electronic devices within LAN or WAN environments.
or even replacement for traditional remote controls. They even
allow remote control use cases that have not been possible
before. Fig. 1 illustrates two of these use cases where a
smartphone is used to control consumer electronic hardware
from a local and a wide area network, respectively.
Furthermore, the mentioned mobile devices are able to offer
innovative network services themselves. For example, they
can make personal information management functionality or
rich media applications available to other network devices.
There are situations where more convenient or better-suited
input equipment for the control of these services are either
preferable or even mandatory. Fig. 2 shows such a scenario in
which the driver of a car controls the smartphone through the
car’s head unit, typically an onboard “thin client” computer
with a larger screen. The latter mirrors the mobile phone’s
user interface, transmitted via a wireless personal or local area
network (WPAN or WLAN), and allows controlling it with
touch gestures. This solution provides a safer and more
Fig. 2. Using specialized input devices to control the mobile device in a
WLAN or WPAN environment.
convenient way for the driver to enjoy the applications
running on the mobile phone.
To realize the described use cases, a remote user interface
system is needed that fulfills the following requirements:
• The complexity of both client and server software should
be low, because in a consumer electronic environment the
majority of devices have reduced processing power.
• The responsiveness of the user interface, although being
transferred over a possibly slow network connection,
should be high for good user experience.
• The remote control protocol’s required bandwidth should
be reduced to accomplish the desired fast reaction times in
slow networks and lower the network traffic in scenarios
where users are charged on a traffic amount basis.
• The transferred remote user interface must be adaptable to
the client device, e.g. allowing high-quality client-side
scaling and rotation-specific layouts.
• To achieve high user acceptance, the system should enable
the usage of native UI-elements on client side to represent a
system-specific design and behavior.
• The server component of the system must be portable to a
variety of platforms including platforms that do not natively
support graphics rendering (for example due to the absence
of a display).
• There must be a possibility for server-initiated updates to
allow asynchronous events occurring on the server to be
propagated to the client.
The aim of the RemoteUI Project is to develop technologies
for realizing a high-performance, portable remote user
interface system satisfying these requirements. In this paper,
we propose the RemoteUI System and present the results of an
in-depth comparison with existing well-known solutions.
The remainder of this paper is structured as follows. In
Section II we discuss existing work in the areas of thin client
computing and remote control solutions. In Section III the
architecture of the RemoteUI System and the functionality of
its protocol are defined. For our experimental evaluation we
implemented prototypes for the proposed system, and two
existing representative solutions, as described in Section IV.
With these prototypes we accomplished different experimental
evaluations introduced in Section V. In Section VI we
describe the setup and the results of the bandwidth and traffic
volume comparison tests. In Section VII the test bed of the
latency tests and experiment results are discussed. We
summarize our observations and conclude in Section VIII.
II. RELATED WORK
Transferring graphical user interfaces from a server to a
client via network connections can be realized through two
major approaches. The first approach originated from the thin
client concept, and typically employs pixel information or
low-level graphic primitives for this purpose. For instance,
systems like Virtual Network Computing (VNC) [1], Muse [2]
and the system proposed by Simoens et al. [13] retrieve the
required bitmap information from the framebuffer at the
server. Systems like the Remote Desktop Protocol (RDP,
based on ITU-T T.128 [3]), THINC [4] and BASS [5] work as
a mirror graphic driver and intercept graphic commands. XServer [6] introduces a graphic abstraction layer and Joveski et
al. [12] intercept the latter for accessing graphic primitives.
Information gathered from these sources is encoded with
special run-length codecs like ZRLE or TRLE [7] or existing
codecs like PNG, JPEG, H.264 or Theora. Client and server
are usually connected persistently over TCP or RTP. These
systems have various advantages. Typically the (thin) client
software has relatively low complexity and displays a pixelaccurate reproduction of the server's UI. Employing optimized
codecs, these systems perform adequately when transferring
video streams. However disadvantages exist regarding client
adaptation, for example client-side scaling is possible but
often not optimal on mobile devices. Additionally the
bandwidth requirements of these solutions are relatively high
and the responsiveness in high-latency environments suffers
from the necessity for full client-server round trips.
There are various applications of thin client systems
especially VNC-based solutions in consumer electronic
environments. Haraikawa et al. propose the µVNC system [8],
which transfers the UI of consumer electronic hardware
through VNC. In this way a video recorder can be controlled
with the help of a regular VNC client on a PC. To lower the
complexity of the server software and supersede the necessity
of a real framebuffer, µVNC uses pre-rendered images.
Mainly for the realization of remote video servers, the RVU
Protocol [9] uses an approach similar to RDP to transfer user
interfaces. Videos can be embedded into these UIs using
specialized video codecs. Consumer electronic devices
implementing such technology, which shall support special
mobile-based UI delivery options in the near future, have
already been shipped. To mirror the user interface of a mobile
consumer electronic device to a car's head-unit, Bose et al.
[10] describe a solution that extends VNC with special carrelated features. The device can be connected via USB,
WLAN or WPAN. The European project Mobithin [11] has
focused on the thin client computing with mobile devices.
Different proposed systems [12]-[14] use parts of the MPEG4
standard to accomplish high performance.
The other major approach utilizes widget descriptions to
transfer user interfaces, allowing a high-quality client-side
rendering and scaling. These descriptions are often formulated
in XML languages supporting specialized elements for
widgets like “ToggleButton” or “Progressbar”. Examples are
HAViML [15], Verhoeven and Dees’ protocol [16],
LRDP [17] and QML [18]. A similar approach explores
HTML to realize and transfer rich user interfaces. Sonnenberg
[19] proposes to utilize HTML for the integration of mobile
devices into cars. Following the Web4CE standard [20], a
consumer electronic device can offer a UPnP Remote UI
service [21] in the form of an XHTML 1.0 page. The Web4CE
standard also addresses a common problem of HTML- and
XML-based solutions that use HTTP as the underlying
protocol. A so-called NotifSocket object is defined that allows
the server to send updates to the client through a persistent
TCP connection. Using HTTP, this can only be realized with
client-side polling. The same problem is addressed with
Websockets [22] that have been already implemented in most
current web browsers, in contrary to NotifSocket. However,
both solutions do not practically support efficient transfer of
binary data to HTML-based clients, which is highly desirable
for the transmission of bitmaps. Standardized JavaScript [23]
clients at the moment can only receive data from these sockets
in text form. Therefore, binary data have to be encoded e.g.
with Base64 [24] before the transfer. Base64 introduces
approximately 33% overhead as 6 input bits are encoded with
8 output bits, which adversely influences the bandwidth
requirements of such a system.
According to Nielsen [25] the user interface of a software
system should react within 100 milliseconds to inform the user
about direct object manipulations, e.g. changing a button’s
color to express its pressed state. If the presentation of results
in response to user input takes longer than 1.0 second,
additional feedback is necessary. While network round-trip
times (RTTs) in local area networks are typically below
100 ms, investigations show that RTTs in wireless wide-are
networks (WWANs) such as UMTS can become considerably
high. For a moving client sending 64-byte packets Wewetzer
et al. [26] determined a mean RTT of about 600 ms. Systems
working with widget descriptions in such scenarios have the
advantage that no round-trip with the server is required to
render the widget behavior needed for instant user feedback.
In the proposed RemoteUI System, we leverage the
advantageous features in existing approaches such as a
persistent connection and efficient widget descriptions to form
a high-performance UI system. Furthermore, we propose a
range of techniques to minimize the required bandwidth and
system latency. To evaluate the effects of these solutions we
compare our system with two representative systems, VNCand HTML-based respectively. Both systems are most widely
used and available on many platforms.
This paper has substantially extended its preliminary
version [27] by expounding the system’s design. Furthermore,
the performance evaluations have been detailed through
extensive experiments. The proposed RemoteUI system is
compared with VNC- and HTML-based systems in terms of
traffic volume and latency wherever appropriate.
III. THE REMOTEUI SYSTEM
The proposed RemoteUI System (RUI) consists of at least
one RemoteUI Server that delivers user interface information.
One or more RemoteUI Clients can connect to this server to
retrieve and display its user interface. In Fig. 3 the
components of the RemoteUI System are shown. We
distinguish between server and client components.
The Connection components establish a connection between
the server and the client to enable the display of the user
interface (client) and the delivery of the user interface (server)
respectively. Different implementations for the Connection
component are possible, e.g. WLAN- or WPAN-based. This
component provides a connection object for the overlaying
Fig. 3. Components of the RemoteUI System (yellow/light) and
application specific components (blue/dark)
tier. To establish a connection, an optional Discovery
component can be used to find devices offering the RemoteUI
functionality.
The Event System uses the established connection to realize
an asynchronous transfer of user interface descriptions, input
events and system commands between the server and the
client. The RUI-Protocol component is a class library
describing the protocol entities. The Controller and Input
components use it to create protocol objects at runtime. The
Event System is responsible for their serialization,
transmission, retrieval and deserialization. We use a platformindependent, binary serialization protocol derived from
HBWSP [28]. This protocol is able to describe arbitrary
objects including the efficient representation of binary data as
required for bitmap transfer.
On the server side, a Controller connects the existing
Business Logic and the RemoteUI System. Its implementation
highly depends on the scenario and the execution environment
in which the RemoteUI System is used. In order to control a
smartphone via RemoteUI, the Controller has to be
implemented as a bridge to the platform’s UI components.
On the client side, the Rendering component is responsible
for displaying the received UI descriptions and commands.
The Input component transfers input events originating from
user interaction through the Event System to the server.
A. UI-Tree and Tree Manipulation
The RemoteUI System uses widget descriptions and
graphic primitives to describe and transfer the user interface.
These graphical objects build the so-called UI-Tree, which is
replicated between server and client. Fig. 4 shows a UI-Tree
(b on the left) and its rendered representation (a). While the
contained image in this example must be transferred as pixel
information, an abstract description of the displayed text and
buttons is sufficient to create a client-side representation of the
user interface. With this method, the required amount of data
can be significantly reduced, as we will show in Section VI.
When a new screen is opened, the server initially transfers
its UI-Tree. The client receives, renders and saves a local copy
of the tree. From then on, the client renders simple widget
behavior like button presses or list scrolling locally. If serverside event-listeners have been registered on a widget, the
client transfers the corresponding events to the server. The
processing of these events usually produces visual results that
(a)
Fig. 4. (a) The left UI-Tree being rendered on the phone, (b) UI-Tree manipulation
shall be displayed through UI-Tree manipulations, which are
triggered by server commands. The client receives and
instantly executes the commands, which includes rendering
the resultant tree. To allow the manipulation, each node in the
UI-Tree carries a unique ID that is used for addressing.
Fig. 4 (b) exemplifies this process. The left tree is transformed
into the right by replacing text and image information in
different nodes.
For certain widgets a round trip with the server is necessary
to determine their appearance. The server for instance defines
the state of a toggle button after it is pressed. During the
following round-trip the button is in a “pending” state. If the
round-trip time is high, this new state should be presented to
the user. We call this phenomenon the “additional-state
problem” that can generally occur in remote UI systems. The
RemoteUI Client in future versions will provide special
modified widgets that are capable of displaying these
additional states.
B. Incremental Transfer and Rendering
Although the amount of data required to express a user
interface with the RemoteUI Protocol is relatively small, there
are situations where the transfer of a complete UI-Tree takes a
long time. This can be the case in scenarios with low available
bandwidth or the UI containing many images. To provide fast
user feedback, the RemoteUI-Server incrementally sends the
UI-Tree as soon as the first nodes become ready for transfer.
The RemoteUI-Client incrementally receives, builds up and
renders the local copy of the tree. This results in a row-by-row
display of the user interface. If the first row of the layout can
be displayed within less than a second, we assume this as
sufficient for the required timely user feedback. Additionally,
a less conspicuous progress indicator can be part of the
interface to indicate a pending receive process.
C. Prioritization
The RemoteUI Protocol offers a mechanism to prioritize
different user interface and command objects. The server
processes and transmits objects with high priority first, which
intentionally causes a delay for lower prioritized objects
especially in low bandwidth scenarios. Because the client is
able to incrementally render these objects, this technology
reduces the user interface latency. In a preliminary approach,
(b)
widgets displaying pixel data (ImageViews) are separated from
the UI-Tree and transferred as lower-prioritized SetData
commands. Part of our research is the development of
algorithms to determine the optimal priority from input
parameters such as visibility, traffic volume, operational
importance, available bandwidth and network latency. By
continuously applying these algorithms at runtime, the system
will adapt to changing environment conditions to offer the
highest possible performance.
D. Caching and Prefetching
Theoretically a UI-Tree does not need to be transferred
more than once. Although there are screen switches that are
realized by replacement of the tree, it is likely that a tree will
be reused once loaded. Avoiding the retransfer of resources
like pixel-images used in a UI-Tree can reduce the required
traffic volume significantly. To profit from this fact, the
RemoteUI System caches resources on the client.
Additionally, during idle-times, a low-priority process
prefetches resources that are likely to be required later on. A
tree that contains a resource that has already been fetched only
needs to contain a reference to it, which reduces the user
interface latency.
IV. PROTOTYPIC IMPLEMENTATIONS
As platform for all prototypic implementations we chose the
Linux-based operating system described by McLoughlin [29]
later on referred to as “target platform”. This system is
available for different kinds of mobile consumer electronic
devices like smartphones, tablets and multimedia hardware.
Additionally a specialized version is available for TV systems
predestining the platform for our prototypic implementations.
A. RemoteUI System Prototype
We realized the RemoteUI System with the components
described in Section III for the above mentioned target
platform. Fig. 5 shows the applications that have been
developed by us. While the RUI Client was realized as a
stand-alone-application, the server functionality was divided
into different applications. It was an important requirement for
us to avoid manipulations of the operating system or the
necessity of gaining root privileges to run our applications.
Access to the user interface on the target platform is allowed
Fig. 5. Deployment of the RemoteUI applications
from the owning application only. Therefore in our system an
application must be enhanced with RemoteUI functionality in
order to become remote controllable. A standard application
project can easily be extended with RemoteUI functionality by
weaving the required function calls and properties into the
application’s bytecode. We developed three prototypic Remote
Controllable Applications. A simple Calculator, a
MediaPlayer and a Demo App. Fig. 6 shows Screenshots of
the Calculator, and a special List Demo Activity that is part of
the Demo App, where the number of list items can be
configured. We used these UIs for our evaluations described
in Sections VI and VII.
On startup a Remote Controllable Application connects
locally to the RUI Server application by inter-process
communication (IPC). The RUI Server in the following works
as relay for the communication with a connected RUI Client.
The implementation uses a Connection module working with
TCP-Sockets and a Zeroconf-based Discovery [30]. Images
are transmitted in display size, encoded with full color
PNG [31].
B. HTML System Prototype
To compare the RemoteUI System with an HTML-based
approach, we additionally implemented a prototypic dynamic
web server for the target platform. The server uses HTTP 1.1
[32] to serve static and dynamic content, using a new TCP
connection for each request. It serves static content like
HTML files from the SD card handling each request in a new
thread. Additionally it supports simple listeners (servlets) that
(a)
(b)
Fig. 6. Screenshots of the applications that we used for the
performance comparisons: (a) Calculator, (b) List Demo
dynamically serve content when being registered for certain
URLs. In contrast to the RemoteUI System's server, this
solution is realized as a library and must be directly integrated
into each application. Inter-process communication is not
required in this way. Also this system does not automatically
translate the native UI into HTML. We expected some
performance benefits in terms of latency from these facts.
We integrated the server into a version of the Calculator
application. Following the Web4CE standard, this Calculator
can be controlled via a static XHTML 1.0 web page shown in
Fig. 7 (a). The page contains JavaScript code that sends an
XMLHttpRequest [33] to a calculation servlet on each button
press, carrying the corresponding button value as the GET
parameter. The servlet on request modifies or evaluates the
equation and updates the original Calculator's text view with
the result. The updated text value is then sent to the requesting
script in a JSON [34] response. The script writes this value
into the respective HTML text area. Thus, the prototype works
similarly to the RemoteUI System, keeping the business logic
itself on the server.
A second web page (Fig. 7 (b)) is an HTML version of the
Demo List view. It shows 100*100 pixels thumbnails that are
dynamically converted by a servlet from the originally
300*300 pixels application resources. Therefore the
thumbnails are encoded and transferred in the same format as
with the RemoteUI System. The list itself has no further
functionality so the HTML page does not contain any
JavaScript code. To measure the performance of the HTML
system on the client, we implemented a small browser
allowing us to process all events during page loads and
XMLHttpRequests. This browser is based on the target
system’s WebView widget that uses the same native browser
library as the built-in system browser.
C. VNC Server and Client
We used a VNC server application running natively on the
target platform. The software requires root privileges to get
access to the device’s framebuffer. We modified the device
accordingly to grant this access. On client-side we used a
modified version of the open-source VNC viewer also used by
Yu et al. [2] where we additionally enabled the portrait modus
for controlled devices and introduced logging for our profiler.
(a)
(b)
Fig. 7. Screenshots of the web browser displaying XHTML
versions of the test applications: (a) Calculator, (b) List Demo
D. Profiler Software
For the latency comparisons described in Section VII we
implemented special profiler software for the target platform.
The profiler is able to measure the duration of arbitrary
processes in a distributed environment. The logged events are
stored in a process-local in-memory list. This list is written
into a CSV-file when the profiling is finished. The profiling
can be controlled through a special application that functions
as profiler frontend on the device itself. We integrated the
profiler library into all involved components.
Fig. 8. Bandwidth comparison for the Calculator
V. EXPERIMENTAL SETUP
We carried out different experiments to compare the
required bandwidth and latency of the RemoteUI System with
those of HTML and VNC. For all tests we used a device with
a resolution of 540*960 pixels, a dual-core 1.2 GHz RISC
CPU and 768 MB RAM to execute the server implementations
(RemoteUI Server, web server and VNC server). The client
device offered a resolution of 600*1024 pixels, a 1 GHz RISC
CPU and 512 MB RAM. The RemoteUI Client, VNC client
and our HTML browser were installed on this device. On both
devices we installed the profiler frontend.
VI. BANDWIDTH AND TRAFFIC VOLUME COMPARISON
A. Experimental Setup
To compare the bandwidth and traffic volume we used the
Calculator applications and the List Demos described in
Section IV. In one experiment we configured the List Demo to
contain 50 items to analyze the effect of scrolling (List Demo
50). In a second experiment the List Demo was displaying
only 9 items (List Demo 9); in all remote systems the 9 items
could be displayed without scrolling. The latter experiment
was required to determine the traffic volume produced during
the latency tests described in Section VII.
With capture software running on the notebook we recorded
the TCP traffic when executing the test applications. For the
VNC connection we chose the full color depth (32 bit per
pixel) and the ZRLE encoding. RemoteUI and HTML also
transferred images in full color (PNG) so the display quality
was comparable. Both test applications were started using the
respective remote clients. A start button on a remote screen
was touched, which opened either the respective view to be
transferred (RemoteUI and VNC) or the web page containing
the screen's replica. After being loaded, the Calculator was
used on the client side to carry out a set of simple calculations.
The List Demo 50 was used to scroll down the list of items.
B. Experimental Results
The results of the bandwidth consumption and data
measurements are shown in Fig. 8, Fig. 9 and Table I. The
bandwidth plots for the Calculator experiment (Fig. 8) show
that after the initial transfer of the UI, RemoteUI and HTML
required only few data to transfer button clicks and the
corresponding text area updates (seconds 10-40). VNC in
contrast uses pixel area updates to transfer the changes in the
text area, which caused considerably more traffic. RemoteUI
Fig. 9. Bandwidth comparison for the List Demo 50 with scrolling
TABLE I
ABSOLUTE TRAFFIC VOLUME OF REMOTEUI, HTML AND VNC
Calculator
List Demo 50
List Demo 9
(with scrolling)
(no scrolling)
VNC (kB)
203.428
4523.162
261.046
HTML (kB)
25.966
1118.706
202.542
RemoteUI (kB)
19.202
684.143
205.712
in this case reduced the required traffic volume compared to
VNC by over 90%. HTML was very efficient as well, with a
reduction of about 87%. The HTML version of the Calculator
uses nearly no styling information and contains only a few
lines of JavaScript code. Nevertheless this simple
implementation already caused about 25% more network
traffic than the RemoteUI System that is able to render the UI
in the system's native style. It is expected that a full-featured
HTML version would require much more data.
A similar result emerged for the List Demo 50 experiment.
While RemoteUI required about 684 kB for the transfer,
HTML without any JavaScript code and minimal styling used
over 1.1 MB. However, Fig. 9 shows that both systems load
all list items in advance so that no traffic is generated from
client-side scrolling events. This is a weak point of VNC
where every scrolling action produced a considerable amount
of pixel data transfer (seconds 5-60). These high data
requirements also degrade the system's responsiveness,
making scrolling a difficult business for the user. We
terminated the experiment after 60 seconds with resulting
4.5 MB of VNC traffic. Under these circumstances, VNC does
not satisfy our requirements for scenarios with low bandwidth
or for users being charged on a traffic volume basis.
Transferring the 9-item list without scrolling in all three
systems led to only the initial transfer of the UI. RemoteUI
and HTML required about 200 kB. VNC, with 260 kB, also
had modest traffic volume requirements. For this
configuration mainly the latency of the systems is interesting.
This is influenced by the systems' designs and
implementations, as examined in the following section.
VII. LATENCY COMPARISON
Equally important to the amount of data transfer is the
system’s latency. Evidently, the faster the reaction times of the
system, the better the user’s experience. Data minimization
can improve the latency of such a system especially in low
bandwidth scenarios. However, other parameters such as
computation cycles, device performance and parallelism of the
implementation highly affect the latency.
A. Experimental Setup
For the latency analysis we conducted 18 test series with 50
single test runs each. A test run consisted of the List Demo 9
(described in Section IV.A) being started by clicking a button
on a client-side start screen. With the help of our profiler we
measured the time between the touch-up event and the first
and last drawing events of the client when rendering the
received UIs. We refer to these times as “First Draw” and
“Full Load” RTTs.
To connect the client and the server, we selected a WLANrouter with IEEE 802.11g support and an extendable Linuxbased firmware. We installed tools for traffic shaping onto this
device. For our WLAN configuration we chose the free
channel 7 and WPA-2 encryption.
We conducted all tests with three different maximum
available bandwidths and two different network latencies,
resulting in six test scenarios. At first we used the WLAN's
native data rate and later simulated a 2.1 Mb/s and a 384 kb/s
connection with the traffic shaping functionality of the router.
We chose these bandwidths to obtain an impression of the
systems’ behavior under lower-speed wireless networking
conditions, and typical WPAN or WWAN data rate
environments. It is noted that we are aware that different radio
access technologies have different protocol stacks and
operation conditions. Thus, the results of that subset of the
tests can only be rough indicators for the behavior in WPANs
or WWANs.
The network latency in the first three test series was the
default latency of the WLAN configuration. With a ping
command we measured an average network RTT of 16 ms. In
the other series with the help of the router firmware we
introduced an additional latency of 292 ms. The resultant
network RTT of 600 ms simulates a moving client in a UMTS
environment as described by Wewetzer et al. [26].
B. Experimental Results
Fig. 10 and Table II show the results of the measurements
conducted with a network RTT of 16 ms. The times for a first
draw event taking place (“First Draw” RTT) are represented
by dark colors. The VNC client lacks an incremental rendering
feature, thus only the complete load times have been
considered. It can be seen that RemoteUI performed
significantly better than VNC: the RTTs for the complete
loads were 40 to 50% faster. Additionally, RemoteUI was able
to present a first rendered result in under a second in all
scenarios. This accelerated rendering is important for the user
feedback and supersedes the necessity for additional feedback
during load times.
Fig. 10. Mean round trip times with std. deviation indicators for starting
the List Demo 9 with a given network RTT of 16 ms
TABLE II
MEAN RTT [MS] FOR STARTING THE LIST DEMO 9 WITH A GIVEN
NETWORK RTT OF 16 MS
WLAN
2.1 Mb/s
384 kb/s
HTML
First Draw
332.83
335.22
334.04
Complete Load
909.93
1263.61
4002.24
RemoteUI
First Draw
966.25
945.24
910.70
Complete Load
1513.57
1578.86
5559.33
VNC
Complete Load
2875.44
3825.09
9074.79
Even faster than RemoteUI was the tested HTML solution.
In all scenarios, the browser presented a first rendering after
about 330 ms. The complete load times varied depending on
network bandwidth but in all scenarios were faster than the
ones of RemoteUI. In the WLAN scenario, the system was
able to present a completely rendered result within less than a
second.
There are different reasons for HTML being faster than
RemoteUI in this scenario. We used a static HTML page with
minimal style sheets to represent the list. The time for
converting the server’s UI into a UI protocol as done by
RemoteUI and VNC is therefore unaccounted in the HTMLbased system. Additionally the HTML client used the browser
library built into the operating system of our target platform.
This library is highly optimized and runs natively on the
processor while the RemoteUI Client and the VNC client are
implemented in code executed by a virtual machine. To
achieve optimal results for the client-side behavior of HTML
we also had to implement special JavaScript code. A typical
browser on devices with touch screen waits after a single
touch for a possible second touch forming a double touch
event. This gesture is used for zooming and therefore must be
considered by the browser. The wait time is typically 300 ms,
during which the standard click event is not yet triggered. This
behavior can only be overridden by introducing non-standard
listeners in JavaScript, which operate on the mobile-browser
specific touchEnd event. This approach however introduces
problems to scrolling gestures. Nevertheless, for this test we
implemented a touchEnd-based trigger to achieve a low RTT.
The results otherwise would have contained the additional
300 ms reaction time.
Fig. 11. Mean Round Trip Times for starting the List Demo 9 with a
given network RTT of 600 ms
TABLE III
MEAN RTT [MS] FOR STARTING THE LIST DEMO 9 WITH A GIVEN
NETWORK RTT OF 600 MS
WLAN
2.1 Mb/s
384 kb/s
HTML
First Draw
919.53
923.32
922.14
Complete Load
2121.72
2442.01
5182.64
RemoteUI
First Draw
1550.85
1530.54
1498.30
Complete Load
2075.02
2165.26
6148.93
VNC
Complete Load
3462.54
4420.19
9664.19
Fig. 11 and Table III show the results for the experiments
conducted with a given network RTT of 600 ms. Under these
conditions the additional network RTT has more or less been
added to the RTTs of RemoteUI and VNC. RemoteUI in this
scenario rendered a first result in about 1.5 seconds whilst
HTML still achieved a first user feedback in under a second.
However, the complete load times for the HTML page now
exceeded the ones of RemoteUI in the WLAN and 2.1 Mb/s
scenarios. The reason lies in the system design of HTML. The
images’ URLs are contained in the HTML code, hence the
browser can only retrieve the images after loading the HTML
document. This introduces a second round trip, causing an
effective network delay of about 1.2 seconds in this scenario.
RemoteUI does not require the latter because the server pushes
the images directly after transferring the UI-Tree.
VIII. CONCLUSIONS
We presented a new system for remote user interfaces that
is applicable for a variety of usage scenarios related to
consumer electronic devices. The system allows the
realization of remote control applications that can efficiently
be used in local and wide area networks accordingly. In order
to achieve high performance regarding required bandwidth
and system responsiveness, we combined different approaches
from thin client computing and widget based UI systems.
These include a persistent connection to realize server-side
updates and the usage of abstract view descriptions to enable
efficient client-side scaling and overall bandwidth reduction.
We further optimized the system taking into account the
context of wireless networks and mobile devices. In contrast
to pixel-based thin clients, our solution is able to render basic
UI behaviors on the client. This technique substantially
increases the system's responsiveness in high-latency network
environments. Additionally our client implementation is much
thinner than traditional web browsers, because it does not
require a JavaScript engine. The latter is indispensable if
HTML is used to realize rich UI functionality
Our experimental results show that the proposed RemoteUI
System outperforms the pixel-based approach represented by
VNC substantially regarding bandwidth, traffic volume and
latency. Compared with a simple HTML solution, RemoteUI's
required bandwidth is equal or lower. In some latency test
scenarios, HTML showed better responsiveness when loading
the web pages representing the user interfaces. However, in
our simplistic implementation, the HTML solution lacks
several features such as real dynamic UI-generation, native UI
behavior and rich capabilities for user interaction. These
features could theoretically be integrated into a specialized
HTML client. The required JavaScript libraries, style sheets
and images would significantly increase the traffic volume.
In our latency tests we demonstrated the positive effect of
our proposed prioritization mechanism. The images in our
example were transferred with lower priority, enabling the
RemoteUI Client to partially render a preliminary UI. The
server automatically pushes the images subsequent to the UITree. This design prevents additional network round trips,
leading to RemoteUI being faster than HTML under high
latency conditions.
In our future work we will further investigate mechanisms
for the proposed prioritization scheme. Future versions of the
RemoteUI System shall e.g. realize automatic adaptation to
changing network conditions. The prioritization scheme will
be designed to incorporate this information. Other planned
actions include the application of more efficient image
encodings and the examination of optimized underlying
transport protocols as a possible replacement for TCP. We
also assume that there is a relation between required
bandwidth, network behavior and energy consumption.
Analyzing this relationship is useful to give evidence to the
applicability of the system in situations with limited energy. In
scenarios with unavoidable high network latencies we plan the
introduction of special client-only view variants as a potential
solution for the additional-state problem.
REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
T. Richardson, Q. Stafford-Fraser, K. Wood, and A. Hopper, “Virtual
network computing,” IEEE Internet Computing, vol. 2, no. 1, pp. 33-38,
2002.
W. Yu, J. Li, C. Hu, and L. Zhong, “Muse: a multimedia streaming
enabled remote interactivity system for mobile devices,” Proc. 10th
International Conference on Mobile and Ubiquitous Multimedia, pp.
216–225, 2011.
ITU-T Rec. T.128, “Multipoint application sharing,” Jun. 2008.
R. A. Baratto, L. N. Kim, and J. Nieh, “THINC: a virtual display
architecture for thin-client computing,” Proc. 20th ACM Symposium on
Operating Systems Principles, pp. 277–290, 2005.
O. Boyaci and H. Schulzrinne, “BASS Application Sharing System,”
Proc. 10th IEEE International Symposium on Multimedia, pp. 277–290,
15-17 Dec. 2008.
R. W. Scheifler and J. Gettys, “The X window system,” ACM
Transactions on Graphics, vol. 5, no. 2, pp. 79–109, Apr. 1986.
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
[29]
[30]
T. Richardson and J. Levine, “The Remote Framebuffer Protocol”, IETF
RFC 6143, Mar. 2011.
T. Haraikawa, T. Sakamoto, T. Hase, T. Mizuno, and A. Togashi,
“µVNC: a proposal for Internet connectivity and interconnectivity of
home appliances based on remote display framework,” IEEE
Transactions on Consumer Electronics, vol. 47, no. 3, pp. 512–519,
Aug. 2001.
RVU Alliance, “RVU Protocol Specification 1.0,” 03 Dec. 2009.
R. Bose, J. Brakensiek, and K. Y. Park, “Terminal mode: transforming
mobile devices into automotive application platforms,” Proc. 2nd
International Conference on Automotive User Interfaces and Interactive
Vehicular Applications, pp. 148–155, 2010.
B. Vankeirsbilck, P. Simoens, L. Deboosere, B. Dhoedt, F. Westphal, T.
Plantier, B. Lécroart, F. Prêteux, A. Dejonghe, and J. Point, “Bringing
thin clients to the mobile world,” Proc. NEM Summit Towards Future
Media Internet, 2008.
B. Joveski, P. Simoens, L. Gardenghi, J. Marshall, M. Mitrea, B.
Vankeirsbilck, F. Preteux, and B. Dhoed, “Towards a multimedia remote
viewer for mobile thin clients,” Proc. of the SPIE, the International
Society for Optical Engineering, vol. 7881, 2011.
P. Simoens, P. Praet, B. Vankeirsbilck, J. De Wachter, L. Deboosere, F.
De Turck, B. Dhoedt, and P. Demeester, “Design and implementation of
a hybrid remote display protocol to optimize multimedia experience on
thin client devices,” Proc. IEEE Australasian Telecommunication
Networks and Applications Conference, pp. 391–396, 2008.
B. Joveski, M. Mitrea, and F. Preteux, “MPEG-4 LASeR - based thin
client remote viewer,” Proc. 2nd European Workshop on Visual
Information Processing (EUVIP), pp. 125–128, 2010.
R. G. Wendorft, R. T. Udink, and M. P. Bodlaender, “Remote execution
of HAVi applications on Internet-enabled devices,” IEEE Transactions
on Consumer Electronics, vol. 47, no. 3, pp. 485–495, Aug. 2001.
R. Verhoeven and W. Dees, “Defining services for mobile terminals
using remote user interfaces,” IEEE Transactions on Consumer
Electronics, vol. 50, no. 2, pp. 535– 542, May 2004.
V. Stirbu, P. Shrubsole, and J. Costa-Requena, “LRDP: The Lightweight
Remote Display Protocol,” IETF Internet draft, 11 Feb. 2005.
V. Stirbu and J. Savolainen, “Hypermedia-Driven Framework for
Scalable and Adaptive Application Sharing,” REST: From Research to
Practice, E. Wilde and C. Pautasso, Eds. Springer New York, pp. 199–
219, 2011.
J. Sonnenberg, “Connecting in-vehicle entertainment with CE devices
via Multi-Platform Web Applications,” Proc. IEEE International
Conference on Consumer Electronics, pp. 515–516, 2011.
CEA R7 Home Network Committee, “Web-based Protocol and
Framework for Remote User Interface on UPnP Networks and the
Internet (Web4CE),” Jan. 2011.
UPnP Forum, “RemoteUIServer:1 Service Template Version 1.01.”
02 Sep. 2004.
I. Fette and A. Melnikov, “The WebSocket Protocol,” IETF Internet
draft, 30 Sep. 2011.
Ecma International, “Standard ECMA-262 - ECMAScript Language
Specification.” Jun. 2011.
S. J. Josefsson “The Base16, Base32, and Base64 Data Encodings.”
IETF RFC 4648, Oct. 2006.
J. Nielsen, Usability engineering, Morgan Kaufmann, p. 135, 1993.
C. Wewetzer, M. Caliskan, K. Meier, and A. Luebke, “Experimental
Evaluation of UMTS and Wireless LAN for Inter-Vehicle
Communication,” Proc. 7th International Conference on ITS
Telecommunications, pp. 1–6, 2007.
D. Thommes, Q. Wang, A. Gerlicher, and C. Grecos, “RemoteUI: A
High-Performance Remote User Interface System for Mobile Consumer
Electronic Devices”, Proc. IEEE International Conference on Consumer
Electronics, pp. 670-671, Jan. 2012. (Best Poster Paper Award Winner)
P. Lin and Q. Chen, “A Distributed Platform Based on HBWSP & XML
for Net Resources Sharing,” Proc. IEEE International Seminar on
Future Information Technology and Management Engineering, pp. 242
–245, 2008.
I. V. McLoughlin, “Open-Source and Consumer Electronics - The Back
Door to World Domination: Why Reinvent the Wheel (Especially When
Nice People Are Giving Wheels Away for Free)?,” IEEE Consumer
Electronics Magazine, vol. 1, no. 2, pp. 53 –58, Apr. 2012.
B. Aboba, E. Guttman, and S. Cheshire, “Dynamic Configuration of
IPv4 Link-Local Addresses.” IETF RFC 3927, May 2005.
[31] W3C, “Portable Network Graphics (PNG) Specification (Second
Edition),“ W3C Recommendation. 10 Nov. 2003.
[32] P. J. Leach, T. Berners-Lee, J. C. Mogul, L. Masinter, R. T. Fielding,
and J. Gettys, “Hypertext Transfer Protocol – HTTP/1.1,” IETF RFC
2616, Jun. 1999.
[33] W3C, “XMLHttpRequest Level 2,” W3C Working Draft. 17 Jan. 2012.
[34] D. Crockford, “The application/json Media Type for JavaScript Object
Notation (JSON),” IETF RFC 4627, Jul. 2006.
BIOGRAPHIES
Daniel Thommes (S’11) is a PhD student in a
collaborative research program of the University of the
West of Scotland (UWS), UK, and the Stuttgart Media
University, Germany. Since 2010 he is Research
Assistant and Lecturer with the Stuttgart Media
University. Previously, he worked for several years as a
Research Assistant and Consulting Software Architect.
His research interests include mobile computing,
wireless/mobile networks, web technologies, software
architecture and usability engineering. He received the Diplom-degree in
Media Technology Engineering from the Ilmenau University of Technology,
Germany, in 2006. Since 2012 Daniel is member of the IEEE Consumer
Electronics Society.
Ansgar Gerlicher (M’06) Prof Ansgar Gerlicher is a
Professor in Mobile Applications and Director of
Research in Mobile Applications & Security with the
Institute of Applied Science, Stuttgart Media University,
Germany. Previously, he worked for several years as a
Software Architect and Project Manager in the
Telecommunication and Automotive Industries. His
research interests include integration of consumer
electronic devices in vehicles and mobile and embedded
software architectures, frameworks and mobile security. He received his PhD
degree in real-time collaboration systems from the London College of
Communication, UArts, UK. Since 2012 he is a member of the IEEE
Consumer Electronics Society.
Qi Wang (S’02-M’06) Dr Qi Wang is a Lecturer in
Computer Networking with the University of the West of
Scotland (UWS), UK. Previously, he was a Postdoctoral
Research Fellow with the University of Strathclyde, UK,
and a Telecommunications engineer with the State Grid
Corporation of China. He received his PhD in mobile
networking from the University of Plymouth, UK, and
his MEng and BEng degrees in electronic and
communication systems from Dalian Maritime
University, China. His research interests include Internet Protocol networks,
wireless/mobile networks, and video networking. Recently, he has been
involved in various international and national projects such as the European
Union FP6 MULTINET project and the UK EPSRC DIAS project. He is the
Principal Investigator of the UK EPSRC project entitled Enabler for NextGeneration Mobile Video Applications, and was a recipient of England ORS
Award. He is on the technical programme committees of numerous
international conferences.
Christos Grecos (M’01-SM’06) Prof Christos Grecos is
a Professor in Visual Communications Standards, and
Head of School of Computing at the University of the
West of Scotland (UWS), UK. He leads the AudioVisual Communications and Networks Research Group
(AVCN) with UWS, and his research interests include
image/video compression standards, image/video
processing and analysis, image/video networking and
computer vision. He has published many research papers in top-tier
international publications including a number of IEEE transactions on these
topics. He is on the editorial board or served as guest editor for many
international journals, and he has been invited to give talks in various
international conferences. He was the Principal Investigator for several
national or international projects funded by UK EPSRC or EU. He received
his PhD degree in Image/Video Coding Algorithms from the University of
Glamorgan, UK.