Interactive Systems Technical Design
Lecture #3
Hybrid computing environments
ISTD 2003
Hybrid Computing Environments
• We live in the fusion of real (physical) and virtual
(computer) worlds
• Some systems add virtual properties to physical
• Some systems add physical dimension to virtual
worlds (e.g. Pick-and-Drop studied here)
• These two approaches are not contradicting nor
mutually exclusive
• It is important to find the right balance between virtual
and physical dimensions
ISTD 2003
Case Studies
Pick-and-Drop: A Direct Manipulation Technique for
Multiple Computer Environments
Sony Computer Science Laboratories, Japan
Augmented Surfaces: A Spatially Continuous Work
Space for Hybrid Computing Environments
Sony Computer Science Laboratories, Japan
ISTD 2003
Pick-and-Drop: A Direct Manipulation
Technique for Multiple Computer
Jun Rekimoto, Sony Computer Science Laboratories, Japan
Proc. 10th Annual Symposium on User Interface Software
and Technology, Banff, Canada
ISTD 2003
Pick-and-Drop: A Direct Manipulation
Technique for Multiple Computer
Jun Rekimoto, Sony Computer Science Laboratories,
Proc. UIST’97
ISTD 2003
Motivation : Multiple Computer Environments
• A number of (dedicated) computers within close
• Desktop PC’s for SW development and office tasks
• Wall-size computers (displays) for collaborative work
• Personal PDA’s for mobile use
• Problems
• User interface: each computer has dedicated tethered
keyboards and pointing devices, ”mouse jungle”
• Easy natural data transfer between computers
ISTD 2003
Survey of Multi-Computer Usage
• Data: 39 employees at Sony’s SW laboratories
• Q1. How many computers do you have on your desktop?
3+ 54%
2 39%
1 8%
0 0%
• Q2. How often do you need to transfer data between
computers on the same desktop?
Very often 69%
Never 3%
Often 25%
Sometimes 3%
Occasionally 0%
• Q3. (Under situation Q2) How do you transfer data?
By hand 63% Shared files 63%
Floppies 20% Other 23%
ISTD 2003
FTP 57%
e-mail 34%
Survey of Multi-Computer Usage (cont.)
• Q4. How often do you need to transfer data from your
computer to another’s computer within a short distance?
Very often 28%
Often 23%
Sometimes 36% Occasionally 5%
Never 5%
• Q5. (Under situation Q4) How do you transfer data?
By hand 54% Shared files 57%
Floppies 11% Other 19%
FTP 38%
e-mail 73%
 Lack of easy direct data transfer user interfaces
such as copy-and-paste or drag-and-drop
between computers
ISTD 2003
Tentative Solutions to Problems
• User interface: each computer has dedicated
keyboards and pointing devices, ”mouse jungle”
 More sophisticated tetherless input device such as a stylus
• Can be shared between many pen sensitive displays
• Data sharing between computers
 ”Pick-and-Drop” direct manipulation technique
ISTD 2003
• Extrapolation of Drag-and-Drop technique
Drag-and-Drop vs Pick-and-Drop
pick up
hold and move (virtually)
 No need to drag (slide) the object on the display surface
 Virtual hold and move allows extended functionality, e.g.
intra-computer operations
ISTD 2003
Remote Copy vs Pick-and-Drop
Conceptual difference between remote copy and Pick-and-Drop
ISTD 2003
State Transition Diagrams
• Pick-and-Drop can coexist with Drag-and-Drop
• Time-out between pen-down and pen-up
• Proximity to screen is detected by combining motion
events and a time-out
ISTD 2003
System Architecture
ISTD 2003
Implementation Details
• Computers and displays
• PDA’s: Mitsubishi Amity
• Desktop screens: Wacom PL300
• Wall-sized display: Wacom MeetingStaff + projector
• Input device: Wacom stylus
• Can distinguish at most 3 different pens simultaneously
• Probably not sufficient for practical applications
• Additional pens via RF tags or IR beacons
ISTD 2003
Implementation Details (cont.)
• Computer network
• Ethernet for desktops and wall-sized display
• WLAN for PDA’s
• Application development in Java
• Object transfers via Java serialization
ISTD 2003
• Each pen is assigned a unique ID
• ID is readable when pen is close to display
• ID represented by modifier buttons in the pens
• PenManager
• Binds object ID’s to pen ID’s (~ pen virtually holds the
• Manages object shadows
• Controls data transfer
ISTD 2003
Object Shadows
(a) Pen contacts the display
(b) Pen is lifted up, but is close to the display
-> if pen has data, object shadow appears to indicate this
(c) Pen is away from the screen -> no object shadow
ISTD 2003
Example Applications
Information exchange between PDA’s and kiosks
• Pick-and-Drop allows easy natural data transfer from one
computer (display) to another
Transfer of data between PDA’s
ISTD 2003
Transfer of data from a ”push
media” information kiosk to PDA
Example Applications (cont.)
Drawing on a wall display with a tablet
• Simple paint editor on a tablet (color, brush style)
• No need for a dedicated tool-palette
ISTD 2003
Example Applications (cont.)
• Information exchange between a computer and a book
ISTD 2003
Example Applications (cont.)
Anonymous Displays
• Several small tablets are used as ”temporal work buffers”
• Pick-and-Drop supports intuitive data transfer without
bothering with each computer’s symbolic name
• In comparison to virtual buffers of GUI’s, physical tablets
ISTD 2003
can be arranged freely on physical desktop
provide more natural spatial interface
allow handling more than two buffers simultaneously
are not resricted by the limited and fixed size of main desktop
Physical vs Symbolic
• Functionally, Pick-and-Drop is no more than remote copy
• However, in terms of UI, Pick-and-Drop is both physical
and visible as opposed to symbolic
• Supports direct manipulation of data instead of needless exchange
of symbolic concepts (e.g. drive and file names)
• Supports collaborative work
ISTD 2003
Discussion (cont.)
Shared Files vs Pick-and-Drop
• Shared files force the user to deal with symbolic concepts
such as machine and file names
• Difficult to keep track of multiple computers with shared
• The unit of data transfer is not necessarily a file, e.g. text
string such as URL
• Shared files a good solution for geographically separated
computers, but not so intuitive between computers within
close proximity
ISTD 2003
Candidate Improvements
• Support for a larger number of identifiable pens with RF
• Support for other file transfer protocols
• Enhanced pen operations such as grouping of objects
• Integration with a video conferencing system for sharing
of data of remote work spaces
• Wireless mouses instead of a pens
Further developed version described in:
Jun Rekimoto, ”A Multiple Device Approach for Supporting
Whiteboard-based Interaction”, Proc. CHI 98.
ISTD 2003
• Pick-and-Drop adds physical dimension to user
• Traditional data transfer methods are too virtual and
hard to learn due to their lack of physical aspects
• Pick-and-Drop does not allow manipulating
objects that are out of the user’s physical reach
• Pick-and-Drop requires a stulys-sensitive surface
for operation
ISTD 2003
Augmented Surfaces: A Spatially Continuous
Work Space for Hybrid Computing
Jun Rekimoto, Sony Computer Science Laboratories, Japan
Masanori Saitoh, Keio University, Japan
Proc. ACM SIGCHI Conference on Human Factors in
Computing Systems (CHI 99), Pittsburg, PA
ISTD 2003
Augmented Surfaces: A Spatially Continuous
Work Space for Hybrid Computing
Jun Rekimoto, Sony Computer Science Laboratories, Japan
Masanori Saitoh, Keio University, Japan
Proc. CHI’99
ISTD 2003
Motivation : Hybrid Computing Environments
• Different types of computers and devices in our
everyday working (and living) environments
Portable computers
Digital whiteboards
Table and wall displays
• Challenges
• Smooth integration of portable/personal and preinstalled/public computers for information exchange
• Connecting physical and digital (virtual) spaces
ISTD 2003
Spatially Continuous Workspace
Evolution of spatially continuous workspaces:
(a) user performs individual tasks with a portable computer
(b) table becomes an extension of the portable computer
(c) pre-installed computer displays (table and wall) serve as shared
workspaces for collaborative tasks
ISTD 2003
Key Features of System Design
Environmental computers as extensions of individual
Supports for links between digital information and
physical objects
Spatially continuous operations
(a) object dragged on computer screen
(b) object ”jumps” to table, when cursor
reaches edge of the screen
(c) object is dragged to the wall
(d) object is linked with a physical object
ISTD 2003
Prototype Environment: InfoTable & InfoWall
ISTD 2003
System Architecture
ISTD 2003
Portable computers and physical objects
have visual markers (printed 2D barcode)
• Problem: how to cover the entire table surface with a
single camera and sufficiently high resolution ?
• Solution: use a combination of two cameras
• Fixed camera: monitors whole table for changed areas
• Pan/Tilt camera: scans the table in 36 (6x6) pieces and
re-visits changed areas for high resolution images
• Added/removed object is registered in few seconds
ISTD 2003
Visual Marker Recognition
Each physical object has a visual marker (2D matrix code)
Markers can identify 224 different objects
Smallest size of a marker is 2cm x 2cm
Recognized entities: ID, marker’s position and orientation
ISTD 2003
InfoTable and InfoWall provide spatially continuous
workspace (low-resolution peripheral information space)
to the laptop (high-resolution focal information space)
(a) object is dragged towards the edge of the screen
(b) object (and cursor) migrates onto InfoTable
(c) object (and cursor) migrates from InfoTable onto InfoWall
ISTD 2003
Anchored Cursor
• Visual feedback: line from laptop to cursor
• Provided when cursor is manipulated outside the
laptop’s screen
• Allows identifying cursors’ owners
• Can be used to show the link between information on
the table and on the laptop
ISTD 2003
Object Aura
• Represents object’s information field (data space)
• Visualizes that object has been recognized by the system
• Digital data can be attached to physical objects by placing
them on the object’s aura
• If object is removed, attached data is saved on network server
• Attached data is re-displayed, when object is placed on InfoTab´le
ISTD 2003
Shared Information Surfaces
• InfoTable and InfoWall provide shared information
space for users
• InfoTable does not have ”top” or ”bottom”
• ”Near” sides are determined from visual markers
• Objects are automatically rotated when brought ”near”
ISTD 2003
Object Migration
• All applications are written in Java
• Object transfer with Java’s object serilization and
RMI (Remote Method Invocation)
• Supported migratable object classes
ISTD 2003
Sound (voice notes)
File short-cuts
Image files
User Experiences and Comments
• Hyperdragging easily understood and accepted
• Some users found moving objects across long distances
• Allows manipulating objects that are out of the user’s
physical reach
• Mapping scale between pointer movement and the
pointing device greatly affects usability
• InfoTable’s 20 dpi vs laptop’s screen’s 100 dpi
• Users appreciated attaching objects onto InfoWall
while sitting at the table
• Integration with e.g. printers and scanners?
ISTD 2003
More Information
Dr. Jun Rekimoto
Interaction Laboratory
Sony Computer Science Laboratories
ISTD 2003

Similar documents