Intersensory Interactions and Human Computer Interaction

Transcription

Intersensory Interactions and Human
Computer Interaction (HCI)
Intersensory Interactions
• Intro and metacognitive gap
• Integrating cogsci theory with design
• Cognitive Architecture
Brian Fisher
– Modularity and multimodal interaction
• Information hiding-- conflict resolution
• Cognitive impenetrability
• Performance differences between modules
• Recalibration
– Spatial indexes in complex environments
THE UNIVERSITY
OF BRITISH COLUMBIA
Key Points for HCI Practice
• The Metacognitive Gap: The need for a
grounded Cognitive Science approach
• Reflective design practice methods
– Integrating CogSci with interaction design
– Iterative design cycle
• Multimodal cue matching within modules
Fels: Design of Interactive Multimodal Graphics
Introduction
History -- why we need to adapt HCI
methods for multimodal interaction
• Examples of extended HCI
– Cognitive architecture: Multimodal displays and
how they are understood
– Situated cognition: embodied interaction with
complex displays
1
HCI History
Ergonomic HCI domain limits
• “Classic” HCI: User/Task/Tool Model
User
– Task, Protocol & GOMS/keystroke analysis
• Command-line, menus, workplace systems
• Theoretical underpinnings
– Cogsci of conscious thought
– Learning, Memory, Reasoning
– Sequences of operations
Evolution towards multimodal displays
Task
Tool
What if I am doing this for fun?
What if I want new insights?
What if I want to communicate with someone?
What if I am exploring a complex environment?
Iterative development cycle
• WIMP interface: visual semantics
– Metaphorical tool icons on desktop
– Direct manipulation
• Information visualization: Information
processing in the visual system
– Visual analogs of information
– Spatial Instruments
Walkthrough or experiment
Design
Test
How?
What?
Implement prototype
Spiral with increasing detail and test specificity
2
Future challenges
• Perception, cognition, and action in an immersive
multimodal environment populated with objects
events and actors
• Applications in entertainment, cognition,
communication
• Blend of virtual and real spaces… with seams
– Are the rules consistent?
Opportunities for creative design
• Environments: Affordances for exploration
– Spatial cognition, human space constancy theory
• Support for creative & logical thinking
– Problem solving, embodied cognition models
• Media-based communication & collaboration
– Metacognition, distributed cognition
• Experience (Kansei) engineering: Moving
beyond usability
– Can users shift between them?
– Can frames support rule shifts?
Effective Interface Design for Rich
Sensory Environments
The interaction between display characteristics and
the information processing characteristics of the
user’s perceptual, motor, and cognitive processes
will largely determine interface performance
Metacognitive Gap
• Intuitions about thoughts,goals and plans (“folk
Psychology”) are reasonably accurate
• Intuitions about how people see, hear, and
remember are very inaccurate
• Lack of awareness of the limits of intuition is
the “Metacognitive gap”
• Design-by-intuition leads to bad user
experience
3
Evolving interaction design models
• Guidelines are (still) inadequate
• User-centred design inadequate for
rich sensory environments
• Need for theory-rich, evidence-based
approach: design for lower-level
processes
• Must integrate with higher-order
cognitive task analysis and usercentred design practices
Test: walkthrough, field study or experiment
Exp:“Lab sense”,
FS: Observation
Information
foraging
Interaction Design
Ideas and hypotheses
Evaluation,
Mapping
• Convincing designers that there is something to
understand—the “metacognitive gap” of folk
Psychology
• Convincing Cognitive Scientists to answer relevant
questions—Complex data displays and multiple
tasks and the Psychophysics reductionist approach
• Integrating research and design—Finding a
common language
Reflective HCI Practice (Schön)
Implement
Bridging the theory/practice gap
(Online) Literature:
Psychology
Kinesiology
Sociology
Anthropology
Architecture…
Fields of interest: Experimental
Psychology:
– Founded~ 100 years ago
– Areas of study
• Psychophysics—Vision, hearing, tactile
senses
• Attention—Endogenous, exogenous,
sustained
• Learning and memory
• Goal is often information processing
algorithms
• Discussed in Module 2
4
Fields of interest: Kinesiology &
related disciplines
Useful things from Psychology
•
Perception and attention
– Visual & auditory acuity & discrimination
– Colour perception
– Salient display changes, change blindness
•
– Neuroscience
– Mechanics
– Anthropometry
Learning and memory
–
–
–
–
•
• Science of human movement
Primacy, recency
Skill acquisition
STM limits
State-dependent learning
Decision making
• Examples: Fitts’ law, GOMS/Keystroke
• Goal is often perceptually guided behaviour
• Discussed in Module 3
– Base-rate neglect
– End effects, biases
Fields of interest: Cognitive Science
• Cogsci Society founded 22 years ago
• Combines Experimental Psychology,
AI, Philosophy and Neurophysiology
• 3 levels of analysis
– Semantics: Intentions, Goals, and
Meanings
– Syntax: Information processing
– Implementation: Neural processing
• Goal is often Cognitive Architecture
Other fields of interest
•
•
•
•
•
Psycholinguistics
Social cognition
Cultural Psychology
Communication theory
Embodied communication,
paralinguistics
• Anthropology
5
Example of metacognitive gap
“Horizontal” Modularity
• The “computer metaphor” of mind
• Model Human Processor (MHP)
• Intuition: Single mental processor reads all
• Serial stages of processing
senses and performs a variety of tasks.
• Cognitive Science: Processing modules
operate in parallel.
Action
• Information flow is Bottom-up
Cognition
– Cognitive impenetrability
– “Seeing is believing”
– Restricted flow of information and control.
Perception
– Processing characteristics are counterintuitive
Horizontal modularity restrictions
Cognitive impenetrability (Pylyshyn,
1984) refers to the inability of observers
to use semantic information (such as
what the person believes or intends to do)
to influence the operation of the input
stage.
Vertical modularity (Fodor)
Cognitive processing
Phoneme
perception
Auditory
localization
Voice
recognition
6
Vertical modularity restrictions
Information encapsulation (Fodor, 1983)
refers to structural barriers within the
cognitive architecture that prevents internal
data stores from being shared between
modules in the same stage of processing.
Events are processed by many channels
• Intuitions about thinking
Cognition
– Fails at low levels
• Cognitive Architecture
Bimodal
speech
Ventral
system
Dorsal
System
…
– Multiple brain areas
– Interconnected
Sensory world
– Informationally encapsulated
– Multimodal inputs parsed from scene
Modularity of reading
•
•
•
•
Name the colour of the text
Respond as quickly as possible
Measure response time
2 trials
Dog
Cat
Fish
Bird
Cow
Horse
Pig
Cow
Fish
7
Green
Red
Orange
Red
Blue
Blue
Orange
Green
Red
Advantages of modular processing
• Download task to module, reduce
cognitive bottlenecks
• Fast, effortless information processing
• Near-optimal information integration
between cues and sensory modalities
– Fuzzy logic cue integration
– Bayesian categorization
Cognitive Impenetrability & Modularity
• Stroop effect
– Reading is data-driven module
– Competition for response
• Other Modularity phenomena
– Modularity of perception for action
– Modularity of visual/auditory integration
– Modularity of eye movement control
– Modularity for Models of Minds
"Civilization advances by extending the
number of important operations which
we can perform without thinking
about them."
Alfred North Whitehead
What are the disadvantages?
8
Disadvantage: Processing rigidity
• Stress within a module is not accessible
• Complex stimuli may be processed
differently in different modules
• Tasks may access different modules with
different performance characteristics
• Virtual environments may introduce
discrepancies that impact different modules
(and tasks) differently
Stress within a module may be
undetected
Action
Cognition
Poor cognitive access to
low-level processes
Perception
Example: VDT stress syndrome
• Users complain of headache, vision
problems etc.
• Reports anecdotal, but reading
impairment is observed
• Also pupillary tremor and regressive
saccades
Reflective HCI Practice (after Schön)
FS: Observation
Information
foraging
Implement
Interaction Design
Evaluation,
Mapping
Psychology
Kinesiology
Sociology
Anthropology
Architecture…
9
Reflective HCI Practice: Foraging
Theory of space constancy in active
vision
FS: Observation
Information
foraging
Psychology
Kinesiology
Sociology
Anthropology
Architecture…
FS: Observation
Information
foraging
Evaluation,
Mapping
–
–
–
–
Efference copy
Passive blur
Lateral masking
Saccadic suppression
Reflective HCI Practice: Hypothesizing
• Cognition needs the illusion of a stable world
• Eye movements should create confusing
image shifts
• Maintaining space constancy requires
Psychology
Kinesiology
Sociology
Anthropology
Architecture…
VDT fatigue study
• Hypothesis “Sampling” raster during
saccades reduces intra-saccadic blur, and
may overload saccadic suppression
• Space constancy perspective allowed us to:
– Isolate the important factors in a complex
situation
– Find a more sensitive task and measure
• Study examines detection of movement
during saccade (w Bridgeman & Macnik)
10
VDT fatigue study recommendations
• Suppression thresholds elevated for
flickering stimuli
• Effect is reduced >250 Hz
• Problems will be greatest
– large saccades
– high-contrast display
• Work arounds include blanking display
during saccades
Reflective HCI Practice: Testing
FS: Observation
Information
foraging
Implement
Interaction Design
Evaluation,
Mapping
Disadvantage: Each module must
solve feature assignment problem.
• Modules can’t accept information from other
modules: Information encapsulation
• Different modules should have access to a
different set of matching cues.
• Illusory conjunctions can occur in
multimodal environments:
– Phoneme perception: The McGurk effect
– Auditory localization: The ventriloquist effect
Psychology
Kinesiology
Sociology
Anthropology
Architecture…
Study: Impact of display errors on
multimodal perception
• Immersive environments typically
have display errors
– Location of events is not precise
– Timing is not precise
– Graphics can be low-fidelity
• As immersive environments add
sound and touch, what will be the
impact of these errors?
11
Reflective HCI Practice:Foraging
FS: Observation
Information
foraging
Psychology
Kinesiology
Sociology
Anthropology
Architecture…
FS: Observation
Information
foraging
Evaluation,
Mapping
• Individual senses adapt to display
• Sensory modalities calibrate each other:
haptics, vision, sound
– Observed actions calibrate visual space (space
constancy)
– Vision calibrates hearing for the location of a
multimodal event
– Sound calibrates vision for the time of a
multimodal event
• Result is an after-effect: remapping of
auditory (visual, haptic) space
Reflective HCI Practice: Hypothesizing
Recalibration by pairing (Epstein, 1975)
Psychology
Kinesiology
Sociology
Anthropology
Architecture…
Impact of information encapsulation
• Multimodal environment with errors in
timing and location
• The same event might give rise to a single
multimodal construct in one task, and two
unimodal events for another.
– Vary location of visual and auditory phonemes
in a simple teleconferencing-style video display
– Vary information carried by using synthetic
speech stimuli (5 levels).
12
Ventriloquism meets the McGurk
effect.
• Vary location of visual and auditory phonemes
in a simple teleconferencing-style video display
• Vary information carried by using synthetic
speech stimuli (5 levels).
• Subjects report sound location and syllable
heard,
• Analyses included testing a variety of
mathematical models of information integration
by fitting free parameters with STEPIT.
FS: Observation
Information
foraging
Interaction Design
Evaluation,
Mapping
• Sensory input from a number of channels
simultaneously
• How stimuli from multiple channels are
matched and partitioned into mental
representations
• How information from multiple senses is
integrated to give rise to trans-modal
mental events
Reflective HCI Practice: Testing
Implement
Use of mathematical modeling tools
allow us to address
Psychology
Kinesiology
Sociology
Anthropology
Architecture…
Results:
• Visual capture of auditory source location, resulting
in a shifting of unimodal auditory location
estimation (ventriloquism after-effect).
• No effect of location difference on phoneme
perception as measured by statistical or modeling
tests.
• No correlation between errors in the two tasks (i.e.
subjects could not selectively attend to the auditory
phoneme on trials when visual capture failed).
• Overall, modularity of phoneme perception is
supported.
13
Disadvantage: Changes in task interact
with modules to change performance
• 2 visual systems—“ventral stream” for
recognition and “dorsal stream” for action.
• Where vs how
• Different impact of illusions
• Lesion data
Functional Neuroanatomy of
perception for action.
2 visual systems—“ventral stream” for cognition and
“dorsal stream” for motor performance.
2 visual systems lesion evidence
2 visual system illusions
lesion
performance
deficits
spared abilities
V1 (blindsight) detection and
identification
pointing
Ventrolateral
occipital (DF)
identification, shape
recognition, object
orientation
object manipulation
(orientation matching,
grip scaling)
Posterior
parietal (RV)
object manipulation identification, shape
(orientation matching, recognition, object
grip scaling)
orientation
stimuli
deficits
spared abilities
Tichner circles size report
grip scaling
displacement
during
saccade
detection of
displacement,
location report
pointing
Moving or offcentre frame
induced motion,
location report
pointing
14
Research with videoconferencing
and abstract displays
• Targeting sound: cognitive better than
motor
– Subs aware of visual and auditory
locations, but point to visual
• Targeting vision with context: Less
feedback is better
– Pointing with no visual feedback better
– Lagged cursor better than unlagged
Interpreting pointing studies
• Pointing studies counterintuitive, but
predicted by response characteristics
of neurons in dorsal/ventral to visual
and auditory stimuli
• More in “Graphics in the Large” panel,
Thursday 3:30
2 visual system illusions
Modularity Recap
stimuli
deficits
spared abilities
Tichner circles size report
grip scaling
displacement
during
saccade
Moving or offcentre frame
Sound with
displaced
visual
distractor
pointing
detection of
displacement,
location report
induced motion,
location report
pointing
pointing
apparent location of
sound
• Displays can interact with active perception
to cause hidden stress: VDT study
• Displays can impact matching cues for
multimodal cue integration
– Support or frustrate recalibration by pairing
– Lead to discrepancies in number and
composition if stimuli in different modules
• Displays can cause disagreement between
motor performance and cognitive measures
• Sometimes removing information helps
15
Extending to complex worlds
Indexical cognition (Pylyshyn)
• Previous studies in simple worlds, with a
few visual and auditory events
• Multimodal environments are complex
– Virtual worlds
– Augmented reality
– Ubiquitous computing
• How are multiple multimodal events dealt
with in the cognitive architecture?
“FINSTs... make thoughts true”
• Perception
–
–
–
–
–
“Hotlink” tokens
Drawn to salient events
Object-centred, “sticky”
Visual routines
Finite number ~ 4
• Cognition
–
–
–
–
–
Maintain object history
Implicit memory of object associations
Sparse cognitive representation
Just-in-time delivery of information
Atom of intentionality
Mental representations of complex worlds
• Cognitive architecture perspective requires
that links be established between lower
level perceptual qualities and cognitive
symbols—i.e. a pointer, called a FINST.
• FINSTing allows us to interact with
perceptual objects and events without the
need for mental images per se.
• Symbolic representation + pointers makes
different predictions than intuitive picture-inthe-head
16
Indexical cognition (Pylyshyn)
Naïve view of FINSTs in Cognitive Arch
Action (motor space)
FINSTs
Phoneme
perception
Auditory
localization
Voice
recognition
Another view of FINSTs
More about FINSTs
Action (motor space)
• FINSTs Link mind & perceptual world
FINSTs
Phoneme
perception
Auditory
localization
Voice
recognition
– Visual routines: (collinear, inside,
subitizing)
– History of an object
– Object-centred, “sticky”
– Drawn to salient changes-- onsets,
luminance increments, oddballs
– Finite number ~ 4-7
– FINSTs + ANCHORs for motor behaviour
17
More about ANCHORs
Multiple object tracking demo
• ANCHORs link mind & action
– Remembered locations for eye
movements
– Direct interaction with items off the retina
– Fast, robust motor performance by action
routines
– Affordances for action
Another trial (Scholl)
Another trial (Scholl)
18
Application
Mental representations of complex
environments
• Cognitive architecture perspective requires that
links be established between lower level perceptual
qualities and cognitive symbols—i.e. a pointer,
called a FINST.
• FINSTing allows us to interact with perceptual
objects and events without the need for mental
images per se.
• Symbolic representation + pointers makes different
predictions than intuitive picture-in-the-head
• Coping with spatial transformations in complex
data spaces
Module advantages
• High-realism interface designs improve
performance by “downloading” information
processing operations to input modules.
• Interaction of display characteristics with
capabilities and characteristics of the
functional architecture will determine
performance.
Module disadvantages
• Coordination
– Distortions in location, timing, and category-relevant
information may lead to the formation of conflicting
representations in different modules.
• Processing inflexibility
– Errors and conflicts within a module can create errors and
increase cognitive load. (CRT flicker example)
• Information hiding
Cognitive impenetrability of modules makes it difficult for
operators to determine the reasons for their poor
performance.
19
Inseparability of Mind & World
•
•
•
•
Embodied cognition-- mind/body
Situated cognition-- mind/world
Distributed cognition-- mind/mind
Ecological theories (Vygotski, Luria,
Bateson, Gibson) can be linked to
sensory phenomena and inform
interaction design
What to expect in the next talk
• More about perceptual and attentive
bottlenecks
• Change blindness
• Three flavours of (in)attention
• Implications for visual displays
Key Points for HCI Practice
• The Metacognitive Gap: The need for a
grounded Cognitive Science approach
• Reflective design practice methods
– Integrating CogSci with interaction design
– Iterative design cycle
• Examples of extended HCI
– Cognitive architecture: Multimodal displays and
how they are understood
– Situated cognition: embodied interaction with
complex displays
Environment/Inhabitant/Representation
• Information visualization
• Perceptual/deictic/situated Cognition
• Cognitive Architecture, perception, attention
• Display as extension of mental model
• Coercive graphics
20
Thirteen Ways of Looking at a Blackbird
Wallace Stevens
I
Among twenty snowy mountains,
The only moving thing
Was the eye of a blackbird.
II
I was of three minds,
Like a tree
In which there are three blackbirds.
III
The blackbird whirled in the autumn
winds.
It was a small part of the pantomime.
IV
A man and a woman
Are one.
A man and a woman and a blackbird
Are one.
V
I do not know which to prefer,
The beauty of inflections
Or the beauty of innuendoes,
The blackbird whistling
Or just after.
VI
Icicles filled the long window
With barbaric glass.
The shadow of the blackbird
Crossed it, to and for.
The mood
Traced in the shadow
An indecipherable cause.
VII
O thin men of Haddam,
Why do you imagine golden birds?
Do you not see how the blackbird
Walks around the feet
Of the women about you?
VIII
I know noble accents
And lucid, inescapable rhythms;
But I know, too,
That the blackbird is involved
In what I know.
IX
When the blackbird flew out of sight,
It marked the edge
Of one of many circles.
X
At the sight of blackbirds
Flying in a green light,
Even the bawds of euphony
Would cry out sharply.
XI
He rode over Connecticut
In a glass coach.
Once, a fear pierced him,
In that he mistook
The shadow of his equipage
For blackbirds.
XII
The river is moving.
The blackbird must be flying.
XIII
It was evening all afternoon.
It was snowing
And it was going to snow.
The blackbird sat
In the cedar-limbs.
21

Intersensory Interactions and Human Computer Interaction

Transcription

Similar documents

Paper - DFKI

Math Filler Games for Kindergarten

Computer

Desktop PC

Jewel of the Nile Graphics Elements

Love Our Kids Gala - Federation Early Learning Services

jd kaplan - GraphicsAD.com

the Brochure!

WWAY station in a box