of an object

The Auditory Perception of
Objects
Anna Preis
Institute of Acoustics
Chair of Environmental Acoustics
Andrzej Klawiter
Institute of Psychology
Chair of Logic and Cognitive Science
Adam Mickiewicz University, Poznań, Poland
November 5, 2008 Stockholm University
Traditional approach to audition
Common sense psychology, renowned researchers
even philosophers are all in agreement on this point:
we hear sounds
Hearing opens up a world of sounds for us,
and that these sounds are recognised by features
such as loudness, pitch and timbre.
Psychoacoustic dogma and its
rejection
 The complete characteristics of audition consists in
accouting for how the auditory system receives,
analyses and transforms
Acoustic waves
sound sensation attributes
 Research on natural audition explains how the
auditory system identifies distant objects by
extracting and processing information contained in
acoustic waves reaching the organism
What can be seen and what can be
heard?
Vision we see objects, but the process is
mediated by the electromagnetic waves
Audition we hear sound sources, but the
process is mediated by the acoustics waves
Both perceptual faculties provide us with
information about objects in an environment
Ecological approach to audition
 W. Gaver: there are two kinds of audition: everyday
listening and musical listening (studied in
traditional psychoacoustics)
 Listener can shift attention and choose the mode of
listening: it is possible to hear any sound in terms of
its source (everyday listening) or in terms of its
sensory qualities (musical listening)
 pros: everyday listening is not directed to sounds
but to their sources
 cons: cannot explain the functiong the auditory
system
Ecological approach to audition
• 1979 –Vanderveer –grouping of sounds
• 1979 – Lederman –roughness
• 1984 –Warren, Verbrugge – bouncing versus breaking of an
object
• 1988 – Gaver –type or size of material
• 1993 – Gaver –everyday listening, musical listening
• 1991 – Li et al. –walking person
• 1998 – Carello at al. – metalic ruler sound
• 2000 – Cabe, Pittenger –empting or filling the glass
• 2002 – Houben – velocity of balls
• 2003 – Rocchesso, Fontana – literature rewiev
http://www.soundobject.org/SobBook
The ecological versus the cognitive
approach to audition
• Ecological approach to
audition - inspired by Gibson’s
ecological psychology
Objects in an environment afford
structured information and there
is no need to process it
• Cognitive approach to
audition - inspired by
Marr’s computational
model of vision
• We need a model which
explains how the system
extracts and processes
auditory information
Cognitive approach to audition
 Our proposal is limited only to listening in the
natural environment
 In such environment the subject is engaged in
natural audition, that is auditory detection of
properties of objects
 Listening to speech and to music are different
tasks and should be explained separately
 To understand the natural audition we have to
model it
Principles of the functioning of the
auditory perceptual system
 The principle of functioning as an audile:
audition alone allows an organism to acquire information about
the attributes of objects in an environment
 The principle of constructing the auditory
representation (image) of an object:
an acoustic signal produced by a moving object carries
information about its movement and other properties. The
structure of this signal, its acoustic characteristics, is the basis
for the auditory representation of the object
 The principle of hierarchical processing of
auditory information:
the auditory detection of attributes of an object is the result of
multilevel information processing
The stages of processing visual and
auditory information
Vision
Audition
Primal sketch,
Acoustic hearing
The basic structure of an auditory stream
21/2-D sketch
Spatial hearing
Localization and distance of the sound
source
3-D model representation
of an object’s shape
Hearing an object
Primary and secondary characteristics of the
object
How do we hear according to this
proposal?
Imagine that we hear a passing car:
at the first stage the auditory
system identifies the attributes of
sound sensation like the loudness,
pitch and timbre of the auditory
stream
How do we hear according to this
proposal?
at the second stage the auditory
system estimates the direction and
distance of the sound source
How do we hear according to this
proposal?
at the third stage, we perceive how
the sound source moves, what its
size, weight, form, velocity,
acceleration, etc. is
Relevant experiments
 A series of experiments is needed to test the
multilevel model of auditory perception
 The subjects listen to environmental sounds
and are asked to judge not the sound
attributes but the attributes and states of the
sources that generated the sounds
 JND in frequency or intensity is replaced by
JND in velocity or mass
Differential velocity threshold
 Differential velocity threshold is independent
of the reference velocity (10, 20, 30 and 40
m/s) of linearly moving sound sources, varying
across listeners from 1.5 to 4.6 m/s
 The experimental results also show the
subject’s preference for the Doppler cue and
weakest sensitivity to the cue related with
interaural time differences
Differential velocity threshold
 In these experiments subjects did not have
any difficulties in answering the question
which of the two sources in a pair moved
faster
 The question referred to the source’s attribute,
its velocity, not to the sound’s attribute
 As can be seen from these experiments, people
perceived the velocity of a source quite well,
by using sound generated by it
wszystkie dostępne
tylko zmiany częstotliwości
tylko zmiany poziomu
tylko zmiany ITD
25
dv [m/s]
20
15
10
5
0
EJ
GH
HH
MW
SH
słuchacz
DJ
JK
CJ
JD
Differential mass threshold
 The falling ball of a given mass was
regarded as a sound source
 The ball was dropped from two
different heights onto two types of floor
material
 The aim of this study was to examine if
the auditory system is capable of
detecting the information about the
mass of the source.
Differential mass threshold
1.5m
85
a)
L A E [dB]
80
75
Hard-45cm
Soft-45cm
Soft-85cm
70
Smean [acum]
0.6
b)
0.5
0.4
400
600
800
1000 1200 1400 1600
M ass of ball [g]
JND of mass [g]
400
350
300
250
200
150
100
Hard-45cm
A verage
Ball 4
Ball 3
Ball 5
400
350
300
250
200
150
100
Soft-45cm
400
350
300
250
200
150
100
Soft-85cm
700
800
900
1000 1100
M ass of standard ball [g]
The JND mass values for all balls and all
subjects within each condition do not change.
The only exception is ball 5 in Soft-45cm condition.
The JND in mass values for all balls and all
subjects decrease as a function of L AE .
For example, the smallest values of L AE, in the
Soft-45cm condition corresponds to the largest
differential mass threshold.
Differential mass threshold
 Subjects did not have any difficulties in
judging which of the two sound sources
in a pair was heavier
 As a result of such an experiment an
analysis can be performed and the
question - which auditory cues derived
from the stimuli are responsible for the
mass perception? – can be answered
Futher developments of cognitive approach
to audition: auditory speech perception vs
auditory control of speech
SUBSYSTEMS PROCESSING
INFORMATION
PERCEPTION
MOTOR
CONTROL
Milner,
VISION
Goodale
(1995)
VENTRAL
STREAM
DORSAL
STREAM
Klawiter, AUDITION
Preis
(2005)
VENTRAL
DORSAL
(T. Kaczmarek -JASA 2005)
M. Ishibashi, A. PreisArchives of Acoustics 2005)
(H. Hafke - JASA 2008)
Proofs of existence of two separate
systems: vision-for-perception and visionfor-action
People with visual deficits
(DF)
Visual illusions
Titchener Circles Illusion Aglioti (1995)
Hollow Face Illusion Kroliczak (2005)
„what we think we ‘see’ is not always what guides
our actions”
Findings that support the hypothesis of
separated perception and motor
control in the auditory modality
B. H. Repp (2001) Phase Correction, Phase Resetting, and Phase Shifts After
Subliminal Timing Perturbations in Sensorimotor Synchronization (Journal of
Experimental Psychology)
B. H. Repp (2005) Does auditory perceptual illusion affect on-line auditory action
control? The case of (de)accentuation and synchronization (Exp Brain Res)
G. Hickok, D. Poeppel (2004) Dorsal and ventral streams: a framework for
understanding aspects of the functional anatomy of language (Cognition)
Investigation to verify the importance of
acoustic feedback in the control of the
fundamental voice frequency Fo
The „perception” task
‫ﻡ‬
Task: Vocalization of vowel /u/ for 5s
Question: whether trial had contained a
pitch shift and what direction it was?
We intend to check the possibility of
a situation when a listener despite
not hearing the pitch shift, performs
the adequate motor reaction.
The „motor response” part
Recordings of the listener’s voice
to further analysis subject’s reactions to
pitch shifts
Experimental details
•9 healthy young adults were tested (4 females 5 males). All the listeners were qualified as
having normal hearing. All of them were trained singers
•The subjects were seated in a sound-treated room
•Both beginning and ending of phonation was marked with a visual stimuli
•Voices were recorded with a Shure SM 58 microphone and processed for auditory feedback
pitch-shifting through a DP2 Ensoniq ultraharmonizer
•The output of the harmonizer was mixed with pink masking noise (75 dB SPL) to partially
mask bone-conducted feedback and presented to the subject over Sennheiser HD 600
headphones
•Voice feedback was pitch-shifted by 9, 19, 50 or 99 cents respectively in both directions.
•Every pitch shift value was repeated 30 times
•An automatic, MIDI based control system was used for pitch shifts in the auditory feedback
signal
•Each block of stimuli consisted of 45 vocalizations in which the pitch shift values were
randomly ordered
Analysis
reaction check
The „perception” part
Percentage values represent the
number of correct answers to the
pitch shift stimuli were calculated
The „motor response” part
Positive reaction:
- deviation in F0 with a magnitude of more
than 2 SDs of the 1000-m
pre-stimulus mean, a duration of at least
200ms in a maximum of 800ms after the
pitch shift
Reaction values in cents were calculated for
responses qualified as „valid” only.
Extracting the fundamental frequency
- algorithm incorporated in the
Praat software
fo frequency
low-pass filtering
Analyzing time changes in fundamental
frequency
- MATLAB environment
time
1000ms
800ms
Example of Fo response of a subject to downward
pitch stimuli
Individual results of the
„perception” task
Individual results of the
„perception” task
Green points indicate pitch changes that were below perceptual threshold
Individual results of the
„reaction” part
Individual results of the
„reaction” part
Green points indicate pitch changes undetected in the „perception” part of experiment
CONCLUSION
The results attained support the hypothesis of two separate
streams existing in auditory information processing.
The experiment showed that a human being is capable of a
motor response to changes in the acoustic signal omitted by
their perception
This leads to the conclusion that the information carried
by the acoustic wave can be used for purposes other
than conscious perception itself
It is also highly probable that certain reactions (to loud noises
for example) are mostly caused by a separate, motororiented, processing stream, and therefore a considerable
amount of psychoacoustic tests results needs to be
reinterpreted.
Klawiter, A.,„On hearing of objects” (in Polish, „O słyszeniu przedmiotów” ):
Poznańskie Studia z Filozofii Humanistyki, 5 (18), Zysk i S-ka, Poznań.
(1999)
Klawiter A., Preis A. (2006). „Percepcja słuchowa przedmiotów szkic teorii i jej
testowanie”, Kolokwia Psychologiczne nr 14, Neuronauka, Instytut Psychologii
PAN, Warszawa, 145-162
Ishibashi M., Preis A. (2005). „Auditory cue for the perception of the weight
of falling balls”, Archives of Acoustics. 30, 139-144.