Aural Limbo

Aural Limbo
Space as a sonic interactive interface
A Thesis Submitted to Parsons School of Design, a division
of New School University, New York in Partial Fulfillment
of the Requirements for the degree of Master of Fine Arts
in Design and Technology.
Mateo Zlatar
2003
Thesis Supervisors: Golan Levin, Mark Stafford
Thesis Advisor: Stephanie Owens
Aural Limbo
Space as a sonic interactive interface
Abstract
Aural Limbo is an interactive installation that uses the
physical relationship between sound and space as a poetic
theme. In this work I seek to create opportunity for
spontaneous interaction in the context of the public space,
engaging passersby in an inhabitable sonic instrument that
uses the body presence and location as variables for the
dynamic transformation of sound.
The physical space acts as host for an aural space composed
of everyday sounds of the cityscape. These sounds represent
the heterogeneity of our environment, the disregarded
result of hundreds of activities happening at the same time,
which cognitively don't have particular significance and
usually is filtered as meaningless acoustic information
( noise ).
The play of this work is based on the idea that we can
discover meaningful sound patterns in noise, in this case
through an interface that reveals them, making us aware
of listening itself, as a creative act.
1
Acknowledgements
I am profoundly grateful to my wife Paz Guzman and my
Family for their support during this process. I also feel
beholden to the people who provided me with advice and
inspiration. Thanks to: Golan Levin, Stephanie Owens, Josh
Goldberg, David Rokeby, Mark Stafford, Colleen Macklin,
Sven Travis, Marko Tandefelt, Camille Utterback, Barbara
Morris, Jose Miguel Tagle, Elaine Castillo Keller, Fang-Yu
Lin, Mimi Chan, Eduardo Matamoros, Juan Herrera, Matias
Martinez, Ian Szydlowsky, Matthew Mohr, and all the DT
Community at Parsons.
2
Table of Contents
Preliminaries
Title
Abstract
Acknowledgements
Table of Contents
Chapter 1. Introduction
1.1. Motivations
1.2. Overview of the Thesis
1.3. Contribution of this Thesis
Chapter 2. Background
2.1 Introduction
2.2 Spaces for sound
2.3 The Aural Perception of Spaces
2.3.1 Space as Sound Points
2.3.2. Sound as Feedback of Space
2.3.3. Spaces with aural directional
messages
2.3.4. Dislocated Perception
2.3.5. Sound as Inner Space
2.4 Sound as Space
2.5 Patterns in Noise
Chapter 3. Methodology
3.1. Introduction
3.2. Preliminary Experiments
3.2.1. Wearable Synth (Fall 2002)
3.2.2. Trigger Space (Spring 2003)
3.2.3. Sonic Arena (Fall 2002)
3.3. Thesis Prototypes
3.3.1. System Overview
3.3.2. Traffic Report (Spring 2003)
3.3.3. Spatial Scrub (Spring 2003)
3
3.3.4. Accumulation
3.3.5. Panning Sound
3.4. Summary of Implementations
3.5. Thesis Installation
3.5.1 Revealing Patterns in Noise.
3.5.2. The Context of the Public Space.
3.5.3. The users experience
Chapter 4. Discussion and Analysis
4.1. The Music of Sound
4.2. Challenges and Pitfalls
Chapter 5. Conclusion
5.1. Conclusions
5.2. Future Directions
Chapter 6. Bibliography
Appendices
Appendix A. Custom Interfaces Screenshots
A.1. Program for Accumulation
A.2. Program Interface for Aural Limbo
A.3. Program Interface for Traffic Report
Appendix B. Supplementary Sketch
Colophon
4
Chapter 1. Introduction
1.1. Motivations
“Whereas colors are present “naturally” in nature, there are
no musical sounds in nature, except in a purely accidental and
unstable way; there are only noises.” Claude Levi Strauss,
“The Raw and the Cooked”
The phenomenon of sound has always intrigued me as a
physical experience. Our bodies absorb the acoustical
stimuli not only by the ears; they can perceive the vibration
of the sound waves traveling through the air and bouncing
between the walls, augmenting our perception, modifying
our mental and physical environments.
Sound as an invisible medium needs to be embodied
through physical interfaces in order to be produced and
manipulated, like the piano or the violin in traditional music,
but these instruments represent just a slice of the entire
audible spectrum, and moreover, they represent the culture
of music, which uses discrete units –timbre, pitch and
duration, as semantic units of the language of music. But
sound itself precedes language and culture as a natural
physical phenomenon.
Fig.1 Sketches of the Watertank Experience.
Years ago, a couple of friends and I went in a field trip to
an abandoned water tank in the middle of the forest. The
tank was three stories tall and about 50 feet in diameter
and was completely empty. Once inside we started throwing
stones at the walls, discovering a huge reverberation of
these percussive sounds bouncing between the walls. We
also tried our voices and all possible sound sources within
the space. Sounds were trapped in the space for several
minutes mounting one on top of the other, giving the
possibility of building chords and creating complex
rhythmical patterns. Suddenly I was aware of how sound
5
propagates through the air and the importance of the space
in this phenomena. I realized that space can act and be
treated as a sonic instrument.
In the present thesis, I explore intrinsic relationships
between Sound and Space; from their concreteness as
primary elements of perception to the representational
aspects they may reflect from culture. In this work I seek
to create a non-conventional sound instrument, which goal
is not only to serve as medium of expression, but ultimately
to make us aware of the act of listening itself.
In large urban cityscapes, such as New York, our acoustic
environment is highly heterogeneous and saturated by
hundreds of activities happening at the same time. In the
impossibility of paying attention to every single stimulus,
we filter out what we consider as meaningless acoustic
information (noise). In this context, the concept of noise
becomes highly relative, because we determine what sounds
are significant in a particular moment. The same sound
may be considered meaningful or meaningless, depending
on the focus of attention.
In the process of this work, I explore this idea of a polarized
noise, using different spaces in where the presence of the
body unchains a series of dynamic modifications in nonmusical sounds, revealing hidden rhythmical patterns
emanating from them. The actual space remains the same
while sound reshapes a virtual space, one that can change
its form, texture and size.
The discovery of this “other” space happens in a similar
way as we may see in digital imagery where things
seamlessly morph from one form to another, for example
an abstract object that resolves in a human head. The
surprise of it lies in the change of meaning that the object
suffers along the way. The moment of the transformation
6
is far more suggestive than the final result.
The word Limbo in its common definition is: An intermediate
place or state, An imaginary place for lost or neglected
things, The state of being disregarded. Most of these
definitions are derived from the theological conception,
which refers to an indeterminate place between earth and
heaven, or in Dante’s words: “the higher state the man can
achieve without god”. The Latin word limbus, refers to “an
ornamental border to a fringe”, or a “band or girdle” that
was chosen by Christian theologians of the Middle Ages to
denote this border region.
In my work, the metaphor of Limbo points to the experience
of a meta-space, one that is revealed through contemplative
and expressive rituals as we may find in sacred spaces,
particularly in rituals where the transformation is driven by
sound. This situation may be found in ancestral rituals,
such as in mantra praying or in African drums, where sound
and space play the role of an interface between man and
his divinities.
1.2. Overview of the Thesis
This Thesis is organized in four chapters. In Chapter 2,
Background, I identify the main variables that may be
involved in the development of this work, which are: The
creation of spaces for sound, The aural perception of spaces,
Sound as Space, and the semantics of sound. For each one
of them I recall the work of modern and contemporary
artists who have addressed such conceptions from a variety
of perspectives, providing documentation and analysis.
Chapter 3, Methodology, documents the prototyping phases
in the development of this Thesis, which is comprised of
five design experiments in interactive sound. The process
7
is exposed through documentation on the design and
programming fronts, as design sketches, flowcharts and
mapping diagrams, explaining functionality, interaction
goals and expectations, that finally are summarized and
evaluated.
Chapter 4 is dedicated to discussion and analysis of the
work outcome, evaluating at all its development stages.
Chapter 5, Conclusions, summarizes the Thesis
contributions, pointing unfulfilled desires and future
directions.
Finally, I provide two technical appendixes, one showing
visual programming diagrams and other for supplementary
sketches.
1.3. Contribution of this Thesis
The development of digital technologies has reduced the
way we process images and sound to the same level of
abstraction and treatment, zeros and ones. Programmability
as an intrinsic computational attribute, allows us to
manipulate and specify our will over several kinds of data.
In this common ground, digital designers have the chance
to approach sonic design problems in the same way as they
do visual information. However, the contemporary practice
of multimedia design has remained centered on visual
communication. The use of sound is generally treated as
support for visual information, but not usually in the
opposite way.
While doing research on Sound in New Media, I noticed
how contemporary art practice has addressed sound in a
completely new way, separating it from the musical tradition,
creating a whole new field for conceptual/concrete
expression which is somehow known as Sound Art. The
choice of sound, as medium for this project, represents for
me an opportunity to incorporate the vocabulary of Sound
8
Art into the realm of computer multimedia.
This thesis may be considered an Art project because it has
aesthetic and philosophical goals, but also represents an
inquiry on human-computer interaction, because it
undertakes the responsibility of adapting an information
system to human needs. The use of ‘invisible computing’
in my work, can be seen as a particular example and possibly
extrapolated to the wide variety of tasks we all do with
computers, such as planning, analyzing, visualizing or
entertaining ourselves, which can happen in an individual
or collective basis, collaborating with physically present or
non-present others. All these tasks don’t require the
sedentary behavior of the desktop. In this regard, this thesis
proposes the use of computational augmented spaces as
a contemporary form of computer interaction.
9
Chapter 2. Background
2.1 Introduction
Through the history of architecture it is interesting to notice
how humans have always modified space to host particular
sonorities; for example small offices wrapped in double
panels absorb sounds, creating perfect spaces for private
conversations, and big temples result in perfect
reverberating spaces for elevated chants and prayers. The
constant association between type of spaces and kinds of
sounds contributes to our cultural responses to places.
Our notion of reality leans in each one of our senses. What
if in a sunny day we enter our homes and instead of what
is expected you hear a storm? How would this intervention
change our perception and behavior in that space? In this
thesis I seek to work on the manipulation of sounds and
space at both physical and semantic levels.
In this inquiry, for me was important to understand how
contemporary art practice has addressed sound in a
completely new way, separating it from the musical tradition,
creating a whole new field for conceptual/concrete
expression, giving form to what we know as “Sound Art”.
This chapter uses works of installation or architectural
intervention and performance that depict one or more of
the three key aspects my work wants to address, which I
summarize as:
› The creation of spaces for sound.
› The aural perception of spaces. (Sound as Space)
› Sonic Representation and the semantics of sound.
10
2.2 Spaces for sound
Through the history of architecture it is interesting to notice
how humans have always modified space to host particular
sonorities; for example small offices wrapped in double
panels absorb sounds, creating perfect spaces for private
conversations, and big temples result in perfect
reverberating spaces for elevated chants and prayers. The
constant association between type of spaces and kinds of
sounds contributes to our cultural responses to places. By
looking at the form and size of a space, one may infer what
kind of sonority it will have, but the visual perception does
not prepares us to the actual aural experience they convey.
Some spaces that may seem to be acoustically neutral or
inert, may surprise us with unexpected sound properties,
the sonic quality of spaces influence our perception of
them, modifying its significance and our behavior in that
space. Is in that discovery that we become aware of listening,
paying attention to the subtle variations and underlying
patterns that sound may reveal.
As an example of this kind of experience I came across a
particular work of the Brazilian Architect Oscar Niemayer,
who during the construction of Brasilia in the sixties,
designed a curved structure that served as cover for the
rain in a public square. This construction that seems to be
very functional also was acoustically designed to amplify
the sounds of people’s footsteps,
Fig.2 Public building in Brasilia by Oscar Niemayer (1960).
11
creating percussive patterns by the reverberation of its
sounds. A friend of mine, who brought me pictures of the
building, described the experience as an unexpected gift,
because she just walk through the space without knowing
what was going to happen. As she discovered these
rhythmical patterns, she also noticed how other people
stayed in the space to play with those sounds, as if the
building were a large-scale sound instrument. A building
that is a sonic instrument.
A different kind of strategy for sound spaces can be found
in the work of the Austrian artist Bernhard Leitner, who
created an architectonic intervention as a way to redirect
natural sounds. In 1997 he was commissioned to produce
a sound work for the Donaueschinger music days, a yearly
contemporary music festival in Donaueschingen Germany.
Leitner choose a public park by a river, which had a pergola
under which the water flows. He did a simple intervention
by hanging up a curved metal sheet, which reflects the
natural sound of water to the inhabitant’s ears. This surface
acts as an amplification device for natural sound, which
can be perceived just by inhabiting the space.
Fig.3 Bernhard Leitner: Wasserspiegel (Water Temple).
Among Leitner’s work we also can find hybrid sound
techniques that combines physical construction with
electronic amplification to redirect and highlight natural
sounds. As an example, Le Cylindre Sonore is a cylindrical
12
structure built by Leitner in Parc de la Villette, a public park
in Paris. A double ring made of concrete host 24
loudspeakers between the walls; microphones in the
surrounding bamboo forest capture and amplify its sound
to the interior of the ring. The electronic augmentation of
the natural sounds creates a paradoxical perception: an
indistinguishable blending of actual and virtual sound
spaces, which give the space an almost magical property,
a “listening building” in its double connotation.
Fig.4 Bernhard Leitner: Le Cylindre Sonore, Parc de la Villette, Paris. 1987
2.3 The Aural Perception of Spaces
"If we were trained to turn mentally towards everything we
hear, we would achieve a sense of spatial correspondence
comparable to visual perception." Bill Fontana.
In the interrelation of spaces and the sounds they produce
or host, we can also reverse the picture by studying how
sounds may reassure or neglect the space that contains
them, modifying our perception of spaces. The work of
Bernhard Leitner provides a good example of the design of
spaces based on the vocabulary of sound. The perception
of these spaces unfolds in time with the movement of lines
or points of sound, which perceptually modify our spatial
13
experience, making the whole body a permeable acoustic
depository, a “big ear” for three-dimensional space.
Here I use the work of Leitner as a way to exemplify and
discuss some strategies for building spaces with sound.
In the spring of 1971, Leitner started a series of practical
investigations in a large hall in New York. These
investigations were based on theoretical projects and
concepts, originally published in Artforum (March 1971).
The following lists the investigations and experiments
conducted by Leitner during the 70’s in chronological order:
Soundcube: Borders of a corridor, Swinging space.
Raum-Wiege (Sound Swing)
Leit-Räume (Guiding Spaces): Sound Gate, Sound Slopes
Gang-Variationen (Corridor Variations)
Liege-Objekte (Lying Within Sound)
Trag-Objekte (Portable Objects)
Vertikale Räume (Vertical Spaces)
Erweitern – Verengen von Raum (Expanding-Contracting
Spaces)
These instruments can be broadly classified as general and
specific ones. “Soundcube” and “Spatial Grid” are general
instruments because they are neutral structural matrices
where to test several different sound movement
specifications in space, analyzing their effect on the
perception of space. In the other hand, specific instruments
are the ones designed specifically for certain pre-defined
and particular sound movements such as: “Sound Swing”,
“Guiding Spaces” and “Corridor Variations.”
All these instruments utilized an electronic switching device
built according to Leitner’s specifications, which allow the
programming of sound sequences for any number between
two and forty loudspeakers.
14
“The Soundcube is an instrument for producing space with
sound. It has a grid of loudspeakers on each of its six walls.
It is visually speaking “neutral”, i.e., without any specific
spatial message. The sound is programmed to travel from
loudspeaker to loudspeaker. The dimensions of the cube
depend on the particular situation. An infinite number of
spaces or spatial sensations can be created. The Soundcube
is a laboratory for studies in the definition and character
of space and for investigation into the relationship between
motions of sound and their audio-physical experience. At
the same time it is a place for demonstrations to the public.”
(Leitner, 18.)
Soundcube and Spatial Grid were never built, although
Leitner designed the sound movement specifications in
custom scores, and punch-card programming.
These instruments served as hypothetical situations were
to develop his theoretical framework. The hypothesis he
was trying to depict, was “Rhythm as Space” referring to
accentuation of individual points in space by the pulsing
of individual speaker units across the six walls. A sequence
of pulses describes direction, creating the illusion of lines
and circles of sound traveling in space, in which intensity,
tempo and duration became critically significant.
2.3.1 Space as Sound Points
Across the diverse experiments conducted by Leitner, we
can distinguish specific elements and variables he used as
a strategy to build spaces with sound, which I summarize
as:
Fig.5 Spatial grid of speakers and
movement of sound in Sound
Cube, Leitner 1971.
› The sequence of loudspeakers. (Direction, trace, line.)
› The speed of traveling sound. (Timing)
› The intensity of sound at each point.
› The quality of sound (Timbre.)
15
The choice of sounds for most of his experiments was
percussion and simple electronic tones. They were used as
ways to demarcate points in space, the message of these
sounds was tied to the perception of dimensions in space,
rather than conveying a message in sounds themselves.
As an example of this, I recall Leitner description of sounds
in “Corridor Variations” (1973)
“The arching example shows how fast, soft beats (small
drum) start forte (T) in A2, lift off with a decrescendo, move
slowly (Q) and piano (V) through the two top loudspeakers
(A0,A7) and continue to A6, increasing in both traveling
speed and intensity before ending in A5 with forte (T). The
perception of the ceiling’s curvature depends upon the
decrescendo/ritardando of the rising line and the
crescendo/accelerando of the falling line (steep arch, flat
arch).” (Leitner, 52.)
2.3.2. Sound as Feedback of Space
Fig.6 Borders of a corridor, Leitner 1971.
Fig.7 Swinging Space, Leitner 1971.
“Borders of a corridor” (fig.6), is a sequence of sound
movements describing three inclined planes in half of the
cube, mirroring the exact same situation in the other half,
creating a central corridor in the middle of these two
movements. In “Swinging space” (fig.2), there are seven
circles in the horizontal axis gradually changing their
inclination angle. In these two early experiences, individual
accentuation of points in space creates the illusion of a
sound line describing movement, establishing dimensions
of height and length, which gives an aural perception of
boundaries of space. Although these boundaries cannot
be experienced at once, they are transformed, repeated
and developed in the dimension of time.
“Sound Swing” (1975) is an actual installation conveying
16
Fig. 8 Sound Swing, Leitner 1975.
the idea of a pendulum of sound. (fig.8) This instrument
consists of four loudspeakers, two of them placed at the
ends of mirroring diagonal platforms, and the other two at
the bottom of the structure, leaving a central space for the
body to walk by or stay. The diagonal wooden panels act
as an acoustical resonance link between the upper and
lower loudspeakers, which distance between them was
empirically determined to convey the perception of
continuous pendulum-like motion.
The program for Leitner instruments was recorded on
punched tape in three different codes: one determining the
sequence, one the intensity and one the speed of motion.
“Working with a visually readable program permits one to
introduce directly corrections such as adding or taking out
a loudspeaker in a particular sequence, or modifying the
intensity of each loudspeaker.” (Leitner, 14.).
Fig.9 An example of Leitner’s
punched card programming.
In particular cases, his instruments allowed the users to
manually select the speed and intensity of the sequences,
which he thinks as “self-adjustable feedback between
person and space.” I think this idea can be extrapolated
to the dynamic possibilities offered by contemporary
computer capabilities. How these experiences may be
enhanced if we give users the control of sound variables
on the fly? For example in “Borders of a Corridor” and
“Sound Swing”, the speed of sound motion (rhythm) may
be coupled to the speed of users motion in the space,
increasing or decreasing according to the different users,
17
or a single user behavior in the space.
Among Leitner’s experiments the one that is closer to the
idea of interactive sound is “Spiraling Space” (1972). In a
hypothetical tube-like corridor, successive rings of speakers
render the space. Sound moves along the ring in a 4-second
revolution and all rings move at the same time. When
somebody stands still under a ring, he can feel the circling
sound around, as he moves forward, the sound becomes
to describe a spiraling effect, mapping walking speed to
the rate of the spiral wavelength.
Fig.10 Model for Spiraling Space. Leitner 1972.
In this work, there is a direct correspondence between the
users movement and the unfolding of the spiral in a way it
can be considered “reactive” because users perceive
according to their own individual displacement in space,
with the possibility of having several people experiencing
their own spiral at the same time.
2.3.3. Spaces with aural directional messages
Fig.11 Sound Gate, Leitner 1971.
“Sound Gate” (1971), is a vertical square structure, which
holds 17 speakers along his perimeter and is the building
block for a series of them, assembled in a way to create a
passageway of sound. The structure is 4 X 4 meters and
has two extensions on the floor. Leitner created two different
programs for it; in one the sound moves from one
loudspeaker to the following next loudspeaker along the
structure, giving a directional message to the gate (fig.11).
In the other program (non-directional) criss-crossing motions
of sound accentuates the gate’s vertical plane, creating a
18
Fig.12 Sound Gate, Leitner 1971.
Non-directional program.
Fig. 13 Sketch for Implementation
of Sound Gates in a public space,
Leitner, 1971.
dynamic sound layer, which is crossed by the body in motion
in any given direction, perceiving the dimension of the
gates physically (fig.12) Leitner made a sketch of how he
sees a sequence of gates implemented in a public space
(passageway of hallway) A directional message in this case
will have a polarized meaning; one encouraging the body
movement if walks in the same direction, and the other
moving against your direction which may be analogized to
the experience of walking against the “wind direction”,
which is not wrong or right, but simply adds this sense of
impulse.
2.3.4. Dislocated Perception
“Ascending and descending lines of sound are
superimposed on a slope. Their angles of inclination
enhance or negate each other. The downward movement
of a person is emphasized by a descending line of sound.
Ascending and descending lines of sound biopsychologically influence the descending person’s reading
of the slope’s actual angle of inclination.” (Leitner, 47)
This kind of distortion of spatial sensation happens in the
match or mismatch between sound movement and spatial
forms, as we may find in “Sound Slopes” (1972). This
phenomena can be used to demonstrate differences
between visual and aural perception, perhaps in the design
Fig. 14 Sound Slopes, Leitner, 1972.
19
of a space that can be perceived completely different aurally.
As an example, I imagine a small cubicle in where sounds
reverberate as if were in a big cathedral. Any given sound
in that space will actually be referring to another space.
The encounter of such contradictory elements allow for the
dislocation of perception.
2.3.5. Sound as Inner Space
One of Leitner’s most interesting observations about the
influence of sound in the body is the fact that we absorb
sound with the entire body and not merely by the ears.
“Lying within sound “ are experiments in which sound is
applied directly or close to the body in a static position.
“In sound objects one listens to the sound, one feels it
wherever the vibrations enter the body and one retraces
the movements mentally. Body position and spatial
movements of sound must relate to each other. Lying down
implies a particular readiness to perceive sound motions
around, along and through the body.” (Leitner, 58.)
Fig. 15 Sketches for Platform for Horizontal
Motions. Leitner, 1976.
Fig. 16 Sketches for Platform for Vertical
Motions. Leitner, 1976.
Among these experiments there is “Platform for horizontal
motions” (1976), which consists of a bench with two
loudspeakers underneath. Staccato-like electronic beats
move in the upper part of the body producing a stretching
effect. The same bench is used but with a set of
loudspeakers positioned underneath the chest and above
in two layers in “Platform for vertical motions” (1976). A
soft, medium fast electronic beat moves between them
provoking the persons breathing to match the sound
movement. (Fig.16)
In these experiments, the motion of sound through the
body can influence biological functions such as breathing.
After a time under the influence of rhythmical patterns,
20
Fig.17 David Rokeby performing with his
Very Nervous System.
breathing falls in synch with them, which can be thought
as an expansion of the body in space through sound, similar
as we may found in the ancestral practice of mantra praying,
a form of meditation where sound leads the ritual as its
“score”.
Sound tends to create a feeling of awareness of an inner
space, which goes beyond the body space. Leitner was
aware of this as he recalls in an interview during the eighties:
“Hearing experiences not only enable us to have a special
spatial experience but also an internal space” (Bernhard
Leitner interviewed by Wolfgang Pehnt during Documenta
1984, Cologne.)
I believe that sound has a strong influence at the spiritual
level, independent of individual religious believes or ritual
practices. David Rokeby, multimedia artist creator of Very
Nervous System, a video tracking software and performance
instrument, comments about this issue in the context of
his interactive sound installation “VNS” (1986-1991).
“The diffuse, parallel nature of the sound interaction and
the intensity of the feedback loop can produce a state that
is almost shamanistic. The self expands, and loses itself to
fill the space and by implication, the world.” (Rokeby)
2.4 Sound as Space
“Like modernism itself the phonograph represented a new
day in aurality through its ability to return virtually any sound
back again and again into the sensorium and into the historical
register.” (Kahn, 5)
As different musical instruments produce different sounds,
spaces have also their own sounds, based on their physical
characteristics. The reproduction of sound by
electromagnetical means allow us to recreate a given
moment in time and space in another time and another
21
space, which suggests the possibility of remote presence.
Recording or transmitting live sounds from one environment
to another, translates a spatial situation, carrying the
representation of one space to another. In this sense,
artificially recreating a space can be seen as the creation
of a virtual space.
An elegant example of this can be found in the work of the
American Sound Artist Alvin Lucier. In his piece “I am Sitting
in a Room” (1970), Lucier reads aloud the following text,
recording his voice in a tape recorder:
“I am sitting in a Room different to the one you are in now.
I am recording the sound of my speaking voice and I am going to play
it back into the room again until the resonant frequencies of the room
reinforce themselves so that any semblance of my speech, with perhaps
the exception of rhythm, is destroyed.
What you will hear, then, are the natural resonant frequencies of the
room articulated by speech.
I regard this activity not so much as a demonstration of a physical fact,
but more as a way to smooth out any irregularities my speech might
Fig.18 Lucier performing
“I am Sitting in a Room”
have.”
The recording is played back into the room through a
loudspeaker, while simultaneously re-recorded using a
microphone. The new recording is played back and rerecorded in a successive series of generations. In each new
recording the natural resonance of the room is captured
and reinserted, amplifying it until we can no longer
distinguish the original text, only retaining the pure
resonance of the room.
By listening the recording of this piece, one can “see” the
space, not in form but as function. “I am sitting in a room”
uses space as a sonic instrument that speaks about itself
as space.
One key aspect about Lucier’s piece is the fact that the
22
room in which he is “… is different from the one you are”
as he reads. I think the translocation of spaces is possible
not only because of the technologies of reproduction, but
because our experiences of space can be recalled by our
imagination.
Aural reproduction suffers of transformations that would
make sounds mere representations of other ones. As an
example, a photographic image can be easily understood
as a representation, because the changes in scale,
perspective, lighting, depth, etc. In an approximation to
achieve realism we can find the example of the trompe
l'oeil in painting, which creates an illusion of reality by
coupling dimension and spatiality.
A particular work of the Canadian artist Janet Cardiff, works
out this conception with sound; “Forty Part Motet” is a
sound installation that replaces the voices of forty choir
voices by forty loudspeakers. This work uses a piece of
secular music created in the Sixteenth century by the English
composer Thomas Tallis. Visitors can perceive the choral
music as a whole while standing in the middle of the space;
as they approach to individual speakers they can distinguish
individual voices, being able to walk through and climb
between the different harmonies and layers of sound. This
piece translates the physical inaccessible inner space of
choral music to an accessible other space, as Cardiff
explains: “Even in a live concert the audience is separated
from the individual voices. Only the performers are able to
hear the person standing next to them singing in a different
harmony. I wanted to be able to 'climb inside' the music
connecting with the separate voices. I am also interested
in how the audience may choose a path through this physical
yet virtual space.”
Fig.19 Installation of Forty Part Motet in
England, by Janet Cardiff.
The translocation of sounds can also contradict or neglect
the space which contains it, as in the work of Bill Fontana.
23
Fontana transmits sounds from one location to another
creating a re-presentation of a distant space, which in the
change of context suffers of an intriguing permutation,
which evokes another physical space. An example of this
is "Sound Island" a sound intervention made by Fontana
at Arc de Triomphe, Paris in commemoration of the 50th
anniversary of the D-day. In this work loudspeakers were
placed in the four façades of the monument, which
transmitted live sounds from the Normandy Coast,
transforming the visual and aural experience of the constant
traffic around the Arc. The harmonic complexity of the
natural sounds of the ocean and crushing waves has the
psycho-acoustic ability to mask other sounds, directing our
attention to them over the overwhelming noise of traffic.
It is interesting to notice how an intervention like this, with
no visual or physical alteration, has the power to completely
transform the notion of a space.
Fig.20 Sound Island, Sound intervention
at the Arc d’Triumph by Bill Fontana. Paris,
1994.
2.5 Patterns in Noise
“As a visually oriented culture our essential responses to the
everyday world are semantic. Everyday sounds are regarded as
not having semantic significance (noise). Noise pollution (with
the exception of sounds that are dangerously loud: close
proximity to a jet aircraft or heavy machinery) can be explained
as a semantic problem. Because sounds must be semanticized
in order to be meaningful, our main aural concerns as a culture
have been language and music. Sounds in themselves have
not been regarded as having communicative effectiveness.”
(Sonic Ecology, Fontana.)
In large urban cityscapes, such as New York, our acoustic
environment is highly heterogeneous and saturated by
hundreds of activities happening at the same time. In the
impossibility of paying attention to every single stimulus,
we filter out what we consider as meaningless acoustic
24
Fig.21 Cover Diagram for “The World
Soundcape Project’s Handbook for Acoustic
Ecology” R. Murray Schafer 1978.
Fig.22 John Cage and one
of his “Prepared Pianos”
information falling in the category of noises. In this context,
the concept of noise becomes highly relative, because we
determine what sounds are significant in a particular
moment. The same sound may be considered meaningful
or meaningless, depending on the focus of our attention.
In 1970, the Canadian composer R. Murray Schafer started
his “World Soundscape Project” in which he collects and
analyses the sounds of a particular environment to examine
interrelations in the biosystem and the techno spheres;
identifying noise pollution sources, and sounds that may
be preserved or encouraged.
In his research he embraces the idea of a sound ecology,
which have influenced many other sound artists and
composers such as Bill Fontana and the Australian
radiophonic artist Paul Carter who states: “[…] Degraded
environments will be sparsely orchestrated and badly tuned,
while relatively undisturbed habitats will be harmonically
subtler and rhythmically more various […]” (Madsen, 2).
For Schafer, the degradation of our soundscapes is the loss
of “resonant wilderness”, the loss of the sacred. His essay
“Radical Radio” suggests the idea of injecting fresh sounds
to the heart of the cities from remote and wild locations.
Schafer’s ecology is practiced in an almost scientific way,
even though the problem is based on an aesthetic
parameter, the one of harmonic degradation.
John Cage was one of the first musical composers who
neglected the traditional western affinity for harmony and
tonality as a means of structuring compositions. One of his
most famous pieces, entitled: 4’33’’ is a score containing
the equivalent time of 4 minutes and 33 seconds of just
silence. Cage thinks that absolute silence doesn’t exist,
which may be inspired by his experience in an anechoic
chamber in 1952, when instead of the expected silence, he
heard two constant sounds; his nervous and circulatory
systems at work. Cage started seeking sound in every single
25
object he found, which can be reflected in his “prepared
pianos”. From them it can be said that Cage wanted to
incorporate musical meaning to commonly unsemanticized
sounds, which in Bill Fontana’s words, is the ultimate
contribution of Sound Art.
“The semantic ambiguity of sound will change when society
develops a capacity to perceive patterns or qualities that
are recognizable as part of a context of meaning, such as
the sound vocabularies of contemporary music and acoustic
art […] The task of acoustic art and acoustic design is to
fundamentally challenge all of the old historical definitions
of noise and the resulting preconceptions that most people
have about the sounds they live with”. (Fontana, 3)
For Cage and Fontana the problem of noise may be
understood as a lack of listening awareness; in the case of
Fontana by creating works that uses the natural environment
as living source of musical information. He assumes that
“at any given moment there will be something meaningful
to hear” and “that music - in the sense of meaningful sound
patterns - is a natural process that is going on constantly.”
A fantastic extrapolation of this idea can be seen in the
movie “Dancing in the Dark” of the Danish filmmaker Lars
von Trier. This movie tells the story of a nearly blind worker
of an industrial factory (Bjork), who discovers in the
machinery sounds “hidden” music that she dances and
sings to. This music is in her mind, but we can hear those
musical compositions made from machine sounds as she
thinks it.
Our memories not only store images, words or smells; they
can recall sounds and music as well. In the same way we
mentally envision images we have never seen before, we
can mentally listen to unheard music.
26
Chapter 3. Methodology
3.1. Introduction
The development of the present Thesis has been supported
by a series of experiments and small projects in physical
interaction with sound. In my intend to incorporate physical
space as an interface, I started from simple electronic
components and sensors, using a chip as the sound source.
For the following stages I switched to prerecorded and live
video as input and MAX/MSP as the programming
environment. The present Chapter expose five design
experiments through documentation on the design and
programming fronts, as design sketches, flowcharts and
mapping diagrams, explaining functionality, interaction
goals and expectations, that finally are summarized and
evaluated.
3.2. Preliminary Experiments
During the Fall of 2002 I was involved in a collaboration
studio called Musical Interfaces which first was conceived
as a fundamental electronics workshop for building sound
interfaces. In this studio I had the opportunity to experiment,
design and build three projects, which helped me
understand the basic issues of physically interacting with
sound.
3.2.1. Wearable Synth (Fall 2002)
In this project I intended to map my own body movements
to basic sound parameters in an Invisible Interface, which
allows the body to move freely without the constraints of
being tied to a device by cables or any other gadget. My
goal was to create a synesthetic relationship between body
language and sound, testing different mappings of
27
Speaker
Amplifier
555 Timer Chip
555 Timer Chip
Fig.23 Configuration of FM Synthesis
in Wearable Synth.
Bend Sensor
Bend Sensor
Bend Sensor
Timer Chip
setup
Fig.24 Sensors positions and chip
location in Wearable Synth.
movements to sound and observing the expressive ranges
I obtain by using the most minimal interface.
To accomplish this I used a 555- timer chip, a small electronic
chip that produces electrical impulses, which can be easily
controlled and modified by varying resistance. In my setup
I implemented two of these chips to produce FM synthesis,
where one chip acts as the modulator of the other. (Fig.X)
The chip setup was attached to the belt in a soft case that
contains the speaker as well. Bend sensors attached to the
elbows, knees or shoulders were connected to the chips,
allowing continuous change in sound by flexing the main
body articulations. In this project I spent a big portion of
the timeframe finding an interest sound in the combination
of fixed resistors and capacitors used. The resulting sound
was a constantly warbling tone.
Although the sound was rich and interesting, the expressive
range was limited and required a great amount of practice
to learn how to control the output in an interesting way.
Additionally, the continuous sound was a problem; I wanted
to use silence as another controllable variable, which I later
implemented using a photocell attached to the inner part
of the leg, so when the legs were close to each other, the
photocell was covered, silencing the synth.
Sometimes the body would adopt awkward positions to
accomplish a particular sound, making the performance
humorous at some points. I realized that the body was
trying to adapt to the instrument requirements to produce
interesting sonorities, rather than using the natural body
inflections to produce a synesthetic sound parallel.
3.2.2. Trigger Space (Spring 2003)
An experiment similar to Wearable Synth was done later in
time, using video input instead of sensors attached to the
body.
The video-input is processed in the computer using Cyclops,
28
and MAX/MSP as the computer vision and sound processing
software respectively. Cyclops allows for the specification
of a grid of custom rows and columns over the image,
outputting integer numbers according to the occlusion of
any of the cells in the grid.
I assigned a different sound to each cell. The visual field of
the camera is converted in a kind of "minefield", in that
way the body can trigger those sounds by just crossing a
cell.
Fig.25 Each cell represent a sampled sound
Fig.26 Preliminar sketch for Sonic Arena.
In this case, there was no restriction in the kind and variety
of sound I can use, but due to the way in which I specified
the sounds, that is "cell-specific", the space becomes fixed
and specific positions in space always trigger the same
sound. This constrain made the interaction a “memory”
game, because once you hear a sound at a specific position
of hands, head or legs, you can repeat the movement or
sequence of them to repeat a series of sound events again.
I plan to develop this prototype further for a live dancesound performance, maybe using two video inputs to map
horizontal and vertical motions.
3.2.3. Sonic Arena (Fall 2002)
Users are confronted with a frame containing fine sand rose
to the level of the hand. As users move the sand, trails are
left behind; this action is captured by a video camera placed
inside the pedestal, pointing to the bottom of the frame,
which is made of transparent acrylic. The trails create
difference in the depth of sand, allowing the light to pass
in different intensities and forms. The sand movements are
mapped to a set of prerecorded samples of abstract sounds,
which fade in and out seamlessly, giving an aural feedback
in real-time to the patterns created in the sand.
This project can be seen as a scaled-down version for my
thesis installation because its input and output are the
same but in an inverted situation. The patterns in the sand
29
Fig.27 Sonic Arena in use.
Fig.28 Video input from the bottom of
the frame as received in the computer
Fig.29 Boundary dimensions of
light trails in the sand box.
being observed from underneath will be finally extrapolated
to people movements being observed from above.
The development of this project went through a series of
stages, because I was learning MAX at the same time, so
it had several instances of programming before getting to
the final interface and its inputs and outputs. This process
started with a mock up of the video input with a basic
“paint” object, which is a simple canvas window in MAX
that returns x and y values for the current mouse position.
The first mapping I tried was using these values to control
a set of cycle~ objects, (MSP built-in tone generator) which
was implemented as LFO (low frequency oscillator). The
parameters being controlled were the LFO rate and LFO
depth, perceived aurally as an oscillating pitch in variable
intensities.
In the following stage I switched to video as the input, using
the boundary dimensions of the light blurbs as the
meaningful values to manipulate sound, so in this case the
size of the trails was the modifier parameter rather than
the form or position. I built a patch using the Jitter object
jit.findbounds to get these values. This object analyses the
video input returning numerical coordinates of the size of
a particular color blurb. Here I mapped these values to a
tone generator. The values cross through a series of
mathematical transformations to be received by a series
of cycle~ and line~ functions. In this case, the
correspondence between sand trails and the sound output
was less obvious, but the sound always had the same
timbre with variations only in pitch and depth similar to
Aural Paint. I realized that would be more interesting to
allow users to cross through different sounds along the
sand field.
A third stage was built using Cyclops a computer vision
extra designed by Erik Singer, which has three modes of
image analysis. The one I found interesting to explore is
30
called “difference” which measures differences in the image,
frame by frame in a matrix grid which can be specified in
columns and rows that returns integer values per each grid
unit for each captured frame.
I specified a 8 X 8 grid and connected each output to a
different sample player object (sfplay~), which reads stored
samples in the hard drive. These sound units were assigned
to a specific grid unit, acting as triggers. The samples where
edited so that they faded in and out in relatively long
phases, and they crossfade seamlessly when the different
units were exited. In this case, the approach chosen was
completely different, because I was using sampled sounds
as a way to convey abstract sounds which were also
synthetic in origin, but hard to specify in MAX using sound
synthesis techniques. The result was clearly richer in terms
of sonority enhancing the experience and making it
interesting for a longer time.
In another version I used another set of sampled sounds
that created a semantic relation between the sand in the
interface and the sand by the sea. The sounds were
recordings of the ocean: the shore water cycle, crushing
waves, birds and wind.
Fig.30 Sonic Arena in use.
The use of samples, compared to synthesized sounds,
represent its advantages and disadvantages. The synthesis
of sounds allows an infinite degree of form manipulation
because it works in the specification of sine waves, which
are the basis of aural perception. At the other hand, sampled
sounds opens the possibility of manipulation at the semantic
31
level (sound narrative) because one can use sounds that
convey linguistic messages such as speech or natural
phenomena like weather, birds, or music. In Sonic Arena I
experimented with sampled sounds in both ways, in an
abstract and figurative way. In this particular piece, the
figurative approach was easily understood and made the
experience more successful from the users point of view,
mostly because the interface reinforced the ocean metaphor,
however I think is still possible to play in the boundaries
of both conceptions, a fluctuation between the recognizable
and the indecipherable, the morphing between noise and
meaning.
3.3. Thesis Prototypes
3.3.1. System Overview
During the development of this project I had to research,
find and learn a series of system software that would allow
me to control sound based on a video input. After a short
period of experimentation with simple electronics I started
using MAX/MSP, which is a visual programming language
developed by Miller Puckette. MAX was originally developed
to manipulate MIDI data; with the addition of MSP, (MAX
Signal Processing), it was possible to specify and manipulate
sinewaves and samples. After trying the software for a
couple of months I felt comfortable enough to start
developing my own tools. Especially important for me was
the possibility to have a video input, through the addition
of computer vision extras.
During the prototyping stages I have worked with three
different computer vision extras. The first one was Cyclops,
designed by Erik Singer. Secondly I tried Jitter, and finally
David Rokeby’s SoftVNS 2.1, which I found to be the best
video tracking – image analysis external for MAX, because
it can combine different analysis modes using a single
32
input.
All the prototypes for this Thesis share the same input –
output system, which is based on single video Input,
MAX/MSP programming, and loudspeaker output, the only
variation is in the Computer Vision externals used in each
one of them.
Fig.31 Single video Input,
MAX/MSP programming, and
loudspeaker output.
3.3.2. Traffic Report (Spring 2003)
In the process of learning the different objects for sound
manipulation in MAX, I came across a playback speed
control of sampled sounds. I built a patch that used a loop
of about 40 seconds long, which I previously edited in
Protools. The same day I went to the Brooklyn Library and
while waiting to cross the street, I saw how the traffic of
vehicles flows in rhythmical cycles. Using a video camera
I recorded a full cycle from no traffic to full circulation to
no traffic again, which later I used as the input for a program
I wrote in MAX.
Fig.32 The surroundings of the Brooklyn Library
33
Fig.33 The pattern of the traffic cycle.
From the video image I analyzed the amount of overall
movement occurring in the image frame by frame. I
connected this value to the speed of the sampled sound;
as a result there was a proportional increase in pitch and
speed of sound according to the traffic cycle. I thought that
this translation can be implemented as a simple feedback
a public building gives to its surrounding environment.
The avenue in a side of the Library has four lanes and many
people wait for several minutes to access the Children’s
section entrance, which may give them time enough to
make the association of traffic and sound if loudspeakers
are mounted in the lateral façade.
3.3.3. Spatial Scrub (Spring 2003)
For this prototype I chose the space of a corridor, in which
there’s only two ways to go, back and forth. Using this
space situation, I created a short segment of sound that
contained a large amount of sound events; chords,
percussion, segments of voice, one after another. This
soundtrack loops in very short cycles, about 100
milliseconds. A video input was used to “track” position in
that space. That position was connected to the main audio
cursor, the one indicating the current position in the sample,
in this way, when people walk through the corridor, they
can “scrub” through the sample as if they were a scanner
or playback head.
Fig.34 The body as the playback head of a sound.
The speed of motion determined the size of the loops in
milliseconds, so the faster the speed, smaller portions were
looped, creating a finer “granularity” in sound. The repetition
of the loops, create a rhythmical pattern of sound, which
degree of musical harmony and beauty depends on the
type of sounds in the soundtrack. For this experiment I
used subtle chords, basses and hi-hats that are continuous
tones, without abrupt cuts, so when looped, they seem to
34
interweave in a rhythmical progression.
As experience, the connection between the size of the
corridor and the size of the sound, is not immediately clear,
but as soon as users stop to listen, the sound keeps looping
in the same position, so when resuming motion, the
connection becomes evident.
Fig.35 The interior hall of the Brooklyn Library
Fig.36 The increases in the delay are
implemented in a logarithmic curve, making the
feedback more “sensible” to small changes in
the environment, and more “stable” at the peak.
3.3.4. Accumulation
In this prototype I wanted to experiment with a feedback
whose changes can only be perceived in a long timeframe.
I placed a video camera observing the Brooklyn Library
main hall from above, (almost plain vertical), so I could see
a large space and the traffic of people happening in it.
The sound used was the ambient sound of the same space.
The video image was analyzed in terms of amount of
movement happening each 15 frames. The number obtained
from that reading was used to control a “delay feedback”
in the sound. A delay feedback in sound is similar to a
“video feedback” as we may find in early video works of
Nam Jun Paik, where the camera points to a monitor
displaying the same image being captured, creating a
spiraled image because of the closed circuit between input
and output. In sound such effect can be accomplished by
reinserting a portion of the input to the output, with a slight
delay. The effect is a constant tone emerging from the
original sound, which changes also according to the input.
The amount of movement in the space was connected to
the amount of feedback in the sound, reflecting the physical
situation. In this way, we can obtain an aural “portrait” of
the space in a given moment. If we compare snapshots
along a day, we would see significant changes in the sound.
This experiment used an extremely long “exposure”, so the
changes of sound may not be perceived instantly, but along
the course of a day.
35
3.3.5. Panning Sound
This prototype was an experiment on tracking positions of
multiple users in space, and also the amount of movement
within zones of the space. Tracking people’s positions
requires certain conditions of light in the space in order to
differentiate people from the background. Also as the
lighting conditions may change along the day, there is the
need for periodic revision of the thresholds that determine
people’s presence. This task is hard, so I spent a great
amount of time figuring out a way to dynamically take
periodic measures of lighting conditions to readjust the
thresholds. There is also the problem of occlusion. When
two people are close to each other, it is almost impossible
for the computer program to differentiate them as two
individuals. Nevertheless, the position and distribution of
users in space is valuable information that can be used in
a variety of ways, for example the panning of sounds
between output channels. As I am currently working with
two output channels, Left and Right, I divided the video
input field in 16 vertical zones, each one reporting movement
in its own space, I assigned to each one of these zones a
numerical value from 0 t o16, each extreme representing
an output channel. The numbers in between represent
intermediate values in the amplitude of each channel. In
this way, if one person is present in the space, its location
pans the sound to the opposite side of which he/she is. As
the user moves around, sound moves accordingly.
Fig.37 Custom tracking interface
36
In the case of a greater number of users, their positions
are averaged for determine the movement of sound. The
movement of sound in space would give a dynamic response
to peoples displacement within the space, creating an
attraction–repulsion relationship between sound and
listeners.
I decided to move sounds in the opposite direction because
this makes the movement more evident, because mapping
to the same position can be easily confused with sounds
being louder by being closer to a channel side, which
happens naturally in the proximity of a sound source.
37
3.4. Summary of Implementations
For this summary I considered the five most relevant
experiments to my final Thesis project, which are: Trigger
Space, Spatial Scrub, Traffic Report, Panning and
Accumulation.
These Prototypes represent a wide range in physical size,
which has helped me understand and consider particular
details relevant for each particular scale. In each one of
them I am using a different mode of Image Analysis, what
was suggested in part by the physical situation itself and
in part for the kind of mapping to sound I wanted to try.
While doing this experiments, I could see how each one of
them had a different timeframe requirement, also due to
the nature of the event, and the type of mapping, that will
define the time of the experience.
Fig.38 Comparison Table
Physical Scale
Body
Corridor
Hall
Building Facade
Large Interior Plaza
Horizontal Position
x, y position /
Movement Amount
Movement
Amount
Movement
Accumulation
Pointer to file time
Panning / Reverb
Speed of soundtrack
Feedback Delay
Amount
Seconds
Seconds / Minutes
Minutes
Hours
Image Analysis
Zones
Mapping to Sound
Trigger Samples
Required Temporality
Milliseconds
38
3.5. Thesis Installation
3.5.1 Revealing Patterns in Noise.
Aural Limbo is a physical space, which act as host for and
aural space composed of everyday sounds of the cityscape.
These sounds represent the homogeneity of our
environment, the disregarded result of hundreds of activities
happening at the same time, which cognitively doesn't
have particular significance, and usually are filtered as
meaningless acoustic information (noise).
A ring printed on the floor demarcates the boundaries of
the sensible space. Within this zone, the presence of the
body, its location and amount of movement unchains a
series of dynamic modifications in non-musical sounds,
revealing hidden rhythmical patterns emanating from them.
These patterns are revealed by the digital manipulation of
sound parameters such as: Speed, Reverb, Pan, Delay
Feedback, and Convolve. These parameters are controlled
by a combination of data input from the physical space,
such as presence in the space, position in the space and
difference between frames (motion).
The actual space remains the same while sound reshapes
a virtual space, one that can change its form, texture and
size. The discovery of this other space happens in a similar
way as we may see in digital imagery where things
seamlessly morph from one form to another, for example
an abstract object that resolves in a human head. The
surprise of it lies in the change of meaning that the object
suffers along the way. The moment of the transformation
is far more suggesting than the final result.
Fig.39 Installation Squeme
39
3.5.2. The Context of the Public Space
Fig.40 Access Lobby at 2W 13th Street
Fig.41 Installation Squeme
Although computers seem to be anywhere present in our
lives, people are not expecting to interact with one in the
middle of the street. What interests me about using public
spaces is that creates opportunity for spontaneous
interaction. This installation takes the form of an
architectonic intervention; in the sense it becomes part of
the building, rather that an Artwork exhibited in a building.
The purpose of this intervention is to engage passersby in
a habitable sonic instrument.
With this goal in mind I searched for possible spaces within
the University facilities, looking for an interior public space
that had pedestrian traffic throughout the day.
After finding and requesting two possible spaces, finally I
got the 2W 13th street access lobby at Parsons. This lobby
is a transitory public space. People are distributed from
here to the gallery, offices and elevators. Its physical
structure as well as its inhabitance and traffic patterns,
makes it a good place to host the interactive experience.
The project is conceived as a non-invasive installation
because it doesn't modify or alter the space in any physical
way. It consists of a demarked circle in the floor, which will
be the interaction zone. (Camera field of view). In the
surrounding space a set of four speakers will output the
sound.
3.5.3. The users experience
The following are desiderata for my installation, in terms
of what I expect to be the users experience from the
aesthetic and functional points of view.
0. The space is empty, people start accessing the building
for class. On their way to the elevator they hear sounds of
the city and other ambient sounds coming from the space.
40
A ring printed in the floor demarcates the center of the
sound projection.
1. While waiting in the elevator line, some of them come
back to the space to see what is this about. Others may
want to try in their way out.
2. When one person enters the space, the sound pans in
the opposite direction of his/her position, as this person
moves around the sound moves from speaker to speaker,
creating an interplay between listener and sound.
3. If more people enter the space, the panning is a result
of the average position in the space. If people are equally
distributed, the pan will stay in the middle. (Equal
amplification in Left and Right Channels).
4. As users move within the area, the sounds are morphed
by adding reverberance, or varying speed. At the peak of
movement the original sounds are not longer recognizable
as such, turning into reverberating patterns of ever changing
rhythms and echoes, behaving as musical elements.
5. When the space is emptied, a new soundtrack is triggered.
There are 4 different soundtracks with different
transformations. This sounds are called randomly so two
users may experience different sounds and its respective
transformations.
41
Chapter 4. Discussion and Analysis
4.1. The Music of Sound
Nowadays, everybody is exposed to the notion of digitally
manipulated images and sounds; they can notice them in
surround cinema, radio commercials, TV and pop music.
As computers penetrated people’s homes with the
availability of digital tools, they have been given the
possibility to store, display and manipulate images and
sound. This is especially common with digital pictures and
movies, where people have started to edit their own home
made movies and pictures. In some cases they also have
become aware of manipulation beyond the basic timebased alterations, introducing changes such as color
correction or the addition of special effects. Why has the
manipulation of images become so popular and accessible,
while sound has not? I think this is not only due to our
nature as culture, which is highly visually oriented but also
because we associate sound with music, with all the
traditions and suppositions this impose.
In the past, interaction with sound was confined to the use
of very specific interfaces, and the developing of particular
skills for each instrument; therefore sound was confined
as a medium for musicians. For the rest of us, interfaces
such as the Violin, are scary, because the expectation is to
produce “music”. Sound have continued to be out of reach
for common people, maybe because is still seen as a skillful
medium, or simply because the development of sonic
interfaces has not addressed this audience. Moreover, the
experience of sound doesn’t even require to involve music
as we may think in traditional music, which is in the
aesthetical order of harmony and virtuosity. As Bill Fontana
has expressed: “ [...] music - in the sense of meaningful
sound patterns - is a natural process that is going on
42
constantly.” The discovery of such patterns is something
that happens in our minds. In this work I assume that that
state of mind can be encouraged through an interface that
reveals such patterns.
The work presented in this Thesis seeks to give its users
the experience of a literal “sound embodiment”, by attaching
sounds to their physical action, allowing them to modify
sounds, revealing significant patterns emanating from them.
In this sense, this work should not be considered as a
precision tool for the manipulation of sound, neither an
instrument for musical performance, but as a physical
experience, which ultimate goal is making them aware of
the act of listening itself. Is in this sense that this work
should be discussed and evaluated.
4.2. Challenges and Pitfalls
In the Chapter 3, Methodology, I presented five different
experiments, each one of them having a different spatial
situation and a different mapping to sound. They, as a set
of experiments, can be seen as a prototype for the actual
installation, because in each one of them I tested, in an
isolated way, a particular aspect of interaction with sound
in space.
It is difficult to create equivalent comparisons between
them. Also, as they required the use of a particular spatial
situation, they were set up for just short periods of time,
having myself as the only experiencer. I think is extremely
important to have test users, I am still looking forward to
get to that point, which would allow me to observe from
outside and introduce changes on the fly. Nonetheless, as
exposed before, I still can think of my experiments in terms
of experience, and tell in which degree they succeed in
creating a connection between a physical situation and
43
sound in an interesting way.
Although in each one of them such connection was present
in one or another way, I think “Spatial Scrub” did it better
in terms of surprise, because the connection was not
immediately evident, neither imperceptible, creating a
moment of uncertainty, which is resolved after a few seconds
of use. In that sense, “Triggers” is too immediately
understood, and even though playful, the experience is
short, depending on how playful is your mood at the
moment. In the other extreme is “accumulation” in which
the required time to get the connection is too long. “Traffic”
is the only one that doesn’t use body, but act in a similar
way as accumulation in the sense it can be a mirror of what
is happening in Space, but in a shorter period of time. In
a way, these two experiments succeed more as
“sonifications” of a situation in space rather than as
interactive experiences.
At this point none of them individually satisfy my
expectations and goals, but they have been extremely
useful as ways to understand the infinite details that have
to be considered and the limitations that need to be sorted.
After conducting those experiments I was able to identify
the required “temporality” for each situation, and see its
relation to the chosen mapping and spatial situation,
realizing that the ideal for the installation would be to have
two different mappings, with different temporalities each.
For example, it is important that users notice an immediate
response to their actions, but also that they discover further
influence in sound as they explore the space.
At this point I am working with a set of soundtracks, each
one of them will be introduced in the piece as users
incorporate, keeping the count of number of users. Their
positions will determine pan distribution, and their amount
of movement the amount of delay feedback.
It is my hope that this combinatory system of mappings
44
will increment the possibilities of exploration, extending
the experience in time and depth.
Chapter 5. Conclusion
5.1. Conclusions
The work of this Thesis starts in the questioning of the
cultural approaches we have to sound, in an attempt to
expand its notion as a primary element of perception and
especially as a physical phenomenon, using the example
of contemporary artworks that have approached sound
from spatial, cognitive, and semantic levels, finding special
influence in the work of Bill Fontana and his writing on
Sound Ecology. Based on this practical and theoretical
background, this Thesis has attempted to incorporate the
vocabulary of Sound Art into the realm of computer
multimedia, making use of its capabilities to display and
manipulate sounds in real time. This proposal is made
concrete by introducing and documenting a series of
experiments in physical interaction with sound. In such
experiments, aesthetic and philosophical goals have been
considered to give form to an interactive sound installation
that uses the context of the public space as an example of
the use of computer out of the usual context of the desktop
setup.
5.2. Future Directions
The final outcome of this work represents just a scratch in
the surface of the idea of physical interaction with sound.
45
One single piece gives the opportunity to explore one or
two aspects of the inquiry at a time, and the technical
constraints make necessary to reframe the ideas to the
actual possibilities. Amongst the unexplored avenues of
this work, I can mention the implementation of sound
spatialization, which gives the possibility of moving sounds
in space, through the use of grids of loudspeakers, through
which sound is distributed by special hardware controlled
by the computer. Also from the input side, the use of the
available computer vision software imposes its own limits.
There is an increasing number of artists developing their
own computer vision programs for their specific needs. In
my case I am using a general video-tracking tool, which
was designed for a wide range of purposes, and whose
stability and robustness is extremely delicate.
Another aspect I evaluated to implement in my piece was
the use of live sound input, through the use of microphones
located in different locations in the city, but that requires
broadcasting technology that is not easily available. I also
thought to use microphones listening to the space of the
installation itself, but in this case there’s the need to sort
the problem of the feedback happening in the proximity of
sound input and output.
The choice of the space for the installation has been made
based on the available facilities of the school. For sure, the
city is plenty of interesting public spaces that would give
different contexts to the piece and levels of spontaneity to
the interaction.
I also envision the use of this kind of interface for live
performance of simultaneous dance improvisation and
sound composition, in which sounds stay in a suspended
animation state until the performer unchains them by the
use of space.
46
Chapter 6. Bibliography
Books
Ando, Yoichi. Architectural Acoustics: Blending Sound
Sources, Sound Fields, and Listeners. AIP
Press, 1998.
Barnes, Ralph M. Motion and Time Study. New York, NY.
John Wiley & Sons, Inc. 1950.
Calvino, Italo. Six Memos for the Next Millennium. New
York, NY. Vintage Books, 1993.
Lévi-Strauss, Claude. The Raw and The Cooked. New
York, NY. Harper & Row, 1964.
Heim, Michael . The Metaphysics of Virtual Reality. Oxford
University Press, 1993.
Kaemmer, John E. Music in Human Life: Anthropological
perspectives in music. Austin, Univerity of Texas
Press, 1993.
Kahn, Douglas. Noise, Water, Meat: A History of Sound in
the Arts. Cambridge, MA. MIT Press, 1999.
Kahn, Douglas. and Gregory Whitehead. Wireless
Imagination: Sound, Radio, and the Avant-Garde.
Cambridge, MA. MIT Press, 1992.
Kobin, Minard . Sound Installation Art. Call number #
ML1380. M55. 1996.
47
Leitner, Bernhard . Sound:Space. New York University Press.
New York, 1978.
McLuhan, Marshall. Understanding Media: The extensions
of man. Cambridge, MA. MIT Press, 1994.
Moore, F. Richard . Elements of Computer Music. Prentice
Hall, Engelwood Clifs, New Jersey, 1990.
Rush,Michael . New Media in Late 20th-Century Art. New
York, Thames & Hudson Inc, 1999.
Journals, Magazines, Electronic Sources and Other
Theses.
Davies, Shaun. Sound in Space: Adventures in Australian
Sound Art. February, 23. 2003
<http://www.autonomus.org/soundsite/csa/eis2content/e
ssays/p34_lost.html>
Eno, Brian. Interview by John Alderman posted in HotWired,
June 5 1996.
<http://hotwired.lycos.com/popfeatures/96/24/eno.tran
script.html>
Fontana, Bill. Sonic Ecology. October,27. 2002
<http://www.kunstradio.at/ZEITGLEICH/CATALOG/ENGLISH/
fontana-e.html>
Fontana, Bill. Sound as Virtual Image. December,8. 2002
<http://www.resoundings.org>
Fontana, Bill. Resoundings. December,8. 2002
<http://www.resoundings.org/Pages/Resoundings.html>
48
Kruger, Myron . Essays.
La Grow Sward, Rosalie . An examination of the Matematical
Systems Used in Selected Compositions of Milton
Babbit and Iannis Xenakis: A Disertation in partial fulfillment
of the requirements for the degree Doctor of Philosophy.
Evanston, Illinois, 1981.
Leitner, Bernhard. Bernhard Leitner Website. January, 11.
2002 <http://www.bernhardleitner.at>
Levant, Yvie. Music as Image: Sound and Language.
<http://interact.uoregon.edu/MediaLit/wfae/readings/L
evant.html>
Levin, Golan . Painterly Interfaces for Audiovisual
Performance. MIT, 2000.
<http://acg.media.mit.edu/golan/thesis>
Rokeby, David . VNS Website.
<http://www.interlog.com/~drokeby/vns.html>
Walters, John L. Sound, Code, Image. Published in EYE26,
1997. Pages 24-35.
49
Appendices
Appendix A. Custom Interfaces Screenshots
A.1. Program for Accumulation
Stereo Delay
Feedback Pacth
50
A.2. Program Interface for Aural Limbo
Audio On/Off
Control
Audio Router
Control
Amplitude
Control
Video Input
Control
Input
Monitor
Pan Process
Effects Mixer
Motion
Monitor
Process
Motion
Main Audio
Amplifier
Mix / Pan Monitor
Random
Track Picker
Data
conversion
Track
Selector
Peak Monitor
Audio
Track
01
Audio
Track
02
Audio
Track
03
Reverb
Patch
Reverb
Patch
Feedback
Patch
Audio
Track
04
Reverb
Patch
Main Audio Mixer
51
A.3. Program Interface for Traffic Report
52
B. Suplemmentary Sketch
Preliminar Sketch for A.L.
53
Colophon
This document was prepared with Macromedia Freehand
and Microsoft Word.The text of this thesis was set in
twelve-point MetaPlus Normal (1993)designed by Erik
Spiekermann. The titles were set in MetaPlus Medium Caps.
54
Mateo Zlatar
[email protected]
2003