here

Drawing Music : An investigation into
the research, design and implementation of an interactive Audio reactive
application intended for real time music visualisation purposes.
Prepared for: Ian Wallman
Prepared by: Juan Hurlé
S.A.E.Oxford
21 October 2010
Student Number: 14049
Word Count: 10.500
Table of Contents
Abstract
1
1
2
INTRODUCTION
2
2
4
BACKGROUND
4
2.1 Definition of Visual Music
4
2.2 History of Visual Music
5
This section comprises a selection of the most influential artists along history who
worked in this topic. A thorough discussion of every single one of them is completely out
of the scope of this project since research has revealed that there has been an immense
interest and work regarding Visual Music since the Renaissance.
5
Early Experiments and Colour Organs
5
Scriabin’s Prometheus
5
Scriabin’s relationship between colour tones and music.
6
Wassily Kandinsky
6
Wassily Kandinsky - Several Circles (1926)
7
Non Narrative Cinema: Walter Ruttman, Hans Richter
7
Mary Ellen Bute
7
Expanded Cinema: John Whitney and sons, Oskar Fischinger, and Norman McLaren.
8
2.2 Music Videos, Visual Music, Digital Art and the Club Era
8
2.3 Current Trends
10
2.3.1 Raster Noton
10
2.3.2 Ryoichi Kurokawa
11
Page i
Ryoichi Kurokawa performing at Mutek Festival 2007
11
*Note
11
3
12
TOOLS
12
3.1 Pure Data
12
3.2 Processing
12
3.3 Open Frameworks
13
3.4 vvvv
13
3.5 Isadora
13
3.6 Max/MSP
13
4
15
Max/MSP/Jitter
15
4.1 Brief Overview of Programming
15
4.1 Brief History of Max
16
4.2 Max
16
4.2.1 Objects
4.3 Max Signal Processing (MSP)
18
4.4 Jitter
18
4.5 Expanding Max
19
4.6 The Reason Behind Choosing Max
19
4.6 Basic Programming using the Max Environment
19
5
17
22
Page ii
PROJECT : Music Visualisation Tool
22
5.1 Introduction to the project
22
5.2 Main Concept
23
5.3 Working Rates
23
5.4 The Core: FFT vs 3 Band EQ
24
5.5 MIDI
25
5.6 The patch explained
25
5.7 The Interface
31
Finished Interface
32
5.8 Known Problems and Bugs
32
5.9 Application Building
32
6
33
CONCLUSION
33
A
35
APPENDIX A: Jitter Matrix System
35
B
37
APPENDIX B: Open GL
37
R&B 40
References and Bibliography
40
Page iii
Abstract
This project tries to give some light to the existent possibilities of audio visual composition, based on the
principles of Visual Music, a subject that has been explored by artists and scientist for centuries. The
body of work includes a background of the subject, a study of the current tools and the development of
an interactive Audio reactive application intended for music visualisation purposes. The project has shown
that Audio Visual creation is more doable that ever, even for non technical users due to the increasing on
computer speeds and the broad range of existing tools, a fact that has been proved by the application
created in the time frame of two months, with no previous experience on the subject itself. Audio Visual
composition will most definitively become a common practice in the following years.
Page 1
1
INTRODUCTION
“However great the power of the ear to stir the emotions, however wideranging the information it receives, [...] the sense of hearing cannot attain that clarity
of consciousness which is native to that of sight.”
(Jenny, H. 1967)
Music, just as any other form of artistic expression, can be distinguished from other disciplines by its delimitation by the medium and amongst other things, the sense by which it is perceived ( Lyons, A. 2003,
p. 175), It is clear that music and sound come to our brain through our hearing. Though this is not exactly
true for certain people who are affected by a particular kind of phenomena called Synaesthesia, a rare
perceptual condition which “mixes” senses, commonly audition and vision. Dr. Richard E. Cytowic defines
this condition in the following manner: “Synaesthesia is a word derived from the Greek words ‘syn‘ meaning together and ‘aesthesis‘ meaning perception. It is used to describe the involuntary physical experience of a cross-modal association. That is, the stimulation of one sensory modality reliably causes a perception in one or more different senses.” (Cytowic, R.E. , 1995, p.1 ). Synaesthesia has always been related to art, specially music, and it has also been used as a theme for both Synaesthetes and nonSyanaesthetes in different research fields such as Psychology or Neurology (Orlandatou, K. 2009 , pp.18
- 36 ) For that vast majority of people that do not “suffer” the effects of this affection, visualising music is
limited to either the imagination of the concepts described in physics related to the propagation of waves,
( concepts which are not accessible to everybody and even difficult to imagine for the most knowledgeable ) or either the use of audio analysis tools that provide the user with a visual representation of sound.
In terms of “real” sound visualisations, the experiments of Hans Jenny on the structure and dynamics of
waves and vibrations are probably one of the most interesting pieces of work regarding the visual component of sound, in other words, the shape of sound. By putting sand over a vibrating membrane driven
by an oscillator he was able to discern different patterns associated with different frequencies and amplitudes , interesting facts were discovered such as the increment of the complexity of the shapes along
frequency and also some interesting oddities like the sound of the vowels of some ancient languages
(Hebrew) created in the sand the shape of their written symbol (Pettersson, P. 2002 ).
Page 2
Back to the theme of Synaesthesia in art, many attempts have been done since ancient times to establish
a link between sight and hearing and this project is no different.
In the next chapter, a background on the subject will be presented, showing the most important
Page 3
2
BACKGROUND
Visual Music is a subject that belongs to the world of Performance Arts, although its recent history puts it
in the context of “club culture” and the so called “Vj” it has many connections with other art forms such
as Painting, Lumia, Colour Music, Expanded Cinema, Live Cinema (which can probably be considered as
a branch of the previous one) or the recently born Digital Art. There is very little written history available (
Makela, M. 2006, pp. 4-6) this might well be because it is not a perfectly defined subject, rather, a blurred
emerging discipline that can shape and adapt itself according to the time frame. During the next sections
of this chapter, an insight into this fascinating topic will be presented, trying to mention most of the relevant names found in the research stage.
2.1 Definition of Visual Music
As Cornelia and Holger Lund express on their book Audio.Visual - On Visual Music and Related Media,
the term Visual Music was first used with some frequency in various discussions of experimental and
specially “absolute” film during the 1920s and 1930s (Lund, C and H. 2009, p.11), later on on the same
book, the author attempts to give a description of the topic stating that Visual Music is the collection of
audiovisual productions pursuing the basic objective of evenly balanced or equilibrated interplay between
visual and acoustic components (Lund, C and H. 2009, p.12). This definition seem to be perfectly suitable
since it is reasonably open ended, enough to adapt itself to future explorations of this form of art and also
fits with the idea of this project’s author.
Page 4
2.2 History of Visual Music
This section comprises a selection of the most influential artists along history who worked in this topic. A
thorough discussion of every single one of them is completely out of the scope of this project since research has revealed that there has been an immense interest and work regarding Visual Music since the
Renaissance.
Early Experiments and Colour Organs
The first references to this “genre” date back to the times of Leonardo da Vinci , who made sketches of
colour organs around the fiftieth century (Lyons, A. 2003, p. 166). At the end of the 19th century two
main experiments were made around Colour Organs, a type of organ which would play with different coloured lights to correspond with the music of the instrument (Bishop, B. 1983, p. 3). These two experiments where made by Bainbridge Bishop and Alexander Remington respectively, the latter published a
book in 1911 titled Colour-Music: The Art of Mobile Colour (Lyons, A. 2003, p.167).
Scriabin’s Prometheus
Scriabin established a relationship between tones in colour and music, which was later applied to his orchestral composition Prometheus:A Poem of Fire (Makela, M. 2006, p.14). The table of tones-music that
he created was inspired by his conversations with Nicholai Rimsky-Korsakov, both of them were regarded
as Syanaesthetes (Lyons, A. 2006, p 168).
Page 5
Scriabin’s relationship between colour tones and music.
(Bower, F. 1995 p.114)
Wassily Kandinsky
Another prominent artist who was also believed to be affected by Synaesthesia and specially the type of
colour hearing (Orlandatou, K. 2009 p. 28) was Wassily Kandinsky the first abstract painter. He was inspired by Wagner’s idea Gesamtkunstwerk, or the total work of art embracing all possible artistic disciplines, and the relationship between musical tones and colour formulated by Scriabin. Kandinsky’s heterogeneous ambivalent creative intentions did not prioritise any specific artistic discipline but rather, utilised diverse techniques which overlapped, in order to achieve a catharsis spiritual feeling opposed to materialism. Therefore, he intended to represent spiritualities thanks to abstract shapes and the use of colours. For this author every artistic field was moving towards the spiritual due to abstraction as he perceived the music of Schoneberg, Wagner, Debussy, Scriabin, Schumann and Mussorgsky.
Spiritual renewal could only emerge from a total blending of all arts, for instances this can be seen in his
investigation application of his idea of the psychological effects of colour which generated a new harmonic theory of forms depicting a contrast due to warm and cold or light and dark tensions so that pure
painting mixtures colour and shapes each with its autonomous life but each conceived into a shared
existence.His desire to gather as many artistic experiences as possible “pure art” can be seen in his first
play for theatre The Yellow Sound or in his essays Concerning the Question of Form or Painting as Pure
Art, both from 1912. (Duchting, H. 2007, pp. 37- 42)
Page 6
Wassily Kandinsky - Several Circles (1926)
Non Narrative Cinema: Walter Ruttman, Hans Richter
The field of experimental cinema expanded the possibilities of exploring this topic. Great examples of this
were Hans Richter and Walter Ruttmann who created Non-Narrative films during the 1920s. The most
prominent pieces of work in this field were Ruttmann’s Berlin- A Symphony of a Great City (1920) and
Diagonal Symphony (1919) by Hans Richter and Viking Eggeling. Dozens of Filmmakers created abstract
films that addressed such a theme, but film is fixed, each showing of a film very like every other, while the
performance of live music is not. (Collopy, F. 2007 p. 2)
Mary Ellen Bute
Between the 1930s and the 1950s Mary Ellen Bute created a series of abstract films combined with music called Seeing Sound. She also wrote several books, one of them titled Light as a Material and its Possible Synchronisation with Sound (1932). Mary Ellen was fascinated by the work of Wassily Kandinsky
and the repercussion of music, as she stated “It was particularly while I listened to music that I felt an
overwhelming urge to translate my reactions and ideas into a visual form that would have the ordered
sequence of music” (Bute, M. E. 1952, p.263 ). She continued working along these lines for the rest of
her career.
Page 7
Expanded Cinema: John Whitney and sons, Oskar Fischinger, and Norman McLaren.
“One no longer can specialise in a single discipline and hope truthfully to ex-
press a clear picture of its relationships with the environment”
(Youngblood, G. 1970, p. 41)
In order to understand this rather complex and broad genre it is necessary to give a definition. As explained by Gene Youngblood the author of the book Expanded Cinema (1970): Like life (Expanded Cinema) its a process of becoming man’s ongoing historical drive to manifest his consciousness outside of
his mind, in front of his eyes. ( Youngblood, G. 1970 p. 41 ). This artistic movement fiercely encourages
the use of computers in the creation of works, an interesting development was the computer image
process called Animac which generated images in conjunction with acoustical or anthropometric analogue systems.
Involved in this sphere we find John Whitney, one of the early pioneers in computer based animation, he
believed that this medium was the definitive medium for this sort of artistic expression. There has been a
certain tradition within his family as his brother and three sons exemplified. His descendants boosted their
father’s work to the max, creating multi-projection shows and even touring live with the Grateful Dead
performing visual shows along their music. This can be the perfect precursor for modern music performances where the visual aspect plays an essential role as music accompaniment.
2.2 Music Videos, Visual Music, Digital Art and the Club Era
Modern examples of Visual Music can be easily associated with electronic music performances and the
so called Digital Art.
Page 8
Music Videos and Visual Music are two different ways of relating music and the filmic image, nevertheless
pinpointing the differences between them seems to be complex, similarities can be found in their nomenclature which is only perceived when we approach the issue at first. As Peter Weibel’s note “Music video”
can be interpreted as “I see music”. Similarly the concept “Visual Music” defines a Synaesthetic experience in which temporarily arrange Sounds and noises result visible.
Music Videos evolve from various traditions, as Jack Banks remarked, Video Clips are sort advertising
films that are linked to the development of music television (Banks, J. 1996)
When young A. Lack became the director of Wasec (Warner Amex Satellite Entertainment Company) he
hired Robert Pittman to include a music video within Wasec which lied to the creation of MTV in 1981
with the aim of advertising by broadcasting Video Clips which seemed a greater scope alternative to concert Tours.
Music Videos offered a compact format for mass consumption, however the only visible reference to the
product appeared at the beginning and at the end of the video clip normally including the artist name,
song and album title. This clips focused not only in the sounds but also in the images presenting the stars
and diminishing the high costs of life performances for producers and the public. From another perspective, Music videos must be categorised in the outer context of music films, both of them present the orientation of the artist and the narrative structure even the video clip need much more economical narration.
Visual Music contemporary concept begins in the 1980s in public venues: Clubs, Discotheques, Festivals
and Raves broadcasting on monitors or showing on screens with projectors and thus arranging the
space.
Thanks to the technology developments as the increase processor performance or raised display card
capacity, there is the chance of digitally produce Visual Music since the late 1990s.
Those responsible for such creation are known as Visual or Video Jockeys (Vjs) who pick up images from
the chosen sources an organise them with the music of an on location Dj; so that the VJ uses techniques
from the DJs as Sampling or Looping, this is the reason why the VJ usually works to the Dj’s principles.
Visual Music can be employed in a commercial context but does not reach the mass targeted audiences
that music video does. As the Video Clip, it can be related to different traditions:
Page 9
The Single Wing Turquoise Birdgroup : Henry Jacobson and Jordan Belson , in their Vortex concerts held
at the Morrison Planetary in San Francisco from 1957 until 1960 employed the totality of the space as
projection area.
The Smart Light Shows of the 1960s derived from experiments of Seymour Locks to electronic music at
San Francisco State College. Similarly Joshua White, William Schwarzback and Thomas Shoesmith,
Joshua Light Show of 1969 included live performances of Bob Dylan, Janis Joplin, Frank Zappa and
many others.
A close practice to Visual Music is Video Art, a discipline that begun on the 1960s. Its pioneers, specially
Nam June Paik and Wolf Vostell discovered new ways of producing, manipulating and transmuting existing images .
A discernible depart from the video clip, is that narrative and figurative aspects remain secondary since
the main objective is to create a specific atmosphere where the images present certain opacity so it does
not place the viewers in a different place in time, rather it aims at provoking the public’s performance.
(Weis, M. in Lund, C&H ,2008 pp. 88-96)
2.3 Current Trends
It is important for the author of the current work to provide a special mention to some audio visual performers of nowadays because of their direct relation with this project. The Raster- Noton Collective and
Ryoichi Kurokawa.
2.3.1 Raster Noton
Raster Noton is a record label as well as a platform for new media. It was formed by Olaf Bender,
Frank Bretchschneider and later Carsten Nicolai. Their work involves real-time audiovisual com
position that have been shown at some of the most important events in this area such as the Visual Music Exhibition held at the Museum of Contemporary Art, Los Angeles 2005. As Alva Noto (also known as Carsten Nicolai) said in an interview to Bleep Blog, that he started as a vis ual artist and while going through some sort of crisis and whilst researching about Time, he dis covered that his real passion was music, and combined both disciplines to create a whole that is bigger than the sum of its parts.
Page 10
2.3.2 Ryoichi Kurokawa
This young Japanese artist explores Time Sculpture, fusion between sound and digitally gener
ated shapes whose movements and deformations respond to dynamics of the music played. He has performed in some of the most respected art exhibitions worldwide in renowned places lake the Tate Modern in London, showing today’s probably most accurate achievement in the field of Synaesthetic art. The fusion between audio and video is so seamless that one can hardly distin
guish one from another. The work of this artist can be considered as the main motivation for this project. One can hardly imagine how all the previously mentioned artist would have reacted to s uch thing if they would have been able to see it.
Ryoichi Kurokawa performing at Mutek Festival 2007
*Note
The previous compilation of artist and work is just a minimum part of what its possible to find regarding
this field. As said before, an exhaustive discussion is well out of the scope of the project, however this
enlightens the fact that this form of artistic expression has been constant since ancient times and one can
presume that it will be still alive for quite some time.
Page 11
“We shape our tools and therefore our tools shape us”
(McLuhan, M. 1964)
3
TOOLS
This section comprises a short list and discussion of some of the most popular multimedia programming
environments, the perfect tool for a cross-media project.
3.1 Pure Data
PD or Pure Data, is the outcome of years of effort of the well known MIT (Massachusetts Institute of
Technology) graduate Miller Puckette (author of one of the most important books in digital signal processing: The Theory and Techniques of Electronic Music ). This program was developed while woking at IRCAM (Institut de Recherche et Coordination Acoustique/Musique) during the 1980s . The program is said
to be the third major branch of the programming languages known as Max ( Max/FTS, Max/MSP , jMax,
etc...) orginally developed by him as well (Zmoelig, I. 2010 ). Because of its condition of Open Source it is
supported by a huge community of developers that contribute making PD bigger every day. It contains
objects for pretty much everything cross-media: Audio, Video, 3D Graphics (extension called GEM) and a
long etc. Plus is totally cross-platforb being available in pretty much every known Operative System (OS).
Pure Data can be expandable since it is based in C code, by writing externals in this language. This is an
incredibly useful tool not to be missed out. One can download it for free at http://puredata.info .
3.2 Processing
Page 12
Processing is and Open Source programming Language and environment based on Java, and created by
Ben Fry and Casey Reas. It is cross-platform and it contains over a hundred libraries for many subjects
such as sound, 3D , video and many more. It is extremely well documented with many sources of information available such as book or full websites dedicated on teaching this interesting work environment.
The kit can be downloaded from the official Processing website. (http://processing.org)
3.3 Open Frameworks
Open Frameworks is a C++ library created by Zach Lieberman, Theodore Watson and Arturo Castro that
is born from the ACU toolkit and the Processing environment, it is also code based and as the creators
themselves say “ It provides a simple and intuitive framework for experimentation”. The library serves as a
bridge with other commonly used libraries such as OpenGL, quicktime, or rtAudio.
(http://OpenFrameworks.cc)
3.4 vvvv
Very similar to Max and PD’s programming style, this platform specific tool (only running under windows
systems) is said to be the much faster in graphic rendering than its cousins Max and PD due to its pixel
and vertex shader capabilities based on DirectX 9. (Blum, F. 2007, p. 32). Another Important advantage of
this platform is its open source condition. That is, free to download and use with the only requisite of acknowledging it. A very tempting tool that deserves to be deeply investigated during further projects. It can
be found and downloaded for free from http://vvvv.org.
3.5 Isadora
Isadora, a program created by Troika Tronix, presumes to be easy, flexible, reliable and powerful. It won
the honorary awar at the Prix Ars Electronica ( AEC Ars Electronica centre, 2004 ) Its function its specially
emphasised on the real-time manipulation of digital video. It contains over a hundred objects called actors that can be arranged on a graphical programming interface and when grouped together can also
perform as User Actors. (Blum, F. 2007 , p.33). The toolkit can be obtained at
http://www.troikatronix.com .
3.6 Max/MSP
Page 13
Max is an Interactive visual programming environment designed for music, audio and media. Originally
developed by Miller Puckette at IRCAM in Paris during the 1980s, but later managed by David Zicarelli in
his own company Cycling ’74. It is a commercial licensed program intended for real-time cross-media
composition featuring three main lines of development, the first is Max, the core set of objects of the program intended for working with time, interfacing and control. The second, MSP (Max Signal Processing)
designed for sound synthesis and manipulation who’s objects feature pretty much everything from filters
to oscillators and envelopes. The last but not least is Jitter, the youngest of the family, added in 2003, it
features more than a hundred objects for real-time manipulation of video and graphics (2D and 3D), it also
includes extensions for the common libraries such as OpenGL (Open Graphics Library) . The full program
is written in C code and brings the possibility of expanding it by writing externals in this type of code.
http://cycling74.com
This application has been chosen to be used in this project, the reasoning behind this decision will be
explained in the next part along a deeper insight of this fantastic tool.
Page 14
“The general precept of any product is that simple should be easy, and hard things should be possible”
(Kay, A. n.d.)
4
Max/MSP/Jitter
In this chapter Max, along a brief overview of programming languages are introduced, after going through
the aspects that make this piece of software so useful, a reasoning and full explanation of why it has been
selected will be presented. It has been written in this specific manner to let the reader familiarise with the
application first, and then, evaluate the pros and cons, and the reason behind the decision as opposed to
the other proposed work environments.
4.1 Brief Overview of Programming
In order to understand the basic principle of Max it is necessary to understand some basic concepts on
programming. Computer Hardware operates on a binary lever, that is, it only understands 0s and 1s,
each of those digits is called a bit. They can be grouped in larger number for instance bytes, which are
groups containing eight bits. This grouping method goes up to billions of bytes (kilobytes, megabytes,
gigabytes...). We call Machine Language to the instructions that the computer hardware understands,
that is, binary instructions. If we had to write a computer program straight into binary code, it would take
a huge amount of time, plus the code would be practically impossible to distinguish since it would be an
overwhelming list of zeroes and ones and finding errors ,a process called debugging, would be pretty
much impossible. That is the main reason behind the common use of high-level programming languages.
These High-Level programming languages consist in a series of instructions, called functions, written in a
more human-like way. Such languages need a compiler to translate these instructions to machine lan-
Page 15
guage. The big advantage of high level programming is that identifying errors becomes much more accessible.
A current trend in programming comprises what is known as Object Oriented Programming, this kind of
languages “allow programmers to build software systems as collections of interacting objects, each of
which models the behaviour of some real world entity” (Pope, 1991). Since the early eighties, numerous
interactive music systems have contained object oriented features. Smalltalk is probably the most prominent representative of a true object oriented language, it has been used to design a number of interactive
music systems such as DMIX (Oppenheim 1990) and Kyma (Scaletti 1987) (Winkler, T. 1998 p. 43) Max is
also an object oriented programming language.
4.1 Brief History of Max
Max was first developed around 1986 by Miller Puckette at IRCAM as a control device for the 4X synthesiser. Max takes its name from Max V. Matthews, an early pioneer of computer based music. In 1987 it
was presented in the form of a concert, including two main pieces: Jupiter by Philippe Manoury and Aloni
by Thierry Lancino. This concert showed for the first time the full capabilities of Max (at that time) as a
scoring follower program based on MIDI ( Musical Instrument Digital Interface). Around 1990, the IRCAM
Signal Processing Workstation (ISPW) was introduced with the aim of replacing the now obsolete 4X and
Puckette adapted the software for such a task. ( Winkler, T. 1998 , p. 18).
Around 1991, David Zicarelli extended its functionality and licensed it with Opcode Systems as a commercial platform, whilst Miller Puckette started working in PD (Pure Data).
Soon MSP was added to the package including signal processing functionality, MSP takes its name from
Max Signal Processing ( Although some might argue that the name comes from Miller S. Puckette’s initials). The Graphic extension called Jitter made its appearance around 2003 in the form of over a hundred visual oriented objects which make Max’s functionality ideal for cross-media work. Finally, the latest
update has been Max for Live, a version of Max that works side to side with the now popular Ableton Live
Digital Audio Workstation (DAW) featuring real-time modification and use of plugins made in this platform.
Max has been the tool of choice of many artist and musicians along its relatively short history due to its
intuitive ease of use. (Blum, F. 2007, pp. 18-19 ).
4.2 Max
Page 16
Max is the environment you use to create visual programs , called patches, plus a set of building blocks
called objects used in those programs. ( Cycling’74 n.d.) It is a C based visual computer programming
language, that is, instead of hand written code, the user interconnects boxes which correspond to functions, this programming style is called visual programming, modular programming or patching, the latter
comest from the analogy to Analogue Modular Synthesis Environments where patch chords were used to
establish connections between different modules. This flow chart-like style of programming has numerous
advantages, specially for the non technical user, due to its readability and it real time capabilities, since
one can connect and disconnect objects in real time being able to see their function independently and
not altering the general structure of the rest of the program. Another prominent advantage is that complexities of the computer and irrelevant details are hidden, allowing humans to interact with computers on
a broad level that is conceptually closer to everyday life (Winkler, T. 1998, p. 49)
4.2.1 Objects
Max programs are created by connecting objects, which is another way of naming algorithms. Custom
objects can be created by combining different objects and then encapsulating them inside a sub patch.
This brings up the issue of reusability of code, sub patcher objects can be used infinite times inside different patches or applications (within Max).Objects have inputs (top) and outputs (bottom), and it is important to know that information is always sent from right to left, that is, the rightmost output of an object will
always be sent first.
4.2.2 Messages
Objects receive messages in order to execute their algorithm. Messages, are data (Numbers) or instructions (words) sent to an object to describe how will it function. (Winkler, T. 1998 p.50)
4.2.3 Connections
Connection are made through the mouse, physically connecting objects by mouse clicks. The connections can be seen in the interface as wires, this make the whole program look like a flow chart diagram.
Page 17
Basic Max patch showing basic objects and messages connected (Tigoe 2007)
4.2.4 Encapsulating
Patches can be organised inside sub patches by creating an object called [p], it can have inputs and outputs and once it is finished it works just the same as any other object. This is a key feature of Max for
reutilising your work in other situations, and it will be be extensively used in this project.
4.3 Max Signal Processing (MSP)
MSP is a set of Max objects intended for audio and signal processing (Cycling ’74). It comprises more
than two hundred object intended for DSP (Digital Signal Processing) purposes. It was added around
1997 based on Puckette’s work for PD. As opposed to Max’s behaviour, which sends message information once at a time, MSP sends continuous streams of data called signals. All MSP objects are differentiated from Max by the syntax of their name, always followed by the symbol ( ~ ) which means signal. For
instance an oscillator object, called Cycle, will appear in Max as [cycle~ ]
4.4 Jitter
Jitter is a set of Max objects for video, graphics and matrix data processing. (Cycling ’74). developed by
David Zicarelli’s company Cycling’74 released in 2003. It processes information using a matrix system,
Page 18
this system is explained in Appendix A. In order to distinguish Jitter objects from the rest, they have the
prefix (jit). As an example, the Jitter Matrix object will appear as [jit.matrix]. Jitter includes several libraries
such as OpenGL (Open Graphics Library) for 3D purposes, this library will be explained in Appendix B.
4.5 Expanding Max
Because it has been written in C code, Max can be highly expandable. One can write externals , that is,
Max compatible objects written in C, these objects can be made by using the Max SDK (software developer kit) which can be download for free from Cycling’74’s website. This has opened the doors for a huge
community library, comprising both professional and amateur users and a large collection of external objects that add functionality to the original program. Some of these developers include the CNMAT (Centre
for New Music and Audio Technologies), responsible for the creation of the OSC (Open Sound Control)
Protocol, a protocol that might become MIDI’s alternative. Apart from this JavaScript and Java codes are
also supported. The last but not least option to expand Max is the use of external hardware prototyping
platforms such as Arduino or Wireframe, which allow to connect analog devices like sensors, lights or
even robotics.
4.6 The Reason Behind Choosing Max
As we have seen so far, Max is an extremely powerful tool but some of its “relatives” are not far away. The
main inconvenient is that from the tool list mentioned in chapter three, is along Isadora the only commercial program, in fact it is pretty unaffordable. When compared to PD or vvvv which are Open Source platforms capable of just as much functions, one can think this is not the best choice, however, the selection
of Max over the other two is based on two main principles: First, vvvv is platform specific, it only runs on
Windows based systems, that makes it non portable, plus vvvv does not have the option of exporting
patches as standalone applications. Second, one of the aims of this project is to learn increase the
author’s skills, and PD has been a studied subject over the past few years of this author’s career and the
intention is to learn as many different programming scenarios as possible, however, this previous PD
knowledge has been a huge advantage in order to understand Max, since both programs are fairly similar.
The other fact that keeps PD out of the scope is that, just as vvvv, it does not let the user export his
patches as standalone applications.
4.6 Basic Programming using the Max Environment
Page 19
4.6.1 Data Types
For the correct use of this piece of software, one must understand the seven basic data types that it
uses.
Integers: abbreviated int.
Floating point numbers: called float
Lists: named list these are groups of two or more numbers, int or float and/or symbols
Symbols: Symbols are words, the abbreviation for this data type is s.
Bang: bang means do it. It is the most widely accepted message in Max and forces any object to perform
its function.
Signal: As explained before, signals are continuous streams of floating point numbers used to represent
audio. Only MSP objects accept this data type.
Matrix: Similar to signals, this data type only works within Jitter objects, it represents arrays of numbers,
these numbers can be char, 8 bit integer numbers, float32, 32 bit floating point numbers and float64, 64
bit floating point numbers. Matrix can basically contain any number that can be arranged into rows and
columns.
4.6.2 Parameters and Arguments
Each object has a number of modifiable parameters which determine its behaviour, they are often variables, that is, they can be changed in real time to for example according to a score or improvisation.
Some parameters can typed directly into the object following the symbol @, this can be useful in order to
establish initialisation parameters. They are normally constants, although some of them can be overwritten by means of a message. (Winkler, T. 1998 p.55)
4.6.3 Order of Execution
Max processes one step at a time, however it is fast enough to make this unnoticeable. In order to avoid
possible confusion Max’s order of execution goes from right to left and bottom to top. Another important
Page 20
fact to mention is that most objects accept a bang message on their leftmost input, in fact, if they have
multiple inputs, the object will not perform its function unless a bang is sent to that input, any message or
number acts as a bang on the leftmost input. An extremely useful object to avoid problems derived from
this is trigger, found both as [t ] or [trigger] followed by the data type to be sent. For instance a [t b i ] object will send first an integer number through its rightmost output, which should be received on its input,
and then a bang through its leftmost output.
Following all the rules presented before, one should be able to program some basic patches on Max/
MSP. In the next chapter the application proposed in this project will be presented, assuming some familiarity with the previously shown concepts.
Page 21
“My computer program is like a piano. I could continue to use it creatively all
my life”
(Whitney, J. in Youngblood, G. 1970, p. 207)
5
PROJECT : Music Visualisation Tool
5.1 Introduction to the project
As we have seen so far, the exploration of Visual Music has been a major concern to artists of all fields
along history. Nowadays the music Industry seems to constantly repeat itself with a set of well known
formulas, being difficult to find something innovative and creative just by the use of traditional methodologies. The vast possibilities that the blend of disciplines brings us, allows the artist to achieve new perspectives and goals. Gene Youngblood states that one no longer can specialise in a single discipline and
hope truthfully to express a clear picture of its relations with the environment. Following this lines Norman
Brown said: Freedom is Fusion.
The field of computer music has special interest for the author of this project, as opposed with what one
might think, this extremely open genre is making some composers lazy, that is, many artists seem to rely
on pre established computer tools, and putting up a show which lacks of the showmanship and “liveness” that a band brings to the scene. Sometimes the music created is extremely beautiful but for the
majority of the audience, watching a man behind a laptop and having no clue of what is going on can
become pretty unattractive. For this reason Visual Music seems like a perfect ally for those type of performances, bringing the audience the possibility of seeing music, which can help giving a better picture of
what the artist thinks.
Page 22
For all this reasons, the author has chosen this topic, apart from being a challenge which will prove a
good understanding of sound, and its possibilities regarding cross-media creation, a possible future step
for electronic music and a little bit closer to what Wagner defined as Gesamkunstwerk.
5.2 Main Concept
The idea behind this project is to create an application in the visual programming environment Max/MSP
which creates abstract visualisations of any given music or sound. This project does not tray to make
faithful representation of the sound, it is not a music analysis tool, however the reaction to the produced
sound must be clear to the eye. In order to achieve this ambitious goal, several premises were mandatory:
First, the music should be the main driving force of those visuals. It could have been relatively easy to
cheat the eye by using triggering techniques such as MIDI or OSC, however the intention is to show the
dynamics of music in a visually pleasant way. The frequency of the sound to be translated into visual
stimulus plays an important role in this task. As we have seen with Synaesthetic artists or Cymatics , it
does not “look” the same a low frequency sound and a high pitched one so we must make a clear difference.
Secondly, the program has to be flexible enough to adapt itself to different kinds of music, as we might
know, the frequency and dynamic range of a funk band is not quite the same as an experimental noise
production. Because of this, the tool must have some sort of frequency and amplitude detection system.
Third: The application needs to have a comprehensive set of controls to let the user change the appearance of the visual display, if this was not the case, the use of this tool would become boring and rapidly
obsolete,
Fourth: It must have an intelligible interface, that allows anyone to rapidly grasp the way to operate it.
5.3 Working Rates
Page 23
One of the biggest challenges of this project was to combine the three working rates inside Max/MSP,
firstly, Max runs at what is called control rate, that means that each object performs its function whenever
a message is sent. Secondly, MSP runs at audio rate, which is commonly 44100 Samples per second,
obviously this can even be faster up to 96000 samples per second, depending on the external audio settings of the audio device. Thirdly Jitter runs at Video rate which is normally 24 frames per second (fps).
when combining the three, one must bare in mind this different speeds of execution.
5.4 The Core: FFT vs 3 Band EQ
Regarding the audio analysis section, two main properties were selected as the main drivers for movement: Frequency and Amplitude. The First experiments showed that the overall amplitude was not
enough descriptive for the pursued goal and it had to be different for the different frequency bands.
Fast Fourier Transform (FFT) seemed to be the most suitable method, since it is the core component
used for spectrum analysers and the visual result is quite pleasing.
The Fast Fourier Transform is a very complex algorithm used for audio analysis purposes, a thorough discussion of this formula is completely out of the scope of the project, however, a basic explanation of its
function is necessary in order to comprehend the mechanism used in the project to extract data from the
audio signals.
FFT derives from the Fourier series described by Jean-Baptiste Fourier around 1822, based on the theory
of periodic signals. It is a numerically intense operation that has not been implemented as a common
function until the late 1980s. It states that periodic waveforms of different amplitudes can be decoded
into frequency and phase components. (Roads, C. 1996. p.1076)
There are several implementations of the Fourier Transform, the Short Time Fourier Transform (STFT), the
Discrete Fourier Transform (DFT) or the Fast Fourier Transform (FFT). They are all base in the same principle: It maps any continuous -time (analog) waveform to a corresponding infinite Fourier series summation
of elementary sinusoidal waves, each at a specific amplitude and phase. (Roads, C. 1996, p. 550)
The FFT is no more than a computer efficient implementation of the DFT. It takes “chunks” of audio called
windows, and performs DFT over them. DFT roughly speaking, passes the signal through a series of “filters” equally spaced covering the whole spectrum, the number of “filters” is given by the FFT size, normally equal to the window size. For instance, if we perform an FFT of 512 samples window size, the
Page 24
audio processed will be 512 samples long and the frequency range will be divided into 512 equally
spaced sections using 512 “filters”.
This type of analysis happens to be quite expensive for the computer, and also, the space between the
“filters” is always equal and can not be changed unless the whole FFt function is changed,that is the reason why it has not been implemented in the project, however, it has been a key part of the research and
has lead into the final version, which uses a three band EQ, since the basic objective was quite the same.
By using a 3 band EQ we obtained the amplitude of the most prominent areas in music, the Lower band,
around 90Hz, a mid band around 400 and 1400 Hz , and a High band above 4000 Hz. This happened to
be enough to create a credible movement in the graphic domain whilst a full FFT gave way too much information to be processed and slowed the performance of the application significantly.
5.5 MIDI
As opposed to what the proposal said, MIDI has not been implemented in the final version. Initially this
protocol was proposed in order to fill the gaps not covered in the audio analysis section, however, the
research has revealed that it goes against the whole nature of the project since the idea is to have sound
as the commander of the images created. MIDI can easily trigger events and help with control but the
project’s main point is to create motion based on the music/sound, for that reason it has been purposely
avoided. Future implementations of the application might include this musical protocol in order to facilitate
the use to the musician in the case of a live performance but at the moment, this is completely out of the
scope of this project.
5.6 The patch explained
In this part the full functionality of the patch will be explained in detail. Max objects will be named using [ ]
to clearly distinguish them and avoid using the word object constantly. The patch is explained assuming
that one is looking at the editable Max/MSP file, not the compiled application, the latter one not modifiable and its contents are hidden form the user.
Three main sections or sub-patches: Audio, Rendering and Shapes.
Page 25
5.6.1 Audio
Audio Section (Edit Mode)
The First section of the patch is quite simple and quite important at the same time. This is the Audio section. It first contains an [ adc~ ], in other words, analog to digital converter, used as an audio input, by
double clicking it or pressing the attached message open, it opens the audio preferences dialg, in order
to configure audio drivers, inputs and outputs.
The audio input can be switched for an [sfplay~ ] which plays audio files (wav, aiff) using two [selector~ ]
(one for each audio channel) it has also got an audio output which its output gain can be controlled and
monitored by an “ableton-like” fader named out.
Page 26
Inside EQ&Send sub-Patch
The most important part of this section is the sub-patch called EQ&Send. Double click it and you will see
its contents. This patch contains three filters constructed using [onepole~ ], a low pass filter. The first one
called [onepole~ 90] corresponds to the low pass filter, having a cut off frequency of 90Hz, the second
one is a band pass filter, which has been constructed using two [onepole~] and subtracting the result of
each other. the first has its cut off located at 1500 Hz, and the signal coming out of this one is filtered
again at 400Hz, by using a [-~ ] which subtracts audio we can obtain a filter which ranges from 400Hz to
1500Hz. The last one is a high pass filter, it uses the same principle as the band pass, subtracting a 4000
Hz low passed signal to the main unfiltered one. The signal of each filter is sent to a three dimensional
shape which we will see in the next sections.
The gain of this signals can be controlled by some rotary controls located in the interface section. The
author has chosen not to have variable filters to avoid an excess of controls, since the amount is already
quite big.
5.6.2 Rendering
Render section
The rendering section is the one in charge of showing the visual content on the screen. It function is
based on OpenGL which is covered in the appendix B for better understanding. This section contains
some advanced graphic concepts.
When looking inside this patch we will see various object with many arguments inside. The first and most
important one is the one called [jit.gl.render edit] this object allows 3D to be drawn (rendered) into a win
Page 27
dow called edit. All the 3D objects named edit will use this renderer to be drawn. This object need a constant pulse to keep drawing and a message called erase, so the screen is erased every time is drawn,
this is called refresh. This is achieved by using a [qmetro 20] which sends bangs every 20 milliseconds,
and those bangs go to [ t b b b erase] which trigers, first an erase message for the rendere, and then 3
bangs. One of this bangs goes to the renderer and the other two are used to activate other sections of
the patch by using a send object called [s draw], There is also a second renderer called preview for the
preview screen. The way the program renders the same thing simultaneously in two different screens is
quite complex and requires to look at the sub patch named render master.
Inside Master Patch
The render master sub-patch is based in OpenGL programming. Most of the functionality of this library
can be performed by Max using [jit.gl.sketch] this sub-patch uses this object to draw all the GL objects,
by calling their names following the message drawobject, it also has an attribute called capture sc_tex
which, as its name suggests, captures the drawn content into a texture called “sc_tex” . If we now look at
the master patch, we will see an object called [jit.gl.texture edit @name sc_text] which works as the destination for that captured texture. Next to it are two video plane object which create a plane for playing
videos within the OpenGL environment, both objects have sc_tex assigned as their texture, which means
that they will show that as a content , the main difference between them is the render destination, one of
them goes to the window called edit, and the other goes to preview.
Page 28
Inside Render Master
Back to the Master sub-patch, we also find a [jit.window] called edit, this is the famous destination for
one of the renderers, this object creates a separate window called edit, it has some messages such as
full scree , floating or visible. Full screen’s function should be easy to guess, floating means that the window will stay on top of all other windows so it does not get hidden, and the last one visible, activates and
deactivates it.
The last important object that we find in this sub-patch is called [jit.gl.handle] this object lets the user interact with the mouse in order to control the position and rotation of the OpenGL shapes. Its main argument is the render destination, in this case the window called preview, this means that by using the
mouse on that window, one can interact directly with the shapes. The output of jit.gl.handle goes into a
gate objet that allows the user to choose what shape to control.
the last uncovered object is jit.gl.texture, an object that simply assigns a texture to the shapes. This is just
an aesthetic possibility, not relevant for the core function of the application.
5.6.3 Shapes
Inside ShapesLow sub-Patch
Page 29
Within the main patch window, we find three sub-patches called ShapesLow, ShapesMid, and ShapesHigh, respectively. These patches are the most important ones of the application since they englobe the
graphic content plus the interpretation of the sound data gathered at the audio section. All three are effectively the same, the only difference between them is that each one receives the audio value from a
band of EQ.
The principle behind this patch is that the audio data is transformed to matrix data in order that Jitter can
interpret it correctly, then, this matrix data is used to fill the content of a 3D polygon by using the gathered
data as coordinates in space.
In order to explain this part, the patch must be divided into two sections, the 3D polygon and the Audioto-matrix data conversion.
Note that the concepts described relate to how Jitter interprets matrices, something that is explained in
detail in appendix A.
• Audio-to-Matrix Data Conversion: While learning about Max, and specially Jitter, one discovers that
there are several object to transform MPS (audio) data into a series of jitter matrices. One of this objects
is jit.catch . We use this object to fill a one plane matrix of 64 by 64 pixels. This matrix is then passed
through an jit.op object, which performs a mathematical operation, in this case a multiplication, and the
value to multiply by by is our amount variable. The result of this operation goes to a gate, that routes it
to three different outputs, each of them corresponding to each dimension on three dimensional space
(x, y , z). In order to achieve this, we use jit.pack, with an argument corresponding to the destination
matrix. This object will send the received data to each plane of the geom matrix , that is, the destination.
• Polygon Section: The second part of this patch contains a 3D matrix of 2X2X2 , this is achieved by
sending the coordinates of the vertex through a jit.fill message to an empty matrix. The destination matrix is called geom which corresponds with the output name of the jit.pack object and it has got an argument saying “interp 1”. This activates the interpolation, which means that the values stores will be
substituted for any incoming new ones. Since the values are substituted we have to make sure that
every time we change the dimension to place the Audio-To-Matrix input, the original geomtry vertex are
sent again, otherwise, the shape will be transformed into something abstract every time one changes
the dimension. In order to view the geometry data as 3D we need a 3D object part of the OpenGl library, that is [jit.gl.nurbs ] this object accepts a message called ctlmatrix which means control matrix,
this allows to use matrix data as vertex position to create 3D geometry. Note that this object is also being sent as matrix data by using the matrixoutput argument, this is only to use [jit.gl.mesh] to create the
Page 30
geometry instead of [jit.gl.nurbs]. The only reason behind this is that [jit.gl.mesh] allows different drawing
modes, like points, lines or quads as an example. This is only due to the aesthetic needs of the project.
5.7 The Interface
The interface was a crucial part of this project. It was necessary for the application in order to be useful to
have an intuitive interface that lead the user to a good comprehension in just a quick look. This as well, is
one of the advantages of Max/MSP over its competitors, it lets you create an interface, hiding all the programming data, making it much easier for non technical users.
The interface is divided into 7 sections, separated of each other by means of colour boxes. The first one
is the screen. The second one is called Screen Controls , this section comprises all the material treated in
the render section, the set of controls is limited to the very minimum necessary. The third one is called
Mouse Handling System, this is a link to [jit.gl.handle]’s selection system, that allows the user to choose
what shape to control by the means of the mouse.
Fourth section comprises the audio controls, where the user can select between file or audio input, open
the audio preferences, change the output gain, and more important, have control over the gain of each
frequency band.
The Fifth, Sixth and Seventh sections are the controls for each 3D polygon. All three are pretty much the
same providing controls for many parameters of those shapes. Some of those parameters can also be
controlled by audio, that is achieved by using an exponential multiple of the amplitude as the value for
certain parameters such as point line, and line width, this can be switched on and off via a toggle button,
with also the option between linear and exponential multiplication of the amplitude.
The controls selected for these shape are pretty basic, but by exploring them, one can get some interesting results. The author would like to avoid a thorough discussion of all the parameters and prefers the
user to play around with them and find out by himself.
Page 31
Finished Interface
5.8 Known Problems and Bugs
Although quite successful, the application is not perfect and it contains some operational mistakes, as
well as some graphical ones.
In the graphic domain, the application has been made in a fifteen inch screen at high resolution, and the
finished application takes most of the screen space. This is a problem when loaded into smaller display
computers since the whole interface does not fit and makes its use a little less comfortable.
Issues known in the functionality are a few: In the Mouse Handling System, if reset is not pressed in the
beginning, the shapes automatically disappear when the mouse click on top of them. This can be easily
resolved by pressing reset. Where this bug comes from is totally unknown for the author but it will be explored for further versions.
Some of the drawing methods of the 3D shapes do not work, for instance, triangles, tri_strip or quads .
5.9 Application Building
The finalised project comes in the for of a Macintosh standalone application. The Windows version has
not been possible to compile due to some incompatibilities in the code. The document also includes the
editable file to be opened inside Max/MSP 5 and a text file containing the code. The whole application
was realised using a 30 day demo version of Max during that period.
Page 32
6
CONCLUSION
Because of the high increment in power that computers have experience in the last few years, and will
continue experiencing in an exponential manner, we can say that we are in the best possible moment to
make visual music happen in its most pure and accurate way. Multimedia software grows faster and becomes more and more powerful and more accessible. As this project shows, creating customised audio/
visual applications can be achieved by individuals with no previous knowledge in the field. The possibilities are almost infinite, and only limited by our imagination.
The audio visual path can lead into some interesting new concepts such as the creation of new interfaces
for music creations, where the 3D graphic content can act as the interaction system between the machine and the user, providing with an intuitive set of controls. For instance, each shape could correspond
to a sound, and its size determine the amplitude and its rotation the frequency. The position in 3D space
could set the amount of output lever for each channel of audi, a very interesting concept for multi-channel
installations. If those fictitious 3D shapes that we are talking about behaved differently depending on the
type of sound they “emitted” the user could easily have an instant feedback on what they do and how to
manipulate them in order to achieve the desired results. This type of graphical approach will be radically
different from what we are used to but at the same time more familiar and intuitive due to its visual feedback. We are not far away from being able to create audio visual compositions without treating each element as a separate entity.
As an evaluation of this project, the author considers it very successful since it perfectly accomplishes the
desired function without much struggle. For further explorations of the same idea, multitrack visualisation
is the next logical step, and once again Max can act as great mediator specially when using its new extension Max for Live. Integrating the visual algorithms inside a familiar DAW is now completely possible.
MIDI could also help, but not when used as a trigger, rather when certain visually aesthetic parameters
Page 33
are coupled with audio effect controls in order to obtain a full symbiosis of both graphical and sonic expression.
As said before, audio visual composition is already here, it is just a matter of time for musicians or visual
artists to embrace this as a completely valid form of artistic expression.
As Henry Jenkins said in his book Convergence Culture : “Convergence is coming, and you better be
ready” (Jenkins, H. 2006, p. 10)
Page 34
A
APPENDIX A: Jitter Matrix System
Jitter, is the extension for graphics now included in the software package Max/MSP. In this appendix, the
matrix system that Jitter uses will be explained in detail since it is quite different from the rest.
A matrix is a grid, with each location containing some information. Horizontal lines are called rows and
vertical lines are called columns. Matrices can be multidimensional, in fact a matrix with rows and columns has two dimensions. Dimensions can go up to 32. The next example show a three dimensional
matrix of 3 by 3 by 3 having a total number of 27 cells.
Multidimensional matrix: 3x3x3 = 27 cells
(cycling ’74, 2004)
Page 35
A computer screen can be understood as a matrix, having rows and columns of pixels. for example,
800x600. Jitter represents video using two dimensional matrices were each cell contains four values, Alpha, Red, Green and Blue (ARGB) in a scale of 0 to 255. ( 8 bit colour). In order to treat each colour value
differently, Jitter introduces the concept of a plane.
Each plane is a two dimensional matrix containing one value per cell, so a video matrix will have four
planes, each of them representing a colour.
Colour Planes (cycling ’74, 2004)
The type of data that a jitter matrix can hold is char, 8 bit integer numbers, float32, 32 bit floating point
numbers, and float64, 64 bit floating point numbers.
The number of planes is defined as an argument followed by the data type i. e. [jit.matrix 4 char 320 240 ]
for a 320 by 240 matrix containing four planes and 8 bit integers on each cell.
Page 36
B
APPENDIX B: Open GL
OpenGL (Open graphics library) is a standard library which specifies a virtual machine for turning lists of
numbers into 2D images. The list may contain the 3D locations of vertices to connect lines or polygons.
Other data can be contained such as coordinates for texture mapping, normals, lighting colours and
some other things.
Geometry
Geometry matrices must contain data in float32 or float64 . Each cell of the matrix represents one vertex,
the image is created by drawing connections between those vertices. The way the shapes are drawn is
set by the drawing modes:
points: Draws each vertex as a point
lines: Connects every pair of vertices with a line.
line_strip: similar to lines.
line_loop: like line_strip but a line segment is drawn connecting the last vertex in the ,matrix back to the
first one.
triangles: draws unconnected triangles.
tri_strip: Draws a strip of connected triangles. Important to know the order of the vertices.
tri_fan: Draws a fan of triangles
quads: Draws unconnected quadrilaterals
Page 37
quad_strip: draws connected quadrilaterals.
polygon: Draws a single polygon using all the vertices in a matrix, if the polygon is not simple and convex
the results are undefined.
tri_grid: If the geometry matrix has two dimensions, triangles are drawn which link each vertex to its
neighbours in the matrix to form a surface
quad_grid: If the geometry matrix has two dimensions, quadrilaterals are drawn which link each vertex to
its neighbours in the matrix to form a surface.
Planes
Geometry matrices can have up to 13 planes of data:
plane 0: x coordinate of vertex
plane 1: y coordinate of vertex
plane 2: z coordinate of vertex
plane 3: s coordinate of texture at vertex, if textured
plane 4: t coordinate of texture at vertex, if textured.
plane 5: x component of lighting normal at vertex.
plane 6: y component of lighting normal at vertex.
plane 7: z component of lighting normal at vertex.
plane 8: red component of vertex colour.
plane 9: Green component of vertex colour.
plane 10: Blue component of vertex colour.
plane 11: alpha component of vertex colour.
plane 12” edge flag for connection to next vertex: if < 0, no line is drawn.
Page 38
Page 39
R&B
References and Bibliography
Books
- Blum, F. 2007, Digital Interactive Installations, 1st edition, VDM Verlag Dr. Muller,
Saarbrucken.
- Duchting, H. 2007, Kandinsky, 1st edition, Taschen , Cologne.
- Fiedler, J. 2006, Bauhaus, 1st edition, Konemann, Bonn.
- Noble, J. 2009, Programming Interactivity, 1st edition, O"Reilly, Sebastopol, CA.
- Jaeger, T. 2005, Live Cinema Unravelled, 1st edition, Self Published.
- Jenkins, H. 2006, Convergence Culture, 1st edition, New York University Press,
New York.
- Jenny, H. 2001, Cymatics - A Study of Wave Phenomena and Vibration, 1st edition,
Macromedia Press,
- Kirk and Hunt, R and A. 1999, Digital Sound Processing for Music and Multimedia,
1st edition, Focal Press, Oxford.
Page 40
- Loy, G. 2006, Musimathics: The Mathematical Foundations of Music, 1st edition,
MIT Press, Cambridge, MA.
- Lund, C and H. 2008, Audio.Visual - On Visual Music and Related Media, 1st edition, Arnoldsche, Germany.
- Martin, S. 2006, Videoart, 1st edition, Taschen, Cologne.
- Noble, J. 2009, Programming Interactivity, 1st edition, O'Reilly, Sebastopol, CA.
- Roads, C. 1996, The Computer Music Tutorial, 1st edition, MIT Press, Cambridge,
MA.
- Roads, C. 2004, Microsound, 1st edition, MIT Press, Cambridge, MA.
- Schiffman, D. 2008, Learning Processing - A Begginer's Guide to Programming
Languages, Animation,and Interaction, 1st edition, Morgan Kauffmann, Burlinton,
MA.
- Sloane, P. 1989, Visual Nature of Color, 1st edition, Design Press, New York.
- Stephenson, G. 1965, Intro to Matrices, Sets and Groups For Science Students, 1st
edition, Dover Publications, Dover, NY.
- Temperley, D. 2001, The Cognition of Basic Musical Structures, 1st edition, MIT
Press, Cambridge, MA.
- Terzidis, K. 2009, Algorithms for Visual Design Using the Processing Language, 1st
edition, Wiley Publishing, Indianapolis, IN.
- Vince, J. 2008, Geometric Algebra for Computer Graphics, 1st edition, SpringerVerlag, New York.
Page 41
- Vnce, J. 2010, Mathematics for Computer Graphics , 3rd edition, Springer , New
York.
- Winkler, T. 1998, Composing Interactive Music: Techniques and Ideas Using Max,
1st edition, MIT Press, Cambridge,MA.
- Youngblood, G. 1970, Expanded Cinema, 1st edition, E P Dutton,
Websites
- Beyst, S. 2001, Visual Music [online], d-sites, Available from:
http://d-sites.net/english/visual%20music.htm [Accessed: 02.10.2010]
- Cycling 74 2010, Max/MSP Documentation [online], Cycling 74, Available from:
http://www.cycling74.com/docs/max5/vignettes/intro/docintro.html [Accessed:
2.10.2010]
- Firth, I. 2007, Music and Colour: A New Approach to the Relationship [online], Music
and Colour , Available from: http://www.musicandcolour.net [Accessed: 20.09.2010]
- Friedlander, p. 1998, Visual Music [online], Paul friedlander, Available from:
http://www.paulfriedlander.com/text/visualmusic.html [Accessed: 20.09.2010]
- Lew, M. 2004, live Cinema: An Instrument for Editing as a Live Performance [online], Massachussets Institute of Technology, Available from:
http://www.musicandcolour.net [Accessed: 4.10.2010]
- McDonell, M. 2002, Visual Music [online], Eircom, Available from:
http://homepage.tinet.ie/~musima/visualmusic/visualmusic.htm [Accessed:
20.09.2010]
Page 42
- 2004, Center for Visual Music [online], Center for Visual Music, Available from:
http://www.centerforvisualmusic.org/ [Accessed: 4.10.2010]
- Vade 2010, Vade [online], Vade, Available from: http://abstrakt.vade.info/ [Accessed: 4.10.2010]
Theses and Dissertations
- Dekker, A. n.d., Synaesthetic Performance In the Club Scene, MA, Netherlands
Media Art Institute.
- Makela, M. 2006, Live Cinema: Language and Elements, MA, Helsinki University of
Art and Design.
- Lyons, A. 1995, Time Space Texture: An Approach to Audio-Visual Composition,
PhD, The University of Sydney
Articles
- Bainbridge, B. 1893, A Souvenir of the Color Organ, With Some Suggestios in Regard to the Soul of the Rainbow and the Harmony of Light. [online], Rhythmic Light,
Available from: www.rythmiclight.com [Accessed: 4.10.2010]
- Jones and Neville, R. and B. 205, Creating Visual Music in Jitter: Approaches and
Techniques, Computer Music Journal, Cambridge, MA., vol. 29, Issue 4. , pp. 54-70
- Wright 1923, The Future of Painting [online], Rhythmic Light, Available from:
www.rythmiclight.com [Accessed: 4.10.2010]
Page 43
Pictures
- Bower, F. 1995, Scriabin's Relationship Between Colour Tones and Music p.114.
- Kandinsky, W. Several Circles 1926 [online], Available from:
http://pippastephens.wordpress.com/2009/12/07/music-to-my-eyes/ [Accessed:
28.10.2010]
- Kurokawa, R. Live Mutek 2005 [online], Available from:
http://dronesandpulses.blogspot.com/2008/12/ryoichi-kurokawa.html [Accessed:
28.10.2010]
Page 44