Slides

The Beauty of the Beast:
Semantic Audio
Stephan Baumann
Competence Center Computational Culture
German Research Center for Artificial Intelligence, Kaiserslautern, Germany
Story #1
1)
2)
3)
4)
5)
6)
7)
8)
9)
10)
11)
12)
13)
14)
15)
16)
17)
18)
19)
20)
21)
2002: my first ISMIR at IRCAM!
2003: invited MIR Research at IRCAM
Buying Hotel Costes Vol.1 at FNAC
2005: Buying Hotel Costes Vol.8
The Think Twice Tune of Ralph Myerz got me in ...
Buying Ralph Myerz at Amazon
Googling Ralph Myerz
Exploring the official Website
Spotting the next concert and venue
Checking friends in Oslo
Booking tickets
Booking airplane
2006: Visiting gig
Taking images
Uploading images to Flickr with tags
Writing a blog posting
Writing a forum posting with link
Norwegian and belgian traffic comes in
Receiving an email of „el presidente“
Chatting about fandom
Receiving the exclusive promo DVD
Transition
4 years!
Semantics
1)
2)
3)
4)
5)
6)
7)
8)
9)
10)
11)
12)
13)
14)
15)
16)
17)
18)
19)
20)
21)
2002: my first ISMIR at IRCAM!
2003: invited MIR Research at IRCAM
Buying Hotel Costes Vol.1 at FNAC
2005: Buying Hotel Costes Vol.8
The Think Twice Tune of Ralph Myerz got me in ...
Buying Ralph Myerz at Amazon
Googling Ralph Myerz
Exploring the official Website
CONTEXT
Spotting the next concert and venue
•
Place
Checking friends in Oslo
•
Time
Booking tickets
•
Mood
Booking airplane
•
Listening Mode
2006: Visiting gig
•
Activity
Taking images
– Hear
– Watch
Uploading images to Flickr with tags
– Record
Writing a blog posting
– Modify
Writing a forum posting with link
– Interact
Norwegian and belgian traffic comes in
– ...
Receiving an email of „el presidente“
• ...
Chatting about fandom
Receiving the exclusive promo DVD
EDITORIAL
•
Title
•
Singer
•
Performer
•
Composer
•
Lyricist
•
Bandmembers
•
Producer
•
...
CONTENT
•
Tempo
•
Metric
•
Lyrics
•
Melody
•
Tonal
•
Genre
•
Harmonic Progression
•
Chord-Options
•
Orchestration
•
Instruments
•
Type of recording
•
Studio
•
...
[Paul Lamere]
[Oscar Celma]
[Oscar Celma]
[Oscar Celma]
Story #2
•
•
•
•
•
•
•
•
•
•
•
•
•
1972 Organ lessons, experiences with music notation and fun
1977 - 1981 Self-taught piano player
1982 - 1993 Cover bands, Soul Funk Jazz, Level42, Chic,
Commodores
1994 Frustration, the keyboard player as a sideman
phenomenom
1997 - 2005 Several projects, HipHop, Lounge, Jazz, leisure
time only
2005 Bass lessons
The Sting 57 Precision Re-Issue
Buying a used bass at Ebay
Annual Music School Concert for the proud parents/kids
Leadsheet „Nur ein Wort“ / Wir-sind-Helden
Buying the CD
Playing a cover version live
2006 Meeting: Wir-sind-Helden at an openhouse event
Transition
34 years!
Semantics
•
•
•
•
•
•
•
•
•
•
•
•
•
1972 Organ lessons, experiences with
music notation and fun
1977 - 1981 Self-taught piano player
1982 - 1993 Cover bands, Soul Funk
Jazz, Level42, Chic, Commodores
1994 Frustration, the keyboard player
as a sidekick phenomenom
1997 - 2005 Several projects, HipHop,
Lounge, Jazz, leisure time only
2005 Bass lessons
The Sting 57 Precision Re-Issue
Buying a used bass at EBAY
Annual Music School Concert for the
proud parents/kids
Leadsheet „Nur ein Wort“ / Wir sind
Helden
Buying the CD
Playing a cover version live
2006 Meeting: Wir sind Helden at an
openhouse event
CONTEXT
•
Place
•
Time
•
Mood
•
Listening Mode
•
Activity
– Hear
– Watch
– Record
– Modify
– Interact
– ...
•
...
EDITORIAL
•
Title
•
Singer
•
Performer
•
Composer
•
Lyricist
•
Bandmembers
•
Producer
•
...
CONTENT
•
Tempo
•
Metric
•
Lyrics
•
Melody
•
Tonal
•
Genre
•
Harmonic Progression
•
Chord-Options
•
Orchestration
•
Instruments
•
Type of recording
•
Studio
•
...
Taste: Snippets of my archive
•
Präludium 1 [J.S. Bach] / Köln Concert [Keith Jarrett]
•
Good Times [Chic] / Rappers Delight [Sugarhill Gang] /
MadeItBack,GoodTimesMix [Beverly Knight]
•
My Funny Valentine, Fabulous Baker Boys Soundtrack [M.Pfeiffer] /
My Funny Valentine [Big Muff]
•
I have seen [Zero7] / La femme d‘argent [Air]
•
Lady [Modjo] / Lady Acoustic Version [Modjo]
•
Eisbär [Nouvelle Vague]
•
A song for sorry angel [Franz Ferdinand, Jane Birkin, S. Gainsbourg]
Taste: Snippets of my archive
•
Le smou A-cappella [Fantastischen 4] / Le smou Instr. [Fantastischen 4]
/ Le smou [Fantastischen 4]
•
Mo money mo problems [Notorious B.I.G] / I‘m coming out [Diana Ross]
•
Roxanne Live 1991 [Sting] / Roxanne Live 2001 [Sting]
•
Jesus Christ Superstar [James Taylor Quartet]
•
Fly like an eagle [Seal, Steve Miller]
•
Imagine Live [Randy Crawford + Yellowjackets]
•
Think Twice [Ralph Myerz and the Jack Herren Band, A special album] /
Think Twice [Ralph Myerz and the Jack Herren Band, Hotel Costes 7]
Interaction drives MIR,LSAS!
•
•
•
•
•
Multimodal queries
Identifying versions
Searching for lyrics
Searching for sheets
...
lean forward
•
•
•
•
•
•
•
Playlists
Collaging
Concerts
RSS-pushes
...
betwixt
Performance: „playing the keys, bass“
Active mashing: „Webcam Karaoke“
Stephan Baumann - Competence Center Computational Culture - LSAS2006
lean backward
LSAS 2006
... Mid-level ... Timbre ... Identification ... Logscale Modulation Frequency Coefficient
...Tempo Feature ... Emotion ... Classification ...
Timbre ... Instruments ... Classification ...
Automatically Describing ... Map ... Fuzzy Logic
... Semantic Description of Music ... Automatic
Extraction ... Structure ... Pitch Class
Distribution ... Browsing inside ... Track ... Case
Study ... Music Warehouses ... Next Generation
... Music Search Engines ... Real-World ...
Sound ... Recognition ... Adapting ... Structure
... Audio Similarity Spaces ...Index-based
Similarity Searching ... Query-by-Content ...
Audio Retrieval ...
Stephan Baumann - Competence Center Computational Culture - LSAS2006
Audio -> Music?
•
•
•
Implementation and Evaluation of a LowPower Sound-Based User Activity
Recognition System, [M. Stäger, P.
Lukowicz, G. Tröster] Proceedings of
the 8th International Symposium on
Wearable Computers, ISWC 2004
Analysis of Chewing Sounds for Dietary
Monitoring [O. Amft, M. Stäger, P.
Lukowicz, G. Tröster] UbiComp 2005
Estimating the Number of Marine
Mammals using Recordings of Clicks from
One Microphone, [X. Halkias and D. Ellis]
ICASSP-2006
Stephan Baumann - Competence Center Computational Culture - LSAS2006
ISMIR 2002-2006: Gossip
•
•
•
•
•
•
•
•
„... forget MIDI...“
2002 [Whitman,Berenzweig]
„ ...P2P, imagine the options and power!“
2002 [Tzanetakis]
„ ...audio similarity, genre classification?... it is done“
2004 [Aucouturier], „...NO!“2004 [Peeters]
„ ... webtext and cultural metadata are hot..“
2004 []
„... awesome melody extraction but the patent ..“
2005 [IDMT]
„ ... cultural metadata is so mainstream ..“
2005 []
„ ... Semantic Web guys are still sleeping.., Web2.0 is taking off...“
2005 [Celma]
„ ... chroma seems to be en-vogue?!...the MFCC trap ...“
2006 [Peeters]
Stephan Baumann - Competence Center Computational Culture - LSAS2006
ISMIR 2002-2005: Snippets
•
•
•
•
Transition from MIDI to AUDIO
MFCC hype
MFCC glass ceiling
Using text - Cultural Metadata
– from the web: label sites, fan sites, etc.
– co-occurences in radio program lists
– Playlists
• Source Separation
– PCA, ICA
• Machine Learning (Yale, Weka, ...)
• etc. etc. etc.
Stephan Baumann - Competence Center Computational Culture - LSAS2006
Stephan Baumann - Competence Center Computational Culture - LSAS2006
ISMIR 2006
•
Specialized vertical work co-exists in peace with holistic and application-oriented
approaches (keyword-counting in a corpus of the proceedings)
Stephan Baumann - Competence Center Computational Culture - LSAS2006
[Extract of final MIREX poster, Stephen Downie et. al]
Stephan Baumann - Competence Center Computational Culture - LSAS2006
Europe: Snippets
• European scene, FP6 projects on MIR
– SemanticHiFi
– SIMAC
• UPF, Sony CSL, IRCAM, ÖFAI, QMU, Digital
Music Centre, IDMT, DFKI, ...
• Major Steps: ML, multimodal, Ph.Ds, transfer
of people, spinoffs
• Pampalk‘s Ph.D list of 2006
• Commercial: Last.fm (UK, Austria, Germany)
Stephan Baumann - Competence Center Computational Culture - LSAS2006
United States: Snippets
•
•
•
•
•
•
•
•
Basic research at MIT medialab (Ellis, Whitman, Berenzweig, Smaragdis)
FX pal (J. Foote)
HP Labs (Beth Logan)
Princeton, CMU, Moodlogic (George Tzanetakis)
Transfer of people and know-how to Google, MERL, startups ...
SUN entered the MIR game (Paul Lamere)
– 3D space for interactive MIR
– http://blogs.sun.com/plamere
– www.snappradio.com
Commercial: Pandora (and more than two dozen Web2.0 startups)
Latest trends (thanks to Paul‘s watchblog)
– Foosic: http://www.foosic.org
– OWL: http://www.owlmm.org
– Echonest: http://echonest.com (Brian Whitman)
Stephan Baumann - Competence Center Computational Culture - LSAS2006
A
Pandora: Music Genome
* Abstract Lyrics
* Accordion Playing
* Acousti-Lectric Sonority
* Acousti-Synthetic Sonority
* Acoustic Bass Solo
* Acoustic Drum Samples
* Acoustic Guitar Accompaniment
* Acoustic Guitar Layering
* Acoustic Piano Accompaniment
* Acoustic Rhythm Guitars
* Acoustic Rhythm Piano
* Acoustic Rock Instrumentation
* Acoustic Sonority
* Afro-Cuban Influences
W
* Aggressive Drumming
* Wah-Wah Guitar
* Aggressive Female Vocalist
* Well-Articulated Acoustic Guitar Solo
* Aggressive Male Vocalist
*
Well-Articulated Alto Sax Solo
* Altered Female Vocal
* Well-Articulated Electric Guitar Solo
* Altered Male Vocal
* Well-Articulated Piano Solo
* Altered Piano Timbres
* Well-Articulated Tenor Sax Solo
* Altered Vocal Sound
* Well-Articulated Trombone Solo
* Ambient Soundscapes
* Well-Articulated Trumpet Solo
* Ambiguous Lyrics
* West Coast Rap Roots
* Angry Lyrics
* Western Classical Influences
* Angular Melodies
* Wet Recording Sound
* Wet Snare
* Atmospheric Production
* World Music Influences
* Avant-garde Leanings
Stephan Baumann - Competence Center Computational Culture - LSAS2006
Future of Semantic Tagging?
• ESP Game [Luis von Ahn]
Stephan Baumann - Competence Center Computational Culture - LSAS2006
Stephan Baumann - Competence Center Computational Culture - Tag der Informatik, 17.11.2006
Space, Time, Content
HEAR, CONDENSE, CREATE
• Information management
• Identity management
• Relationship management
TASTE
• Music listening
• Musicians
Stephan Baumann - Competence Center Computational Culture - LSAS2006
Self promotion: BluetunA
Self promotion: BluetunA
Stephan Baumann - Competence Center Computational Culture - LSAS2006
Stephan Baumann - Competence Center Computational Culture - LSAS2006
Mainstream vs. Niche
• You can sell hope!
... there is a business model ...
– The New Yorker, 2006: The Formula
(e.g. PolyphonicHMI->Platinum Blue)
– Composition of webservices
(e.g. http://foafing-the-music.iua.upf.edu)
– a lot of recommendation startups ...
• So there is/will be funding :)
Stephan Baumann - Competence Center Computational Culture - LSAS2006
To read? .. Beyond audio but ...
www.computationalculture.org
www.hardbloggingscientists.de
Stephan Baumann
Thanks to all the contributors ...