Beethoven, Bach und Billionen Bytes Thomas Duis Meinard Müller

Transcription

Beethoven, Bach und Billionen Bytes
Musik trifft Informatik
Meinard Müller
Universität des Saarlandes
und MPI Informatik
Thomas Duis
Hochschule für Musik
Saar
Wissenschaftsmatinée
November 2010
Thomas Duis
Konzertpianist
1998 Professor HfM Saar
2004 Rektor HfM Saar
Zahlreiche CD-Produktionen
Meinard Müller
2007 Habilitation
Bonn, Germany
2007 MPI Informatik
Saarland, Germany
2010: Mozart
mit Mannheimer Streichquartett
2010: George Enescu (Violine/Viola)
mit Laurent Albrecht Breuninger
Multimodal Music Processing
Sheet Music (Image)
MIDI
Singing / Voice (Audio)
MusicXML (Text)
Music
Senior Researcher
Multimedia Information Retrieval
& Music Processing
Music Literature (Text)
Music Film (Video)
5 PhD Students
Research Goals
Music Information Retrieval (MIR) → ISMIR
Analysis of music signals
(harmonic, melodic, rhythmic, motivic aspects)
Design of musically relevant audio features
Tools for multimodal search and interaction
CD / MP3 (Audio)
Piano Roll Representation
Dance / Motion (Mocap)
Piano Roll Representation
Player Piano (1900)
Player Piano (1900)
Player Piano (2000)
Piano Roll Representation (MIDI)
J.S. Bach, C-Major Fuge
(Well Tempered Piano, BWV 846)
Time
Pitch
Query:
Query:
Goal: Find all occurrences of the query
Goal: Find all occurrences of the query
Matches:
Audio Data
Memory Requirements
Various interpretations – Beethoven‘s Fifth
1 Bit
=
1: on
0: off
1 Byte
=
8 Bits
Bernstein
1 Kilobyte (KB)
=
1 Thousand Bytes
Karajan
1 Megabyte (MB)
=
1 Million Bytes
1 Gigabyte (GB)
=
1 Billion Bytes
1 Terabyte (TB)
=
1000 Billion Bytes
Scherbakov (piano)
MIDI (piano)
Memory Requirements
Music Synchronization: Audio-Audio
12.000 MIDI files
<
350 MB
One audio CD
≃
650 MB
Two audio CDs
>
1 Billion Bytes
1000 audio CDs
≃
Billions of Bytes
Beethoven‘s Fifth
Karajan
Scherbakov
Beethoven‘s Fifth
Feature extraction: chroma features
Karajan
Karajan
Scherbakov
Scherbakov
Synchronization: Karajan → Scherbakov
Time (seconds)
Time (seconds)
Time (seconds)
Time (seconds)
Cost matrix
Cost-minimizing alignment path
Karajan
Karajan
Scherbakov
Scherbakov
Application: Interpretation Switcher
Music Synchronization: MIDI-Audio
Music Synchronization: MIDI-Audio
Music Synchronization: Scan-Audio
MIDI = metadata
Automated annotation
Audio recording
Sonification of annotations
Scanned Sheet Music
Symbolic Note Events
Scanned Sheet Music
OMR
Correspondence
Correspondence
Audio Recording
Audio Recording
Scanned Sheet Music
High
Qualtity
OMR
Correspondence
Scanned Sheet Music
OMR
Correspondence
Audio Recording
Application: Score Viewer
„Dirty“
but hidden
High
Qualtity
Audio Recording
Music Processing
Coarse Level
Fine Level
What do different versions have in
common?
What are the characteristics of a
specific version?
Music Processing
Music Processing
Coarse Level
Fine Level
Coarse Level
Fine Level
common?
specific version?
common?
specific version?
What makes up a piece of music?
What makes music come alive?
Identify despite of differences
Identify the differences
Performance Analysis
Music Processing
Coarse Level
Fine Level
common?
specific version?
Identify despite of differences
Identify the differences
Example tasks:
Audio Matching
Cover Song Identification
Example tasks:
Tempo Estimation
1. Capture nuances regarding tempo, dynamics,
articulation, timbre, …
2. Discover commonalities between different
performances and derive general performance
rules
3. Characterize the style of a specific musician
(``Horowitz Factor´´)
Schumann: Träumerei
Score (reference):
Performance:
Performance:
Time (seconds)
Time (seconds)
Score (reference):
Score (reference):
Strategy: Compute score-audio synchronization
and derive tempo curve
Performance:
Tempo Curve:
Musical tempo (BPM)
Time (seconds)
Musical time (measures)
Score (reference):
What can be done if no reference is available?
Tempo Curves:
Musical tempo (BPM)
Musical tempo (BPM)
Tempo Curves:
Music Processing
Music Processing
Relative
Absolute
Relative
Absolute
Given: Several versions
Given: One version
Given: One version
Comparison of extracted
parameters
Direct interpretation of extracted
parameters
Music Processing
Music Processing
Relative
Absolute
Relative
Absolute
Given: One version
Given: One version
parameters
parameters
parameters
parameters
Extraction errors have often no
consequence on final result
Extraction errors immediately
become evident
Extraction errors have often no
consequence on final result
Extraction errors immediately
become evident
Example tasks:
Music Synchronization
Genre Classification
Example tasks:
Music Transcription
Tempo Estimation
Tempo Estimation and Beat Tracking
Measure
Tactus (beat)
Example 1: Queen – Another One Bites The Dust
Tatum (temporal atom)
Time (seconds)
Example 1: Queen – Another One Bites The Dust
Example 2: Chopin – Mazurka Op. 68-3
Pulse level: Quarter note
Tempo:
???
Time (seconds)
Example 2: Chopin – Mazurka Op. 68-3
Which temporal level?
Pulse level: Quarter note
Tempo:
Local tempo deviations
50-200 BPM
Sparse information
(e.g., only note onsets available)
Tempo (BPM)
Tempo curve
200
Vague information
(e.g., extracted note onsets corrupt)
50
Time (beats)
Local Energy Curve: Note Onset Positions
Energy
Energy
Local Energy Curve:
Time (seconds)
Time (seconds)
Spectrogram
Compressed Spectrogram
Steps:
Steps:
1. Spectrogram
2. Log Compression
Frequency (Hz)
Frequency (Hz)
1. Spectrogram
Time (seconds)
Time (seconds)
Steps:
Steps:
1. Spectrogram
2. Log Compression
3. Differentiation
1.
2.
3.
4.
Frequency (Hz)
Difference Spectrogram
Spectrogram
Log Compression
Differentiation
Accumulation
Novelty Curve
Time (seconds)
Time (seconds)
Novelty Curve
Local Average
Steps:
Steps:
1.
2.
3.
4.
1.
2.
3.
4.
5.
Spectrogram
Log Compression
Differentiation
Accumulation
Novelty Curve
Time (seconds)
Time (seconds)
Spectrogram
Log Compression
Differentiation
Accumulation
Normalization
Intensity
Intensity
Tempo (BPM)
Intensity
Tempo (BPM)
Tempo (BPM)
Intensity
Tempo (BPM)
Time (seconds)
Novelty Curve
Predominant Local Pulse (PLP)
Time (seconds)
Motivic Similarity
Motivic Similarity
Beethoven‘s Fifth (1st Mov.)
Motivic Similarity
Beethoven‘s Fifth (3rd Mov.)
Motivic Similarity
Beethoven‘s Fifth (3rd Mov.)
Thanks
Prof. Dr. Michael Clausen
(Bonn University)
Dipl.-Inform. Christian Fremerey
(Bonn University)
Dipl.-Inform. David Damm
(Bonn University)
Dipl.-Inform. Sebastian Ewert
(Bonn University)
Dipl.-Ing. Peter Grosche
(Saarland University)
Dipl.-Math. Verena Konz
(Saarland University)
PD Dr. Frank Kurth
(Fraunhofer-FKIE, Wachtberg )
Beethoven‘s Appassionata
Conclusions
Conclusions
Computer Sciene
Computer Sciene
Music
Information Retrieval
Information Retrieval
Music Analysis
Multimedia
Multimedia
Pattern Matching
Pattern Matching
Signal Processing
Signal Processing
Music Enjoyment
User Interfaces
User Interfaces
Music Education
Advanced Music Access
Conclusions
Computer Sciene
Conclusions
Music
Computer Sciene
Music

Beethoven, Bach und Billionen Bytes Thomas Duis Meinard Müller

Transcription

Similar documents

Lanxess Lustran Polymers

1. - DROPS

Treuchtlinger Kurier

Konstantin Scherbakov