Slides - Workshop on Functional Art, Music, Modeling and Design

Exploring Melody Space in a
Live Context Using Declarative
Functional Programming
FARM Workshop at ICFP 2014, Gothenburg
Thomas G. Kristensen, uSwitch Ltd, London
Composer is a simple, responsive and extensible system
utilising logic programming to allow novices to explore
and learn music rules
Background
Background
offline
online
Background
programmers
offline
online
musicians
Background
programmers
offline
online
musicians
Background
programmers
offline
online
Shasheela
musicians
Background
offline
online
programmers
musicians
Shasheela
FHarm
Background
programmers
musicians
offline
Shasheela
FHarm
online
Principled
approach
Background
programmers
musicians
offline
Shasheela
FHarm
online
Principled
approach
Graphical
score grammars
Background
programmers
musicians
offline
Shasheela
FHarm
online
Principled
approach
Graphical
score grammars
•Anders: Composing music by composing rules (Ph.D. thesis)
•Koops, Magalhãe and de Haas: A functional approach to automatic melody harmonisation
•Aaron, Blackwell, Hoadley and Regan: A principled approach to developing new languages for live coding
•Stead, Blackwell and Aarong: Graphic score grammars for end-users
Melody rules
C♯ D♯
• Tonic note
• Mode
• Cadence
F♯
G♯ A♯
C♯ D♯
3
C
1
2
D
E
F
7
4
5
6
G
A
B
8
C
D
E
Melody rules
C♯ D♯
• Tonic note
• Mode
• Cadence
F♯
G♯ A♯
C♯ D♯
3
C
1
2
D
E
F
7
4
5
6
G
A
B
8
C
D
E
Architecture
JVM
Instrument state
loop
OSC listener
Composer loop
OSC server
Overtone loop
SuperCollider
Architecture
JVM
Instrument state
loop
I
OSC listener
OSC server
Composer loop
II
a.
Overtone loop
IV
SuperCollider
III
V
Logic programming
(scaleo next-note scale-rest notes-rest)))
([note [0 . scale-rest] notes]
(fresh [next-note]
(semitone note next-note)
(scaleo next-note scale-rest notes))))
Logic programming
With this rule, it is possible for us to write a logic program to
generate C-major.
(run* [notes]
(scaleo :C3 major-scale notes)
(counto notes 8))
;; => ([:C3 :D3 :E3 :F3 :G3 :A3 :B3 :C4])
As there is no orientation on our computation, it is also possible for
us to go the other way, and re-create the binary pattern of a scale
given a concrete instance of a scale
(run* [tonic-note pattern]
(scaleo tonic-note pattern
[:C3 :D3 :E3 :F3 :G3 :A3 :B3 :C4]))
;; => ([:C3 (1 0 1 0 1 1 0 1 0 1 0 1 1 . _0)])
As a melody in C OMPOSER is just a permutation of a scale, with
the added restriction that the first and the last note must be the first
and last note of the scale, we can start generating melodies that
conform to the scale rule.
(run 3 [m1 m2 m3 m4 m5 m6 m7 m8]
(scaleo next-note
scale-rest
notes-rest)))
As there is no orientation
on our computation,
it is also
possible for
. scale-rest]
notes]
us([note
to go the[0other
way, and re-create
the binary pattern of a scale
[next-note]
given (fresh
a concrete
instance of a scale
(semitone note next-note)
(run* [tonic-note
pattern]
(scaleo
next-note scale-rest notes))))
Logic programming
(scaleo tonic-note pattern
:E3 :F3
:A3write
:B3 a
:C4]))
With this rule, [:C3
it is :D3
possible
for :G3
us to
logic
;; => ([:C3
(1 0 1 0 1 1 0 1 0 1 0 1 1 . _0)])
generate
C-major.
program to
As a melody
in C OMPOSER is just a permutation of a scale, with
(run*
[notes]
the added
restriction
the first and the
last note must be the first
(scaleo
:C3that
major-scale
notes)
and last(counto
note of the
scale,
we can start generating melodies that
notes
8))
conform
to the :D3
scale:E3
rule.:F3 :G3 :A3 :B3 :C4])
;;
=> ([:C3
(run 3 [m1 m2 m3 m4 m5 m6 m7 m8]
As there
is no[n1
orientation
ourn6computation,
it is also possible for
(fresh
n2 n3 n4onn5
n7 n8]
us to go the other
way,
and
re-create the binary pattern of a scale
(scaleo
:C3
major-scale
given a concrete instance
scale
[n1 of
n2a n3
n4 n5 n6 n7 n8])
(permuteo [m1 m2 m3 m4 m5 m6 m7 m8]
(run* [tonic-note pattern]
[n1 n2 n3 n4 n5 n6 n7 n8])
(scaleo
pattern
(==tonic-note
m1 :C3)
(==[:C3
m8 :C4)))
:D3 :E3 :F3 :G3 :A3 :B3 :C4]))
;; => ([:C3
:E31 :F3
;;
([:C3 :D3
(1 0
0 1 :G3
1 0:A3
1 0:B3
1 0:C4]
1 1 . _0)])
;;
[:C3 :E3 :D3 :F3 :G3 :A3 :B3 :C4]
;; a melody
[:C3 :F3
:E3 :G3is:A3
:C4])
As
in C:D3
OMPOSER
just:B3
a permutation
of a scale,
with
the
added restriction that the first and the last note must be the first
1 A few basic macros and functions from core.logic are used in the code
and
last note of the scale, we can start generating melodies that
examples presented. defne defines a goal function using pattern matching;
conform
to all
thesolutions
scale rule.
run* returns
to a system, possibly never halting; run returns
no more than a given number of solutions to a system; fresh introduces
(run
[m1 m2
m3variable.
m4 m5 For
m6a m7
m8]
a new 3
unbound
logic
more
detailed overview of core.logic
the di
was th
expre
melod
To
extens
as the
for th
betwe
progra
often
neede
It
interfa
an iOS
intera
for no
to the
send O
Af
chose
caden
in a l
interfa
with a
of the
chang
key th
The
definition
can
be done
recursively
over itthe
binary
sequence
(scaleo
next-note
scale-rest
notes-rest)))
As there
is no orientation
on our
computation,
is
also
possible
for
1
defining
scale,
as
demonstrated
here
. scale-rest]
notes]
us([note
to gothe
the[0
other
way,
and re-create
the:binary pattern of a scale
(fresh
[next-note]
given
a
concrete
instance of ascale
scale notes]
(defne scaleo [tonic-not
Logic programming
(semitone note next-note)
([note
[1 . scale-rest]
[note . ()]])
(run*
[tonic-note
pattern]
(scaleo
next-note scale-rest notes))))
([note
[1 . scale-rest]
[note . notes-rest]]
(scaleo
tonic-note pattern
[next-note]
[:C3
:E3 :F3
:A3write
:B3 a
:C4]))
With(fresh
this rule,
it is :D3
possible
for :G3
us to
logic program to
(semitone
;; => ([:C3
(1 0 1 0 1note
1 0 next-note)
1 0 1 0 1 1 . _0)])
generate
C-major.
(scaleo next-note scale-rest notes-rest)))
([note
[0 .in scale-rest]
As
a melody
C OMPOSER isnotes]
just a permutation of a scale, with
(run*
[notes]
(fresh
[next-note]
the added
restriction
the first and the
last note must be the first
(scaleo
:C3that
major-scale
notes)
note
next-note)
and last(counto
note (semitone
of the
scale,
we can
start generating melodies that
notes
8))
(scaleo
next-note scale-rest notes))))
conform
to the
scale:E3
rule.
;;
=> ([:C3
:D3
:F3 :G3 :A3 :B3 :C4])
With
rule,
possible
us to write a logic program to
(run this
3 [m1
m2 itm3is m4
m5 m6 for
m7 m8]
As there
is no[n1
orientation
ourn6computation,
it is also possible for
(fresh
n2 n3 n4onn5
n7 n8]
generate
C-major.
us to go the other
way,
and
re-create the binary pattern of a scale
(scaleo
:C3
major-scale
(run*
given [notes]
a concrete instance
scale
[n1 of
n2a n3
n4 n5 n6 n7 n8])
(scaleo(permuteo
:C3 major-scale
[m1 m2 m3notes)
m4 m5 m6 m7 m8]
(run*(counto
[tonic-note
notes pattern]
8))[n1 n2 n3 n4 n5 n6 n7 n8])
(scaleo
;; => ([:C3
:D3
:E3
:F3 :G3pattern
:A3 :B3 :C4])
(==tonic-note
m1 :C3)
(==[:C3
m8 :C4)))
:D3 :E3 :F3 :G3 :A3 :B3 :C4]))
As
is no orientation
our
possible for
;;there
:D3
:E31 :F3
:A3
;;
=> ([:C3
([:C3
(1 0
0on
1 :G3
1 0computation,
1 0:B3
1 0:C4]
1 it1 is. also
_0)])
[:C3other
:E3 way,
:D3 :F3
:A3 :B3
:C4] pattern of a scale
us;;to go the
and :G3
re-create
the binary
;; a amelody
[:C3 :F3
:E3
:C4])
As
ininstance
C:D3
OMPOSER
is:A3
just:B3
a permutation
of a scale, with
given
concrete
of a:G3
scale
the
added restriction that the first and the last note must be the first
1
(run*
pattern]
A few[tonic-note
basic macros and
functions from core.logic are used in the code
and
last
note
of
the
scale,
wea goal
canfunction
start generating
melodies that
(scaleo
tonic-note
pattern
examples
presented.
defne defines
using pattern matching;
conform
to all
thesolutions
scale :D3
rule.
run* returns
to a:E3
system,
never
halting;
run returns
[:C3
:F3possibly
:G3 :A3
:B3
:C4]))
more
than a(1
given
fresh introduces
;;no =>
([:C3
0 1number
0 1 1of0solutions
1 0 1 to
0 a1 system;
1 . _0)])
(run
3
[m1
m2
m3
m4
m5
m6
m7
m8]
a new unbound logic variable. For a more detailed overview of core.logic
the diG
was
ingth
expre
ram
melod
in a
ofToc
extens
char
as the
for th
3.2
betwe
progra
Muc
often
perf
neede
prog
It
cert
interfa
on b
antoiOS
a
intera
by a
forsect
no
to the
up b
send O
them
Af
the
chose
was
caden
inexpr
al
mel
interfa
with T
a
ofexte
the
as t
chang
fortht
key
Logic programming
(ns composer.composer
(:refer-clojure :exclude [==])
(:require [clojure.core.async :refer [go >! <!]]
[clojure.core.logic :refer :all]
[clojure.core.logic.pldb :refer :all]))
(defn scale-from-tones [tone-types]
(take 25
(->> tone-types
(map {:semitone [1]
:tone [0 1]
:minor-third [0 0 1]})
flatten
butlast
(cons 1)
cycle)))
(def major-scale
(scale-from-tones
[:tone :tone :semitone
(def harmonic-minor-scale
(scale-from-tones
[:tone :semitone :tone
(def natural-minor-scale
(scale-from-tones
[:tone :semitone :tone
(def locrian-mode
(scale-from-tones
[:semitone :tone :tone
(def mixolydian-mode
(scale-from-tones
[:tone :tone :semitone
(def scale-modes
[[:major-scale
[:harmonic-minor-scale
[:natural-minor-scale
[:locrian-mode
[:mixolydian-mode
(defne scaleo [base-note scale notes]
([note [1 . scale-rest] [note . ()]])
([note [1 . scale-rest] [note . notes-rest]]
(fresh [next-note]
(semitone note next-note)
(scaleo next-note scale-rest notes-rest)))
([note [0 . scale-rest] notes]
(fresh [next-note]
(semitone note next-note)
(scaleo next-note scale-rest notes))))
(defn key-restriction
[instrument-state s1]
(if-let [key (:key instrument-state)]
(all (== key s1))
succeed))
(defn compositions
[instrument-state & [n]]
(with-db
semitone-facts
(if n
(run n [melody2]
(logic-program instrument-state melody2))
(run* [melody2]
(logic-program instrument-state melody2)))))
(defn- random-composition
[instrument-state]
(rand-nth
(or (seq (compositions instrument-state 1024))
[[]])))
;; Loop
:tone :tone :tone :semitone]))
:tone :semitone :minor-third :semitone]))
:tone :semitone :tone :tone]))
:semitone :tone :tone :tone]))
:tone :tone :semitone :tone]))
major-scale]
harmonic-minor-scale]
natural-minor-scale]
locrian-mode]
mixolydian-mode]])
(db-rel semitone note-1 note-2)
(def keys-from-c
[:C3 :C#3 :D3 :D#3 :E3 :F3 :F#3 :G3 :G#3 :A3 :A#3 :B3
:C4 :C#4 :D4 :D#4 :E4 :F4 :F#4 :G4 :G#4 :A4 :A#4 :B4
:C5])
(def semitone-facts
(reduce
(fn [db [note-1 note-2]]
(db-fact db semitone note-1 note-2))
empty-db
(partition 2 1 keys-from-c)))
(defn scale-restriction
[instrument-state scale-type]
(if (:scale instrument-state)
(all (membero [(:scale instrument-state) scale-type] scale-modes))
succeed))
(defn cadence-restriction
[instrument-state m7 s2 s4 s5]
(case (:cadence instrument-state)
:perfect
(all (== m7 s5))
:plagal
(all (== m7 s4))
:just-nice (all (== m7 s2))
nil
succeed))
(defn- logic-program
[instrument-state melody2]
(fresh [melody
m1 m2 m3 m4 m5 m6 m7 m8
scale
s1 s2 s3 s4 s5 s6 s7 s8
base-note scale-type]
(key-restriction instrument-state s1)
(== melody [m1 m2 m3 m4 m5 m6 m7 m8])
(== scale [s1 s2 s3 s4 s5 s6 s7 s8])
(== m1 s1)
(== m8 s8)
(cadence-restriction instrument-state m7 s2 s4 s5)
(== melody2 [m1 m2 m3 m4 m5 m6 m7 m1])
(scale-restriction instrument-state scale-type)
(scaleo base-note scale-type scale)
(permuteo scale melody)))
(defn- same-melody-params?
[instrument-state-1 instrument-state-2]
(let [non-melody-keys [:speed :gaps]]
(= (apply dissoc instrument-state-1 non-melody-keys)
(apply dissoc instrument-state-2 non-melody-keys))))
(defn composer-loop
"Listens for new instrument states on instrument-state-ch and emits a
random melody to melody-ch. The loop terminates when
instrument-state-ch closes.
Changes to :speed or :gaps does not compose a new melody, but alters
the timing of the existing."
[instrument-state-ch melody-ch]
(go
(loop [prev-instrument-state nil
prev-composition
nil]
(when-let [instrument-state (<! instrument-state-ch)]
(let [gaps (for [i (range 8)] (get (:gaps instrument-state) i 0.5))
speed (:speed instrument-state)
new-melody (if (same-melody-params? prev-instrument-state
instrument-state)
(:melody prev-composition)
(random-composition instrument-state))
new-composition {:gaps gaps
:speed speed
:melody new-melody}]
(>! melody-ch new-composition)
(recur instrument-state
new-composition))))))
The system
The system
Experiments
• Goal: a reactive system
• Experiment 1: What is the size of the
melody space and how long does it take to
enumerate it?
• Experiment 2: What is a reasonable bound
on the search space to achieve
responsiveness?
Experiment 1
Melody space
Execution time (ms)
Melodies/second
No scale
Any tonic note
C
–
pc
–
pc
Major scale
Any tonic note
C
–
pc
–
pc
258
–
–
9, 360
4, 299
2, 177
120
278
431
258
–
–
258
–
–
258
–
–
1, 560
3, 852
404
720
294
2, 448
e of the melody space for different instrument configurations and the time it takes to enumerate that space in C OM
notes available to the engine, which accounts for the bound of 258 . “pc” is short for “perfect cadence”.
Execution time for bounded melody space searches
100000
Experiment 2
No scale
Major scale, any tonic note, any cadence
Major scale, C, any cadence
Major scale, C, perfect cadence
Execution time (ms)
100000
10000
1000
100
10
1
2
8
32
128
512
Search space bound
2048
8192
32768
Conclusion
• Composer demonstrates it is possible to
build a responsive interactive system with
extremely small and succinct core
• The declarative nature of the core
implementation makes it possible to extend
the terminology to other types of music
Future work
• Proper sampling of search space
• Labeled interface
• Non-Western music
• User testing