Bode Lecture Games, Decisions, and Control

Transcription

Bode Lecture
Games, Decisions, and Control
Fifty years back, fifty years forward
TAMER BASAR
¸
Dept ECE and CSL, UIUC
[email protected]
43rd IEEE Conf Decision and Control
Atlantis, THE BAHAMAS
December 17, 2004
Decision and Control
Decision
Control
December 17, 2004 -- Bode Lecture
Games, Decisions and Control
Decisions
Control
Games
Hendrik Wade Bode
December 24, 1905 - June 21, 1982
Hendrik Wade Bode
December 24, 1905 - June 21, 1982
Primary/Secondary
school in Urbana Leal / Urbana High
Hendrik Wade Bode
December 24, 1905 - June 21, 1982
Attended school in
Urbana Leal / Urbana High
Was denied admission to UIUC, because he
was too young (14)!
Hendrik Wade Bode
December 24, 1905 - June 21, 1982
Attended school in
Urbana
He later received an
honorary Sc.D. degree
from UIUC (1977)
What was the control field like 50 years ago?
We did not yet have
•  State space theory
•  Kalman filtering
•  Maximum principle
•  Dynamic programming
(almost)
We did not yet have
•  State space theory
•  Kalman filtering
•  Maximum principle
•  Dynamic
programming
But we did have
(in addition to Nyquist, Black, Bode, Wiener, ...)
•  Game theory December 17, 2004 -- Bode Lecture
John von Neumann
1903 - 1957
But we did have
•  Game theory
His 1947 book with O. Morgenstern
Theory of Games and
Economic Behavior December 17, 2004 -- Bode Lecture
But we did have
•  Game theory
John Nash
1928 - Cooperative and
noncooperative games
(Nash solutions)
~ 1950
Lloyd Shapley
1923 - But we did have
•  Game theory
Coalition formations
and stochastic games
~ 1953
But we did have
•  Game theory
•  Decision theory
Abraham Wald
1902 - 1950
His 1947 book on
Sequential Analysis --- sequential tests of statistical hypotheses
But we did have
•  Game theory
•  Decision theory
•  Linear programming
George Dantzig
1914 -
Mid 1950 s
•  1953/54 -- RAND Corp*
Dynamic Programming
-- book in 1957
multi-stage decision
processes / dynamic
rescheduling under
uncertainty
Richard Bellman
1920--1984 *Others at RAND in 50 s: Hestenes, LaSalle, Blackwell, Fleming, Berkovitz
Mid 1950 s
•  1953/54 -- RAND Corp
Dynamic Programming
•  1954 -- RAND Corp
Differential Games
--- book in 1965*
2-person 0-sum games
of pursuit-evasion type
--- precursors of MP, DP,
principle of optimality
Rufus Isaacs
1914 --1981 *Review
by Y.C. Ho in TAC 65 (501-3)
RAND Memoranda
A Game of Aiming and
Evasion: General
Discussion and the
Markman s Strategies
Rufus Isaacs
RM-1385
24 November 1954
Mid 1950 s
•  1953/54 -- RAND Corp
Dynamic Programming
•  1954 -- RAND Corp
Differential Games
•  1956 -- Soviet Union
Optimal Control
-- book in 1961
Mathematical Theory
of Optimal Processes -- Maximum Principle
Lev S. Pontryagin
1908 --1988 (& V.G. Boltyanski, R.V.Gamkrelidze, E. F. Mishchenko)
Pursuit-Evasion
Game
u
xp
xe
w
Pursuer: dxp / dt = f(xp, u), t ≥ 0, u(t) ∈ U
Evader: dxe / dt = f(xe, w), t ≥ 0, w(t) ∈ W
minµ maxν | xp(T) - xe(T) |
for fixed T
⇑
⇑
u= µ (•) v= ν(•)
⇑
{xp(t), xe(t)} or {xp(t), xe(t), w(t)} or {xp(t), xe(0)} December 17, 2004 -- Bode Lecture
Pursuit-Evasion
Game
u
xp
xe
w
Pursuer: dxp / dt = f(xp, u), t ≥ 0, u(t) ∈ U
Evader: dxe / dt = f(xe, w), t ≥ 0, w(t) ∈ W
minµ maxν | xp(T) - xe(T) |
for fixed T
OR
minµ maxν { inf {t : | xp(t) - xe(t) | < ε } }
Termination / Capturability
What is a (0-Sum) Differential Game?
Generalized optimal control problem with two inputs and conflicting objectives
dx / dt = f(x, u, w)
t≥0
x(0) = x0
T
J(u,w) = q(x(T), T) + ∫0 g(x,u,w) dt minimize wrt u / maximize wrt w
u(t) ∈ U, w(t) ∈ W, x(t) ∈ X, T = inf{t: x(t) ∈S}
What is a (0-Sum) Differential Game?
Generalized optimal control problem with two inputs and conflicting objectives
dx/dt = f(x, u, w)
t≥0
x(0) = x0
T
J(u,w) = q(x(tf), tf) + ∫0 g(x,u,w) dt minimize wrt u / maximize wrt w
u(t) ∈ U, w(t) ∈ W, x(t) ∈ X, T = inf{t: x(t) ∈S}
Applications Scenarios
•  Games of Pursuit (of kind and of degree)
–  Homicidal chauffeur
–  Dolichobrachistochrone
–  Isotropic rocket
–  Game of two cars
–  Maritime collision avoidance
–  Dogfight (two-target games)
•  Worst-Case Design (robust control, estimation, ID)
Main Common Feature
Principle of optimality (Bellman) /
Tenet of transition (Isaacs)
An optimal policy has the property that whatever the initial
state and initial decision(s) are, the remaining decisions must
constitute an optimal policy with regard to the state
resulting from the first decision. [Bellman 57, p. 83]
x(t)
(strong) time consistency
0
t
T
Tenet of Transition (Isaacs, p. 67)
… we are dealing with a family of games based on different starting
points. Consider an interlude of time in midplay. At its commencement the
path has reached some definitive point. Consider all possible x which may
be reached at the end of the interlude for all possible choices of u and w.
Suppose that for each endpoint, the game beginning there has already
been solved (V is known there). Then the payoff resulting from each
choice of u and w will be known, and they are to be so chosen as to render
it minimax. When we let the duration of the interlude approach zero, ….
V(x(t+Δ), t+Δ)
x(t)
0
t
t+Δ
T
Tenet of Transition (Isaacs, p. 67)
… we are dealing with a family of games based on different starting
points. Consider an interlude of time in midplay. At its commencement the
path has reached some definitive point. Consider all possible x which may
be reached at the end of the interlude for all possible choices of u and w.
Suppose that for each endpoint, the game beginning there has already
been solved (V is known there). Then the payoff resulting from each
choice of u and w will be known, and they are to be so chosen as to render
it minimax. When we let the duration of the interlude approach zero, …
==> Leads to as sufficient condition (under perfect state information) to HJB / HJI equation
Hamilton-Jacobi-Bellman/Isaacs Eq
HJB / HJI
1805 - 1865
W. R. Hamilton
1804 - 1851
PDE as a
sufficient condition
for optimality Karl G.J. Jacobi
sol {dV(x,t) / dt + incremental cost } = 0
& BC Hamilton-Jacobi-Bellman/Isaacs Eq
1805 - 1865
W. R. Hamilton
1873 - 1950
1804 - 1851
Karl G.J. Jacobi
Constantin Carathéodory
sol {dV(x,t) / dt + incremental cost } = 0
& BC HJI Equation
minu maxw or maxw minu
∂V / ∂t + ∇ xV⋅ f(x, u, w) + g(x,u,w) sol {dV(x,t) / dt + incremental cost } = 0
& BC
HJI Equation
infu supw
UPPER V
or
supw infu
LOWER V
∂V / ∂t + ∇ xV⋅ f(x, u, w) + g(x,u,w) sol { dV(x,t) / dt + incremental cost } = 0
& BC Isaacs Condition
minu maxw ≡ maxw minu
SADDLE POINT !
∂V / ∂t + ∇xV⋅ f(x, u, w) + g(x,u,w) sol { dV(x,t) / dt + incremental cost } = 0
Existence, uniqueness, smoothness?
& BC
(60 s, 70 s: Berkovitz, Fleming, Friedman, Krasovskii, Subbotin, …)
Viscosity Solution of HJI
SP {∂V / ∂t + ∇xV⋅ f(x, u, w) + g(x,u,w) } = 0
& BC
Viscosity / Minimax solution if V is only continuous
(Based on Crandall-Lions 83; Ishii 84 / Subbotin 80
also Clarke, Clarke-Vinter 83; Fleming 69; Kruzkov 69)
Computational tools: based on viscosity, minimax, viability
(90s: Bardi, Falcone, Soravia / Cardaliaquet, Quincampoix, Saint-Pierre)
From one-sided control to ZSDG
•  Synthesis of FB control laws from OL only
if there is a SP, and even then with care
Necessary condition for OL SP
(u*, w*) = SP sol H(t, x*, p, u, w)
H(t, x, p, u, w) = g(x, u, w) + p' f (x, u, w) d p' / dt = -∇x H(t, x*, p, u*, w*) & BC
dx* / dt = f (x*, u*, w*) & IC
From one-sided control to ZSDG
•  Synthesis of FB control laws from OL only
if there is a SP, and even then with care
Relationship with V of HJI
(u*, w*) = SP sol H(t, x*, p, u, w)
H(t, x, p, u, w) = g(x, u, w) + p' f (x, u, w)
p'(t) = ∇x V(x*, t)
or set inclusion if V is not smooth
(based on Barron-Jensen 86, Clarke-Vinter 87; also Berkovitz 89)
From one-sided control to ZSDG •  Synthesis of FB control laws from OL only if there is a SP,
and even then with care
•  Singular surfaces --- manifolds on which
–  Saddle-point controls are not uniquely
determined by the OL necessary conditions, or
–  V is not continuously differentiable, or
–  V is not continuous
From one-sided control to ZSDG •  Synthesis of FB control laws from OL only if there is a SP,
and even then with care
•  Singular surfaces (of co-dimension one)
transition dispersal equivocal universal focal
line
line
line
line
line
switching
envelope
Who knows what?
What information is available to each player
(active or passive)
–  Open-loop : only x0
–  Closed-loop state : at time t: x(s), s≤ t
–  Closed-loop state no-memory : at time t: x(t)
–  State with fixed delay θ : at time t: x(s), s≤ t-θ
–  Past actions with delay : for PI w(s), s≤ t-θ
⇒
Information spaces : NI and NII
Strategies
Mappings from information spaces to control
(action) spaces
µ : NI → U
ν : NII → W u = µ (ηI)
w = υ (ηII) ηI = η1(x,w)
ηII = η2(x,u) u = µ (w)
w = υ (u)
solvability of the loop equations
Loop Equations and Solvability
dx / dt = f(x,u, w)
u = µ (ηI)
w = υ (ηII) X
W
U
⇓
u = µ (w, x0)
U
⇓
u = û (x0 )
υ
W
µ
w = υ (u, x0)
⇓
w = ω (x0)
(ηI, ηII) ⇒ (µ ∈ M, υ ∈V) ⇒ (µ ∈M, υ ∈ V)
Open-loop: (µ, υ) are constant maps
Else, closed-loop, if loop eqs admit unique sols
u = µ (w, x0)
U
⇓
u = û (x0 )
υ
W
µ
w = υ (u, x0)
⇓
w = ω (x0)
(ηI, ηII) ⇒ (µ ∈ M, υ ∈V) ⇒ (µ ∈M, υ ∈ V)
(µ, υ) ⇒ (µ, υ) ⇒ (û, ω) -- OL representation
Generate the same state trajectory and same cost value J (µ, υ) = J (µ, υ) = J (û, ω) u = µ (w, x0)
U
⇓
u = û (x0 )
υ
W
µ
w = υ (u, x0)
⇓
w = ω (x0)
(ηI, ηII) ⇒ (µ ∈ M, υ ∈V) ⇒ (µ ∈M, υ ∈ V)
If (µ*, υ*) is a CL SP solution, and DG admits an OL
SP solution, then (û*, ω*) is the OL SP solution
u = µ (w, x0)
U
⇓
u = û (x0 )
υ
W
µ
w = υ (u, x0)
⇓
w = ω (x0)
(ηI, ηII) ⇒ (µ ∈ M, υ ∈V) ⇒ (µ ∈M, υ ∈ V)
Two DGs (M1, V), (M2, V), M1 ⊆ M2,
• Existence of SP for DG2 ⇒ SP for DG1
• • Upper/lower value of J1 ≥ U/L value of J2
u = µ (w, x0)
U
⇓
u = û (x0 )
υ
W
µ
w = υ (u, x0)
⇓
w = ω (x0)
(ηI, ηII) ⇒ (µ ∈ M, υ ∈V) ⇒ (µ ∈M, υ ∈ V)
Two DGs (M1, V1), (M2, V2), M1 ⊆ M2,
• • • Upper value of J1 ≥ Upper value of J2
u = µ (w, x0)
U
⇓
u = û (x0 )
υ
W
µ
w = υ (u, x0)
⇓
w = ω (x0)
Rapprochement of Control & DG
Robust Control in late 80 s
plant uncertainty, w
dx / dt = f(x,u, w)
u
P
+
C / µ
y
sensor noise /
uncertainty
w
Rapprochement of Control & DG
Robust Control in late 80 s
plant uncertainty, w
dx / dt = f(x,u, w)
u
P
+
C / µ
sensor noise /
uncertainty
w
y
L(µ,w) = ||z(x, u, w)|| / ||w|| infµ supw L(µ,w) = γ* opt. disturbance attenuation December 17, 2004 -- Bode Lecture
H∞-Optimal Control
M1 ⊆ M2 L(µ,w) = ||z(x, u, w)|| / ||w|| ⇓ ⇒
*
infµ supw L(µ,w) = γ
*
*
γ1 ≥ γ2 0
γ*
OL
{x(t)}
M1 ⊆ M2 L(µ,w) = ||z(x, u, w)|| / ||w|| ⇓ ⇒
*
*
*
γ1 ≥ γ2 0
γ*
{x(t), w(t)}
{x(t-θ)}
{x(t)}
OL
M1 ⊆ M2 L(µ,w) = ||z(x, u, w)|| / ||w|| ⇓ ⇒
*
*
*
γ1 ≥ γ2 0
γ*
{x(t), w(t)}
{x(t)}
{x(t-θ)}
sampled state
OL
M1 ⊆ M2 L(µ,w) = ||z(x, u, w)|| / ||w|| ⇓ ⇒
*
*
*
γ1 ≥ γ2 0
γ*
{x(t), w(t)}
{x(t)}
{x(t-θ)}
{y(s), s≤t}
sampled state
OL
M1 ⊆ M2 L(µ,w) = ||z(x, u, w)|| / ||w|| ⇓ ⇒
*
*
*
γ1 ≥ γ2 0
γ*
{x(t), w(t)}
{x(t)}
OL
{y(s), s≤t}
sampled sampled /
state
quantized y
{x(t-θ)}
How to compute µγ
L(µ,w) = ||z(x, u, w)|| / ||w||
⇒
infµ supw L(µ,w) = γ*
L(µγ,w) ≤ γ γ > γ*
⇔ a related ZSDG (parametrized by γ) J(µ,w) = ||z(x, u, w)||2 - γ2 ||w||2 --> infµ supw
Given M, find the smallest γ such that the upper
value of J is zero (bounded), and the corresponding
µγ, for γ > γ*.
How to compute µγ
The ZSDG:
J(µ,w) = ||z(x, u, w)||2 - γ2 ||w||2
dx / dt = f(x,u, w), x(0) = 0, µ∈ M
⇓
• HJI PDE for the case ηI = {x(t)} or {x(t), w(t)}
⇒ State FB controller parametrized by γ
• • Obtain a representation in M for smallest γ
-- certainty equivalent control December 17, 2004 -- Bode Lecture
Certainty Equivalence
From state FB to measurement FB
State FB: µγ(x, t), Vγ(x,t) value function (cost-to-go) x
0
Wγ(y[0,t],t, x)
t
Vγ(x,t)
cost-to-come
(information state)
T
Certainty Equivalence
From state FB to measurement FB
State FB: µγ(x, t), Vγ(x,t) value function (cost-to-go) x
0
Wγ(y[0,t],t, x)
cost-to-come
t
Vγ(x,t)
maxx { Wγ(y[0,t],t, x) + Vγ(x,t)}
T
⇒ xˆ (unique) ˆ
t)
⇒ µγ(x,
An Example: State FB
dx / dt = (u + w) x ,
|z|2 = x4 + u2, T=∞
HJI: x4 - (1/4) [∇xV]2 x2 (1- γ-2) = 0
⇒ Vγ(x) = [γ / √(γ2- 1) ] x2
γ > 1
µγ(x) = - [γ / √(γ2- 1) ] x2 υγ(x) = - [γ-1 / √(γ2- 1) ] x2 ⇒ dx / dt = - [γ-1 √(γ2- 1) ] x3
GAS December 17, 2004 -- Bode Lecture
A non-smooth V
dx / dt = (u + w) x ,
|z|2 = x2 + u2, T=∞
HJI: x2 - (1/4) [∇xV]2 x2 (1- γ-2) = 0
⇒ Vγ(x) = [2γ / √(γ2- 1) ] |x|
γ > 1 viscosity solution
µγ(x) = - [γ / √(γ2- 1) ] |x| υγ(x) = - [γ-1 / √(γ2- 1) ] |x|
⇒ dx / dt = - [γ-1 √(γ2- 1) ] x2 sgn(x)
GAS December 17, 2004 -- Bode Lecture
Non-smooth V / viscosity solution
dx / dt = (u + w) x
T= ∞
J = E[ ||x||2 + ||u||2 - γ2 ||w||2 ]
Perturb HJI by a small 2nd-order term:
x2 - (1/4) [∇xV]2 x2 (1- γ-2) - ε2 [∇xxV] = 0
ε → 0 ⇒ Vγ(x) = [2γ / √(γ2- 1) ] |x|
γ > 1 viscosity solution
Worst-Case Designs
Are DG-based designs really conservative?
Worst-Case Designs
No! Provided that M and W are picked properly
Worst-Case Designs
No!
Provided that M and W are picked properly
An intelligent worst-case design controller
uses all the available and real-time accessible
/ learnable information on the uncertainty and takes measures to safeguard the performance
only against conditionally unknown uncertainty.
Stochastics
w → (w, ξ) complete statistical description
known to both players
Loop equations
u = µ (w, ξ)
w = υ (u, ξ)
• Relationship between CL and OL policies is no
longer valid. No OL/CL representations.
Stochastics
Loop equations
u = µ (w, ξ)
w = υ (u, ξ)
• Relationship between CL and OL policies is no longer valid
•• Certainty equivalence generally fails
--- even if both players have access to same measurements
Stochastics
Loop equations: u = µ (w, ξ)
w = υ (u, ξ)
••• Very few existence, uniqueness, characterization results,
unless information on ξ is nested (one player knows everything relevant on ξ the other player knows)
-- even when goals are aligned (teams)
Stochastics
Loop equations: u = µ (w, ξ)
w = υ (u, ξ)
••• Very few existence, uniqueness, characterization results,
unless information on ξ is nested
•••• Iterated second-guessing
Games could be more tractable than teams
ξ1
υ
ξ2
w
+
y
µ
u
L±(ξ, u, w) = (u - w)2 ± k (w - ξ1)2 J±(µ , υ) = E [ L± (ξ, µ(w + ξ2), υ(ξ1))]
(ξ1, ξ2) jointly Gaussian, all scalar
Games could be more tractable than teams
ξ1
υ
ξ2
w
+
y
µ
L±(ξ, u, w) = (u - w)2 ± k (w - ξ1)2 u
minµ maxυ J-(µ , υ) --- unique SP, linear* *CDC 71
minµ minυ J+(µ , υ) --- nonlinear, not known*
*Witsenhausen
68
Many Players / Multi-Criteria
Generalized optimal control problem with multiple (N) inputs and multiple (N) objectives
dx / dt = f(x, u), x(0) = x0, u = (u1,..,uN)
Ji(u) = qi(x(T), T) + ∫0T gi(x,u) dt → minimize wrt ui with other controls, u-i, fixed
Nash eqm u*: min Ji(ui, u-i*) → ui*
Characterization of NE
Counterpart of HJI Equation (coupled PDEs)
sol {∂Vi / ∂t + ∇ xVi⋅ f(x, u) + gi(x,u)} =0 & BC
static NE
{µi(x(t), t)} state FB NE
Characterization of NE
Counterpart of HJI Equation (coupled PDEs)
sol {∂Vi / ∂t + ∇ xVi⋅ f(x, u) + gi(x,u)} =0 & BC
static NE
{µi(x(t), t)} state FB NE
For a general information structure ({ηi}→ {Mi})
Loop equations
ui = µ i(u-i),
µi
∈ Mi
NE in {Mi} ⇔ NE in {Mi}
, i=1,.., N
Richer Structure / Issues
•  OL NE is not a representation of CL FB NE
•  Certainty equivalence does not hold
•  Informational non-uniqueness is a critical
issue (in the deterministic case)
•  Informational non-uniqueness is a critical
issue (in the deterministic case)
•  Existence of value (and its precise definition)
is a delicate issue
some other talk ……
Next 50 Years ?
Back to Hendrik Bode
Citation for Edison Medal (1969)
For fundamental contributions to the arts of
communications, computation and control;
…………. December 17, 2004 -- Bode Lecture
Back to Hendrik Bode
Citation for Edison Medal (1969)
For fundamental contributions to the arts of
communications, computation and control;
…………. Rapprochement of the 3 C s
Next 50 Years ?
Decisions
Control
Games
Next 50 Years ?
•  Communication and computation constraints
(as part of allowable strategies / control laws)
-- quantization, delay, loss, tradeoffs among
different tasks
Next 50 Years ?
different tasks
•  Computational tools December 17, 2004 -- Bode Lecture
Next 50 Years ?
different tasks
•  Computational tools •  Expansion of scope of applications
Next 50 Years ?
different tasks
•  Computational tools •  Expansion of scope of applications
Last year s CDC: Entanglement of Communication & Control
Next 50 Years ?
•  Allow for parallel developments / competing
approaches
Next 50 Years ?
approaches
•  Diversity is good!
Next 50 Years ?
approaches
•  Allow curiosity to drive research to new
frontiers
Next 50 Years ?
approaches
•  Allow curiosity to drive research to new
frontiers
•  Information and Feedback will still be with us
for the next 50 years
URL for this Bode Lecture
http://decision.csl.uiuc.edu/~tbasar/talks.html
(after December 19, 2004)

Bode Lecture Games, Decisions, and Control

Transcription

Similar documents

brochure - YPH Welding Supplies

Exercise 8: Frequency Response

The Water Route 2016

here - Kevin Fronczak

brian bode - Oya Salon

09-19-13 - St. Joseph, MO

Speaker Information Agenda

Survival impact of lung transplantation for COPD

Upcoming Sunday Services QUUF Christmas Day Potluck Dinner

June 2010 Church Bell