Design of Algorithms by Induction

Design of Algorithms by
Induction
Bibliography: [Manber]- Chap 5
Algorithm Analysis <-> Design
• Given an algorithm, we can
– Analyze its complexity
– Proof its correctness
• But:
– Given a problem, how do we design a
solution that is both correct and efficient ?
• Is there a general method for this ?
Induction proofs vs
Design of algorithms by induction (1)
• Induction used to prove that a statement P(n)
holds for all integers n:
– Base case: Prove P(0)
– Assumption: assume that P(n-1) is true
– Induction step: prove that P(n-1) implies P(n) for all
n≥0
– Strong induction: when we assume P(k) is true for
all k<=n-1 and use this in proving P(n)
Induction proofs vs
Design of algorithms by induction (2)
• Induction used in algorithm design:
– Base case: Solve a small instance of the problem
– Assumption: assume you can solve smaller
instances of the problem
– Induction step: Show how you can construct the
solution of the problem from the solution(s) of the
smaller problem(s)
Design of algorithms by induction
• Represents a fundamental design principle that
underlies techniques such as divide & conquer,
dynamic programming and even greedy
• Question: how to reduce the problem to a
smaller problem or a set of smaller problems ?
(n -> n-1, n/2, n/4, …?)
• Examples:
–
–
–
–
–
The successful party problem [Manber 5.3]
The celebrity problem [Manber 5.5]
The skyline problem [Manber 5.6]
One knapsack problem [Manber 5.10]
The Max Consecutive Subsequence [Manber 5.8]
The successful party problem
• Problem: you are arranging a party and have a list of n
persons that you could invite. In order to have a
successful party, you want to invite as many people as
possible, but every invited person must be friends with at
least k of the other party guests. For each person, you
know his/her friends. Find the set of invited people.
Successful party - Example
K=3
Ann
Bob
Finn
Chris
Ellie
Dan
Successful party – Direct approach
•
•
Direct approach for solving: remove persons
who have less than k friends
But: which is the right order of removing ?
1. Remove all persons with less than k friends, then
deal with the persons that are left without enough
friends ?
2. Remove first one person, then continue with
affected persons ?
Instead of thinking about our algorithm
as a sequence of steps to be executed,
think of proving a theorem that the algorithm exists
The design by induction approach
• Instead of thinking about our algorithm as a
sequence of steps to be executed, think of
proving a theorem that the algorithm exists
• We need to prove that this “theorem” holds for a
base case, and that if it holds for “n-1” this
implies that it holds for “n”
Successful party - Solution
• Induction hypothesis: We know how to find the
solution (maximal list of invited persons that
have at least k friends among the other invited
persons), provided that the number of given
persons is <n
• Base case:
– n<= k: no person can be invited
– n=k+1: if every person knows all of the other persons,
everybody gets invited. Otherwise, no one can be
invited.
Successful party - Solution
• Inductive step:
– Assume we know how to select the invited persons
out of a list of n-1, prove that we know how to select
the invited persons out of a list of n, n>k+1
– If all the n persons have more than k friends among
them, all n persons get invited
– Else, there exists at least one person that has less
than k friends. Remove this person and the solution is
what results from solving the problem for the
remaining n-1 persons
Successful party - Conclusions
• Designing by induction got us to a correct
algorithm, because we designed it proving its
correctness
– Every invited person knows at least k other invited
– We got the maximum possible number of invited
persons
• The best way to reduce a problem is to eliminate
some of its elements.
– In this case, it was clear which persons to eliminate
– We will see examples where the elimination is not so
straightforward
The Celebrity problem
• Problem: A celebrity in a group of people is someone
who is known by everybody but does not know anyone.
You are allowed to ask anyone from the group a
question such as “Do you know that person?” pointing to
any other person from the group. Identify the celebrity (if
one exists) by asking as few questions as possible
• Problem:
• Given a n*n matrix “know” with know[p, q] = true if p
knows q and know[p, q] = false otherwise.
• Determine whether there exists an i such that:
– Know[j; i] = true (for all j, j≠ i) and Know[i; j] = false (for all j, j≠i )
Celebrity - Solution 1
• Brute force approach: ask questions arbitrary,
for each person ask questions for all others =>
n*(n-1) questions asked
Celebrity - Solution 2
• Use induction:
• Base case: Solution is trivial for n=1, one person is a
celebrity by definition.
• Assumption: Assume we leave out one person and that
we can solve the problem for the remaining n-1 persons.
– If there is a celebrity among the n-1 persons, we can find it
– If there is no celebrity among the n-1 persons, we find this out
• Induction step:
• For the n'th person we have 3 possibilities:
– The celebrity was among the n-1 persons
– The n’th person is the celebrity
– There is no celebrity in this group
Celebrity - Sol 2- Induction step
• We must solve the problem for n persons, assuming that
we know to solve it for n-1
• We leave out one person.
• We choose this person randomly – let it be the n’th
person
• Case 1: There is a celebrity among the n-1 persons, say
p. To check if this is also a celebrity for the n'th person
– check if know[n, p] and not know[p, n]
• Case 2: There is no celebrity among the n-1 persons. In
this case, either person n is the celebrity, or there is no
celebrity in the group.
– To check this we have to find out if know[i, n] and not know[n, i],
for all i <> n.
Celebrity – Solution 2
Function Celebrity_Sol2(n:integer) returns integer
if n = 1 then return 1
else
p = Celebrity_Sol2(n-1);
if p != 0 then
// p is the celebrity among n-1
if( knows[n,p] and not knows[p,n] ) then
return p
end if
end if
forall i = 1..n-1
if (knows[n, i] or not knows[i, n]) then
return 0 // there is no celebrity
end if
end for
return n
// n is the celebrity
Celebrity – Solution 2 Analysis
T(n)
Function Celebrity_Sol2(n:integer) returns integer
if n = 1 then return 1
else
p = Celebrity_Sol2(n-1);
if p != 0 then
// p is the celebrity among n-1
if( knows[n,p] and not knows[p,n] ) then
return p
end if
end if
forall i = 1..n-1
if (knows[n, i] or not knows[i, n]) then
return 0 // there is no celebrity
end if
end for
return n
// n is the celebrity
T(n-1)
O(n)
Celebrity – Solution 2 Analysis
• T(n)=T(n-1) + n
• T(1)=1
• T(n)=1+2+3+ +n=n*(n+1)/2
• T(n) is O(n2)
• We have reduced a problem of size n to a problem of
size n-1. We then still have to relate the n-th element to
the n-1 other elements, and here this is done by a
sequence which is O(n), so we get an algorithm of
complexity O (n2), which is the same as the brute force.
• If we want to reduce the complexity of the algorithm to
O(n), we should have T(n)=T(n-1)+c
Celebrity – Solution 3
• The key idea here is to reduce the size of the
problem from n persons to n-1, but in a clever
way – by eliminating someone who is a noncelebrity.
• After each question, we can eliminate a person
– if knows[i,j] then i cannot be a celebrity => elim i
– if not knows[i,j] then j cannot be a celebrity => elim j
Celebrity - Solution 3
• Use induction:
• Base case: Solution is trivial for n=1, one person is a
celebrity by definition.
• Assumption: Assume we leave out one person who is
not a celebrity and that we can solve the problem for the
remaining n-1 persons.
– If there is a celebrity among the n-1 persons, we can find it
– If there is no celebrity among the n-1 persons, we find this out
• Induction step:
• For the n'th person we have only 2 possibilities left:
– The celebrity was among the n-1 persons
– There is no celebrity in this group
Celebrity - Sol 3- Induction step
• We must solve the problem for n persons, assuming that
we know to solve it for n-1
• We leave out one person. In order to decide which
person to elim, we ask a question (to a random person i
about a random person j).
• The eliminated person is e=i or e=j, and we know that e
is not a celebrity
• Case 1: There is a celebrity among the n-1 persons that
remain after eliminating e, say p. To check if this is also a
celebrity for the person e
– check if know[e, p] and not know[p, e]
• Case 2: There is no celebrity among the n-1 persons. In
this case, there is no celebrity in the group. It is no need
any more to check if e is a celebrity !
Celebrity – Solution 3
Function Celebrity_Sol3(S:Set of persons) return person
if card(S) = 1 return S(1)
pick i, j any persons in S
if knows[i, j] then // i no celebrity
elim=i
else // if not knows[i, j] then j no celebrity
elim=j
p = Celebrity_Sol3(S-elim)
if p != 0 and knows[elim,p] and not knows[p,elim]
return p
else
return 0 // no celebrity
end if
end if
end function Celebrity_Sol3
Celebrity – Solution 3 Analysis
Function Celebrity_Sol3(S:Set of persons) return person
if card(S) = 1 return S(1)
pick i, j any persons in S
if knows[i, j] then // i no celebrity
elim=i
else // if not knows[i, j] then j no celebrity
elim=j
p = Celebrity_Sol3(S-elim)
if p != 0 and knows[elim,p] and not knows[p,elim]
return p
else
return 0 // no celebrity
end if
end if
end function Celebrity_Sol3
T(n)=T(n-1)+c
Algorithm is O(n)
Celebrity - Conclusions
• The size of the problem should be reduced (from
n to n-1) in a clever way
• This example shows that it sometimes pays off
to expend some effort ( in this case – one
question) to perform the reduction more
effectively
The Skyline Problem
• Problem: Given the exact locations and heights
of several rectangular buildings, having the
bottoms on a fixed line, draw the skyline of these
buildings, eliminating hidden lines.
The Skyline Problem
• A building Bi is represented as a triplet (Li, Hi, Ri)
• A skyline of a set of n buildings is a list of x coordinates and the
heights connecting them
• Input (1,11,5), (2,6,7), (3,13,9), (12,7,16), (14,3,25), (19,18,22),
(23,13,29), (24,4,28)
• Output (1, 11), (3, 13), (9, 0), (12, 7), (16, 3), (19, 18), (22, 3), (23,
13), (29, 0)
Skyline – Solution 1
• Base case: If number of buildings n=1, the
skyline is the building itself
• Inductive step: We assume that we know to
solve the skyline for n-1 buildings, and then we
add the n’th building to the skyline
Adding One Building to the Skyline
• We scan the skyline, looking at one horizontal
line after the other, and adjusting when the
height of the building is higher than the skyline
height
Worst case:
Bn
O(n)
Skyline – Solution 1 Analysis
T(n)
Algorithm Skyline_Sol1(n:integer) returns Skyline
if n = 1 then return Building[1]
else
sky1 = Skyline_Sol1(n-1);
sky2 = MergeBuilding(Building[n], sky1);
return sky2;
end algorithm
T(n-1)
T(n)=T(n-1)+O(n)
O(n2)
O(n)
Skyline – Solution 2
• Base case: If number of buildings n=1, the
skyline is the building itself
• Inductive step: We assume that we know to
solve the skylines for n/2 buildings, and then we
merge the two skylines
Merging two skylines
• We scan the two skylines together, from left to
right, match x coordinates and adjust height
where needed
n1
n2
Worst case:
O(n1+n2)
Skyline – Solution 2 Analysis
T(n)
Algorithm Skyline_Sol2(left, right:integer) returns
Skyline
if left=right then return Building[left]
else
middle=left+(left+right)/2
sky1 = Skyline_Sol2(left, middle);
sky2 = Skyline_Sol2(middle+1, right);
sky3 = MergeSkylines(sky1, sky2);
return sky3;
end algorithm
T(n/2)
T(n/2)
O(n)
T(n)=2T(n/2)+O(n)
O(n log n)
Skyline - Conclusion
• When the effort of combining the subproblems
cannot be reduced, it is more efficient to split
into several subproblems of the same type which
are of equal sizes
– T(n)=T(n-1)+O(n)
– T(n)=2T(n/2)+O(n)
• This technique is
… O(n2)
… O(n log n)
Divide and conquer
Efficiently dividing and combining
• Solving a problem takes following 3 actions:
– Divide the problem into a number of subproblems
that are smaller instances of the same problem.
– Solve the subproblems
– Combine the solutions to the subproblems into the
solution for the original problem.
• Execution time is defined by recurrences like
– T(size)=Sum (T(subproblem size)) + CombineTime
– Choosing the right subproblems sizes and the right
way of combining them decides the performance !
Recurrences
• T(n)=O(1), if n=1
• For n>1, we may have different recurrence relations, according to
the way of splitting into subproblems and of combining them. Most
common are:
– T(n)=T(n-1)+O(n)
… O(n2)
– T(n)=T(n-1)+O(1)
… O(n)
– T(n)=2T(n/2)+O(n)
… O(n log n)
– T(n)=2T(n/2)+O(1)
… O(n)
– T(n)=a*T(n/b) + f (n) ;
The Knapsack Problem
• This is one of the multiple variants of the
knapsack problem, that applies to packaging
goods in standard containers of a shipping
company that have to be exactly filled up
• The problem: Given an integer K and n items of
different sizes such that the i’th item has an
integer size ki, find a subset of the items whose
sizes sum to exactly K, or determine that no
such subset exist
• P(n,K)
• P(i,k) – the first i items and a knapsack of size k
Knapsack - Try 1
• Induction hypothesis: We know how to solve P(n-1, K)
• Base case: n=1: there is a solution if the single element
is of size K
• Inductive step:
• Case 1: P(n-1,K) has a solution: we simply do not use the n’th item,
we already have the solutiom
• Case 2: P(n-1,K) has no solution: this means that we must use the
n’th item of size kn. This implies that the rest of the items must fit into
a smaller knapsack of size K- kn.We have reduced the problem to
two smaller subproblems: P(n-1, K) and P(n-1, K-kn).
• In order to solve this, we need to strenghten the hypothesis
Knapsack - Try 2 – Solution 1
• Induction hypothesis: We know how to solve P(n-1, k)
for all 0<=k<=K
• Base case: n=1: there is a solution if the single element
is of size k
• Inductive step:
• P(n-1, k) and P(n-1, k-kn).
• We have reduced the problem of size n to 2 problems of
size n-1
• T(n)=2 T(n-1) + O(1)
• T(n) is O(2n)
Knapsack – Solution 2
• Solution 1 solved 2n problems P(i,k)
• Actually, the total number of distinct
problems P(i,k) is n*K
• 2n – n*K is redundant work !
• Solution: we store all the known results of
all problems P(i,k) in an n*K matrix
Knapsack - Conclusions
• Divide-and-conquer algorithms partition the
problem into disjoint subproblems, solve the
subproblems, and then combine their solutions to
solve the original problem.
• When the subproblems overlap, a divide-andconquer algorithm does more work than
necessary, repeatedly solving the common
subsubproblems.
• In order to avoid repeated work, the solutions of
subproblems are saved and when the same
subproblem appears again the saved solution is
used
• This technique is Dynamic programming
Finding the Maximum Consecutive
Subsequence
• Problem: Given a sequence X = (x1, x2, …, xn) of (not
necessarily positive) real numbers, find a subsequence
xi; xi+1; … ; xj of consecutive elements such that the sum
of the numbers in it is maximum over all subsequences
of consecutive elements
• Example: The profit history (in billion $) of the company
ProdIncCorp for the last 10 years is given below. Find the maximum
amount that ProdIncCorp earned in any contiguous span of years.
y1
y2
y3
y4
y5
y6
y7
y8
y9
y10
2
-3
1.5
-1
3
-7
0.5
0.5
1
-2
Max consec subsequence – Try 1
• Base case: n=1: we have a single element and it is the
max subsequence
• Induction hypothesis: We know how to find the
maximum subsequence in a sequence of length < n.
• We know(we assume) that the maximum consecutive
subsequence of x1;…; xn-1 is xi… xj , j<=n-1
• We have to find the maximum consecutive subsequence
of x1;…; xn-1; xn
– We distinguish 2 cases:
• Case 1: j=n-1 (the max subseq is a suffix of the given one)
• Case 2: j<n-1 (the max subseq is no suffix of the given one)
Max consec subsequence – Try 1
• Case 1: j=n-1 (the max subseq is a suffix of the given one)
n-1
n
y1
y2
y3
y4
y5
2
-3
1.5
-1
3
If x[n] >0 then the max seq will be xi;… xj; xn
n-1
n
y1
y2
y3
y4
y5
y6
2
-3
1.5
-1
3
-7
If x[n] <0 then the max seq remains xi;… xj
Max consec subsequence – Try 1
• Case 2: j<n-1 (the max subseq is not a suffix)
n-1
n
y1
y2
y3
y4
y5
y6
y7
2
-3
1.5
-1
3
-7
0.5
n-1
n
y1
y2
y3
y4
y5
y6
y7
y8
y9
2
-3
1.5
-1
3
-7
0.5 0.5
8
We need to know also
the maximum subsequence that is a suffix !
Max consec subsequence – Try 2
• Stronger Induction hypothesis: We know how to find
the maximum subsequence in a sequence of length < n
and the maximum subsequence that is a suffix
• Base case: when n=0, both subsequences are empty
(their sums are 0)
• Inductive step: We add xn to the max suffix. If this sum
is bigger than the global max subsequence, then we
update it (as well as the new suffix). Otherwise we retain
the previous max subsequence. We also need to find the
new max suffix, and add xn to its old value. If this sum
results negative, we take the empty set as max suffix.
Max consec subseq - Solution
Algorithm Max_Subsequence(IN: X[1..n], OUT: GlobalMax, SuffixMax)
begin
if (n=0) then
GlobalMax:=0;
SuffixMax:=0;
else
Max_Subsequence(X[1..n-1], GlobalMax1, SuffixMax1);
GlobalMax:=GlobalMax1;
if x[n] + SuffixMax1 > GlobalMax1 then
SuffixMax := SuffixMax1 + x[n];
GlobalMax :=
SuffixMax;
else if x[n] + SuffixMax1 > 0 then
SuffixMax := SuffixMax1 + x[n];
else
SuffixMax := 0;
end
T(n)=T(n-1)+c
Algorithm is O(n)
Max consec subseq - Solution
Algorithm Max_Subsequence(X,n)
Input: X (array of length n)
Output: Global_Max (The sum of the maximum subsequence)
begin
Global_Max:= 0;
Suffix_Max := 0;
for i=1 to n do
if x[i] + Suffix_Max > Global_Max then
Suffix_Max := SuffixMax + x[i];
Global_Max :=
Suffix_Max;
else if x[i] + Suffix_Max > 0 then
Suffix_Max := Suffix_Max + x[i];
else
Suffix_Max := 0;
end
It was straightforward to rewrite the solution in a non-recursive way !
Max consec subseq - Conclusion
• Sometimes we cannot prove that P(<n) => P(n)
• We can add an additional statement Q and prove easier
• (P(< n) and Q(< n) ) => P(n)
• Q is a property of the solution, property that we must
discover
• Attention: Q(n) must become part of the induction
hypothesis, we must in fact prove:
• (P(< n) and Q(< n) ) => ( P(n) and Q(n))
This is Strengthen the Induction Hypothesis
Conclusions (1)
• What is Design by induction ?
• An algorithm design method that uses the idea
behind induction to solve problems
– Instead of thinking about our algorithm as a sequence of steps
to be executed, think of proving a theorem that the algorithm
exists
• We need to prove that this “theorem” holds for a base case, and
that if it holds for “n-1” this implies that it holds for “n”
Conclusions (2)
• Why/when should we use Design by induction ?
– It is a well defined method for approaching a large variety of
problems (“where do I start ?”)
• Just take the statement of the problem and consider it is a theorem
to be proven by induction
– This method is always safe: designing by induction gets us to a
correct algorithm, because we designed it proving its correctness
– Encourages abstract thinking vs coding => you can handle the
reasoning in case of complex algorithms
• A[1..n]; A[n/4+1 .. 3n/4]
• L, R; L=(R+3L+1)/4, R=(L+3R+1)/4
– We can make it also efficient (see next slide)
Conclusions (3)
• The inductive step is always based on a reduction
from problem size n to problems of size <n. How to
efficiently make the reduction to smaller problems:
– Sometimes one has to spend some effort to find the suitable
element to remove. (see Celebrity Problem)
– If the amount of work needed to combine the subproblems is not
trivial, reduce by dividing in subproblems of equal size – divide
and conquer (see Skyline Problem)
– A reduction to several subproblems might eventually compute
the same subproblems again and again. In this case, when the
subproblems are not disjoint, try to store these intermediate
results – dynamic programming. (see Knapsack Problem)
Conclusions (4)
• What if induction does not succeed for the given
problem statement?:
– A straightforward reduction might not give sufficient information
to extend the result, in this case one can try to strengthen the
induction hypothesis. (see Maximum Subsequence Problem)
• Try to find relevant properties of the solution and add these to the
induction hypothesis