Numerical Solution of Linear Equations & Power Method

Transcription

Solution of System of Linear Equations
&
Eigen Values and Eigen Vectors
P. Sam Johnson
March 30, 2015
P. Sam Johnson (NITK)
Solution of System of Linear Equations & Eigen Values and Eigen Vectors
March 30, 2015
1/43
Overview
Linear systems
Ax = b
(1)
occur widely in applied mathematics.
They occur as direct formulations of “real world” problems ; but more
often, they occur as a part of numerical analysis.
There are two general categories of numerical methods for solving (1).
Direct methods are methods with a finite number of steps ; and they end
with the exact solution provided that all arithmetic operations are exact.
Iterative methods are used in solving all types of linear systems, but they
are most commonly used for large systems.
March 30, 2015
2/43
Overview
Iterative methods are faster than the direct methods. Even the round-off
errors in iterative methods are smaller.
In fact, iterative method is a self correcting process and any error made
at any stage of computation gets automatically corrected in the
subsequent steps.
We start with a simple example of a linear system and we discuss
consistency of system of linear equations, using the concept of
determinant.
Two iterative methods – Gauss-Jacobi and Gauss-Seidel methods are
discussed, to solve a given linear system numerically.
March 30, 2015
3/43
Overview
Eigen values are a special set of scalars associated with a linear system of
equations. The determination of the eigen values and eigen vectors of a
system is extremely important in physics and engineering, where it is
equivalent to matrix diagonalization and arises in such common
applications as stability analysis, the physics of rotating bodies, and small
oscillations of vibrating systems, to name only a few.
Each eigen value is paired with a corresponding vector, called eigen
vector.
We discuss properties of eigen values and eigen vectors in the lecture. Also
we give a method, called Rayeigh’s Power Method, to find out the
largest eigen value in magnitude (also called, dominant eigen value). The
special advantage of the power method is that the eigen vector corresponds
to the dominant eigen value is also calculated at the same time.
March 30, 2015
4/43
An Example
A shopkeeper offers two standard packets because he is convinced that
north indians each more wheat than rice and south indians each more rice
than wheat. Packet one P1 : 5kg wheat and 2kg rice ; Packet two P2 :
2kg wheat and 5kg rice. Notation. (m, n) : m kg wheat and n kg rice.
Suppose I need 19kg of wheat and 16kg of rice. Then I need to buy x
packets of P1 and y packets of P2 so that x(5, 2) + y (2, 5) = (10, 16).
Hence I have to solve the following system of linear equations
5x + 2y
= 10
2x + 5y
= 16.
Suppose I need 34 kg of wheat and 1 kg of rice. Then I must buy 8
packets of P1 and −3 packets of P2 . What does this mean? I buy 8
packets of P1 and from these I make three packets of P2 and give them
back to the shopkeeper.
March 30, 2015
5/43
Consistency of System of Linear Equations – Graphically
(1 Equation, 2 Variables)
The solution of the equation
ax + by = c
is the set of all points satisfy the equation forms a straight line in the
plane through the point (c/b, 0) and with slope −a/b.
Two lines
parallel – no solution
intersect – unique solution
same – infinitely many solutions.
March 30, 2015
6/43
Consistency of System of Linear Equations
Consider the system of m linear equations in n unkowns
a11 x1 + a12 x2 + · · · + a1n xn = b1
a21 x1 + a22 x2 + · · · + a2n xn = b2
..
.
.
.
. + .. + · · · + ..
= ..
am1 x1 + am2 x2 + · · · + amn xn = bm .
Its matrix form is

a11
 a21

 ..
 .
a12
a22
..
.
...
...
  
x1
b1
a1n
 x2   b2
a2n 
  
..   ..  =  ..
.  .   .
...
am1 am2 . . . amn
xm





bm .
That is, Ax = b, where A is called the coefficient matrix.
March 30, 2015
7/43
To determine whether these equations are consistent or not, we find the
ranks of the coefficient matrix A and the augmented matrix


a11 a12 . . . a1n b1
 a21 a22 . . . a2n b2 


K = .
..
..
..  = [A b].
 ..
.
···
.
. 
am1 am2
. . . amn bm
We denote the rank of A by r (A).
1. r (A) 6= r (K ), then the linear system Ax = b is inconsistent and has
no solution.
2. r (A) = r (K ) = n, then the linear system Ax = b is consistent and
has a unique solution.
3. r (A) = r (K ) < n, then the linear system Ax = b is consistent and
has an infinite number of solutions.
March 30, 2015
8/43
Solution of Linear Simultaneous Equations
Simultaneous linear equations occur quite often in engineering and science.
The analysis of electronic circuits consisting of invariant elements, analysis
of a network under sinusoidal steady-state conditions, determination of the
output of a chemical plant, finding the cost of chemical reactions are some
of the problems which depend on the solution of simultaneous linear
algebraic equations, the solution of such equations can be obtained by
direct or iterative methods (successive approximation methods).
Some direct methods are as follows:
1. Method of determinants – Cramer’s rule
2. Matrix inversion method
3. Gauss elimination method
4. Gauss-Jordan method
5. Factorization (triangulization) method.
March 30, 2015
9/43
Iterative Methods
Direct methods yield the solution after a certain amount of computation.
On the other hand, an iterative method is that in which we start from an
approximation to the true solution and obtain better and better
approximations from a computation cycle repeated as often as may be
necessary for achieving a desired accuracy.
Thus in an iterative method, the amount of computation depends on the
degree of accuracy required.
For large systems, iterative methods are faster than the direct methods.
Even the round-off errors in iterative methods are smaller. In fact,
iteration is a self correcting process and any error made at any stage of
computation gets automatically corrected in the subsequent steps.
March 30, 2015
10/43
Diagonally Dominant
Definition
An n × n matrix A is said to be diagonally dominant if, in each row, the
absolute value of each leading diagonal element is greater than or equal to
the sum of the absolute values of the remaining elements in that row.


10 −5 −2
The matrix A =  4 −10 3  is a diagonally dominant and the
1
6
10


2 3 −1
matrix A = 5 8 −4 is not a diagonally dominant matrix.
1 1 1
March 30, 2015
11/43
Diagonal System
Definition
In the sytem of simultaneous linear equations in n unknowns AX = B, if A
is diagonally dominant, then the system is said to be diagonal system.
Thus the system of linear equations
a1 x + b1 y + c1 z
= d1
a2 x + b2 y + c2 z
= d2
a3 x + b3 y + c3 z
= d3
is a diagonal system if
|a1 | ≥ |b1 | + |c1 |
|b2 | ≥ |a2 | + |c2 |
|c3 | ≥ |a3 | + |b3 |.
March 30, 2015
12/43
The process of iteration in solving AX = B will converge quickly if the
cofficient matrix A is diagonally dominant.
If the coefficient matrix is not diagonally dominant we must rearrange the
equations in such a way that the resulting coefficient matrix becomes
dominant, if possible, before we apply the iteration method.
Exercises
1. Is the system of equations diagonally dominant? If not, make it diagonally dominant.
3x + 9y − 2z
=
10
4x + 2y + 13z
=
19
4x − 2y + z
=
3.
x + 6y − 2z
=
5
4x + y + z
=
6
−3x + y + 7z
=
5.
2. Check whether the system of equations
is a diagonal system. If not, make it a diagonal system.
March 30, 2015
13/43
Jacobi Iteration Method (also known as Gauss-Jacobi)
Simple iterative methods can be designed for systems in which the
coefficients of the leading diagonal are large as comparted to others.
We now describe three such methods. Gauss-Jacobi Iteration Method
Consider the system of linear equations
a1 x + b1 y + c1 z
= d1
a2 x + b2 y + c2 z
= d2
a3 x + b3 y + c3 z
= d3 .
If a1 , b2 , c3 are large as compared to other coefficients (if not, rearrange
the given linear equations), solve for x, y , z respectively. That is, the
cofficient matrix is diagonally dominant.
March 30, 2015
14/43
Then the system can be written as
x
=
y
=
z
=
1
(d1 − b1 y − c1 z)
a1
1
(d2 − a2 x − c2 z)
b2
1
(d3 − a3 x − b3 y ).
c3
(2)
Let us start with the initital approximations x0 , y0 , z0 for the values of
x, y , z repectively. In the absence of any better estimates for x0 , y0 , z0 ,
these may each be taken as zero.
Substituting these on the right sides of (2), we gt the first approximations.
This process is repeated till the difference between two consecutive
approximations is negligible.
March 30, 2015
15/43
Exercises
3. Solve, by Jacobi’s iterative method, the equations
20x + y − 2z
=
17
3x + 20y − z
=
−18
2x − 3y + 20z
=
25.
4. Solve, by Jacobi’s iterative method correct to 2 decimal places, the equations
10x + y − z
=
11.19
x + 10y + z
=
28.08
−x + y + 10z
=
35.61.
5. Solve the equations, by Gauss-Jacobi iterative method
10x1 − 2x2 − x3 − x4
=
3
−2x1 + 10x2 − x3 − x4
=
15
−x1 − x2 + 10x3 − 2x4
=
27
−x1 − x2 − 2x3 + 10x4
=
−9.
March 30, 2015
16/43
Gauss-Seidel Iteration Method
This is a modification of Jacobi’s Iterative Method.
a1 x + b1 y + c1 z
= d1
a2 x + b2 y + c2 z
= d2
a3 x + b3 y + c3 z
= d3 .
If a1 , b2 , c3 are large as compared to other coefficients (if not, rearrange
the given linear equations), solve for x, y , z respectively.
Then the system can be written as
x
=
y
=
z
=
1
(d1 − b1 y − c1 z)
a1
1
(d2 − a2 x − c2 z)
b2
1
(d3 − a3 x − b3 y ).
c3
(3)
March 30, 2015
17/43
Here also we start with the initial approximations x0 , y0 , z0 for x, y , z
respectively, (each may be taken as zero).
Subsituting y = y0 , z = z0 in the first of the equations (3), we get
x1 =
1
(d1 − b1 y0 − c1 z0 ).
a1
Then putting x = x1 , z = z0 in the second of the equations (3), we get
y1 =
1
(d2 − a2 x1 − c2 z0 ).
b2
Next subsituting x = x1 , y = y1 in the third of the equations (3), we get
z1 =
1
(d3 − a3 x1 − b3 y1 )
c3
and so on.
March 30, 2015
18/43
That is, as soon as a new approximation for an unknown is found, it is
immediately used in the next step.
This process of iteration is repeatedly till the values of x, y , z are obtained
to desired degree of accuracy.
Since the most recent approximations of the unknowns are used which
proceeding to the next step, the convergence in the Gauss-Seidel
method is twice as fast as in Gauss-Jacobi method.
March 30, 2015
19/43
Convergence Criteria
The condition for convergence of Gauss-Jacobi and Gauss-Seidel methods
is given by the following rule:
The process of iteration will converge if in each equation of the system,
the absolute value of the largest co-efficient is greater than the sum of the
absolute values of all the remaining coefficients.
That is, if the coefficient matrix is a diagonal dominant matrix, then the
process of iteration will converge.
For a given system of linear equations, before finding approximations by
using Gauss-Jacobi / Gauss-Seidel methods, convergence criteria should be
verified. If the coefficient matrix is a not diagonal dominant matrix,
rearrange the equations, so that the new coefficient matrix is diagonal
dominant. Note that all systems of linear equations are not diagonal
dominant.
March 30, 2015
20/43
Exercises
6. Apply Gauss-Seidel iterative method to solve the equations
20x + y − 2z
=
17
3x + 20y − z
=
−18
2x − 3y + 20z
=
25.
7. Solve the equations, by Gauss-Jacobi and Gauss-Seidel methods (and compare the values)
27x + 6y − z
=
85
x + y + 54z
=
110
6x + 15y + 2z
=
72.
8. Apply Gauss-Seidel iterative method to solve the equations
10x1 − 2x2 − x3 − x4
=
3
−2x1 + 10x2 − x3 − x4
=
15
−x1 − x2 + 10x3 − 2x4
=
27
−x1 − x2 − 2x3 + 10x4
=
−9.
March 30, 2015
21/43
Relaxation Method
a1 x + b1 y + c1 z
= d1
a2 x + b2 y + c2 z
= d2
a3 x + b3 y + c3 z
= d3 .
We define the residuals Rx , Ry , Rz by the relations
Rx
= d1 − a1 x − b1 y − c1 z
Ry
= d2 − a2 x − b2 y − c2 z
Rz
= d3 − a3 x − b3 y − c3 z.
(4)
March 30, 2015
22/43
To start with we assume that x = y = z = 0 and calculate the initial
residuals.
Then the residuals are reduced step by step, by giving increments to
the variables.
We note from the equations (4) that if x is increased by 1 (keeping y and
z constant), Rx , Ry and Rz decrease by a1 , a2 , a3 respectively. This is
shown in the following table along with the effects on the residuals when y
and z are given unit increments.
δx = 1
δy = 1
δz = 1
δRx
−a1
−b1
−c1
δRy
−a2
−b2
−c2
δRz
−a3
−b3
−c3
The table is the negative of transpose of the coefficient matrix.
March 30, 2015
23/43
At each step, the numerically largest residual is reduced to almost zero.
How can one reduce a particular residual?
To reduce a particular residual, the value of the corresponding variable is
changed.
For example, to reduce Rx by p, x should be increased by p/a1 .
When all the residuals have been reduced to almost zero, the increments
in x, y , z are added separately to give the desired solution.
Verification Process : The residuals are not all negligible when the
computed values of x, y , z are substituted in (4), then there is some
mistake and the entire process should be rechecked.
March 30, 2015
24/43
Convergence of the Relaxation Method
Relaxation method can be applied successfully only if there is at least one
row in which diagonal element of the coefficient matrix dominate the
other coefficients in the corresponding row.
That is,
|a1 | ≥ |b1 | + |c1 |
|b2 | ≥ |a2 | + |c2 |
|c3 | ≥ |a3 | + |b3 |
should be valid for at least one row.
March 30, 2015
25/43
Exercises
9. Solve by relaxation method, the equations
9x − 2y + z
= 50
x + 5y − 3z
= 18
−2x + 2y + 7z
= 19.
10. Solve by relaxation method, the equations
10x − 2y − 3z
= 205
−2x + 10y − 2z
= 154
−2x − y + 10z
= 120.
March 30, 2015
26/43
For any n × n matrix A, consider the polynomial
λ − a11 −a12
−a21 λ − a22
χA (λ) := |λI − A| = ···
···
−an1
−an2
···
···
···
···
−a1n
−a2n
···
λ − ann
.
(5)
Clearly this is a monic polynomial of degree n. By the fundamental
theorem of algebra, χ(A) has exactly n (not necessarily distinct) roots.
χA (λ)
χA (λ) = 0
the roots of χA (λ)
distinct roots of χA (λ)
the
A
the
the
the
characteristic polynomial of
characteristic equation of A
characteristic roots of A
spectrum of A
March 30, 2015
27/43
Few Observations on Characteristic Polynomials
The constant terms and the coefficient of λn−1 in χA (λ) are (−1)n |A|
and tr (A).
The sum of the characteristic roots of A is tr (A) and the product of
the characteristic roots of A is |A|.
Since λI − AT = (λI − A)T , characteristic polynomials of A and AT
are the same.
Since λI − P −1 AP = P −1 (λI − A)P, similar matrices have the same
characteristic polynomials.
March 30, 2015
28/43
Definition
A complex number λ is an eigen value of A if there exists x 6= 0 in Cn
such that Ax = λx. Any such (non-zero) x is an eigen vector of A
corresponding to the eigen value λ.
When we say that x is an eigen vector of A we mean that x is an eigen
vector of A corresponding to some eigen value of A.
λ is an eigen value of A iff the system (λI − A)x = 0 has a non-trivial
solution.
λ is a characteristic root of A iff λI − A is singular.
Theorem
A number λ is an eigen value of A iff λ is a characteristic root of A.
March 30, 2015
29/43
The preceding theorem shows that eigen values are the same as
characteristic roots. However, by ‘the characteristic roots of A’ we mean
the n roots of the characteristic polynomial of A whereas ‘the eigen values
of A’ would mean the distinct characteristic roots of A.
Equivalent names:
Eigen values
Eigen vectors
proper values, latent roots, etc.
characteristic vectors, latent vectors, etc.
March 30, 2015
30/43
Properties of Eigen Values
1. The sum and product of all eigen values of a matrix A is the sum of
principal diagonals and determinant of A, respectively. Hence a
matrix is not invertible if and only if it has a zero eigen value.
2. The eigen values of A and AT (the transpose of A) are same.
3. If λ1 , λ2 , . . . , λn are eigen values of A, then
(a)
(b)
(c)
(d)
1/λ1 , 1/λ2 , . . . , 1/λn are eigen values of A−1 .
kλ1 , kλ2 , . . . , kλn are eigen values of kA.
kλ1 + m, kλ2 + m, . . . , kλn + m are eigen values of kA + mI.
λp1 , λp2 , . . . , λpn are eigen values of Ap , for any positive integer p.
4. The transformation of A by a non-singular matrix P to P −1 AP is
called a similarity transformation.
Any similarity transformation applied to a matrix leaves its eigen
values unchanged.
5. The eigen values of a real symmetric matrix are real.
March 30, 2015
31/43
Cayley - Hamilton Theorem
Theorem
For every matrix A, the characteristic polynomial of A annihilates A.
That is, every matrix satisfies its own characteristic equation.
Main uses of Cayley-Hamilton theorem
1. To evaluate large powers A.
2. To evaluate a polynomial in A with large degree even if A is singular.
3. To express A−1 as a polynomial in A whereas A is non-singular.
March 30, 2015
32/43
Dominant Eigen Value
We have seen that the eigen values of an n × n matrix A are obtained by
solving its characteristic equation
λn + an−1 λn−1 + cn−2 λn−2 + · · · + c0 = 0.
For large values of n, polynomial equations like this one are difficult and
time-consuming to solve.
Moreover, numerical techniques for approximating roots of polynomial
equations of high degree are sensitive to rounding errors. We now look at
an alternative method for approximating eigen values.
As presented here, the method can be used only to find the eigen
value of A, that is, largest in absolute value – we call this eigen value the
dominant eigen value of A.
March 30, 2015
33/43
Dominant Eigen Vector
Dominant eigen values are of primary interest in many physical
applications.
Let λ1 , λ2 , . . . , and λn be the eigen values of an n × n matrix A.
λ1 is called the dominant eigen value of A if
|λi | > |λj |,
For example, the matrix
i = 2, 3, . . . , n.
1 0
has no dominant eigen value.
0 −1
The eigen vectors corresponding to λ1 are called dominant eigen vectors
of A.
March 30, 2015
34/43
Power Method
In many engineering problems, it is required to compute the numerically
largest eigen values (dominant eigen values) and the corresponding eigen
vectors (dominant eigen vectors).
In such cases, the following iterative method is quite convenient which is
also well-suited for machine computations.
Let x1 , x2 , . . . , xn be the eigen vectors corresponding to the eigen values
λ1 , λ2 , . . . , λn .
We assume that the eigen values of A are real and distint with
|λ1 | > |λ2 | > · · · > |λn |.
March 30, 2015
35/43
As the vectors x1 , x2 , . . . , xn are linearly independent, any arbitrary column
vector can be written as
x = k1 x1 + k2 x2 + · · · + kn xn .
We now operate A repeatedly on the vector x.
Then
Ax = k1 λ1 x1 + k2 λ2 x2 + · · · + kn λn xn .
Hence
Ax = λ1 k1 x1 + k2
λn
λ2
x2 + · · · + kn
xn .
λ1
λ1
and through iteration we obtain,
r
r λ2
λn
r
A x = λ1 k1 x1 + k2
x2 + · · · + kn
xn
λ1
λ1
(6)
provided λ1 6= 0.
March 30, 2015
36/43
Since λ1 is dominant, all terms inside the brackets have limit zero except
the first term.
That is, for large values of r , the vector
r
r
λ2
λn
k1 x1 + k2
x2 + · · · + kn
xn
λ1
λ1
will converge to k1 x1 , that is the eigen vector of λ1 .
The eigenvalue is obtained as
(Ar +1 x)p
n→∞ (Ar x)p
λ1 = lim
p = 1, 2, . . . , n
where the index p signifies the pth component in the corresponding vector.
That is, if we take the ratio of any corresponding components of Ar +1 x
and Ar x, this ratio should therefore have a limit λ1 .
March 30, 2015
37/43
Linear Convergence
Neglecting the terms of λ2 , λ3 , . . . , λn in (6) we get
Ar +1 x
λ2 Ar x
− x1
− x1 =
λ1 k1 λr1
k1 λ1r +1
which shows that the convergence is linear with convergence ratio
λ2 .
λ1 Thus, the convergence is faster than if this ratio is small.
March 30, 2015
38/43
Working Rule To Find Dominant Eigen Value
Choose x 6= 0, an arbitrary real vector.
Generally, we choose x as
 
 
1
1
1 , or, 0 .
1
0
In other words, for the purpose of computation, we generally take x with 1
as the first component.
We start with a column vector x which is as near the solution as possible
and evaluate Ax which is written as
λ(1) x (1)
after normalization. This gives the first approximation λ(1) to the eigen
value and x (1) is the eigen vector.
March 30, 2015
39/43
Rayeigh’s Power Method
Similarly, we evaluate Ax (2) = λ(2) x (2) which gives the second
approximation. We repeat this process till [x (r ) − x (r −1) ] becomes
negligible.
Then λ(r ) will be the dominant (largest) eigen value and x (r ) , the
corresponding eigen vector (dominant eigen vector).
The iteration method of finding the dominant eigen value of a square
matrix is called the Rayeigh’s power method.
At each stage of iteration, we take out the largest component from y (r ) to
find x (r ) where
y (r +1) = Ax (r ) ,
r = 0, 1, 2, . . . .
March 30, 2015
40/43
To find the Smallest Eigen Value
To find the smallest eigen value of a given matrix A, apply power method
to the matrix A−1 .
Find the dominant eigen value of A−1 and then take is reciprocal.
Exercise
11. Say true or false with justification: If λ is the dominant eigen value of
A and β is the dominant eigen value of A − λI , then the smallest
eigen value of A is λ + β.
March 30, 2015
41/43
Exercises
12. Determine the largest eigen value and the corresponding eigen vector
of the matrix
5 4
.
1 2
13. Find the largest eigen value and the corresponding eigen vector of the
matrix


2 −1 0
−1 2 −1
0 −1 2
using power method. Take [1, 0, 0]T as an initial eigen vector.
14. Obtain by power method, the numerically dominant eigen value and
eigen vector of the matrix


15 −4 −3
−10 12 −6 
−20 4 −2.
March 30, 2015
42/43
References
1. Richard L. Burden and J. Douglas Faires, Numerical Analysis Theory and Applications, Cengage Learning, Singapore.
2. Kendall E. Atkinson, An Introduction to Numerical Analysis, Wiley
India.
3. David Kincaid and Ward Cheney, Numerical Analysis Mathematics of Scientific Computing, American Mathematical
Society, Providence, Rhode Island.
4. S.S. Sastry, Introductory Methods of Numerical Analysis, Fourth
Edition, Prentice-Hall, India.
March 30, 2015
43/43

Numerical Solution of Linear Equations & Power Method

Transcription

Similar documents

Pure Mathematics

Math Analysis Honors – Worksheet 44 Multivariate Linear Systems

Package 'RcppEigen'

CL7103 SYSTEM THEORY QUESTON BANK UNIT I

Lesson 5 Enrich Fractional Equations

Restraining Order Against Sam Lufti

phase plane analysis of a peculiar case of ecological ammensalism

Solving Systems of equations Video Lessons Directions: Watch all