Teshale Shinato

SOME ITERATIVE METHODS FOR SOLVING SYSTEM OF
NONLINEAR EQUATIONS
MSc PROJECT
TESHALE SHINATO
MAY 2016
HARAMAYA UNIVERSITY, HARAMAYA
Some Iterative Methods for Solving System of Nonlinear Equations
A Project Submitted to the Department of Mathematics,
Postgraduate Program Directorate
HARAMAYA UNIVERSITY
In Partial Fulfillment of the Requirements for the Degree of
MASTER OF SCIENCE IN MATHEMATICS
(NUMERICAL ANALYSIS)
Teshale Shinato
May 2016
Haramaya University, Haramaya
HARAMAYA UNIVERSITY
Postgraduate Program Directorate
I hereby certify that I have read and evaluated this Project titled ‘Some Iterative Methods for
Solving System of Nonlinear Equations’ prepared under my guidance by Teshale Shinato. I
recommend that it be submitted as fulfilling the project requirement.
Seleshi Demie (PhD)
Advisor
__________________
Signature
___________________
Date
As member of the board of Examiners of the M.Sc Project Open Defense Examination, we
certify that we have read and evaluated the Project prepared by Teshale Shinato and examined
the candidate. We recommend that the Project be accepted as fulfilling the Project requirement
for the degree of Master of Science in Mathematics (Numerical Analysis).
Chairperson
________________
Signature
________________
Date
________________
Internal Examiner
________________
Signature
________________
Date
________________
External Examiner
________________
Signature
________________
Date
Final approval and acceptance of the project is contingent upon the submission of its final
copy to the Council of Graduate Studies (CGS) through the candidate’s department or school
of graduate committee (DGC or SGC).
ii
STATEMENT OF THE AUTHOR
By my signature below, I declare that this Project is my own work. I have followed all ethical
and technical principles of scholarship in the preparation, and compilation of this Project. Any
scholarly matter that is included in the Project has been given recognition through citation.
This Project is submitted in partial fulfillment of the requirements for the degree of Master of
Science in Mathematics (Numerical Analysis) at the Haramaya University. The Project is
deposited in the Haramaya University Library and is made available to borrowers under the
rules of the Library. I solemnly declare that this Project has not been submitted to any other
institution anywhere for the award of any academic degree, diploma or certificate.
Brief quotations from this Project may be made without special permission provided that
accurate and complete acknowledgement of the source is made. Requests for permission for
extended quotations from or reproduction of this Project in whole or in part may be granted by
the Head of the Department when in his or her judgment the proper use of the material is in
the interest of scholarship. In all other instances, however, permission must be obtained from
the author of the Project.
Name
Date
Department:
Mathematics
Signature
iii
ABBREVIATIONS
BM
Broyden’s method
NI
Number of Iterations
NM
Newton’s method
PCM-1
The predictor corrector method-1
PCM-2
The predictor corrector method-2
SNLEs
System of nonlinear equations
iv
BIOGRAPHICAL SKETCH
The author was born in 1989 on April 16 in SNNP Regional State, Hadiya Zone, Gibe wereda,
Megacho kebele. He attended his primary education at Muma primary school and joined
Yekatit 25/67 Secondary School for grades 9 and 10, and Wachamo Comprehensive
Secondary and Preparatory School for grades 11 and 12 which are found at the Hosanna
town. He then joined Dire Dawa University in 2007 and received Bachelor of Education
degree in Mathematics in July, 2009.
He has been employed in Southern Regional State, Hadiya Zone as a High school teacher on
August 15, 2009 and stayed at work for four years in the school.
After working for four years in that school, he then joined Postgraduate Program at Haramaya
University, College of Natural and Computational Sciences, Department of Mathematics on
October, 2013 to pursue a program of study for MSc degree in Mathematics (Numerical
Analysis).
v
ACKNOWLEDGEMENT
I would like to express my sincere gratitude to my advisor Dr. Seleshi Demie next to
Almighty God for his profound commitment and encouragements he has shown throughout
my study period. I pay respect and express indebtedness to him because of his inspiring
guidance, consistent supervision especially his suggestions in selection of tools from the grass
root of the proposal and at every phase of this project work. Without his knowledge, direction,
guidance, and all of his help, this project would not have been achieveable.
My gratitude goes to Gibe Woreda, particularly, to the Woreda’s Education Sector for the
chance they have given me without any opposition and accepting my questions for joining the
postgraduate program to study M.Sc. in mathematics and for their provision of the necessary
support to complete the postgraduate study at Haramaya University.
I would also like to acknowledge how thankful I am for my Mom and Dad. Without their
unconditional love and support, and always being by my side with prayer and encouragement
throughout the years, it would have been so much difficult to be successful. Also I have a
heartfelt gratitude for all my brothers and sisters especially for the two elder brothers, Tade
and Fike for their support in all necessary conditions. I strongly understood that it was
impossible to achieve a lot of my goals without their support.
vi
TABLE OF CONTENTS
STATEMENT OF THE AUTHOR
iii
ABBREVIATIONS
iv
BIOGRAPHICAL SKETCH
v
ACKNOWLEDGEMENT
vi
LIST OF TABLES
ix
ABSTRACT
x
1.INTRODUCTION
1
1.1.Background of the Study
1
1.2.Statement of the Problem
3
1.3.Objectives of the Study
4
2.LITERATURE REVIEW
6
2.1.Iterative Methods for Solving SNLEs
6
2.2.Quadrature Formula Based Methods for Solving SNLEs
8
3.MATERIALS AND METHODS
10
4.SOME PRELIMINARY CONCEPTS
11
4.1.Some Relevant Terms
11
4.2.Some Quadrature Formulas
16
5.SINGLE STEP ITERATIVE METHODS FOR SOLVING SNLEs
17
5.1.Newton Method
17
5.2.Broyden’s Method
21
6.TWO ITERATIVE METHODS WITH TWO STEPS FOR SOLVING
SNLEs
27
6.1.Predictor-corrector Methods 1 and 2
27
6.2.Convergence Analysis of the Two Methods
30
Continues…
vii
6.2.1.Proof of cubic order of convergence for algorithm 6.2
30
6.2.2.Proof of cubic order of convergence for algorithm 6.4
36
7.NUMERICAL RESULTS OF THE FOUR METHODS
45
8.SUMMARY, CONCLUSION AND RECOMMENDATION
61
8.1.Summary
61
8.2.Conclusion
62
8.3.Recommendation
62
9.REFERENCES
63
10.APPENDICES
66
viii
LIST OF TABLES
Table
Page
1: Iteration result of Newton’s method for Example 7.1
46
2: The iteration result of PCM-1 for Example 7.1
49
3: The iteration result of PCM-2 for Example 7.1
51
4: The summary of the iteration results for Example 7.1 by Broyden’s method
53
5: Comparison of BM, NM, PCM-1 and PCM-2 for Example 7.1
53
6: The summary of iteration results of Newton’s method for Example 7.2
54
7: The summary of iteration results for PCM-1 in Example 7.2
55
8: The iteration result of PCM-2 for the system given in Example 7.2
55
9: The iteration result of the Broyden’s method for the system given in Example 7.2
55
10: Comparison of BM, NM, PCM-1 and PCM-2 for Example 7.2
56
11: Iteration results of BM, NM, PCM-1 and PCM-2 for Example 7.3
57
12: Iteration results of BM, NM, PCM-1 and PCM-2 for Example 7.4
58
13: Iteration results of BM, NM, PCM-1 and PCM-2 for Example 7.5
59
14: Iteration results of BM, NM, PCM-1 and PCM-2 for Example 7.6
60
ix
Some Iterative Methods for Solving System of Nonlinear Equations
ABSTRACT
System of nonlinear equations arises in different areas of applied sciences such as engineering,
physics, chemistry, medicine, and robotics. They appear also in many geometric computations
such as intersections, minimum distance, and when solving initial or boundary value problems in
ordinary or partial differential equations. Solving system of nonlinear equations is difficult when
compared to solving its linear counterpart and finding solutions of nonlinear problems is
important in the study of numerical analysis. Iterative methods such as Newton’s method,
Broyden’s method, Adomian decomposition method, Steepest Descent method, Homotopy
perturbation method, predictor-corrector methods, etc. are among the methods used to solve
nonlinear equations. This project investigated solutions of SNLEs by using Newton’s and
Broyden’s methods which are single step methods and two methods with two steps each which are
predictor-corrector method-1 and 2 (PCM-1 and 2). For both PCM-1 and PCM-2, Newton’s
method served as a predictor and algorithms 6.3 and 6.5 served as corrector, respectively. The
comparisons made among these methods were based on the number of iterations they took to
converge to the solution by considering numerical examples from SNLEs. The results indicated
that PCM-1 and PCM-2 took less number of iterations to converge and hence they performed
better than Newton and Broyden’s methods.
x
1. INTRODUCTION
1.1.
Background of the Study
Numerical analysis is the area of mathematics that creates, analyzes, and implements
algorithms for solving numerical problems. Such problems originate generally from real-world
applications of algebra, geometry and calculus. These problems also occur throughout the
natural sciences, social sciences, engineering, medicine, and business.
During the past half-century, the growth in power and availability of digital computers has led
to an increasing use of realistic mathematical models in science and engineering, and highly
sophisticated methods of numerical analysis have been needed to solve these more detailed
mathematical models of the world.
The formal academic area of numerical analysis varies from quite theoretical mathematical
studies (Atkinson and Han, 2001) to computer science issues (Edwards, 1997). With the
growth in importance of using computers to carry out numerical procedures in solving
mathematical models of the world, an area known as scientific computing or computational
science has taken shape during the 1980s and 1990s. This area looks at the use of numerical
analysis (Fosdick et al., 1996) from a computer science perspective. It is concerned with using
the most powerful tools of numerical analysis, computer graphics, symbolic mathematical
computations, and graphical user interfaces to make it easier for a user to set up, solve, and
interpret complicated mathematical models of the real world. Therefore, numerical analysis is
one of the fields of mathematics which uses different types of methods to solve complicated
mathematical problems in the areas stated above.
There are different kinds of challenging problems in science, engineering and other fields of
study which need to be solved. The most important problem among those problems is finding
the solution for system of nonlinear equations. There are simple formulas for solving linear
and quadratic equations; and there are also somewhat complicated formulas for cubic
equations. Besides polynomial equations, we have many problems in scientific and
engineering applications that involve the functions of transcendental nature.
2
Numerical methods are often used to obtain approximated solutions of some complicated
problems because those problems are not possible to obtain exact solution by usual algebraic
processes.
The solutions of systems of equations have a well-developed mathematical and computational
theory when solving linear systems, or a single nonlinear equation. The situation becomes
complicated when the equations in the system do not exhibit nice linear or polynomial
properties. In this case, both the mathematical theory and computational practices are far from
complete understanding of the solution process. So systems of nonlinear equations are one of
such types of problems. They arise in various domains of practical importance such as
engineering, medicines, chemistry, and robotics. They appear also in many geometric
computations such as intersections, minimum distance, and when solving initial or boundary
value problems in ordinary or partial differential equations.
As it is mentioned above, solving systems of nonlinear equations is more difficult when
compared with the linear counterpart. So finding solutions of these problems is one of the
most important parts of the studies in numerical analysis and because of this reason different
mathematicians developed different iterative methods to solve problems of systems of
nonlinear equations. For example, there are the following types of methods: methods derived
from Quadrature formulas, Newton’s method, Broyden’s method, Adomian decomposition
method, Steepest Descent method, Homotopy perturbation method, predictor-corrector
methods, etc.
This project paper mainly focused on Newton’s method, Broyden’s method and two iterative
methods with two steps each (predictor-corrector methods 1 and 2) for solving system of
nonlinear equations.
Historically Newton’s method was developed by Sir Isaac Newton around 1600. Most experts
agree that the road for the Newton’s method was opened by Francois Vieta in 1600
(Deuflhard, 2012). Isaac Newton was using this method to approximate numerical solutions of
scalar nonlinear equations. One of his first applications of this was on the numerical solution
of a cubic polynomial equation 𝑓(𝑥) = 𝑥 3 − 2𝑥 − 5 = 0 at 𝑥 (0) = 2 (Deuflhard, 2012). This
3
method later developed by Thomas Simpson after the derivatives concepts introduced.
Simpson played a great role for leaping Newton’s method from using on scalar equations to
system of equations (Deuflhard, 2012).
Broyden’s method is an iterative method developed by Charles Broyden around 1665. The
reason for the development of this method was the drawbacks in Newton’s method. Broyden
thought (Deuflhard, 2012) that the evaluation of Jacobian matrix at every iteration is a difficult
task and he need to overcome this problem. Then Broyden started the iteration by computing
the whole Jacobian only at the first iteration and then to use the rank-one updated approximate
matrix at the other iterations.
The Predictor-corrector methods 1 and 2 are iterative methods which are derived from closedopen quadrature formulas by using Taylor polynomial. It is well known that the quadrature
rules play an important and significant role in the evaluation of the integrals (Burden and
Faires, 2001; Noor, 2007a, b; Podisuk et al., 2007; Darvishi and Barati, 2004; Frontini and
Sormani, 2004; Cordero and Torregrosa, 2006; Cordero and Torregrosa, 2007; Darvishi and
Barati, 2007; Babajee et al., 2008). It has been shown that the quadrature formulas can be used
to develop some iterative methods for solving a system of nonlinear equations. The two
iterative methods with two steps each in this project paper are also methods developed from
quadrature formulas.
1.2.
Statement of the Problem
System of nonlinear equations is a problem that arises in different areas of studies as
mentioned in section 1.1. Finding solutions for such problems are not easy when compared
with other problems such as solving single nonlinear equations, linear equations, quadratic
equations, polynomial equations, equations of transcendental nature, system of linear
equations, and so on. Because of this a number of mathematicians tried to find different
iterative methods to solve problems of system of nonlinear equations. From those methods,
this project focused on four iterative methods from which two of them are methods with a
single step (i.e., Newton and Broyden’s methods) and the remaining two methods are methods
4
with two-steps each (i.e., predictor-corrector methods 1 and 2). For both predictor-corrector
methods 1 and 2, the predictor step is a Newton’s method and the corrector steps are the two
iterative methods, which were derived from quadrature formulas using Taylor series as it has
been shown in algorithms 6.3 and 6.5 of chapter-6.
So these are methods to solve SNLEs of the form:
𝑓1 (𝑥1 ,
𝑥2 ,
𝑓2 (𝑥1 , 𝑥2 ,
𝑓3 (𝑥1 ,
𝑥3 , ⋯ ,
𝑥𝑛 ) = 0
𝑥3 , ⋯ ,
𝑥𝑛 ) = 0
𝑥3 , ⋯ , 𝑥𝑛 ) = 0
𝑥2 ,
⋮
𝑓𝑛 (𝑥1 ,
𝑥2 ,
𝑥3 , ⋯ ,
𝑥𝑛 ) = 0
where each function 𝑓𝑖 can be thought of as mapping a vector X = [𝑥1 , 𝑥2 , 𝑥3 , ⋯ 𝑥𝑛 ]𝑇 of the
𝑛 dimensional space Rn into the real line R, for 𝑖 = 1,2,3, … . The general system of nonlinear
equations in 𝒏 unknowns can be alternatively represented by a function F, mapping from R n
into Rn, as
𝑓1 (𝑥1 , 𝑥2 , 𝑥3 , … … , 𝑥𝑛 )
0
𝑓2 (𝑥1 , 𝑥2 , 𝑥3 , … … , 𝑥𝑛 )
0
F(𝑥1 , 𝑥2 , 𝑥3 , … … , 𝑥𝑛 ) = 𝑓3 (𝑥1 , 𝑥2 , 𝑥3 , … … , 𝑥𝑛 ) = 0
⋮
⋮
[
0]
[𝑓𝑛 (𝑥1 , 𝑥2 , 𝑥3 , … … , 𝑥𝑛 )]
In a vector notation system, equation (2) can be written as: F(X) = 0
The functions, f1(x1, x2, …, xn), f2(x1, x2, ..., xn), f3(x1, x2, . . . ., xn), . . .,and fn(x1, x2, …, xn)
are called the coordinate functions of F(X).
The problem of solving a nonlinear system of equations is equivalent to the statement:
For F: D⊆Rn → Rn , find x ∗ ∈ Rn such that F (x ∗ )= 0, where F(x) is some nonlinear vectorvalued function with n unknowns.
1.3.
Objectives of the Study
The general objective of this project is to study four iterative methods (i.e., Newton’s method,
Broyden’s method, predictor-corrector method-1 and predictor-corrector method-2), which are
5
used to solve SNLEs and to compare the performance of these methods one with the other
based on the number of iterations taken to reach to the approximate solution.
The study is intended to achieve the following specific objectives;
 To derive the iterative schemes for the four methods.
 To prove the cubic order of convergence for the four methods.
 To use these methods to solve system of nonlinear equations.
 To compare the four methods based on the number of iterations they took to converge
to the solution.
6
2. LITERATURE REVIEW
2.1. Iterative Methods for Solving SNLEs
In order to solve the problems of SNLEs encountering in everyday life activities, many
numerical iterative methods were developed by different researchers. In this section we deal
with some of these methods.
Darvishi (2009) used a two-step higher order Newton-like method for solving systems of
nonlinear equations. The method considers the problem of finding a real root of a function F:
̅ of the system of nonlinear equations F(x) = 0 of n equations
Rn →Rn, that is a real solution, 𝒙
with n variables. This solution can be obtained as a fixed point of some function G: Rn →Rn by
means of the fixed point iteration method.
Darvishi and Barati (2007) derived a super cubic iterative method from the Adomian
decomposition method to solve systems of nonlinear equations. The authors showed that the
method is third-order convergent using classical Taylor expansion but Babajee et al. (2008)
introduced a note on the local convergence of iterative methods based on Adomian
decomposition method and 3-node quadrature rule. They formulated their method by using
Ostrowski’s technique based on point of attraction to solve system of nonlinear equations. The
authors showed that the method of Darvishi and Barati (super cubic iterative methods to solve
systems of nonlinear equations) was in fact the fourth order convergent not the third order.
They pointed out that the new method is better than that of the classic Darvishi and Barati
methods after the proof of point of attraction theorem.
Noor et al. (2015) introduced a new family of iterative methods to solve a system of nonlinear
equations, F(X) = 0 by using a new decomposition technique which is quite different from
Adomian decomposition method. In the implementation of Adomian decomposition, one has
to calculate the derivatives of the so-called Adomian polynomial, which is itself a difficult
problem. To overcome the difficulty of Adomian method they used the decomposition of
Daftardar-Gejji and Jafari (2006) as developed by Noor et al. (2006) to suggest new iterative
methods for solving system of nonlinear equations. Their method does not involve the higher-
7
order differentials of the functions and is very simple as compared with the Adomian
decomposition.
Furthermore, by improving Newton’s method, Chun (2006) has presented a new iterative
method to solve nonlinear equations. By using Adomain decomposition, Darvishi et al.
(2007a; b) have constructed new methods, and Golbabai and Javidi (2007) have applied the
homotopy perturbation method to build a new family of Newton-like iterative methods for
solving system of nonlinear equations. Ozel (2010) has considered a new decomposition
method for solving the system of nonlinear equations. Hafiz and Bahgat (2012) modified
Householder iterative method for solving system of nonlinear equations.
Broyden, C. (1965) formulated a very important iterative method, the so called a class of
iterative methods for solving system of nonlinear equations. The reason that made him to
introduce this method is because of the challenges that faced him when solving problems of
system of nonlinear equations by using Newton’s method. He was well aware of the
shortcomings of Newton’s method and thought how to overcome the problem, especially, the
problem in computing the Jacobian matrix and finding its inverse at each and every iteration
which was difficult, time consuming and expensive operation, particularly, during solving
large sized system of nonlinear equations. This method has preferred to use an updating
approximation matrix instead of the Jacobian matrix. By doing this, Broyden simplified the
cost of computation of Jacobian matrix in the iteration process of solving SNLEs. There are
advantages and some drawbacks of this method. The advantage of Broyden’s method is the
reduction of computations, specifically the way that the inverse of the approximation matrix
can be computed directly from the previous iteration for the next step reduces the number of
computations in comparison to Newton’s method. The drawback of this method is that the
method does not converge quadratically. This means that more iteration number may be
needed to reach the solution, when compared to the number of iterations that the Newton
method requires.
Anuradha and Jaiswal, (2013) formulated the third and fourth order iterative methods for
solving systems of nonlinear equations. The main objective of their work was to extend the
third order iterative method for solving single nonlinear equations into system of nonlinear
equations. They were also motivated to develop the method which improves the order of
8
convergence of Newton’s method with minimum number of functional evaluations.
To
achieve this goal, Anuradha and Jaiswal used a weighted method. To compare their
computational efficiency they used not only traditional ways but also recently introduced
flops-like based concepts. Finally, from their work they found that the efficiency of
quadratically convergent multidimensional method is not satisfactory in the most of the
practical problems so that the method extended from the third order method for solving single
nonlinear equations into multivariate case to construct the fourth order convergence by using
weighted method concept and it is much more better than the other compared methods. The
sense of their efficiency index was not dependent on only the number of functional evaluations
but also on the number of operations per iterations. They have given the comparison of
efficiencies based on flops and functional evaluations. They showed that the traditional
efficiency index of the fourth order method is same but flops-like efficiency index of their
method dominates the other methods.
Hosseini (2010) developed an improved homotopy analysis method for solving a system of
nonlinear algebraic equations and compared this method with Newton-Raphson method and
homotopy perturbation method. They have shown that this proposed method is very effective
and convenient in performance than the other two methods, namely Newton-Raphson and
homotopy perturbation. The proposed method converges fast to the solution as compared with
Newton-Raphson and homotopy perturbation methods.
2.2.
Quadrature Formula Based Methods for Solving SNLEs
Mamat et al. (2014) formulated a trapezoidal Broyden’s method for solving SNLEs. In this
method, the authors used the trapezoidal rule to solve the problems and in the method, the two
point predictor-corrector approach was used, where the Broyden’s method is the predictor and
the proposed method is the corrector. They compared the performance of the proposed method
with three Newton-like methods namely Newton’s method (NM), Broyden’s method (BM)
and Fixed-Newton’s method (FM) and showed the performance of the proposed method to be
better than the methods compared with it because of taking fewer numbers of iterations until
the process converges. Hafiz and Bahgat (2012) proposed and analyzed a new method the so
called an efficient two step-iterative method for solving system of nonlinear equations. This is
9
a predictor-corrector method developed by using the weight combination mid-point,
trapezoidal and quadrature formulas. They used a Newton’s method as a predictor step and the
new method as the corrector step of their method. Hafiz and Bahgat compared their method
with a Homotopy perturbation and Newton-Raphson methods based on their performance.
Some numerical examples were used to show the efficiency of the method relative to the
compared methods. The result of the comparison indicates that the method developed by Hafiz
and Mohamed is very effective and provided highly accurate results in a less number of
iterations than that of Homotopy perturbation’s method and Newton-Raphson method.
Frontini and Sormani, (2004) developed third-order methods from quadrature formulae for
solving SNLEs. Their method is the extension from single dimensional to n-dimensional
problems of system of nonlinear equations. They made a modification on a Newton’s method
based on quadrature formulas of order at least one, which produces iterative methods with
order of convergence three. In this method the authors have given a general error analysis
providing the higher order of convergence. The effectiveness of their methods is tested by
taking some counter examples from system of nonlinear equations. The comparison is also
method with some other existing methods to show how fast and effective are the methods.
Hassan and Waziri (2015) presented a Broyden-like update via some quadrature formulas for
solving system of nonlinear equations. Their method is a new alternative approximation based
on the quasi-Newton approach for solving SNLEs using the average of midpoint and
Simpson’s quadrature formulas. The goal of this method is to enhance the efficiency of
Broyden’s method by reducing the number of iterations it takes to reach the solution. The
authors have given the local convergence analysis and computational results showing the
relative efficiency of the proposed method. The computational experiment of the method is
given by using some numerical examples and compared to the existing classical methods.
Finally, the overall results of this new method imply that the proposed method is effective and
better than that of original Broyden’s method.
In this project, four iterative methods (Newton’s method, Broyden’s method, predictorcorrector methods 1 and 2) to solve a system of nonlinear equations were discussed and finally
some numerical examples were used to demonstrate the performance of the methods and
compared them depending up on the number of iterations.
10
3. MATERIALS AND METHODS
This project work has been developed through
 a number of reference books and materials which are available in the library of
Haramaya and Addis Ababa universities and also collected some quite relevant
information from the soft copies, internet, etc. Related journals, seminars and projects
were examined in detail and used to consolidate all the entire frameworks and skeleton
of the project.
 Some necessary definitions, theorems, and examples are contained in this project to
illustrate the essential idea of iterative methods for solving system of nonlinear
equations.
 MATLAB programs were used to minimize the high consumption of time and energy
that could happen during computations by hands and to accomplish the work easily.
11
4. SOME PRELIMINARY CONCEPTS
In this chapter, some preliminary concepts, relevant definitions and well known results, such
as norm, convex set, Frèchet derivative, lipschitz continuity, convergence of iterative methods,
stopping criteria, and some quadrature formulas have been presented to be helpful throughout
the study of this project.
4.1.
Some Relevant Terms
Iterative methods:
The term iterative method refers to a procedure that is repeated over and over again, to find the
root of an equation or find the solution of a system of equations. It is a technique that uses
successive approximations to obtain more accurate solutions to systems of equations at each
step. In a computational mathematics, an iterative method is a mathematical procedure that
generates a sequence of improving approximate solutions for a class of problems. A specific
implementation strategy of an iterative method, including the termination criteria, is an
algorithm of the iterative method.
Algorithms:
In mathematics and computer science, an algorithm is a step-by-step procedure for
calculations. Algorithms are used for calculation, data processing, and automated reasoning.
More precisely, an algorithm is an effective method expressed as a finite list of well-defined
instructions for calculating a function starting from an initial state and initial input perhaps
empty. The instructions describe a computation that, when executed, will proceed through a
finite number of well-defined successive states, eventually producing "output" and terminating
at a final ending state.
12
Vector Norm:
A vector norm on Rn is a function,‖. ‖ from Rn into R with the following properties:
(i)
‖X‖ ≥ 0 for all X∈Rn
(ii)
‖X‖ = 0 if and only if X=0
(iii) ‖𝛼X‖ = |𝛼|‖X‖ for all 𝛼 ∈R and X∈Rn
‖X + 𝑌‖ ≤ ‖𝑋‖ + ‖𝑌‖ for all X, 𝒀 ∈ Rn
(iv)
Vectors in Rn are column vectors, and it is convenient to use the transpose notation when a
vector is represented in terms of its components. For example, the vector
𝑥1
𝑥2
X= 𝑥3 , can be written as X= (𝑥1 , 𝑥2 , 𝑥3 , … , 𝑥𝑛 )T.
⋮
[𝑥𝑛 ]
If X= (𝑥1 , 𝑥2 , 𝑥3 , … , 𝑥𝑛 )T ∈Rn is a vector, then
1
(a) 𝑙𝑝 -norm of X is,
‖X‖p = {∑𝑛𝑖=1|𝑥𝑖 |𝑝 }𝑝 , 1≤ 𝑝 < ∞
If p = 1, then ‖X‖1 = |𝑥1 | + |𝑥2 | + |𝑥3 | + ⋯ + |𝑥𝑛 |.
(b) The 𝑙2 −norm for the vector X is called the Euclidean norm because it represents the
length of the vector denoted by
‖X‖ = ‖X‖2 = √|𝑥1 |2 + |𝑥2 |2 + |𝑥3 |2 + ⋯ + |𝑥𝑛 |2
(c) 𝑙∞ - Norm of X represents the absolute value of the largest component in the vector X.
It is denoted by,
‖X‖∞ = max |𝑥𝑖 |
1≤𝑖≤𝑛
Matrix Norm:
A matrix norm on the set of all n×n matrices is a real valued function, ‖. ‖ defined on this
set, satisfying the following properties for all n×n matrices A and B and a real number 𝛽:
(i)
‖A‖ ≥ 0
(ii)
‖A‖ = 0, If and only if A is 0, the matrix with all entries equal to zero.
(iii)
‖𝛽A‖ = |𝛽|‖A‖ for all 𝛽 ∈ R,
(iv)
‖A + B‖ ≤ ‖A‖ + ‖B‖,
13
(v)
‖AB‖ ≤ ‖A‖‖B‖.
The distance between 𝑛 × 𝑛 matrices A and B with respect to this matrix norm can be given
as‖A − B‖.
If A = [𝑎𝑖𝑗 ] is an n×n matrix, then one can define the norm of matrix A as:
(a) 𝑙1− norm
‖A‖1 = max ∑n𝑖=1|𝑎𝑖𝑗 |
1≤𝑗≤3
(b) 𝑙∞ − norm
‖A‖∞ = max ∑𝑛𝑗=1|𝑎𝑖𝑗 |
1≤𝑖≤𝑛
(c) 𝑙2 − norm
2
‖A‖2 = √∑𝑛𝑖=1 ∑𝑛𝑗=1(𝑎𝑖𝑗 ) = √∑𝑛𝑖=1{(𝑎𝑖1 )2 + (𝑎𝑖2 )2 + (𝑎𝑖3 )2 + ⋯ + (𝑎𝑖𝑛 )2 }
Sometimes there are functions that can be written in a matrix notation form when the functions
are as in the case of system of equations. So system of nonlinear equations having of the form
F(x1,x2 , x3 ,. . .,xn)= [f1(x1, x2, …, xn), f2(x1, x2, …, xn), . . .,fn(x1, x2, …, xn) ]T can be one of
such equations and written in a matrix form as:
f1 (x1 , x2 , x3 , … … , xn )
x1
f2 (x1 , x2 , x3 , … … , xn )
x2
(x
)
f
,
x
,
x
,
…
…
,
x
F(X) = 3 1 2 3
n , where X= x3
⋮
⋮
[xn ]
[fn (x1 , x2 , x3 , … … , xn )]
Jacobian Matrix:
A jacobian matrix is an 𝑛x𝑛 matrix of first order partial derivatives of component functions of
a system of nonlinear equations that can be written as
𝜕𝑓1
𝜕𝑓1
𝜕𝑓1
𝜕𝑥1
𝜕𝑓2
𝜕𝑥2
𝜕𝑓2
𝜕𝑥3
𝜕𝑓2
𝜕𝑥1
𝜕𝑓3
𝜕𝑥2
𝜕𝑓3
𝜕𝑥3
𝜕𝑓3
𝜕𝑥2
𝜕𝑥3
⋮
⋮
⋮
⋮
⋮
⋮
𝜕𝑓𝑛
𝜕𝑓𝑛
[𝜕𝑥1
𝜕𝑓𝑛
𝜕𝑥2
𝜕𝑥3
F ′ (X) = J(X) = 𝜕𝑥
1
…
𝜕𝑓1
𝜕𝑥𝑛
𝜕𝑓2
x1
x2
…
, where X = x3
𝜕𝑥𝑛
⋮
… ⋮
[x n ]
… ⋮
𝜕𝑓𝑛
…
𝜕𝑥𝑛 ]
…
𝜕𝑥𝑛
𝜕𝑓3
14
Convex set:
Let K be a non-empty set in Rn. The set K is said to be a convex set if for all U and V which
are subsets of K and for t ∈ [0, 1], then (1 - t) U + tV is again a subset of K.
Fréchet-Differentiability:
A mapping F : D⊆ Rn →Rn is said to be Frèchet-differentiable at X∈int(D), if there exists a
linear operator F′ (X) ∈ L(Rn) such that for any h ∈ Rn, then
lim
‖F(X+h)−F(X)−F′ (X)h‖
‖h‖
h→0
= 0.
Lipschitz Continuity:
A mapping F: D ⊆ Rn → Rn, is said to be Lipschitz continuous on D0 ⊆ D, if there exists a
constant L ≥ 0, such that, ∀ x1, x2∈D0,
‖F(x1 ) − F(x2 )‖ ≤ L‖x1 − x2 ‖
Order of Convergence:
Let 𝜶 be a root of the function F(X) = 0 and suppose X (k−2), X (k−1) , X (k) and X (k+1) are the
last four consecutive iterations which are very closer and closer to the true value 𝜶 of an
iterative method. Then, the computational order of convergence ρ of the iterative method can
be approximated by using the formula
ρ≈
ln(‖X (k+1) − X (k) ‖⁄‖X (k) − X (k−1) ‖)
ln(‖X (k) − X (k−1) ‖⁄‖X (k−1) − X (k−2) ‖)
There are different orders of convergences. These are as defined below.
 A sequence X (k+1) is said to be linearly convergent if X (k+1) converges to the actual root 𝜶
with order 𝑝 = 1, for a constant 𝛿 < 1 such that
‖X (k+1) − 𝛼‖
‖X (k+1) − 𝛼‖
lim
= lim
=𝛿
𝑘→∞ ‖X (k) − 𝛼‖𝑝
𝑘→∞ ‖X (k) − 𝛼‖1
 A sequence X (k+1) is said to be quadratically convergent if X (k+1) converges to the actual
root 𝜶 with order 𝑝 = 2 such that
‖X (k+1) − 𝛼‖
=𝛿
𝑘→∞ ‖X (k) − 𝛼‖2
lim
 A sequence X (k+1) is said to be super linearly convergent if
‖X (k+1) − 𝛼‖
=0
𝑘→∞ ‖X (k) − 𝛼‖2
lim
15
The higher the value of the order of convergence p is, the more rapid the convergence of the
sequence is.
Stopping (or Termination) Criteria:
The iterative method should be stopped if the approximate solution is accurate enough. A
good termination criterion is very important, because if the criterion is too weak the solution
obtained may be useless, whereas if the criterion is too severe the iteration process may never
stop or may cost too much work. The iteration process would be stopped when iteration error
is just small enough. The difference between a computed iterate and the true solution of a
linear system, measured in some vector norm. Mathematically;
Let {X (k) }𝒌≥𝟎 be a sequence in Rn convergent to 𝜶, and ∈>0 be a given tolerance. Then, one of
the following conditions can be used as termination (stopping) criteria:
‖F(X (k) )‖ < 𝜀
‖X (k) − X (k−1) ‖ < 𝜀
Or,
‖𝐗 (𝐤) −𝐗 (𝐤−𝟏) ‖
‖𝐗 (𝐤) ‖
< 𝜀, X (k) ≠ 0
The main issue of the iterative method is to check or to prove if the sequence really converges
to a fixed point X ∗ . If not, we say that the method diverges. If the method diverges no matter
how close the initial approximation X (0) to the fixed point X ∗ is, we say that the algorithm is
unstable. Otherwise, the algorithm could be stable for smaller errors 𝑒 (𝑘) = X (k) − X ∗ but it
could diverges for larger errors 𝑒 (𝑘) .
Convergence of Iterative Methods:
An iterative method is said to be convergent if the corresponding sequence converges for the
given initial approximations. Or, a sequence is said to be convergent if it has a limit. An
iterative method converges for fixed initial guesses if and only if X (𝑘) = X ∗ 𝑎𝑛𝑑 𝐹(X ∗ ) = 0.
Banach Lemma: Let C∈ 𝑅 𝑛×𝑛 with ‖𝐶‖ < 1, then I + C is invertible and
1
1
≤ ‖(I + C)−1 ‖ ≤
1 + ‖𝐶‖
1 − ‖𝐶‖
16
Existence and Uniqueness of Solutions for SNLEs:
It is often difficult to determine the existence or number solutions to SNLEs. For system of
linear equations the number of solutions must be zero, one or infinitely many. But nonlinear
equations can have any number of solutions. Thus, determining existence and uniqueness of
solutions is more complicated for nonlinear system of equations than for linear equations.
Although it is difficult to make any assertion about solution of nonlinear equations, there are
nevertheless some useful local criteria that guarantee existence of a solution. The simplest for
these is for one-dimensional problems, for which the Intermediate Value Theorem provides a
sufficient condition for a solution, which says that if f is continuous on a closed interval [a, b],
and c lies between f(a) and f(b), then there is a value x*∈[a,b] such that f(x*) = c. Thus, if f(a)
and f(b) differ in sign, then by taking c = 0 in the theorem we can conclude that there must be
a root within the interval [a,b]. Such an interval [a, b] for which the sign of f differs at its
endpoints is called a bracket for a solution of the one-dimensional nonlinear equation f(x) = 0.
Note: There is no simple analog for n dimensions.
The nonlinear equations can have any number of solutions. It can have a simple as well as
multiple roots.
4.2.
Some Quadrature Formulas
The following are some useful quadrature formulas (Burden and Faires, 2001; Podisuk et al.,
2007), which are needed in the study of the next chapters.
One point open quadrature formula (Midpoint rule) is given by
𝑏
𝑎+𝑏
∫𝑎 𝑓(𝑥)𝑑𝑥 ≈ (𝑏 − 𝑎)𝑓 (
2
)
(4.1)
Two point closed quadrature formula (Trapezoidal rule) is also given by:
𝑏
∫𝑎 𝑓(𝑥)𝑑𝑥 ≈
(𝑏−𝑎)
2
[𝑓(𝑎) + 𝑓(𝑏)]
(4.2)
A two point closed-open quadrature formula is given by:
𝑏
∫𝑎 𝑓(𝑥)𝑑𝑥 ≈
(𝑏−𝑎)
4
𝑎+2𝑏
[𝑓(𝑎) + 3𝑓 (
3
)]
(4.3)
Another two point closed-open quadrature formula is given by:
𝑏
∫𝑎 𝑓(𝑥)𝑑𝑥 ≈
(𝑏−𝑎)
4
2𝑎+𝑏
[3𝑓 (
3
) + 𝑓(𝑏)]
(4.4)
17
5. SINGLE STEP ITERATIVE METHODS FOR SOLVING SNLEs
As it is discussed earlier in the background section, there are a number of iterative methods
developed by a number of researchers to solve system of nonlinear equations. Some of these
methods are methods containing a single step or they are methods containing two or more
steps. Here in this chapter, the two most popular numerical iterative methods with a single step
are discussed.
5.1.
Newton Method
Newton’s method is one of the most popular iterative methods that is used to approximate the
roots of a given system of nonlinear equations. The historical road to Newton’s method is
interesting in its own right, setting aside the technical details. Sir Isaac Newton did indeed
play a role in its development, but there were certainly other key players. Despite the
uncertainties, most experts agree that the road begins with a perturbation technique for solving
scalar polynomial equations, pioneered by Francois Vieta in 1600 (Deuflhard, 2012). This
method used one decimal place of the computed solution on each steps, and converged
linearly (Deuflhard, 2012). In 1664 Isaac Newton learned of Vieta’s method, and by 1669 he
had improved it by linearizing the resulting equations (Deuflhard, 2012). One of his first
applications of this newfangled scheme was on the numerical solution of the cubic polynomial
equation 𝑓(𝑥) = 𝑥 3 − 2𝑥 − 5 = 0. By using an initial guess 𝑥 (0) = 2, he created an iterative
method by substituting 𝑥 (𝑘) = 𝑥 (𝑘−1) + δ and solved for δ (neglecting terms higher than the
first order), and repeated the process, etc, which produced a better and better approximation to
the true root (Deuflhard, 2012). Newton realized that by keeping all decimal places of the
corrections, the number of accurate places would double, which means quadratic convergence.
However, there is no evidence that Newton incorporated derivatives in his method. In 1690,
Joseph Raphson improved Newton’s method of 1669 by making the scheme fully iterative,
making computation of successive polynomials unnecessary (Deuflhard, 2012). But it was in
1740 when the derivative aspect of ”Newton’s Method” was brought to the fore by Thomas
Simpson (Deuflhard, 2012). Simpson extended Newton’s formulation from 1669 by
introducing derivatives, and wrote the mathematical formula for the iterative method for both
a scalar nonlinear equation, and a system of two equations and two unknowns (Deuflhard,
18
2012). For a scalar equation and some initial guess 𝑥 (0) , this formulation was given by the
equation,
𝑥 (𝑘) = 𝑥 (𝑘−1) −
𝑓(𝑥 (𝑘−1) )
,
𝑓 ′ (𝑥 (𝑘−1) )
where 𝑘 = 1,2, …
Thus, it was Simpson who made the leap from using this procedure on scalar equations to
systems of equations.
Systems of equations arise so frequently in not only just pure mathematics, but also in the
physical science and engineering disciplines. The physical laws that direct mathematicians’
and engineers’ research can almost always be transformed from a continuous system to a
discrete system of equations, and more often than not, they are nonlinear. In fact, nonlinear
systems abound in mathematical problems. This makes the phenomena being modeled more
diverse and interesting, but, the price to pay is that finding solutions are nontrivial. The two
main classes of methods used for solving nonlinear algebraic equations are Newton and quasiNewton method. Equivalently, it could be said that the two classes of methods are derivative
and derivative-free, respectively.
Assume F: D⊆Rn→Rn is r- times Fréchet differentiable function in D and has a unique
solution, X (∗) ∈ Rn which is the zero of the given system of nonlinear equations, then, for
any x (k+1), x (k) ∈D performing the Taylor expansion of F at point X (k) , we have:
1 ′′ (k)
2
F (X )(X (∗) − X (k) )
2!
1
1
3
4
+ F ′′′ (X (k) )(X (∗) − X (k) ) + F (iv) (X (k) )(X (∗) − X (k) ) + ⋯
3!
4!
F(X (∗) ) = F(X (k) ) + F ′ (X (k) )(X (∗) − X (k) ) +
2
⟹ F(X (∗) ) = F(X (k) ) + F ′ (X (k) )(X (∗) − X (k) ) + O (‖X (∗) − X (k) ‖ ),
where F ′ (X (k) ) is a Jacobian matrix associated with F.
2
The Newton’s method is based on a linear approximation, so that O(‖X (∗) − X (k) ‖ ) and
higher order terms are neglected; i.e., if we take the first two terms of the Taylor’s series
expansion we have:
F(X (∗) ) ≈ F(X (k) ) + F ′ (X (k) )(X (∗) − X (k) )
Since X (∗) is the root of F, then we obtain
19
F(X (k) ) + F ′ (X (k) )(X (∗) − X (k) ) = 0
⟹ F ′ (X (k) )(X (∗) − X (k) ) = −F(X (k) )
Multiplying both sides by the inverse of F ′ (X (k) ) if it exists, will give the following result:
(F ′ (X (k) ))
−1
−1
F ′ (X (k) )(X (∗) − X (k) ) = − (F ′ (X (k) ))
⟹ X (∗) − X (k) = − (F ′ (X (k) ))
−1
⟹X (∗) = X (k) − (F ′ (X (k) ))
−1
F(X (k) )
F(X (k) )
F(X (k) )
−1
⟹ x (k+1) = X (k) − F ′ (X (k) ) F(X (k) )
(5.1)
where, F ′ (X k ) is a Jacobian Matrix evaluated at a point X 𝑘 , for k= 0, 1, 2, …, and take
x (k+1) = X (∗) .
So equation (5.1) is called Newton’s method for solving SNLEs F(X) = 0 and is generally
expected to give a quadratic convergence. That means, if an initial vector X (0) is chosen such
that it is sufficiently close to the true root X (∗) and J(x) is nonsingular and Lipschitz
continuous for all 𝑥 close to X (∗) , then the Newton’s method enjoys a quadratic convergence,
meaning the error at iteration k is proportional to the square of the error at iteration k-1
(Martinez, 2000).
Quadratic Convergence Analysis of Newton’s Method:
The Newton method is best known for its quadratic convergence. Toward this, we first prove a
useful lemma.
Lemma 1: Let F: Rn →Rn be continuously differentiable in an open convex set D⊆Rn.
Suppose a constant 𝛾 exists such that ‖𝐹 ′ (𝑥) − 𝐹 ′ (𝑦)‖ ≤ 𝛾‖𝑥 − 𝑦‖ for all 𝑥, 𝑦 ∈ 𝐷.
𝛾
Then, ‖𝐹(𝑥) − 𝐹(𝑦) − 𝐹 ′ (𝑦)(𝑥 − 𝑦)‖ ≤ 2 ‖𝑥 − 𝑦‖2
Proof: By the line integration,
1
𝐹(𝑥) − 𝐹(𝑦) = ∫ 𝐹 ′ (𝑦 + 𝑡(𝑥 − 𝑦)) (𝑥 − 𝑦)𝑑𝑡
So, 𝐹(𝑥) − 𝐹(𝑦) − 𝐹
′ (𝑦)(𝑥
− 𝑦) =
0
1 ′
∫0 𝐹 (𝑦
+ 𝑡(𝑥 − 𝑦) − 𝐹 ′ (𝑦)) (𝑥 − 𝑦)𝑑𝑡
1
′
⟹ ‖𝐹(𝑥) − 𝐹(𝑦) − 𝐹 (𝑦)(𝑥 − 𝑦)‖ ≤ ∫‖𝐹 ′ (𝑦 + 𝑡(𝑥 − 𝑦) − 𝐹 ′ (𝑦))‖ ‖(𝑥 − 𝑦)‖𝑑𝑡
0
20
1
𝛾
≤ ∫0 𝛾𝑡‖𝑥 − 𝑦‖2 𝑑𝑡 = 2 ‖𝑥 − 𝑦‖2
Theorem: Let F: Rn →Rn be continuously differentiable in an open convex set D⊆Rn. Assume
that ∃𝜉 ∈ Rn and 𝛽, 𝛾 > 0 such that

F(𝜉) = 0

F ′ (𝜉)−1 exists

‖F ′ (𝜉)−1 ‖ ≤ 𝛽 and

‖F ′ (𝑥) − F ′ (𝑦)‖ ≤ 𝛾‖𝑥 − 𝑦‖ for 𝑥, 𝑦 in a neighborhood of 𝜉. Then ∃∈> 0 such that
for all 𝑥0 ∈ (𝜉 , ∈), the sequence {𝑥 (𝑘) } defined by 𝑥 (𝑘+1) = 𝑥 (𝑘) + 𝑠𝑘 and
𝐹 ′ (𝑥 (𝑘) )𝑥 (𝑘) = −F(𝑥 (𝑘) ) is well defined and converges to 𝜉 and satisfies,
‖𝑥 (𝑘+1) − 𝜉‖ ≤ 𝛽𝛾‖𝑥 (𝑘) − 𝜉‖
2
Proof:
By continuity of F ′ , choose ∈≤ 𝑚𝑖𝑛 {𝛾,
𝑥 ∈ 𝑁(𝜉,
1
2𝛽𝛾
} so that 𝐹 ′ (𝑥) is nonsingular for all
∈). For 𝑘 = 0, ‖‖𝑥 (0) − 𝜉‖‖ <∈.
1
So, ‖F ′ (𝜉)−1 (𝐹 ′ (𝑥 (0) ) − F ′ (𝜉))‖ ≤ ‖F ′ (𝜉)−1 ‖‖𝐹 ′ (𝑥 (0) ) − F ′ (𝜉)‖ ≤ 𝛽𝛾‖𝑥 (0) − 𝜉‖ ≤ 2.
−1
By Banach Lemma, ‖F ′ (𝜉)−1 ‖ = ‖[F ′ (𝜉)−1 + (𝐹 ′ (𝑥 (0) ) − F ′ (𝜉))] ‖
≤
‖F ′ (𝜉)−1 ‖
≤ 2‖F ′ (𝜉)−1 ‖ ≤ 2𝛽
1 − ‖F ′ (𝜉)+−1 (𝐹 ′ (𝑥) − F ′ (𝜉))‖
Now, 𝑥1 − 𝜉 = 𝑥 0 − 𝜉 − F ′ (𝑥 (0) )−1 𝐹 ′ (𝑥 (0) ) = 𝑥 0 − 𝜉 − F ′ (𝑥 (0) )−1 (𝐹(𝑥 (0) ) − F(𝜉))
= F ′ (𝑥 (0) )−1 [F(𝜉) − 𝐹(𝑥 (0) ) − 𝐹 ′ (𝑥 (0) )(𝜉 − 𝑥 (0) )]
𝛾
So, ‖𝑥1 − 𝜉‖ ≤ ‖F ′ (𝑥 (0) )−1 ‖‖F(𝜉) − 𝐹(𝑥 (0) ) − 𝐹 ′ (𝑥 (0) )(𝜉 − 𝑥 (0) )‖ ≤ 2𝛽 2 ‖𝜉 − 𝑥 (0) ‖
2
= 𝛽𝛾‖𝜉 − 𝑥 (0) ‖ (by Banach Lemma)
≤ 𝛽𝛾𝜖‖𝑥 (0) − 𝜉‖ ≤
The proof is completed by induction.
1 (0)
‖𝑥 − 𝜉‖ ≤ 𝜖
2
2
21
Note: The above theorem shows that the Newton’s method converges quadratically if F ′ (𝜉) is
non-singular and if the starting point is very close to 𝜉.
Advantages and Disadvantages of Newton’s method:
One of the advantages of Newton’s method is that it is not too complicated. The major
disadvantage associated with this method is that Jacobian matrix as well as its inversion has to
be calculated for each iteration. Calculating both Jacobian matrix and its inverse can be quite
time consuming depending on the size of SNLEs. Another problem that we may be challenged
with when using Newton’s method fails to converge. If Newton’s method fails to converge,
this will result in an oscillation between points.
5.2.
Broyden’s Method
Charles Broyden was a Physicist working with nonlinear reactor models for the English
electric industry company in 1960’s in Leicester (Griewank, 2012). He had to solve a problem
involving non-linear algebraic equations. Broyden was well aware of the shortcomings of the
Newton’s method and thought of the way to overcome them. When trying to solve nonlinear
systems that arose from the discretization of these models, he realized that instead of
repeatedly evaluating and factoring the Jacobian matrix in Newton’s method, he could use
secant information (i.e. function value diff erences and the previous solution information) to
directly compute an approximation matrix B (k) to the Jacobian (Griewank, 2012). In the onedimensional case, the problem is trivial because a unique approximation B(k) can be
computed by dividing the function diff erence by the solution diff erence. Thus, letting 𝑦 (𝑘) =
F(X (k) ) − F(X (k−1) ) and 𝛿 (𝑘) = X (k) − X (k−1) ,
B(k) =
𝑦 (𝑘)
≈ F ′(X (k) )
(𝑘)
𝛿
For multi-dimensional system of nonlinear equations, Broyden realized that he doesn’t
necessarily need to work with the true inverse of Jacobian matrix, but with a suitable
approximation matrix B (k) . His method is the generalization of the secant method to multiple
dimensions. Newton's method for solving the equation F(X) = 0 uses the Jacobian matrix at
every iteration and computing this Jacobian is a difficult and expensive operation. So the idea
22
behind Broyden's method is to compute the whole Jacobian only at the first iteration, and to do
a rank-one update at the other iterations. His method suggests that the Jacobian approximation
from the previous iteration, B(k−1), could be updated with a certain rank-one update. This
update would satisfy not only the secant condition, but also the condition that B(k) w =
𝑇
B(k−1) w for all directions w ∈ 𝑅 𝑛 orthogonal to 𝛿 (𝑘) (i.e.(𝑤 (𝑘) ) 𝛿 (𝑘) = 0) (Griewank, 2012).
The rank-one update is the case when an n×n matrix B has the form, B = A + u ⊗ v, where
u and v are given vectors in Rn, and u ⊗ v is the outer product of u and v, defined by
u1 v1 u1 v2 ⋯ u1 vn
u
u2 v2 ⋯ u2 vn
2 v1
u ⊗ v = uv T = [ ⋮
⋮
⋮ ]
⋱
un v1 un v2 ⋯ un vn
This modification of A to obtain B is called a rank-one update. This is because u ⊗ v has rank
one, since every column of u ⊗ v is a scalar multiple of u.
Now to gain a sense of how this method is developed, consider a linear approximation of F(X)
about some X (k) .
Suppose F: D⊆Rn→Rn is r- times Fréchet differentiable function in D and has a unique
solution, X (∗) ∈ Rn which is the zero of the given system of nonlinear equations, then, for
any x (k+1), x (k) ∈D after performing the Taylor expansion of F at some point X (k) , we have
the following condition.
F(X (k−1) ) ≈ F(X (k) ) + B(k) (X (k−1) − X (k) ), where B(k) = F′(X (k) ) and k = 1, 2, . . .
⟹ F(X (k) ) − F(X (k−1) ) = B(k) (X (k) − X (k−1) )
(5.2)
Let 𝛿 (𝑘) = X (k) − X (k−1) and 𝑦 (𝑘) = F(X (k) ) − F(X (k−1) ).
Now equation (5.2) becomes,
B(k) 𝛿 (𝑘) = 𝑦 (𝑘)
(5.3)
Equation (5.3) (Sauer, 2006) is called the secant relation. In order to determine a unique B (k) ,
Broyden pointed that since there is no new information obtained from the orthogonal
complement of 𝛿 (𝑘) , then there is no reason why B(k−1) would change in a direction w,
𝑇
where (𝛿 (𝑘) ) (𝑤) = 0. So rather than computing B(k) from the scratch, the previous
23
approximation B (k−1) should be updated such that both the secant condition and equation (5.4)
is satisfied.
B(k) w = B(k−1) w
(5.4)
An approximate Jacobian satisfying both equations (5.3) and (5.4) is given by
𝑇
B
(k)
=B
(k−1)
+
( 𝑦 (𝑘) −B(k−1) 𝛿 (𝑘−1) ) (𝛿 (𝑘−1) )
(5.5)
𝑇
(𝛿 (𝑘−1) ) 𝛿 (𝑘−1)
Finally, the iterative solution for Broyden’s method becomes:
X (k+1) = X (k) − (B(k) )
−1
(F(X (k) ))
(5.8)
Broyden’s method (Griewank, (1986)) converges super linearly.
Superlinear Convergence Analysis of Broyden’s Method
To prove the superlinear convergence of this method, let’s first see the following lemmas.
Lemma 2: Let F:Rn →Rn be continuously differentiable in an open convex set D⊆Rn. Suppose
a constant 𝛾 exists such that ‖𝐹 ′ (𝑥) − 𝐹 ′ (𝑦)‖ ≤ 𝛾‖𝑥 − 𝑦‖ for all 𝑥, 𝑦 ∈ 𝐷.
Then it holds that for all 𝑥, 𝑦, 𝜉 ∈ 𝐷,
𝛾
‖𝐹(𝑥) − 𝐹(𝑦) − 𝐹 ′ (𝜉)(𝑥 − 𝑦)‖ ≤ (‖𝑥 − 𝜉‖ + ‖𝑦 − 𝜉‖)‖𝑥 − 𝑦‖
2
Proof: By the line integral
1
‖𝐹 ′ (𝑥) − 𝐹 ′ (𝑦) − 𝐹 ′ (𝜉)(𝑥 − 𝑦)‖ = ‖∫‖𝐹 ′ (𝑦 + 𝑡(𝑥 − 𝑦) − 𝐹 ′ (𝜉))‖ ‖(𝑥 − 𝑦)‖𝑑𝑡‖
0
1
1
≤ 𝛾‖𝑥 − 𝑦‖ ∫‖𝑦 + 𝑡(𝑥 − 𝑦) − 𝜉‖ 𝑑𝑡 ≤ 𝛾‖(𝑥 − 𝑦)‖ ∫{𝑡‖𝑥 − 𝜉‖ + (1 − 𝑡)‖𝑦 − 𝜉‖}𝑑𝑡
0
≤
0
𝛾
(‖𝑥 − 𝜉‖ + ‖𝑦 − 𝜉‖)‖𝑥 − 𝑦‖
2
Lemma 3: Let F: Rn →Rn be continuously differentiable in an open convex set D⊆Rn.Suppose
a constant 𝛾 exists such that ‖𝐹 ′ (𝑥) − 𝐹 ′ (𝑦)‖ ≤ 𝛾‖𝑥 − 𝑦‖ for all 𝑥, 𝑦 ∈ 𝐷. The for X (k+1) ,
X (k) ∈ D, holds that
‖𝐵 (𝑘+1) − 𝐹′(𝜉)‖ ≤ ‖𝐵 (𝑘) − 𝐹′(𝜉)‖ + ‖X (k+1) − 𝜉‖ + ‖X (k) − 𝜉‖
Proof: By definition,
24
𝐵
(𝑘+1)
=𝐵
(𝑘)
−𝐹
′ (𝜉)
=𝐵
(𝑘)
−𝐹
′ (𝜉)
(∆F(X (k) ) − 𝐵 (𝑘) (𝑠𝑘 ))(𝑠𝑘 )T
+
(𝑠𝑘 )T (𝑠𝑘 )
(∆F(X
(𝑠𝑘 )(𝑠𝑘 )T
(𝑠𝑘 )(𝑠𝑘 )T
(𝐼 −
)
−
𝐹′(𝜉)
(𝐼
−
)
+
(𝑠𝑘 )T (𝑠𝑘 )
(𝑠𝑘 )T 𝑠𝑘
(k)
) − 𝐹′(𝜉)) (𝑠𝑘 )(𝑠𝑘 )T
(𝑠𝑘 )T (𝑠𝑘 )
(𝑠 )(𝑠 )T
(∆F(X(k) )−𝐹′ (𝜉))(𝑠𝑘 )(𝑠𝑘 )T
Taking norm, ‖𝐵 (𝑘+1) − 𝐹 ′ (𝜉)‖ ≤ ‖𝐵 (𝑘) − 𝐹 ′ (𝜉)‖ ‖𝐼 − (𝑠𝑘 )T (𝑠𝑘 )‖ + ‖
𝑘
𝑘
(𝑠𝑘 )T (𝑠𝑘 )
‖
(𝑠 )(𝑠 )T
But, ‖𝐼 − (𝑠𝑘 )T 𝑘 ‖ ≤ 1
𝑘
(𝑠𝑘 )
Therefore, the 3rd term is estimated by,
(∆F(X (k) ) − 𝐹 ′ (𝜉)) (𝑠𝑘 )(𝑠𝑘 )T
{[F (X(k+1) − F(X(k) ) − F′ (𝜉)(X(k+1) − X(k) ))]} (𝑠𝑘 )T
‖
‖=‖
‖
(𝑠𝑘 )T (𝑠𝑘 )
(𝑠𝑘 )T (𝑠𝑘 )
𝛾
≤ 2 (‖X (k+1) − 𝜉‖ + ‖X (k) − 𝜉‖) (By lemma 2)
Theorem: Let F: Rn →Rn be continuously differentiable in an open convex set D⊆Rn. Suppose
∃𝜉 ∈Rn, 𝛽, 𝛾 > 0 such that

F(𝜉) = 0

F ′ (𝜉)−1 exists

‖F ′ (𝜉)−1 ‖ ≤ 𝛽 and

‖F ′ (𝑥) − F ′ (𝑦)‖ ≤ 𝛾‖𝑥 − 𝑦‖ for 𝑥, 𝑦 in a neighborhood of 𝜉. Then ∃𝛿1 , 𝛿2 > 0
such that, if ‖X (0) − 𝜉‖ < 𝛿1 and ‖𝐵 (0) − 𝐹 ′ (𝜉)‖ < 𝛿2 , then the Broyden’s method is
well defined, converges to 𝜉, and satisfies
‖X (k+1) − 𝜉‖ ≤ 𝐶𝑘 ‖X (k) − 𝜉‖,
With lim Ck = 0 (Superlinear convergence)
k→∞
1
2𝛿
Proof: Choose 𝛿2 ≤ 6𝛽 and 𝛿1 ≤ 5𝛾
25
‖F ′ (𝜉)−1 𝐵 (0) − 𝐼‖ ≤ 𝛽𝛿2 ≤
By Banach Lemma (B (0) )
−1
1
6
exists. So 𝑥 (1) can be defined furthermore,
−1
‖F′ (𝜉)−1 ‖
−1
𝛽
‖(B(0) ) ‖ = ‖F ′ (𝜉) + (B(0) − F ′ (𝜉)) ‖ ≤ 1−‖F′ (𝜉)−1 ‖‖B(0)−F′ (𝜉)‖ ≤ 1−𝛽𝛿 .
2
‖𝜖1 ‖ = ‖x(1) − 𝜉‖ = ‖x(0) − (B(0) )
−1
(F(x(0) ) − F(𝜉)) − 𝜉‖
= ‖−(B(0) )−1 [𝐹(x (0) ) − F(𝜉) − B (0) (x (0) − 𝜉)]‖
= ‖(B(0) )−1 [𝐹(x (0) ) − F(𝜉) − F′ (𝜉)(x (0) − 𝜉) + (F′ (𝜉) − B (0) )(x (0) − 𝜉)]‖
≤
𝛽
𝛾
𝛽
[𝛾, 𝑣𝑜𝑒𝑟2𝛿1 + 𝛿2 ]‖𝜖0 ‖
[ ‖𝜖0 ‖2 + 𝛿2 ‖𝜖0 ‖] ≤
1 − 𝛽𝛿2 2
1 − 𝛽𝛿2
1
𝛽
6
6
1
≤
( 𝛿2 ) ‖𝜖0 ‖ ≤ 6 1 ‖𝜖0 ‖ ≤ ‖𝜖0 ‖
1 − 𝛽𝛿2 5
2
1−65
From lemma 2,
𝛾
‖𝐵 (1) − 𝐹 ′ (𝜉)‖ ≤ ‖𝐵 (0) − 𝐹 ′ (𝜉)‖ + (‖x(1) − 𝜉‖ + ‖x(0) − 𝜉‖)
2
𝛾 3
𝛾 32
3
3
≤ 𝛿2 + ( ‖𝜖0 ‖) ≤ 𝛿2 (1 +
) = (1 + ) 𝛿2 ≤ 𝛿2
2 2
225
10
2
1
Thus, ‖F ′ (𝜉)−1 𝐵 (0) − 𝐼‖ ≤ 2𝛽𝛿2 ≤ 3.
By Banach Lemma (B (0) )
−1
‖(B (0) ) ‖ ≤
−1
exists, then
‖F′ (𝜉)−1 ‖
1−‖F′ (𝜉)−1 ‖‖B(1)−F′ (𝜉)‖
≤
𝛽
1−2𝛽𝛿2
3
≤ 𝛽
2
The following estimation can now be made,
‖𝜖2 ‖ = ‖x(2) − 𝜉‖ = ‖x(1) − (B (1) )
−1
(F(x(1) ) − F(𝜉)) − 𝜉‖
= ‖−(B(1) )−1 [𝐹(x (1) ) − F(𝜉) − B (1) ϵ1 ]‖
= ‖(B(1) )−1 [𝐹(x (1) ) − F(𝜉) − F′ (𝜉)ϵ1 + (F′ (𝜉) − B (1) )ϵ1 ]‖
=
3𝛽 𝛾
3
3𝛽 𝛾 𝛿1 3
[‖𝜖1 ‖2 + 𝛿2 ‖ϵ1 ‖] =
[
+ 𝛿 ] ‖ϵ1 ‖
2 2
2
2 2 2 2 2
26
≤
3𝛽𝛿2 𝛾 1 2 3
1 16
1
‖ϵ0 ‖ ≤ ‖ϵ0 ‖
[
+ ] ‖ϵ1 ‖ ≤
2 2 2 5𝛾 2
4 10
2
Continuing
𝛾
≤ ‖B(2) − 𝐹′(𝜉)‖ ≤ ‖B1 − 𝐹′(𝜉)‖ + (‖ϵ1 ‖ + ‖ϵ2 ‖)
2
≤
13𝛿2 𝛾 3
3 𝛾 31 2
3 1 3
1 2
+ ( ‖ϵ1 ‖) ≤ 𝛿2 (1 +
+
) ≤ 𝛿2 (1 +
+
) ≤ 𝛿2 (2 − ( ) ) ≤ 2𝛿2
10
2 2
10 2 2 2 5𝛾
10 2 10
2
The proof is complete by mathematical induction.
Advantages and Disadvantages of Broyden’s method:
The main advantage of Broyden’s method is the reduction of computations; more specifically,
the way the inverse of the approximation matrix, (B (k) )
previous iteration, (B (k−1) )
−1
−1
can be computed directly from the
reduces the number of computations needed for this method in
comparison to NM. The disadvantage of this method is that it does not converge quadratically.
This means that more iteration may be needed to reach the solution, when compared to the
number of iterations Newton method requires. Another disadvantage of this method is that it is
not self-correcting. This means that in contrast to NM, it does not correct itself round of errors
with consecutive iterations. This may cause only a slight inaccuracy in the iteration compared
to NM, but the final iteration result will be the same.
27
6. TWO ITERATIVE METHODS WITH TWO STEPS FOR SOLVING
SNLEs
In this chapter, two iterative methods with two-steps each, commonly known as predictorcorrector methods for solving SNLEs F(X) = 0 are derived and analyzed. The closed-open
quadrature rules are used to derive the correctors but the predictor for both cases is the
Newton’s method. It is shown that both methods are cubic convergent. The Taylor series
technique is used for the proof of convergence analysis of the two methods. To illustrate the
performance of the two methods some numerical examples are solved. The comparison of the
two methods with Newton’s method and Broyden’s method is also given.
6.1.
Predictor-corrector Methods 1 and 2
Consider the system of nonlinear equations
F(X) = 0
(6.1)
where X = (x1 , x2 , x3 , … , xn )𝑇 ∈ Rn . Suppose that D⊆Rn be open and convex set in Rn.
Let F: D⊆ Rn → Rn be r- times Fréchet differentiable function in D and 𝛽 ∈ Rn be a zero of
the given system of nonlinear equations. For any X, X (k) ∈ D, the Taylor formula for the
function F can be written as:
1 ′′ (k)
2
F (X )(X − X (k) )
2!
1
1
1
3
4
5
+ F ′′′ (X (k) )(X − X (k) ) + F (iv) (X (k) )(X − X (k) ) + F (v) (X (k) )(X − X (k) )
3!
4!
5!
1
(r−1)
+⋯+
F (r−1) (X (k) )(X − X (k) )
(r − 1)!
F(X) = F(X (k) ) + F ′ (X (k) )(X − X (k) ) +
1 (1−t)(r−1)
+ ∫0
F (r) (X (k) + t(X − X (k) ))(X − X (k) )
(r)
dt
(6.2)
F(X) = F(X (k) ) + ∫0 F ′ (X (k) + t(X − X (k) )) (X − X (k) )𝑑𝑡
(6.3)
(r−1)!
where 𝑡 ∈ [0,1].
If we take 𝑟 = 1, then equation (6.2) becomes
1
If we approximate integral in equation (6.3) by using closed-open quadrature formula based on
equation (4.3), we have
28
1
1
∫0 F ′ (X (k) + t(X − X (k) )) (X − X (k) )dt ≈ 4 [F ′ (X (k) ) + 3F ′ (
X(k) +2X
3
)] (X − X (k) )
(6.4)
Substituting equation (6.4) into equation (6.3), we have:
X(k) +2X
1
F(X) = F(X (k) ) + 4 [F ′ (X (k) ) + 3F ′ (
3
)] (X − X (k) )
(6.5)
From equation (6.1), F(X) = 0 based on which equation (6.5) becomes
1 ′ (k)
X (k) + 2X
(k)
′
0 = F(X ) + [F (X ) + 3F (
)] (X − X (k) )
4
3
Subtracting F(X (k) ) from both sides, we have
X(k) +2X
1
[F ′ (X (k) ) + 3F ′ (
4
3
)] (X − X (k) ) = − F(X (k) )
X(k) +2X
⟹ [F ′ (X (k) ) + 3F ′ (
3
)] (X − X (k) ) = −4F(X (k) )
If we multiply both sides by the inverse of F ′ (X (k) ) + 3F ′ (
−1
′
[F (X
(k)
X (k) + 2X
) + 3F (
)]
3
′
′
= −4 [F (X
(k)
X(k) +2X
3
), it becomes
X (k) + 2X
[F ′ (X (k) ) + 3F ′ (
)] (X − X (k) )
3
X (k) + 2X
) + 3F (
)]
3
−1
F(X (k) )
′
−1
⟹X−X
(k)
′
= −4 [F (X
(k)
X (k) + 2X
) + 3F (
)]
3
′
X(k) +2X
⟹ X = X (k) − 4 [F ′ (X (k) ) + 3F ′ (
3
−1
)]
F(X (k) )
F(X (k) )
(6.6)
Based on equation (6.6), we have the following algorithm.
Algorithm 6.1: For a given initial approximation value X (0) , then the (𝑘 + 1)𝑡ℎ approximate
value, X = X (k+1) of SNLEs can be computed by using the iterative scheme:
X(k) +2X(k+1)
X (k+1) = X (k) − 4 [F ′ (X (k) ) + 3F ′ (
3
−1
)]
F(X (k) ),
where k = 0, 1, 2, …
This algorithm is an implicit type method. To solve the problems of SNLEs by using
algorithm 6.1, we have to find the solution of the given problem implicitly, which by itself is a
difficult problem. To avoid such difficulties we use the predictor and corrector concepts.
Therefore, we use the Newton’s method as a predictor and the method in algorithm 6.1 as a
corrector. This gives the following predictor – corrector method for solving SNLEs.
Algorithm 6.2: For a given initial value X (0) the approximate solution X (k+1) can be
computed by the following iterative schemes.
Predictor step:
29
Y (k) = X(k) − (F′ (X(k) ))
−1
F(X(k) ),
Corrector step:
X (k+1) = X (k) − 4 × [F ′ (X (k) ) + 3F ′ (
X(k) +2 𝐘(𝐤)
3
−1
F(X (k) ), k = 0,1,2, …
)]
Algorithm 6.2 is one of the four methods on which this project paper focused to solve SNLEs.
This method is referred to as predictor-corrector method-1 (PCM-1).
Again approximating the integral given in (6.3) by another Closed-open quadrature formula
(based on (4.16)), we have:
1
2X(k) +X
1
∫0 F ′ (X (k) + t(X − X (k) )) (X − X (k) ) 𝑑𝑡 ≈ 4 [3F ′ (
3
) + F ′ (X)] (X − X (k) )
(6.7)
Replacing (6.7) in (6.3), we obtain
2X(k) +X
1
F(X) = F(X (k) ) + 4 [3F ′ (
3
) + F ′ (X)] (X − X (k) ).
(6.8)
It is clear that F(X) = 0 from equation (6.1). Hence, equation (6.8) becomes:
1
2X (k) + X
0 = F(X (k) ) + [3F ′ (
) + F ′ (X (k) )] (X − X (k) )
4
3
Now, this equation can be written as
1
2X (k) + X
′
[3F (
) + F ′ (X)] (X − X (k) ) = −F(X (k) )
4
3
2X(k) +X
⟹[3F ′ (
3
) + F ′ (X)] (X − X (k) ) = −4F(X (k) )
Multiplying both parts of this equation by the inverse of [3F ′ (
−1
2X (k) + X
[3F (
) + F ′ (X)]
3
′
2X(k) +X
3
) + F ′ (X)], it becomes
2X (k) + X
[3F ′ (
) + F ′ (X)] (X − X (k) )
3
−1
2X (k) + X
= −4 [3F (
) + F ′ (X)]
3
′
2X(k) +X
⟹ X − X(k) = −4 [3F′ (
⟹ X = X (k) − 4 [3F ′ (
3
2X(k) +X
3
F(X (k) )
) + F′ (X)]
) + F ′ (X)]
−1
−1
F(X(k) )
F(X (k) )
(6.9)
From this relation, we can arrive at the following two-step iterative scheme for solving
SNLEs.
30
Algorithm 6.3: The approximate solution X = X (k+1) is calculated by using the following
iterative schemes if the initial approximation value for the problem is given by X (0) .
−1
X
(k+1)
=X
(k)
2X (k) + X (k+1)
− 4 [3F (
) + F ′ (X (k+1) )]
3
′
F(X (k) ),
𝑘 = 0, 1,2, …
Algorithm 6.3 is also implicit type method. We remove the implicit problem by using NM as a
predictor step in predictor-corrector method for solving SNLEs F(X) = 0.
Algorithm 6.4: For a given initial value, X (0) , the approximate solution, X (k+1), can be
computed by using the following iterative procedure.
Predictor step:
Y (k) = X(k) − (F′ (X(k) ))
−1
F(X(k) )
Corrector step
−1
X (k+1)
2X (k) + Y (k)
(k)
′
= X − 4 [3F (
) + F ′ (Y (𝐤) )]
3
F(X (k) ),
𝑘 = 0, 1,2, …
Algorithm 6.4 is the second iterative method on which this paper focused on to solve system
of nonlinear equations. We refer to it as predictor-corrector method-2 (PCM-2).
6.2.
Convergence Analysis of the Two Methods
Here the order of convergence for both methods given in algorithm 6.2 (PCM-1) and
algorithm 6.4 (PCM-2) are shown by proving theorems 6.1 and 6.2.
6.2.1.
Proof of cubic order of convergence for algorithm 6.2
Theorem 6.1 (Noor and Waseem, 2008): Let F: D⊆ Rn → Rn , be r - times Fréchet
differentiable function in an open and convex set D⊆ Rn containing the root 𝜶 of F(X) = 0.
Suppose that F ′ (X (k) ) is continuous and nonsingular in D. If the initial approximate value X (𝟎)
is sufficiently close to the exact root 𝜶, then the iterative method defined by Algorithm 6.2
converges to 𝜶 with cubic order of convergence and satisfies the error equation
X(k) +2𝐘(𝐤)
)] e(k+1)
3
[F ′ (X (k) ) + 3F ′ (
−1
= [F ′′ (X (k) ) (F ′ (X (k) ))
3
4
F ′′ (X (k) ] (e(k) ) +O(‖e(k) ‖ )
(6.10)
31
Proof: The proof is adopted from Noor and Waseem, 2008.
From Algorithm 6.2, the corrector method is given by the formula
X
(k+1)
=X
(k)
′
− 4 [F (X
(k)
)+
But, Y (k) = X (k) − (F ′ (X (k) ))
X(k) +2 𝐘(𝐤)
3F ′ (
3
−1
−1
F(X (k) )
)]
(6.11)
F(X (k) )
(6.12)
e(k) = X (k) − 𝜶
Defining,
⟹X (k) = e(k) + 𝜶
(6.13)
But from the equation, X
(k+1)
=X
(k)
′
− 4 [F (X
(k)
)+
X(k) +2𝐘(𝐤)
3F ′ (
)]
3
X(k) +2𝐘(𝐤)
X (k+1) − X (k) = −4 [F ′ (X (k) ) + 3F ′ (
3
−1
F(X (k) ) we have:
−1
F(X (k) )
)]
Using (6.13) in the place of X (k) , it becomes
⟹X
(k+1)
⟹(X
− (e
(k+1)
(k)
′
+ 𝜶) = −4 [F (X
− 𝜶) − e
(k)
(k)
′
= −4 [F (X
)+
(k)
X(k) +2𝐘(𝐤)
3F ′ (
)]
3
)+
−1
F(X (k) )
X(k) +2𝐘(𝐤)
3F ′ (
)]
3
−1
F(X (k) )
If e(k) = X (k) − 𝜶, then X (k+1) − 𝜶 = e(k+1) .
So the equation reduces to
X(k) +2𝐘 (𝐤)
⟹e(k+1) − e(k) = −4 [F ′ (X (k) ) + 3F ′ (
3
−1
)]
F(X (k) )
(6.14)
X(k) +2𝐘 (𝐤)
If we pre-multiply both parts of the eq’n (6.14) by F ′ (X (k) ) + 3F ′ (
[F ′ (X (k) ) + 3F ′ (
3
), we obtain
X (k) + 2Y (k)
)] (e(k+1) − e(k) )
3
−1
′
= −4 [F (X
′
⟹ [F (X
(k)
(k)
X (k) + 2𝐘 (𝐤)
X (k) + 2𝐘 (𝐤)
) + 3F (
)] [F ′ (X (k) ) + 3F ′ (
)]
3
3
′
F(X (k) )
X (k) + 2Y (k)
) + 3F (
)] (e(k+1) − e(k) ) = −4F(X (k) )
3
′
X(k) +2Y(k)
⟹[F ′ (X (k) ) + 3F ′ (
3
X(k) +2Y(k)
)] e(k+1) = [F ′ (X (k) ) + 3F ′ (
3
)] e(k) − 4F(X (k) ) (6.15)
Using the equation (6.2) with X = α,
F(X) = F(X (k) ) + F ′ (X (k) )(X − X (k) ) +
1 ′′ (k)
1
2
3
4
F (X )(X − X (k) ) + F ′′′ (X (k) )(X − X (k) ) + O (‖X − X (k) ‖ )
2!
3!
32
4
where, O (‖X − X (k) ‖ ) is a truncation error with order four, we obtain
F(𝛂) = F(X (k) ) + F ′ (X (k) )(𝛂 − X (k) ) +
1 ′′ (k)
1
2
3
F (X )(𝛂 − X (k) ) + F ′′′ (X (k) )(𝛂 − X (k) )
2!
3!
4
+O (‖𝛂 − X (k) ‖ )
Since 𝛂 is the root of F, then F(𝛂) = 0.
2
1
⟹0 = F(X (k) ) + F ′ (X (k) )(𝛂 − X (k) ) + 2! F ′′ (X (k) )(𝛂 − X (k) ) +
3
1
F ′′′ (X (k) )(𝛂 − X (k) ) +
3!
4
O (‖𝛂 − X (k) ‖ )
From the definition given in (6.13) above, we know that
e(k) = X (k) − 𝜶
⟹ −e(k) = 𝜶 − X (k)
Now substituting −e(k) instead of 𝜶 − X (k) in the above equation, we get
0 = F(X (k) ) + F ′ (X (k) )(−e(k) ) +
1 ′′ (k)
1
2
3
F (X )(−e(k) ) + F ′′′ (X (k) )(−e(k) ) + O(‖−e(k) ‖4 )
2!
3!
⟹ 0 = F(X (k) ) − F ′ (X (k) )(e(k) ) +
1 ′′ (k) (k) 2
1
3
4
F (X )(e ) − F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )
2!
3!
2
1
⟹ F(X (k) ) = F ′ (X (k) )(e(k) ) − 2! F ′′ (X (k) )(e(k) ) +
1
3!
3
4
F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ ) (6.16)
−1
Now multiplying equation (6.16) with (F ′ (X (k) )) the product becomes as follows.
(F ′ (X (k) ))
−1
F(X (k) )
−1
= (F ′ (X (k) ))
[F ′ (X (k) )(e(k) ) −
−1
⟹(F ′ (X (k) ))
+
1 ′′ (k) (k) 2
1
3
4
F (X )(e ) + F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )]
2!
3!
−1
F(X (k) ) = (F ′ (X (k) ))
−1
1
F ′ (X (k) )(e(k) ) − 2! (F ′ (X (k) ))
F ′′ (X (k) )(e(k) )
2
1 ′ (k) −1 ′′′ (k) (k) 3
4
(F (X )) F (X )(e ) + O (‖e(k) ‖ )
3!
(F ′ (X (k) ))
−1
−1
1
−1
1
F(X (k) ) = (e(k) ) − 2! (F ′ (X (k) ))
+ 3! (F ′ (X (k) ))
F ′′′ (X (k) )(e(k) )
3
X(k) +2𝐘 (𝐤)
3
2
4
+ O (‖e(k) ‖ )
Now applying a Taylor series in (6.2) for F ′ (
F′ (
F ′′ (X (k) )(e(k) )
X(k) +2Y(k)
X(k) +2𝐘 (𝐤)
) = F ′ (X (k) ) + F ′′ (X (k) ) (
3
3
(6.17a)
) at the point X (k) , we have
1
X(k) +2𝐘 (𝐤)
− X (k) ) + 2! F ′′′ (X (k) ) (
3
− X (k) )
2
33
3
4
1
X (k) + 2Y (k)
X (k) + 2Y (k)
+ F (iv) (X (k) ) (
− X (k) ) + O (‖
− X (k) ‖ )
3!
3
3
2
′
= F (X
(k)
′′
) + F (X
(k)
2
1
2
) ( (−X (k) + Y (k) )) + F ′′′ (X (k) ) ( (−X (k) + Y (k) ))
3
2!
3
3
4
1 (iv) (k) 2
2
(k)
(k)
(k)
(k)
+ F (X ) ( (−X + Y )) + O (‖ (−X + Y )‖ )
3!
3
3
F′ (
X(k) +2𝐘(𝐤)
3
2
1
4
) = F ′ (X (k) ) + 3 F ′′ (X (k) ) ((Y (k) − X (k) )) + 2! × 9 F ′′′ (X (k) )(Y (k) − X (k) )
+
1
3!
×
8
27
3
4
F (iv) (X (k) )(Y (k) − X (k) ) + O (‖e(k) ‖ )
2
(6.17b)
It is clear that we use Y (k) instead of X (k+1) to separate X (k+1) of a predictor step in
algorithms (6.2) and (6.4) from that of the corrector step. So we can write X (k+1) − X (k)
instead of Y (k) − X (k) for the following case such that
−1
X (k+1) = X (k) − (F ′ (X (k) ))
F ′ (X (k) ), or
−1
X (k+1) − X (k) = − (F ′ (X (k) ))
F ′ (X (k) )
(6.18)
Now substituting (6.18) in eq’n (6.17b), we obtain
F′ (
X(k) +2𝐘 (𝐤)
3
−1
2
) = F ′ (X (k) ) + 3 F ′′ (X (k) ) (− (F ′ (X (k) ))
−1
4
+ F ′′′ (X (k) ) (− (F ′ (X (k) )) F ′ (X (k) ))
18
F ′ (X (k) ))
2
3
−1
8 (iv) (k)
4
+
F (X ) (− (F ′ (X (k) )) F ′ (X (k) )) + O (‖e(k) ‖ )
162
′
=F (X
(k)
2
′′
) − 3 F (X
(k)
′
) ((F (X
(k)
−1
))
′
F (X
(k)
4
′′′
)) + 18 F (X
(k)
′
) ((F (X
(k)
))
−1
2
′
F (X
(k)
))
3
−1
8 (iv) (k)
4
−
F (X ) ((F ′ (X (k) )) F ′ (X (k) )) + O (‖e(k) ‖ )
162
−1
X (k) + 2Y (k)
2
F (
) = F ′ (X (k) ) − F ′′ (X (k) ) ((F ′ (X (k) )) F ′ (X (k) ))
3
3
′
2
′′′
+ F (X
9
(k)
′
) ((F (X
(k)
))
−1
2
′
F (X
(k)
)) −
4
81
F
(𝐢𝐯)
(X
(k)
′
) ((F (X
Now using equation (6.17a) in (6.19), we have
(k)
−1
))
3
′
(k)
4
F (X )) + O (‖e(k) ‖ )
(6.19)
34
X(k) +2𝐘 (𝐤)
F′ (
3
2
) = F ′ (X (k) ) − 3 F ′′ (X (k) ) ((e(k) ) −
−1
3
1
(k)
( )
′
(
F
(
X
))
F′′′ (X k )(e(k) )
3!
1
( )
(F′ (X k ))
2!
−1
2
F′′ (X(k) )(e(k) ) +
4
+ O (‖e(k) ‖ ))
−1
−1
2
1
1
2
3
4
+ F ′′′ (X (k) ) ((e(k) ) − (F ′ (X (k) )) F ′′ (X (k) )(e(k) ) + (F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ ) )
9
2!
3!
−
−1
4 (iv) (k)
1
2
F (X ) ((e(k) ) − (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
81
2!
3
−1
1
3
4
4
+ (F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )) + O (‖e(k) ‖ )
3!
2
−1
1
= F ′ (X (k) ) − 3 F ′′ (X (k) )(e(k) ) + 3 F ′′ (X (k) ) (F ′ (X (k) ))
F′′(X (k) )(e(k) )
2
−1
1
3
4
− F ′′ (X (k) ) (F′(X (k) )) F′′′(X (k) )(e(k) ) + O (‖e(k) ‖ )
9
−1
2
1
2
2
+ F ′′′ (X (k) ) [(e(k) ) − (e(k) ) (F ′ (X (k) )) F′′(X (k) )(e(k) )
9
2!
+
−1
1 (k)
3
4
(e ) (F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )
3!
2
−1
1 ′ (k) −1
1
3
2
(F (X )) F′′(X (k) )(e(k) ) + ((F ′ (X (k) )) F′′(X (k) )(e(k) ) )
2!
4
−1
−1
1
2
3
− [(F ′ (X (k) )) F ′′ (X (k) )(e(k) ) ] [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ]
12
−1
1
4
+ (F ′ (X (k) )) F ′′′ (X (k) )(e(k) )
3!
−1
−1
1
3
2
− [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ] [(F ′ (X (k) )) F ′′ (X (k) )(e(k) ) ]
12
+
+
−
2
−1
1
3
4
[(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ] + O (‖e(k) ‖ )]
36
−1
−1
4 (iv) (k)
1
1
2
3
F (X ) ((e(k) ) − (F ′ (X (k) )) F ′′ (X (k) )(e(k) ) + (F ′ (X (k) )) F ′′′ (X (k) )(e(k) )
81
2!
3!
4
3
4
+ O (‖e(k) ‖ )) + O (‖e(k) ‖ )
X(k) +2𝐘 (𝐤)
⟹ F′ (
3
2
1
−1
) = F ′ (X (k) ) − 3 F ′′ (X (k) )(e(k) ) + 3 F ′′ (X (k) ) (F ′ (X (k) ))
F′′(X (k) )(e(k) )
2
2
35
−1
1
2
3
2
− F ′′ (X (k) ) (F′(X (k) )) F′′′(X (k) )(e(k) ) + F ′′′ (X (k) )(e(k) )
9
9
−1
−1
1
1
2
3
− F ′′′ (X (k) )(e(k) ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) ) + F ′′′ (X (k) )(e(k) ) (F′(X (k) )) F ′′′ (X (k) )(e(k) )
9
27
2
−1
−1
1
1 ′′′ (k)
3
(k)
(k) 2
+ F ′′′ (X (k) )(e(k) ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) ) +
F (X ) ((F ′ (X (k) )) F ′′(X )(e ) )
9
18
−1
−1
1 ′′′ (k)
2
3
F (X ) [(F ′ (X (k) )) F ′′ (X (k) )(e(k) ) ] [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ]
54
−1
1
4
+ F ′′′ (X (k) ) (F′(X (k) )) F ′′′ (X (k) )(e(k) )
27
−1
−1
1
3
2
− F ′′′ (X (k) ) [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ] [(F ′ (X (k) )) F ′′ (X (k) )(e(k) ) ]
54
−
+
2
−1
1 ′′′ (k)
3
F (X ) [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ]
162
−
−1
4 (iv) (k)
1
2
F (X ) ((e(k) ) − (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
81
2!
3
−1
1
3
4
4
+ (F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )) + O (‖e(k) ‖ )
3!
3
−1
1
Let O (‖e(k) ‖ ) = − 9 F′′(X (k) ) (𝐅 ′ (X (k) ))
−
F ′′′ (X (k) )(e(k) )
3
−1
1 ′′′ (k) (k)
2
F (X )(e ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
9
+ ⋯ + (−
−1
4 (iv) (k)
1
2
F (X )) ((e(k) ) − (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
81
2!
3
−1
1
3
4
4
+ (F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )) + O (‖e(k) ‖ )
3!
Then,
⟹ F′ (
X(k) +2𝐘 (𝐤)
3
2
2
2
1
−1
) = F ′ (X (k) ) − 3 F ′′ (X (k) )(e(k) ) + 3 F ′′ (X (k) ) (F ′ (X (k) ))
3
+ 9 F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )
F ′′ (X (k) )(e(k) )
2
(6.20)
Now using equations (6.16) and (6.20) in equation (6.15), we have
′
[F (X
′
(k)
= F (X
X (k) + 2Y (k)
X (k) + 2Y (k)
(k+1)
′
(k)
′
) + 3F (
)] e
= [F (X ) + 3F (
)] (e(k) ) − 4F(X (k) )
3
3
(k)
′
)(e
(k)
X (k) + 2Y (k)
) + 3F (
) (e(k) ) − 4F(X (k) )
3
′
36
= F ′ (X (k) )(e(k) )
−1
2
1
2
+3 [F ′ (X (k) ) − F ′′ (X (k) )(e(k) ) + F ′′ (X (k) ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
3
3
2
2
3
+ F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )] (e(k) )
9
1
1
2
3
4
−4 [F ′ (X (k) )(e(k) ) − F ′′ (X (k) )(e(k) ) + F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )]
2!
3!
= F ′ (X (k) )(e(k) ) + 3F ′ (X (k) )(e(k) ) − 2F ′′ (X (k) )(e(k) )
2
2
3
4
+ F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )
3
2
2
3
4
−4F′ (X(k) )(e(k) ) + 2F′′ (X(k) )(e(k) ) − F′′′ (X(k) )(e(k) ) + O (‖e(k) ‖ )
3
+F ′′ (X (k) ) (F ′ (X (k) ))
−1
F ′′(X
(k) )(e(k) )3
Finally, after cancellation of like terms, we obtain the following.
′
[F (X
(k)
X (k) + 2Y (k)
) + 3F (
)] e(k+1)
3
′
−1
= [F ′′ (X (k) ) (F ′ (X (k) ))
3
4
F′′(X (k) )] (e(k) ) + O (‖e(k) ‖ )
Therefore, this is the proof of algorithm (6.2). From this proof of error equation, one can see
that Algorithm (6.2) has a cubic order of convergence.
6.2.2.
Proof of cubic order of convergence for algorithm 6.4
Theorem 6.2 (Noor and Waseem, 2008): Let F: D⊆ Rn → Rn , be r- times Fréchet
differentiable function in an open and convex set D⊆ Rn containing the root 𝜶 of F(X) = 0.
Suppose that F ′ (X (k) ) is continuous and nonsingular in D. If the initial approximate value X (𝟎)
is sufficiently close to the exact root 𝜶, then the iterative method defined by Algorithm 6.4
converges to 𝜶 with cubic order of convergence and satisfies the error equation
2X (k) + Y (k)
[3F ′ (
) + F ′ (Y (k) )] e(k+1)
3
−1
= [F ′′ (X (k) ) (F ′ (X (k) ))
3
4
F ′′ (X (k) ] (e(k) ) + O (‖e(k) ‖ )
Proof: The proof is adopted from Noor and Waseem, 2008.
From Algorithm (6.4) the method is given by the formula:
(6.21)
37
2X(k) +𝐘 (𝐤)
X (k+1) = X (k) − 4 [3F ′ (
3
) + F ′ (Y (k) )]
−1
wher𝑒 Y (k) = X (k) − (F ′ (X (k) ))
−1
F(X (k) )
(6.22)
F ′ (X (k) ).
Here we use Y (k) for X (k+1) to show the difference between X (k+1) of predictor step from
X (k+1) of the corrector step.
Now from equation (6.22), we have
2X(k) +𝐘 (𝐤)
X (k+1) − X (k) = −4 [3F ′ (
3
) + F ′ (Y (k) )]
−1
F(X (k) )
(6.23)
But from equation (6.13), we know that
X (k) = e(k) + 𝜶 ⟹ X (k) − 𝜶 = e(k)
−1
⟹X
(k+1)
− (e
(k)
2X (k) + Y (k)
+ 𝜶) = −4 [3F (
) + F ′ (Y (k) )]
3
′
F(X (k) )
−1
⟹ (X
(k+1)
− 𝜶) − e
(k)
2X (k) + Y (k)
= −4 [3F (
) + F ′ (Y (k) )]
3
′
2X(k) +𝐘 (𝐤)
⟹ e(k+1) − e(k) = −4 [3F ′ (
3
Pre-multiplying both sides by[3F ′ (
) + F ′ (Y (k) )]
2X(k) +𝐘 (𝐤)
3
−1
F(X (k) )
F(X (k) )
(6.24)
) + F ′ (Y (k) )], we obtain
2X (k) + Y (k)
[3F (
) + F ′ (Y (k) ] (e(k+1) − e(k) ) = −4F(X (k) )
3
′
2X(k) +Y(k)
⟹ [3F ′ (
3
2X(k) +𝐘 (𝐤)
) + F ′ (Y (k) )] e(k+1) = [3F ′ (
2X(k) +Y(k)
Now applying a Taylor series on F ′ (
F′ (
3
3
) + F ′ (Y (k) )] e(k) − 4F(X (k) )
) at the point X (k) , we have:
2X (k) + Y (k)
2X (k) + Y (k)
) = F ′ (X (k) ) + F ′′ (X (k) ) (
− X (k) )
3
3
2
3
1 ′′′ (k) 2X (k) + Y (k)
1 (iv) (k) 2X (k) + Y (k)
(k)
+ F (X ) (
− X ) + F (X ) (
− X (k) )
2!
3
3!
3
4
2X (k) + Y (k)
+O (‖
− X (k) ‖ )
3
2
′
= F (X
(k)
′′
) + F (X
(k)
−X (k) + Y (k)
1
−X (k) + Y (k)
)(
) + F ′′′ (X (k) ) (
)
3
2!
3
(6.25)
38
3
4
1
−X (k) + Y (k)
−X (k) + Y (k)
+ F (iv) (X (k) ) (
) + O (‖
‖ )
3!
3
3
2X (k) + Y (k)
1
1
2
F (
) = F ′ (X (k) ) + F ′′ (X (k) )(Y (k) − X (k) ) + F ′′′ (X (k) )(Y (k) − X (k) )
3
3
18
′
+
1 (iv) (k)
3
4
F (X )(Y (k) − X (k) ) + O (‖e(k) ‖ )
162
We can write the above equation as follows by replacing Y (k) by X (k+1)
F′ (
2X (k) + Y (k)
1
1 ′′′ (k)
2
) = F ′ (X (k) ) + F ′′ (X (k) )(X (k+1) − X (k) ) +
F (X )(X (k+1) − X (k) )
3
3
18
3
1
4
+ 162 F (iv) (X (k) )(X (k+1) − X (k) ) + O (‖e(k) ‖ )
(6.26)
Now substituting equation (6.18) in equation (6.26), we have
−1
2X (k) + Y (k)
1
F (
) = F ′ (X (k) ) + F ′′ (X (k) ) (− (F ′ (X (k) )) F ′ (X (k) ))
3
3
′
2
−1
−1
1
1 (iv) (k)
( )
( )
+ F ′′′ (X (k) ) (− (F ′ (X (k) )) F ′ (X (k) )) +
F (X ) (− (F′ (X k )) F′ (X k ))
18
162
3
4
+O (‖e(k) ‖ )
2X(k) +Y(k)
F′ (
1
3
′′′
+ 18 F (X
1
( )
−1
) = F ′ (X (k) ) − 3 F ′′ (X (k) ) ((F′ (X k ))
(k)
′
) ((F (X
4
+O (‖e(k) ‖ )
(k)
))
−1
F′ (X(k) ))
2
′
F (X
(k)
1
)) − 162 F
(iv)
(X
(k)
′
) ((F (X
(k)
−1
))
3
′
F (X
(k)
))
(6.27)
Again substituting equation (6.17a) in equation (6.27), one can obtain the following result:
F′ (
2X (k) + Y (k)
)=
3
1
1 ′ (k) −1 ′′ (k)
1 ′ (k) −1 ′′′ (k)
2
3
4
F ′ (X (k) ) − F ′′ (X (k) ) ((e(k) ) − (F (X )) F (X ) (e(k) ) + (F (X )) F (X ) (e(k) ) + O (‖e(k) ‖ ))
3
2!
3!
−1
−1
1
1
1
2
3
4
+ F ′′′ (X (k) ) ((e(k) ) − (F ′ (X (k) )) F ′′ (X (k) )(e(k) ) + (F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ ))
18
2!
3!
2
39
−1
−1
1 (iv) (k)
1
1
2
3
F (X ) ((e(k) ) − (F ′ (X (k) )) F ′′ (X (k) )(e(k) ) + (F ′ (X (k) )) F ′′′ (X (k) )(e(k) )
162
2!
3!
−
4
3
4
+ O (‖e(k) ‖ )) + O (‖e(k) ‖ )
−1
2X (k) + Y (k)
1
1
2
F (
) = F ′ (X (k) ) − F ′′ (X (k) )(e(k) ) + F ′′ (X (k) ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
3
3
6
′
−1
1 ′′ (k)
3
4
( )
( )
F (X ) (F′ (X k )) F′′′ (X k )(e(k) ) + O (‖e(k) ‖ )
18
−
2
1
−1
1
+ 18 F ′′′ (X (k) ) [(e(k) ) − 2! (e(k) ) (F ′ (X (k) ))
−1
1
(e(k) ) (F ′ (X (k) ))
3!
−1
1
((F ′ (X (k) ))
4
1
[(F ′ (X (k) ))
12
(F ′ (X (k) ))
3!
4
−1
1
(F ′ (X (k) ))
2!
3
F ′′ (X (k) )(e(k) ) +
2
2
2
F ′′ (X (k) )(e(k) ) ] [(F ′ (X (k) ))
−1
3
F ′′′ (X (k) )(e(k) ) ] +
4
F ′′′ (X (k) )(e(k) ) −
1
−1
1
−1
[(F ′ (X (k) ))
12
[(F ′ (X (k) ))
36
3
F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ ) +
F ′′ (X (k) )(e(k) ) ) −
−1
−1
1
2
F ′′ (X (k) )(e(k) ) +
−1
3
F ′′′ (X (k) )(e(k) ) ] [(F ′ (X (k) ))
3
2
2
F ′′ (X (k) )(e(k) ) ] +
4
F ′′′ (X (k) )(e(k) ) ] + O (‖e(k) ‖ )] +. . . .
−1
2X (k) + Y (k)
1
1
2
F (
) = F ′ (X (k) ) − F ′′ (X (k) )(e(k) ) + F ′′ (X (k) ) (F ′ (X (k) )) F′′(X (k) )(e(k) )
3
3
6
′
−
−1
1 ′′ (k)
1
3
2
( )
( )
F (X ) (F′ (X k )) F′′′ (X k )(e(k) ) + F ′′′ (X (k) )(e(k) )
18
18
−
−1
−1
1 ′′′ (k) (k)
1 ′′′ (k) (k)
2
3
F (X )(e ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) ) +
F (X )(e ) (F ′ (X (k) )) F ′′′ (X (k) )(e(k) )
36
108
2
−1
−1
1 ′′′ (k)
1 ′′′ (k)
(k)
(k)
(k) 3
(k)
(k)
(k) 2
′
′′
′
′′
+ F (X ) (F (X )) F (X )(e ) + F (X ) [(F (X )) F (X )(e ) ]
36
72
−1
−1
1 ′′′ (k)
2
3
−
F (X ) [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ] [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ]
216
−1
1 ′′′ (k)
4
+
F (X ) (F ′ (X (k) )) F ′′′ (X (k) )(e(k) )
108
−1
−1
1 ′′′ (k)
3
2
−
F (X ) [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ] [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ]
216
40
+
2
−1
1 ′′′ (k)
3
4
F (X ) [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ] + O (‖e(k) ‖ ) + ⋯
648
3
Suppose that O (‖e(k) ‖ ) = − 18 F ′′ (X (k) ) (F′ (X k ))
1
( )
−1
F′′′ (X k )(e(k) )
( )
3
−1
1 ′′′ (k) (k)
2
F (X )(e ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
36
−1
1 ′′′ (k) (k)
3
+
F (X )(e ) (F ′ (X (k) )) F ′′′ (X (k) )(e(k) )
108
−
+⋯+
2
−1
1 ′′′ (k)
3
4
F (X ) [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ] + O (‖e(k) ‖ ) + ⋯
648
Then the above equation becomes
−1
2X (k) + Y (k)
1
1
2
F (
) = F ′ (X (k) ) − F ′′ (X (k) )(e(k) ) + F ′′ (X (k) ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
3
3
6
′
+
1
18
2
3
F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )
(6.28)
Now using equation (6.16) and (6.28) in (6.25), we have
[3F ′ (
2X (k) + Y (k)
2X (k) + Y (k)
) + F ′ (Y (k) )] e(k+1) = [3F ′ (
) + F ′ (Y (k) )] e(k) − 4F(X (k) )
3
3
2X(k) +Y(k)
⟹[3F ′ (
3
2X(k) +Y(k)
[3F ′ (
3
1
2X(k) +Y(k)
) + F ′ (Y (k) )] e(k+1) = 3F ′ (
3
) (e(k) ) + F ′ (Y (k) )(e(k) ) − 4F(X (k) )
1
) + F ′ (Y (k) )] e(k+1) = 3 [F ′ (X (k) ) − 3 F ′′ (X (k) )(e(k) ) +
F ′′ (X (k) ) (F ′ (X (k) ))
6
−1
2
1
2
3
1
F ′′′ (X (k) )(e(k) ) ] + O (‖e(k) ‖ )
F ′′ (X (k) )(e(k) ) + 18 F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )] (e(k) ) +
F ′ (Y (k) )(e(k) )
1
2
−4 [F ′(X (k) )(e(k) ) − 2! F ′′ (X (k) )(e(k) ) +
3!
3
4
(6.29)
But by applying a Taylor series in equation (6.2) for F′(Y(k) ) at a point X (k) , one can obtain
the following.
F ′ (Y (k) ) = F′(X (k) ) + F ′ ′(X (k) )(Y (k) − X (k) ) +
1 ′′ (k)
2
3
F ′(X )(Y (k) − X (k) ) + O (‖Y (k) − X (k) ‖ )
2!
−1
Since Y (k) = X (k+1) and X (k+1) − X (k) = − (F ′ (X (k) ))
−1
F ′ (Y (k) ) = F′(X (k) ) + F ′ ′(X (k) ) (− (F ′ (X (k) ))
F ′ (X (k) ) then,
F ′ (X (k) ))
41
2
−1
1
3
+ F ′′ ′(X (k) ) (− (F ′ (X (k) )) F ′ (X (k) )) + O (‖e(k) ‖ )
2!
−1
𝐹 ′ (Y (k) ) = F ′ (X (k) ) − F ′ ′(X (k) ) ((F ′ (X (k) ))
1
′′
+ 2! F ′(X
(k)
′
) ((F (X
(k)
−1
))
F ′ (X (k) ))
2
′
F (X
(k)
3
)) + O (‖e(k) ‖ )
(6.30)
From equation (6.17a), we have
(F ′ (X (k) ))
−1
1
F(X (k) ) = e(k) −
−1
+ 3! (F ′ (X (k) ))
1 ′ (k) −1 ′′ (k) (k) 2
(F (X )) F (X )(e )
2!
F ′′′ (X (k) )(e(k) )
3
4
+ O (‖e(k) ‖ )
(*)
Now substitute (*) in equation (6.30), the result becomes
F ′ (Y (k) ) =
F ′ (X (k) ) − F′′(X (k) ) ((e(k) ) −
+
+
1 ′ (k) −1 ′′ (k) (k) 2
(F (X )) F (X )(e )
2!
1 ′ (k) −1 ′′′ (k) (k) 3
4
(F (X )) 𝐅 (X )(e ) + O (‖e(k) ‖ ))
3!
−1
−1
1 ′′ (k)
1
1
2
3
F ′(X ) ((e(k) ) − (F ′ (X (k) )) F ′′ (X (k) )(e(k) ) + (F ′ (X (k) )) F ′′′ (X (k) )(e(k) )
2!
2!
3!
+ O (‖e
(k) 4
2
F ′ (Y (k) ) = F ′ (X (k) ) − F ′′ (X (k) )(e(k) ) +
−
3
‖ )) + O (‖e(k) ‖ )
−1
1 ′′ (k)
2
F (X ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
2!
−1
1
4
3
F′′(X (k) ) (F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )
3!
42
+
1
2!
2
F′′′ (X(k) ) [(e(k) ) −
+
+
−
+
−
+
1
3!
1
2!
1
3!
1
2!
(e(k) ) (F′ (X(k) ))
(e(k) ) (F′ (X(k) ))
(F′ (X(k) ))
12
1
1
[(F′ (X(k) ))
(F′ (X(k) ))
12
1
36
−1
[(F′ (X(k) ))
′
(k)
[(F (X ))
3
3
−1
1
4
4
((F′ (X(k) ))
2
F′′ (X(k) )(e(k) ) ] [(F′ (X(k) ))
F′′′ (X(k) )(e(k) )
−1
2
F′′′ (X(k) )(e(k) ) + O (‖e(k) ‖ )
−1
3
′′′
(k)
F (X
−1
2
2
F′′(X(k) )(e(k) ) )
3
F′′′ (X(k) )(e(k) ) ]
4
F′′′ (X(k) )(e(k) ) ] [(F′ (X(k) ))
F ′ (Y (k) ) = F ′ (X (k) ) − F ′′ (X (k) )(e(k) ) +
−
F′′ (X(k) )(e(k) )
F′′(X(k) )(e(k) ) +
−1
−1
−1
−1
(k) 3
2
−1
2
F′′ (X(k) )(e(k) ) ]
3
4
)(e ) ] + O (‖e(k) ‖ )] + O (‖e(k) ‖ )
−1
1 ′′ (k)
2
( )
( )
F (X ) (F′ (X k )) F′′ (X k )(e(k) )
2!
−1
1
3
4
F′′(X(k) ) (F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )
3!
−1
1
2
2
F′′′(X(k) )(e(k) ) − F′′′(X(k) )(e(k) ) (F′ (X(k) )) F′′ (X(k) )(e(k) )
2!
4
−1
1
3
4
+ F′′′(X(k) )(e(k) ) (F′ (X(k) )) F′′′ (X(k) )(e(k) ) + O (‖e(k) ‖ )
12
+
1
2
−1
−1
1
1
3
2
+ F′′′(X (k) ) (F ′ (X (k) )) F′′(X (k) )(e(k) ) + F′′′(X (k) ) ((F ′ (X (k) )) F′′(X (k) )(e(k) ) )
4
8
−1
−1
1
2
3
− F′′′(X (k) ) [(F ′ (X (k) )) F ′′ (X (k) )(e(k) ) ] [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ]
24
−1
1
4
+ F′′′(X (k) ) (F ′ (X (k) )) F ′′′ (X (k) )(e(k) )
12
−1
−1
1
3
2
− F′′′(X (k) ) [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ] [(F ′ (X (k) )) F ′′ (X (k) )(e(k) ) ]
24
+
2
−1
1
3
4
F′′′(X (k) ) [(F ′ (X (k) )) F ′′′ (X (k) )(e(k) ) ] + O (‖e(k) ‖ )
72
3
1
( )
−1
Assume that, O (‖e(k) ‖ ) = − 3! F′′(X (k) ) (F′ (X k ))
F′′′ (X(k) )(e(k) )
3
43
−1
1
2
− F′′′(X (k) )(e(k) ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
4
+⋯+
1
72
(k)
F′′′(X
′
(k)
) [(F (X ))
−1
′′′
(k)
F (X
(k) 3
2
4
)(e ) ] + O (‖e(k) ‖ )
1
( )
−1
Then, F ′ (Y (k) ) = F ′ (X (k) ) − F ′′ (X (k) )(e(k) ) + 2! F ′′ (X (k) ) (F′ (X k ))
2
1
( )
F′′ (X k )(e(k) )
3
+ F′′′ (X(k) )(e(k) ) + O (‖e(k) ‖ )
(6.31)
2!
Now, by replacing equation (6.31) in equation (6.29) in the place of F ′ (Y(k) ), we have
2X (k) + Y (k)
[3F (
) + F ′ (Y (k) )] e(k+1)
3
′
−1
1
1
2
= 3 [F ′ (X (k) ) − F ′′ (X (k) )(e(k) ) + F ′′ (X (k) ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
3
6
1
2
3
+ F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )] (e(k) )
18
−1
1
2
+ [F ′ (X (k) ) − F ′′ (X (k) )(e(k) ) + F ′′ (X (k) ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
2!
1
2
3
+ F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )] (e(k) )
2!
1
1
2
3
4
−4 [F ′ (X (k) )(e(k) ) − F ′′ (X (k) )(e(k) ) + F ′′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )]
2!
3!
2X (k) + Y (k)
⟹ [3F (
) + F ′ (Y (k) )] e(k+1)
3
′
−1
1
2
2
= 3F ′ (X (k) )(e(k) ) − F ′′ (X (k) )(e(k) ) + F ′′ (X (k) ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) )
2
1
3
2
+ F ′′′ (X (k) )(e(k) ) + F ′ (X (k) )(e(k) ) − F ′′ (X (k) )(e(k) )
6
−1
1
+ 2! F ′′ (X (k) ) (F ′ (X (k) ))
2
+2F ′′ (X (k) )(e(k) ) −
2
3
1
3
F ′′ (X (k) )(e(k) ) + 2! F ′′′ (X (k) )(e(k) ) − 4F ′ (X (k) )(e(k) )
2 ′′′ (k) (k) 3
4
F (X )(e ) + O (‖e(k) ‖ )
3
44
2X (k) + Y (k)
⟹ [3F ′ (
) + F ′ (Y (k) )] e(k+1) = (4F ′ (X (k) )(e(k) ) − 4F ′ (X (k) )(e(k) ))
3
+(
−1
−1
1 ′′ (k)
1
3
3
F (X ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) ) + F ′′ (X (k) ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) ) )
2!
2!
2
2
+ (−2F ′′ (X (k) )(e(k) ) + 2F ′′ (X (k) )(e(k) ) )
1
1
2
3
3
3
4
+ [ F ′′′ (X (k) )(e(k) ) + F ′′′ (X (k) )(e(k) ) − F ′′′ (X (k) )(e(k) ) ] + O (‖e(k) ‖ )
6
2
3
After cancellation of like terms, we obtain
−1
2X (k) + Y (k)
3
4
[3F ′ (
) + F ′ (Y (k) )] e(k+1) = F ′′ (X (k) ) (F ′ (X (k) )) F ′′ (X (k) )(e(k) ) + O (‖e(k) ‖ )
3
Therefore, this is the proof for error equation of Algorithm (6.4). From this proof one can
understand that Algorithm (6.4) has cubic order of convergence.
45
7. NUMERICAL RESULTS OF THE FOUR METHODS
In this chapter, some numerical examples are given to show the performance of the four
methods (Newton, Broyden, PCM-1 and PCM-2 methods) and the number of the iteration
results of the one method is compared to the other among them. For the sake of comparison,
the given examples are also solved by the four methods. Each value of the iteration of the
methods is given in a table and then the comparison is made on the other table for all methods
in common.
Now all the examples given below are tested with a precision 𝜀 = 10−4 and the stopping
criteria of the iteration process for the given examples is ‖X (k) − X (k−1) ‖2 < 𝜀.
To show how the methods perform, some of the examples are solved manually and the rest are
solved by using MATLAB codes given to the methods to ease the work.
Example 7.1
Consider the following system of nonlinear equations (Hafiz and Bahgat, (2012)) and find the
approximate solution X (k+1) of the system if the initial value
x (0) = [0.1 0.1 −0.1]𝑇 .
𝑥12
F(X) =
[
3𝑥1 − cos(𝑥2 𝑥3 ) − 0.5
0
− 81(𝑥2 + 0.1)2 + sin(𝑥3 ) + 1.06
= [0]
10𝜋 − 3
−𝑥1 𝑥2
0
e
+ 20𝑥3 +
]
3
 Solution by using Newton’s method
The Newton’s method is given by:
−1
X (k+1) = X (k) − (F ′ (X (k) ))
F(X (k) )
Now evaluating F(X) and F′(X) at x (0) , we have
3(0.1) − cos(0.1 × −0.1) − 0.5
−1.2000
(0.1) − 81(0.1 + 0.1)2 + sin(−0.1) + 1.06
F(x (0) ) =
= [−2.2698]
10𝜋 − 3
−(0.1×0.1)
8.4620
e
+
20(−0.1)
+
[
]
3
2
46
F
′ (X)
3
2𝑥1
=[
−𝑥2 e−𝑥1𝑥2
𝑥3 sin(𝑥2 𝑥3 )
−162(𝑥2 + 0.1)
−𝑥1 e−𝑥1 𝑥2
𝑥2 sin(𝑥2 𝑥3 )
cos(𝑥3 ) ]
20
3
−0.1 × sin(0.1 × −0.1)
2 × 0.1
−162(0.1 + 0.1)
F (x ) = [
−(0.1×0.1)
−0.1 × e
−0.1 × e−(0.1×0.1)
3.0000
0.0010
−0.0010
= [ 0.2000 −32.4000 0.9950 ]
−0.0990 −0.0990 20.0000
′
(0)
0.1 × sin(0.1 × −0.1)
cos(−0.1)
]
20
Then,
−1
X (1) = X (0) − (F ′ (X (0) ))
X
(1)
F(X (0) )
0.1
3.0000
= [ 0.1 ] − ([ 0.2000
−0.1
−0.0990
0.0010
−32.4000
−0.0990
−0.0010 −1
−1.2000
0.4999
])
×
[
]
=
[
0.9950
−2.2698
0.0195 ]
20.0000
8.4620
−0.5215
Taking the same procedure for the next iterations, their values are given in the following table.
Table 1: Iteration result of Newton’s method for Example 7.1
k
(𝒌)
(𝒌)
‖𝐱(𝐤) − 𝐱(𝐤−𝟏) ‖𝟐
(𝒌)
0
𝒙𝟏
0.1000
𝒙𝟐
0.1000
𝒙𝟑
-0.1000
__
1
0.4999
0.0195
-0.5215
0.5866
2
0.5000
0.0016
-0.5236
0.0180
3
0.5000
0.0000
-0.5236
0.0016
4
0.5000
0.0000
-0.5236
0.0000
From the table 1 it is clear that the iteration process repeats the same result after the third
iteration because the solution in this project is approximated using 4 decimal places after the
decimal point. The norm when 3rd iteration subtracted from the 4th is zero so that this result is
less than the given 𝜺. One can understand from this the solution of the given system has
reached its converging point and so that the approximate solution for the system
is[0.5000
0.0000
−0.5236]𝑡 .
47
 Solution by using the predictor-corrector method-1 (PCM-1) (or Algorithm 6.3)
Now by using the iterations of Newton’s method illustrated in the table-1 as a predictor step
for respective iterations of the corrector step of PCM-1, it is possible to find the approximate
solution of the system of nonlinear equations given in example 7.1 by PCM-1 (the method in
Algorithm 6.3) as follows.
The formula of this method to solve such problems is given by
Predictor step:
Y (k) = X(k+1) = X(k) − (F′ (X(k) ))
−1
F(X(k) )
(Newton’s method)
And,
The corrector step:
X (k+1)
X (k) + 2 Y(k)
(k)
′
(k)
′
= X − 4 × [F (X ) + 3F (
)]
3
−1
F(X (k) ),
where, k = 0,1,2, …
Now the preceding conditions before coming to the iteration process is finding F ′ (X) of the
X(k) +2 𝐘(𝐤)
given system and then evaluating F(X) and F ′ (X) at x (0) . Again evaluating 3F ′ (
at x (0) and Y (0) . That is:
F ′ (X) = [
3
2𝑥1
−𝑥2 e−𝑥1𝑥2
𝑥3 sin(𝑥2 𝑥3 )
−162(𝑥2 + 0.1)
−𝑥1 e−𝑥1 𝑥2
𝑥2 sin(𝑥2 𝑥3 )
cos(𝑥3 ) ]
20
3(0.1) − cos(0.1 × −0.1) − 0.5
−1.2000
(0.1) − 81(0.1 + 0.1)2 + sin(−0.1) + 1.06
F(x (0) ) =
= [−2.2698]
10𝜋 − 3
−(0.1×0.1)
8.4620
e
+
20(−0.1)
+
[
]
3
3
−0.1 × sin(0.1 × −0.1) 0.1 × sin(0.1 × −0.1)
2 × 0.1
−162(0.1 + 0.1)
cos(−0.1)
F ′ (x (0) ) = [
]
−0.1 × e−(0.1×0.1)
−0.1 × e−(0.1×0.1)
20
3.0000
0.0010
−0.0010
= [ 0.2000 −32.4000 0.9950 ]
−0.0990 −0.0990 20.0000
2
And
3
)
48
X (k) + 2 Y
3F (
3
(k)
′
X (0) + 2 Y
) = 3F (
3
(0)
′
)
But Y (0) is the first iteration value of Newton’s method so that,
0.4999
Y (0) = X (1) = [ 0.0195 ] (From Newton’s iteration)
−0.5215
So, 3F ′ (
X(k) +2 Y(k)
3
1.0998
[ 0.1390 ]
−1.1430
3
= 3 × F′
(
= 3[
X(0) +2 Y(0)
) = 3F ′ (
)= 3×
)
= 3 × F′(0.3666 , 0.0463, −0.3810)
)
3
2 × 0.3666
−0.0463 × 𝑒 −0.3666×0.0463
9
= [ 2.1996
−0.1366
3
0.1
0.4999
[ 0.1 ]+ 2×[ 0.0195 ]
−0.5215
F ′ ( −0.1
3
0.0202
−71.1018
−1.0813
−0.381 × sin(0.0463 × −0.381)
−162(0.0463 + 0.1)
−0.3666 × 𝑒 −0.3666×0.0463
0.0463 × sin(0.0463 × −0.381)
]
cos(−0.3810)
20
−0.0025
2.7849 ]
60.0000
Now the corrector step becomes,
⟹ X (1)
X
(1)
X (1)
X (0) + 2 Y(0)
(0)
′
(0)
′
= X − 4 × [F (X ) + 3F (
)]
3
0.1
3
0.001
= [ 0.1 ] − 4 ([ 0.2
−32.4
−0.1
−0.099 −0.099
−1.2000
× [−2.2698]
8.4620
0.1
12
= [ 0.1 ] − [ 2.3996
−0.1
−0.2356
0.0202
−103.5018
−1.1803
−1
F(X (0) )
−0.001
9
0.995 ] + [ 2.1996
20
−0.1366
0.0202
−71.1018
−1.0813
−0.0025 −1
2.7849 ])
60.0000
−4.8
−0.0035 −1
3.7799 ] × [−9.0792]
33.848
80
0.5000
X (1) = [ 0.0061 ]
−0.5233
Following the same manner for the next iterations, the solution is given in table 2.
49
Table 2: The iteration result of PCM-1 for Example 7.1
(𝒌)
K
(𝒌)
𝒙𝟏
(𝒌)
𝒙𝟑
-0.1000
‖𝐱 (𝐤) − 𝐱 (𝐤−𝟏) ‖𝟐
__
0
0.1000
𝒙𝟐
0.1000
1
0.5000
0.0061
-0.5233
0.5899
2
0.5000
0.0000
-0.5236
0.0016
3
0.5000
0.0000
-0.5236
0.0000
From this table we can see that the iteration process repeated the same result after the 2nd
iteration and the Euclidean norm of the difference of 2nd iteration from 3rd is 0.
This implies that the iteration converged to its approximate solution so that the approximate
solution for the SNLEs given in Example 7.1 is [0.5000
0.0000
−0.5236]𝑇 .
 Solution by using the predictor-corrector method-2 (PCM-2) (or Algorithm 6.5)
In this case also we are going to find an approximate solution for the given in Example 7.1.
This method also gives the same approximate solution like other methods listed above.
As PCM-1 did, this method also has the predictor and corrector steps with the form as follows.
−1
Predictor step: Y (k) = X (k) − (F ′ (X (k) ))
F(X (k) )
In this case also the predictor step is the Newton’s method. So all the values of Y (k) are the
iteration values of Newton’s method.
Corrector step:
X
(k+1)
=X
(k)
−
2X(k) +𝐘 (𝐤)
4 [3F ′ (
)
3
′
+ F (Y
(k)
−1
)]
F(X (k) ), 𝑘 = 0, 1,2, …
To find the approximate solution using this method we have given the initial approximation
value of the system x (0) = [0.1 0.1 −0.1]𝑇 and Y (0) = [0.4999
0.0195
−0.5215]T .
The value of F(X (0) ) is already calculated for the above methods and has the value,
F(X
(0)
−1.2000
) = [−2.2698].
8.4620
2X(k) +Y(k)
Now evaluating 3F ′ (
3
) for k=0, we have:
50
3F ′ (
0.1000
0.4999
2 × [ 0.1000 ] + [ 0.0195 ]
−0.1000
−0.5215
3
2X (0) + Y (0)
) = 3 × F′
3
(
0.6999
[ 0.2195 ]
−0.7215
3
= 3 × F′
)
(
)
= 3 × F ′ (0.2333, 0.0732, −0.2405 )
But, F
′ (X)
=[
3
2𝑥1
−𝑥2 e−𝑥1 𝑥2
3 × F ′ (0.2333,
𝑥3 sin(𝑥2 𝑥3 )
𝑥2 sin(𝑥2 𝑥3 )
−162(𝑥2 + 0.1)
cos(𝑥3 ) ] then,
−𝑥1 𝑥2
−𝑥1 e
20
−0.2405 )
0.0732,
3
= 3[
2 × 0.2333
−0.0732 × e−0.2333×0.0732
9.0000
= [ 1.3998
−0.2160
0.0126
−84.1752
−0.6879
−0.2405 × sin(0.0732 × −0.2405) 0.0732 × sin(0.0732 × −0.2405)
−162(0.0732 + 0.1)
cos(−0.2405)
]
−0.2333 × e−0.2333×0.0732
20
−0.0039
2.9136 ]
60
And,
F ′ (Y (0) ) = F′(0.4999, 0.0195, −0.5215)
3
2 × 0.4999
=[
−0.0195 × e−0.4999×0.0195
′
F (Y
(0)
3
) = [ 0.9998
−0.0193
−0.5215 × sin(0.0195 × −0.5215) 0.0195 × sin(0.0195 × −0.5215)
−162(0.0195 + 0.1)
cos(−0.5215)
]
−0.4999×0.0195
−0.4999 × e
20
0.0053
−19.3590
−0.4951
−0.0002
0.8671 ]
20
Here again,
2X(0) +Y(0)
3F ′ (
3
) + F ′ (Y (0) )
9.0000
= [ 1.3998
−0.2160
0.0126
−84.1752
−0.6879
12
= [ 2.3996
−0.2353
0.0179
−103.5342
−1.1830
−0.0039
3
2.9136 ] + [ 0.9998
60
−0.0193
−0.0041
3.7807 ]
80.0000
Now the first iteration of PCM-2 becomes,
0.0053
−19.3590
−0.4951
−0.0002
0.8671 ]
20
51
−1
X (1)
2X (0) + Y (0)
(0)
′
= X − 4 [3F (
) + F ′ (Y (0) )]
3
X (1)
0.1000
12
= [ 0.1000 ] − 4 × ([ 2.3996
−0.1000
−0.2353
F(X (0) )
0.0179
−103.5342
−1.1830
−0.0041 −1
−1.2000
3.7807 ]) × [−2.2698]
80.0000
8.4620
0.5000
X (1) = [ 0.0061 ]
−0.5233
Following the same manner as it is done in the 1st iteration, the solution is given in table 3.
Table 3: The iteration result of PCM-2 for Example 7.1
(𝒌)
K
(𝒌)
𝒙𝟏
𝒙𝟐
(𝒌)
𝒙𝟑
‖𝐱 (𝐤) − 𝐱 (𝐤−𝟏) ‖𝟐
0
0.1000
0.1000
-0.1000
___
1
0.5000
0.0061
-0.5233
0.5899
2
0.5000
0.0000
-0.5236
0.0061
3
0.5000
0.0000
-0.5236
0.0000
As it is shown in the table above, the iteration process for the approximate solution repeats the
same iteration values after the 2nd iteration and the 2-norm for the difference of the 2nd
iteration from the 3rd iteration is zero. From these things one can understand that the iteration
process has reached the convergence point.
Therefore, the approximate solution by using the PCM-2 for the system given in Example 7.1
0.5000
is X = [ 0.0000 ].
−0.5236
 Approximate solution for Example 7.1 by Broyden’s method
This iterative method is given by the formula:
X (k+1) = X (k) − (B(k) )
−1
(F(X (k) ))
52
But B(𝑘) = B(𝑘−1) +
𝐹(X(k) )(𝛿 (𝑘−1) )
𝑇
𝑇
(𝛿 (𝑘−1) ) (𝛿 (𝑘−1) )
, 𝑘 = 0, 1, 2, …, and B(𝑘) is an approximate updating
matrix instead of Jacobian matrix which has nearly the same value with Jacobian matrix. The
initial updating matrix B (0) is always the same with the value of the Jacobian matrix evaluated
at X (0) .
So having this concept the SNLEs given in Example 7.1 is solved as follows for the same
initial value X (0) .
−1.2000
Then, F(X (0) ) = [−2.2698]
8.4620
And, B (0) = F ′ (x (0) ) =
3
2 × 0.1
[
−0.1 × e−(0.1×0.1)
3.0000
= [ 0.2000
−0.0990
−0.1 × sin(0.1 × −0.1) 0.1 × sin(0.1 × −0.1)
−162(0.1 + 0.1)
cos(−0.1)
]
−(0.1×0.1)
−0.1 × e
20
0.0010
−32.4000
−0.0990
−0.0010
0.9950 ]
20.0000
From equation (4.19) we have B(k) 𝛿 (𝑘) = −𝐹(X (k) ), where 𝛿 (𝑘) = X (k+1) − X (k) .
⟹B(0) 𝛿 (0) = −𝐹(X (0) )
3.0000
[ 0.2000
−0.0990
𝛿 (0)
0.0010
−32.4000
−0.0990
3.0000
= ([ 0.2000
−0.0990
−0.0010
−1.2000
(0)
0.9950 ] 𝛿 = − [−2.2698]
20.0000
8.4620
0.0010
−32.4000
−0.0990
−0.0010 −1
1.2000
0.3999
0.9950 ]) × [ 2.2698 ] = [−0.0805]
20.0000
−8.4620
−0.4215
But, 𝛿 (0) = X (1) − X (0)
⟹X
(1)
=X
(0)
+𝛿
(0)
0.1000
0.3999
0.4999
= [ 0.1000 ] + [−0.0805] = [ 0.0195 ]
−0.1000
−0.4215
−0.5215
Continuing the iteration in the above manner and using B(𝑘) = B(𝑘−1) +
k= 1, 2, 3, …, the solution at each iteration is given in the following table.
𝐹(X(k) )(𝛿 (𝑘−1) )
𝑇
𝑇
(𝛿 (𝑘−1) ) (𝛿 (𝑘−1) )
for
53
Table 4: The summary of the iteration results for Example 7.1 by Broyden’s method
(𝒌)
k
‖𝐱(𝐤) − 𝐱(𝐤−𝟏) ‖𝟐
(𝒌)
(𝒌)
𝒙𝟏
𝒙𝟑
𝒙𝟐
0
0.1000
0.1000
-0.1000
___
1
0.4999
0.0195
-0.5215
0.5866
2
0.5000
0.0088
-0.5232
0.0108
3
0.5000
0.0008
-0.5234
0.0080
4
0.5000
0.0000
-0.5236
0.00082462
5
0.5000
0.0000
-0.5236
0.0000
Thus from the table above, the norm is equal to zero at the 5th iteration. This implies that the
SNLEs given in Example 7.1 has converged to its approximate solution and so that the system
0.5000
has the approximate solution X =[
].
0
−0.5236
 Comparison of the results of the four methods listed above
The following table gives the comparison of the methods how fast they converge to
approximate solution based the number of iterations they perform.
Table 5: Comparison of BM, NM, PCM-1 and PCM-2 for Example 7.1
F(
X)
NI
Broyden method(BM)
Newton method(NM)
𝑥1
𝑥2
𝑥3
𝑥1
𝑥2
𝑥3
𝑥1
𝑥2
𝑥3
𝑥1
𝑥2
1
2
0.4999
0.5000
0.0195
0.0088
-0.5215
-0.5232
0.4999
0.0195
0.0016
-0.5215
0.5000
0.0061
-0.5233
0.5000
0.0061
-0.5233
-0.5236
0.5000
0.0000
-0.5236
0.5000
0.0000
-0.5236
3
4
5
0.5000
0.5000
0.5000
0.0008
0.0000
0.0000
-0.5234
-0.5236
-0.5236
0.5000
0.0000
0.0000
-0.5236
0.5000
0.0000
-0.5236
0.5000
0.0000
-0.5236
0.5000
0.5000
PCM-1(Alg.6.3)
PCM-2(Alg.6.5)
𝑥3
-0.5236
From the above table, it’s very important to see that the same given SNLEs has different
number of iterations for different methods. As it illustrated above Broyden method takes 5
iterations for the system to converge and the Newton’s method takes 4 iterations for the same
system. But the PCM-1 and PCM-2 methods take only three iterations until the system comes
to converge to the approximate solution.
54
Therefore, based on the number of iterations that the four methods perform, it’s possible to say
that the PCM-1 and PCM-2 methods are better than that of the other two methods compared
with them.
Example 7.2
Solve the following system of nonlinear equations which has the exact solution [1 1]𝑇 using
initial approximation X (0) = [0.5 0.5]T (Hafiz and Mohammed, (2012)).
The system is given as,
F2 (X) = (
𝑥12 − 10𝑥1 + 𝑥22 + 8
0
)=[ ]
0
𝑥1 𝑥22 + 𝑥1 − 10𝑥2 + 8
 Solution by Newton’s method
Just as it is done in Example 7.1, we follow the same manner to solve this system and its
solution for the iterations is given in the following table.
The above iteration results are shortly summarized in the table below.
Table 6: The summary of iteration results of Newton’s method for Example 7.2
(𝒌)
(𝒌)
‖𝐱(𝐤) − 𝐱(𝐤−𝟏) ‖𝟐
k
𝒙𝟏
𝒙𝟐
0
0.5000
0.5000
___
1
0.9377
0.9392
0.6201
2
0.9987
0.9984
0.0850
3
1.0000
1.0000
0.0021
4
1.0000
1.0000
0.0000
From the table above the norm is zero at the 4th iteration and the sequence repeats the same
results after the 3rd iteration. This implies that the system F2 (X) has reached the point of
convergence. Therefore, the system has the exact solution [1 1]𝑇 .
55
 Solution by PCM-1
Table 7: The summary of iteration results for PCM-1 in Example 7.2
(𝒌)
(𝒌)
k
‖𝐱 (𝐤) − 𝐱 (𝐤−𝟏) ‖
𝒙
𝒙
𝟏
𝟐
0
0.5000
0.5000
____
1
0.9911
0.9900
0.6937
2
1.0000
1.0000
0.0134
3
1.0000
1.0000
0.0000
𝟐
The norm at the 3rd iteration is equal to zero. This implies that the sequence is converged to
the solution and therefore, the system F2 (X) has the exact solution X = [1 1]T .
 The solution of F2 by PCM-2
Table 8: The iteration result of PCM-2 for the system given in Example 7.2
(𝒌)
(𝒌)
‖𝐱 (𝐤) − 𝐱 (𝐤−𝟏) ‖𝟐
k
𝒙𝟏
𝒙𝟐
0
0.5000
0.5000
___
1
0.9841
0.9483
0.6598
2
1.0000
1.0000
0.0541
3
1.0000
1.0000
0.0000
From the table given above the norm being zero at the 3rd iteration implies that the sequence of
the iteration converged to the solution. So the value X = [1 1]𝑇 is the exact solution of the
system given in Example 7.2.
 Solution by Broyden’s method
Table 9: The iteration result of the Broyden’s method for the system given in Example 7.2
(𝒌)
(𝒌)
‖𝐱(𝐤) − 𝐱(𝐤−𝟏) ‖𝟐
K
𝒙𝟏
0
0.5000
0.5000
___
1
0.9377
0.9392
0.6201
2
0.9912
0.9901
0.0738
3
0.9998
0.9997
0.0129
4
1.0000
1.0000
0.000036056
5
1.0000
1.0000
0.0000
𝒙𝟐
56
The norm at the 5th iteration is zero. This indicates that the system has converged to its
solution.
Therefore, the given system of nonlinear equations has the exact solutionX = [1 1]𝑇 .
The following table gives the comparison of the methods how fast they converge to solution
based on the number of iterations they perform for the given example 7.2.
Table 10: Comparison of BM, NM, PCM-1 and PCM-2 for Example 7.2
𝐅𝟐 (X)
Broyden method(BM)
Newton method(NM)
PCM-1(Alg.6.3)
𝑥2
PCM-2(Alg.6.5)
NI
𝑥1
𝑥2
𝑥1
𝑥2
𝑥1
𝑥1
𝑥2
1
0.9377
0.9392
0.9377
0.9392
0.9911
0.9900
0.9841
0.9483
2
0.9912
0.9901
0.9987
0.9984
1.0000
1.0000
1.0000
1.0000
3
0.9998
0.9997
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
4
1.0000
1.0000
1.0000
1.0000
5
1.0000
1.0000
From the above table, it’s very important to see that the same given SNLEs has different
number of iterations for different methods. As it illustrated above Broyden method takes 5
iterations for the system to converge and the Newton’s method takes 4 iterations for the same
system. But the PCM-1 and PCM-2 methods take only three iterations until the system comes
to converge to the exact solution of the system given in Example 7.2.
Therefore, based on the number of iterations that the four methods perform, it’s nice to
suggest that the PCM-1 and PCM-2 methods are better than the other two methods compared
with them.
Here below Examples 7.3 to 7.6 are given to show more on the performance of the four
methods. These problems are solved using different MATLAB codes given for the four
methods and these codes are written in appendices A, B, C, …, H in chapter-10.
Example 7.3 [Darvishi and Barati, (2007)]: Solve the following system of nonlinear equations
at initial approximate value(0.5 0.5 0.5)𝑇 .
57
𝑥12 + 𝑥22 + 𝑥32 − 1 = 0
2𝑥12 + 𝑥22 − 4𝑥3 = 0
3𝑥12 − 4𝑥22 + 𝑥32 = 0
Solution:
The solution of this system is given in table-11 for all methods together. The system is solved
using MATLAB codes that are given in Appendices A, B, C and D.
Table 11: Iteration results of BM, NM, PCM-1 and PCM-2 for Example 7.3
F(X)
Broyden method(BM)
Newton method(NM)
PCM-1(Alg.6.3)
PCM-2(Alg.6.5)
NI
𝑥1
𝑥2
𝑥3
𝑥1
𝑥2
𝑥3
𝑥1
𝑥2
𝑥3
𝑥1
𝑥2
𝑥3
1
0.7500
0.6500
0.3500
0.7500
0.6500
0.3500
0.6906
0.6263
0.3429
0.7171
0.6374
0.3465
2
0.6885
0.6275
0.3444
0.7001
0.6289
0.3426
0.6983
0.6285
0.3426
0.6983
0.6285
0.3426
3
0.6985
0.6280
0.3428
0.6983
0.6285
0.3426
0.6983
0.6285
0.3426
0.6983
0.6285
0.3426
4
0.6982
0.6286
0.3426
0.6983
0.6285
0.3426
5
0.6983
0.6285
0.3426
6
0.6983
0.6285
0.3426
From the above table, we can see that the given system has different number of iterations
when solved by using different methods. As it is illustrated above, Broyden’s method took 6
iterations for the system to converge and the Newton’s method took 4 iterations. But both
PCM-1 and PCM-2 methods took only three iterations until the system comes to converge to
the approximate solution and the approximate solution of the given system is
0.6983
X = [0.6285]
0.3426
Therefore, the PCM-1 and PCM-2 methods took less number of iterations than others. This
implies both PCM-1 and PCM-2 methods are effective and better than Newton and Broyden’s
methods.
Example 7.4[Darvishi and Barati, (2004)]: Solve the following system of nonlinear equations
at initial approximate value(2.5 0.5 1.5)𝑇 .
𝑥12 + 𝑥22 + 𝑥32 − 9 = 0
𝑥1 𝑥2 𝑥3 − 1 = 0
𝑥1 + 𝑥2 − 𝑥32 = 0
58
Solution:
Using the same way of solving as we have done above, the solution for this case is also solved
using MATLAB codes and the only iteration results for all methods are given in table-12
below. This system of nonlinear equations has been solved using the same codes in appendices
A, B, C and D. The only changes made for this example is the use of different SNLEs with
different initial approximation value.
Table 12: Iteration results of BM, NM, PCM-1 and PCM-2 for Example 7.4
F(X)
Broyden method(BM)
Newton method(NM)
PCM-1(Alg.6.3)
PCM-2(Alg.6.5)
NI
𝑥1
𝑥2
𝑥3
𝑥1
𝑥2
𝑥3
𝑥1
𝑥2
𝑥3
𝑥1
𝑥2
𝑥3
1
2.5129
0.2114
1.6581
2.5129
0.2114
1.6581
2.4892
0.2459
1.6536
2.5003
0.2296
1.6559
2
2.4888
0.2461
1.6541
2.4917
0.2424
1.6535
2.4914
0.2427
1.6535
2.4914
0.2427
1.6535
3
2.4914
0.2429
1.6535
2.4914
0.2427
1.6535
2.4914
0.2427
1.6535
2.4914
0.2427
1.6535
4
2.4914
0.2427
1.6535
2.4914
0.2427
1.6535
5
2.4914
0.2427
1.6535
The results of the iteration process show from this table that the required approximate solution
for the system given in Example 7.4 is[2.4914
0.2427
1.6535]T . Another thing what we
can understand depending up on the number of iterations from the table-12 is also that there
are differences in using the given methods. The methods with less number of iterations are
better than that of methods took large number of iterations.
In this case also the PCM-1 and PCM-2 are relatively better than Newton and Broyden’s
methods. They took fewer numbers of iterations until they reach the point of convergence.
Example 7.5 [Hafiz and Bahgat, (2012)] Consider the following system of nonlinear functions
and find the solution of the system if the initial approximate solution 𝑥 (0) = [10 6 − 5]𝑇 .
𝑓1 (𝑥, 𝑦, 𝑧) = 15𝑥 + 𝑦 2 − 4𝑧 − 13 = 0
F(X) = {𝑓2 (𝑥, 𝑦, 𝑧) = 𝑥 2 + 10𝑦 − 𝑒 −𝑧 − 11 = 0
𝑓3 (𝑥, 𝑦, 𝑧) = 𝑦 3 − 25𝑧 + 22 = 0
Solution:
The following table gives the iteration results of this problem for all methods given above and
these results are found by using MATLAB codes in appendices A, B, C and D in chapter-10.
59
Table 13: Iteration results of BM, NM, PCM-1 and PCM-2 for Example 7.5
F(X)
BM
NM
PCM-1
PCM-2
NI
x
y
z
x
Y
z
x
Y
z
x
y
z
1
-0.0513
2.9983
-3.4472
-0.0513
2.9983
-3.4472
0.6441
0.9756
-3.1375
0.2857
2.1708
-3.2292
2
0.3572
1.7757
-3.3363
1.1912
-0.5845
-1.9070
0.8464
0.5118
0.5857
0.8117
-0.4824
0.8327
3
1.5987
-3.1866
-2.4532
1.1080
-0.2296
0.8874
1.0422
1.0307
0.9238
1.0437
1.0274
0.9220
4
-0.0691
-4.0377
-1.7322
1.1378
1.0118
0.8874
1.0421
1.0311
0.9238
1.0421
1.0311
0.9238
5
-0.5645
-3.1357
-1.4904
1.0421
1.0320
0.9239
1.0421
1.0311
0.9238
1.0421
1.0311
0.9238
6
-0.4704
-3.8333
-1.1061
1.0421
1.0311
0.9238
7
-0.2893
-3.6157
-0.8011
1.0421
1.0311
0.9238
8
2.9731
-0.1980
4.7720
9
2.9326
2.0404
3.9224
10
0.7844
1.0117
1.2238
11
0.9634
1.2176
0.8498
12
1.0141
1.0853
0.9036
13
1.0423
1.0308
0.9240
14
1.0415
1.0329
0.9219
15
1.0421
1.0311
0.9238
16
1.0421
1.0311
0.9238
As it is demonstrated in this table, the methods took different number of iterations until the
process converged to the approximate solution.
In general, the two predictor-corrector methods PCM-1 and PCM-2 are fast convergent
methods when compared with that of Newton and Broyden’s methods as we have seen in all
given examples. Both methods took a less number of iterations in all discussed examples.
Example 7.6(Gilberto, 2004): Find the approximate solution for the following system of
nonlinear equations if the initial approximation of the system is given by X (0) = (2 1)T .
𝑥12 + 𝑥22 − 50 = 0
𝑥1 . 𝑥2 − 25 = 0
60
Solution: This system of nonlinear equations has been solved using the same codes in
appendices E, F, G and H. The only changes made for this example is the use of different
SNLEs with different initial approximation value.
Table 14: Iteration results of BM, NM, PCM-1 and PCM-2 for Example 7.6
F(x)
BM
NM
PCM-1
PCM-2
NI
𝑥1
𝑥2
𝑥1
𝑥2
𝑥1
𝑥2
𝑥1
𝑥2
1
9.3333 8.8333 9.3333 8.8333
3.8163
3.4829
5.0640
4.6354
2
3.8772 3.4127 6.0428 5.7928
5.0176
4.9065
5.0714
4.9285
3
4.7441 4.3844 5.1337 5.0087
5.0185
4.9815
5.0238
4.9762
4
5.2024 49415
5.0317 4.9692
5.0062
4.9938
5.0079
4.9921
5
5.1149 4.8781 5.0156 4.9844
5.0021
4.9979
5.0026
4.9974
6
5.1042 4.8956 5.0078 4.9922
5.0007
4.9993
5.0009
4.9991
7
5.0585 4.9416 5.0039 4.9961
5.0002
4.9998
5.0003
4.9997
8
5.0375 4.9625 5.0020 4.9980
5.0000
5.0000
5.0000
5.0000
9
5.0228 4.9772 5.0010 4.9990
5.0000
5.0000
5.0000
5.0000
10
5.0142 4.9858 5.0005 4.9995
11
5.0087 4.9913
5.0002
4.9998
12
5.0054 4.9946
5.0000
5.0000
13
5.0033 4.9967
5.0000
5.0000
14
5.0021 4.9979
15
5.0013 4.9987
16
5.0008 4.9992
17
18
19
5.0005 4.9995
5.0003 4.9997
5.0002 4.9998
20
5.0000 5.0000
21
5.0000 5.0000
As it is demonstrated in the above table, the methods took different number of iterations until
the process of iteration converged to the approximate solution [5.0000
5.0000]T . From this
table Broyden took 21 iterations, Newton method took 13 iterations and both PCM-1 and 2
took 9 iterations.
In general, the two predictor-corrector methods PCM-1 and PCM-2 are fast convergent
methods when compared with that of Newton and Broyden’s methods as we have seen in all
given examples. Both methods took a less number of iterations in all discussed examples.
61
8. SUMMARY, CONCLUSION AND RECOMMENDATION
8.1.
Summary
System of nonlinear equations arises in different areas of applied sciences such as engineering,
physics, medicines, chemistry, and robotics. They appear also in many geometric
computations such as intersections, minimum distance, and when solving initial or boundary
value problems in ordinary or partial differential equations.
Solving system of nonlinear equations is much more difficult when compared with solving the
linear counterpart and finding solutions of these problems is one of the most important parts of
the studies in numerical analysis. Because of this reason a number of mathematicians
developed different iterative methods to solve problems of systems of nonlinear equations.
Some important methods for solving system of nonlinear equations are methods like Newton’s
method, Steepest Decent method, Broyden’s method, Predictor-corrector methods, etc are a
few of them.
Mainly, this project paper focused on Newton’s method, Broyden’s method and two iterative
methods with two steps each (predictor-corrector methods 1 and 2) for solving system of
nonlinear equations. The Predictor-corrector methods 1 and 2 are methods derived from
closed-open quadrature formulas by using Taylor polynomial.
The general aim of this project is to study iterative methods: Newton’s method, Broyden’s
method, predictor-corrector method-1 and predictor-corrector method-2 which use to solve
system of nonlinear equations and to compare the performance of all these methods one with
the other based on the number of iterations which took until they reach to the solution.
The iterative schemes for these methods has been given and discussed how to derive them.
Some numerical examples have been given to indicate the performance of the methods and
showed the number of iterations until the process come to converge to the approximate
solution in a 4-decimal place rounding. From the comparison made among the given methods,
the PCM-1 and PCM-2 were necessarily better performing methods than the others.
62
8.2.
Conclusion
In this project four iterative methods for solving system of nonlinear equations have been
discussed in detail and the iterative schemes for each method are derived. These methods are
Newton’s method, Broyden’s method, PCM-1 and PCM-2. The comparison among the four
methods is made based on the number of iterations they took until they converge to the true
solution and it is shown from all the test problems that PCM-1 and PCM-2 are better
performing methods than that of the Newton and Broyden’s methods. This means that in all
computations performed, it is PCM-1 and PCM-2 that took fewer numbers of iterations than
Newton and Broyden’s methods took until converging to the solution.
So this is the implication for both methods PCM-1 and PCM-2 to be more important to use
them than other methods which are compared to them. Not only this, but also both PCM-1 and
PCM-2 being cubic convergent makes them better than those Newton and Broyden’s methods
for Newton’s method is quadratically convergent and Broyden’s method super linearly
convergent.
8.3.
Recommendation
Based on the performance of the given methods, the following recommendations are
suggested.
 From iterative methods for solving SNLEs some are NM, BM, PCM-1 and PCM-2.
Among these methods, using PCM-1 and 2 help the user to reach to the solution after a
few numbers of iterations when compared with NM and BM.
 Individuals interested to work on SNLEs using one of the four methods can benefit from
the result and the MATLAB codes are presented in appendices to use by changing the
necessary parts of the codes depending up on the size of the system of nonlinear equations
they solve.
 It is necessary to show the way how to choose initial approximate values if individuals
who need to study more on SNLEs to begin the iteration when they need to find a solution
and it is also recommended to have a theory how a comparison is made between two or
more iterative methods.
63
9. REFERENCES
Andreas Griewank, 1986. The global convergence of Broyden-like methods with a suitable
line search. Journal of Australia, Mathematics Soc. Ser. B, 28:75.
Anuradha Singh and Jaiswal, J. P. 2013. Third and fourth-order iterative methods for solving
systems of nonlinear Equations. Mathematics Subject Classification 2000: 65H10,
65Y20.
Aslam Noor, M. 2007. Fifth-order convergent iterative method for solving nonlinear equations
using quadrature formula. Journal of Mathematics Control Science Application, 1:
241-249.
Aslam Noor, M. 2007. Some applications of quadrature formulas for solving nonlinear
equations. Nonlinear Analysis Forum, 12 (1): 91-96.
Atkinson, K. and Han, W. 2001. Theoretical Numerical Analysis. A Functional Analysis
Framework. Springer-Verlag.
Babajee, D.K.R., Dauhoo, M.Z., Darvishi, M.T. and Barati, A.2008. A note on the local
convergence of iterative methods based on Adomian decomposition method and 3node quadrature rule. Applied Mathematics and Computation, 200: 452–458.
Broyden,C.1965. A class of iterative methods for solving system of non-linear equations.
Computational Mathematics, 19:577-593
Burden, R.L. and Faires, J.D. 2001. Numerical Analysis, 7th Edition. PWS Publishing
Company, Boston.
Burden, R.L., Faires, J.D. 2005. Numerical Solutions of Nonlinear Systems of Equations.
Numerical Analysis, Belmount: Thomson Brooks/Cole (pp. 597-640).
Chun, C. 2006. A new iterative method for solving nonlinear equations. Journal of applied
mathematical and computation, 178(2): 415-422.
Cordero, A. and Torregrosa, J.R. 2006. Variants of Newton’s method for functions of several
variables. Applied Mathematics Computation, 183:199-208.
Cordero, A. and Torregrosa, J.R. 2007. Variants of Newton’s method using fifth-order
quadrature formulas. Applied Mathematics Computation, 190: 686_698.
Courtney Remani, 2013. Numerical Methods for Solving Systems of Nonlinear Equations.
Lakehead University, Thunder Bay, Ontario, Canada.
64
Daftardar-Gejji, V. and Jafari, H. 2006. An iterative method for solving nonlinear functional
equations. Journal of Applied Mathematics, 316: 753–763.
Darvishi, M. T. and Barati, A. 2007b. Super cubic iterative methods to solve systems
nonlinear equations. Journal of applied mathematical and computation, 188: 16781685.
Darvishi, M. T., and Barati A. 2007a. A third-order Newton-type method to solve systems of
nonlinear equations. Journal of applied mathematical and computation, 187: 630-635.
Darvishi, M.T. 2009. A two-step high order Newton-like method for solving systems of
nonlinear equations. International Journal of Pure and Applied Mathematics,
57(4):543-555.
Darvishi, M.T. and Barati, A. 2004. A fourth-order method from quadrature formulae to solve
systems of nonlinear equations. Applied Mathematics Computation, 188: 257-261.
Deuflhard, 2012. A short history of Newton's method. Documenta Mathematica, Optimization
stories: 25-30.
Edwards, Jr. 1997. The Historical Development of the Calculus. Springer-Verlag.
Fosdick, L., Jessup, E., Schauble, C., and G. Domik, G., 1996. An Introduction to HighPerformance. Scientific Computing, MIT Press.
Frontini, M. and Sormani, E. 2004. Third-order methods from quadrature formulae for solving
systems of nonlinear equations. Applied Mathematics and Computation, 149: 771-782.
Gautschi, W. 1997. Numerical Analysis: An Introduction, Birkhäuser.
Gilberto Urroz E., 2004. Solution of non-linear equations. Applied Mathematics, 9-16.
Golbabai, A., and Javidi, M. 2007. A new family of iterative methods for solving system of
nonlinear algebraic equations. Journal of applied mathematical and computation, 190:
1717-1722.
Griewank, A., 2012. Broyden updating, the good and the bad! Documenta Mathematica,
Optimization stories, 301–315.
Hafiz, M. A. and Mohamed Bahgat, S. M. 2012. An Efficient Two-step Iterative Method for
Solving System of Nonlinear Equations. Journal of Mathematics Research,
doi.org/10.5539/jmr.v4n4p28.
Hafiz, M.A., and Bahgat, M.S.M., 2012. Extended and modified Halley's iterative method for
solving nonlinear system of equations. Journal of Mathematical and Computational
Sciences, 1(2):1-14.
65
Hassan, M. and Yusuf Waziri, M., 2015. On Broyden-like update via some quadrature
formulas for solving nonlinear systems of equations. Turkish Journal of Mathematics,
39:335-345.
Hosseini, S.M. 2010. Improving homotopy analysis method for system of nonlinear algebraic
equations. Journal of Advanced Research in Applied Mathematics, 4 (2):22-30.
Mamat, M., Kabiru Muhammad and Muhammad Waziri, Y. 2014. Trapezoidal Broyden’s
method for solving systems of nonlinear equations, Applied Mathematical Sciences, 6
(8): 251 – 260.
Martinez, J.M., 2000. Practical quasi-Newton methods for solving nonlinear systems. Journal
of Computational and Applied Mathematics, 124:97-121.
Michael Heath, T., 2002. Scientific computing: An introductory survey. Applied Mathematics
Computation. University of Illinois at Urbana-Champaign, 48-53.
Muhammad Aslam Noor, Muhammad Waseem and Khalida Inayat Noor, 2015. New iterative
technique for solving a system of nonlinear equations. Applied Mathematics and
Computation, 271: 446- 466.
Noor, M.A. and Noor, K.I., 2006. Three-step iterative methods for nonlinear equations,
Applied Mathematics and Computation, 183: 322–327.
Noor, M.A. and Waseem, M., 2008. Iterative methods for solving a system of nonlinear
equations. Computers and Mathematics with Applications, 57:101-106.
Noor, M.A., Noor, K.I., Mohyud-Din, S.T. and Shabbir, A. 2006. An iterative method with
cubic convergence for nonlinear equations, Applied Mathematics and Computation,
183:1249–1255.
Ozel, M. 2010. A New Decomposition Method for Solving System of Nonlinear Equations.
Journal of applied mathematical and computation, 15: 89-95.
Podisuk, M., Chundang, U., Sanprasert, W. 2007. Single-step formulas and multi-step
formulas of the integration method for solving the initial value problem of ordinary
differential equation. Applied Mathematics Computation, 190: 1438-1444.
Timothy Sauer, 2006. Numerical Analysis. Addison-Wesley, Boston, MA.
Traub, J.F. 1964. Iterative Methods for the Solution of Equations. Prentice-Hall Englewood
Cliffss, New Jersey, USA.
Weerakoon, S. and Fernando, T.G.I. 2000. A variant of Newton.s method with accelerated
third-order convergence. Applied Mathematics Lett. 13: 87-93.
66
10. APPENDICES
The following are MATLAB codes for Newton, PCM-1, PCM-2 and Broyden’s methods.
Appendix- A
MATLAB code for Newton’s method:
File name: nm3x3.m
%Newton method to solve nonlinear system of equations f(X)=0 with three equations and
three unknowns
%input f and x0
%x0-initial guess a column vector in R^3
%e.g. x0=[0.1 0.1 -0.1]';
%f-a column vector valued function from R^3 into R^3
%e.g.
f=[3*x-cos(y*z)-0.5;x^2-81*(y+0.1)^2+sin(z)+1.06;exp(-x*y)+20*z+((10*sym(pi)3)/3)];
function nm3x3(f,x0)
syms x y z
maxiter=100;
jb=jacobian(f,[x y z]);
A=subs(f,[x y z],x0);
B=subs(jb,[x y z],x0);%the jacobian matrix of f evaluated at initial guess x0
x1=x0-inv(B)*A;
z1=ones(3,100);
z1(1:3,1)=x0;
z1(1:3,2)=x1;
k=1;
while ((norm(z1(1:3,k+1)-z1(1:3,k),inf)>=10^-4)&(k<=maxiter))
A=subs(f,[x y z],z1(1:3,k+1));
B=subs(jb,[x y z],z1(1:3,k+1));
z1(1:3,k+2)=z1(1:3,k+1)-inv(B)*A;
k=k+1;
end
if k>maxiter
disp('too many iterations')
else
disp('Number of iteration k=')
disp(k)
iter=k;
disp('value of X(j) at iteration j where j=0,1,2,...,k')
disp(z1(1:3,1:k+1))
disp('stopping condition ||X(j)-X(j-1)|| for j=1,...,k')
for j=1:k
disp(norm(z1(1:3,j+1)-z1(1:3,j)))
end
disp('Solution='),disp(z1(1:3,k+1))
67
soln=subs(f,[x y z],z1(1:3,k+1));
disp('check for solution')
disp('f[X(k)]=')
disp(soln)
end
Appendix- B
MATLAB code for PCM-1
File name: pcm-1.m
%predictor corrector method-1 to solve nonlinear system of equations f(X) =0 with three
equations and three unknowns
%input f and x0
%x0-initial guess a column vector in R^3
%e.g. x0=[0.1 0.1 -0.1]';
%f-a column vector valued function from R^3 into R^3
%e.g. f=[3*x-cos(y*z)-0.5;x^2-81*(y+0.1)^2+sin(z)+1.06;exp(-x*y)+20*z+((10*sym(pi)3)/3)];
function pcm_1(f,x0)
syms x y z
maxiter=100;
jb=jacobian(f,[x y z]);
fx0=subs(f,[x y z],x0);
jbx0=subs(jb,[x y z],x0);
y0=x0-inv(jbx0)*fx0;
jbxy0=subs(jb,[x y z],(x0+2*y0)/3);
x1=x0-4*inv(jbx0+3*jbxy0)*fx0;
w1=ones(3,100);
w1(1:3,1)=y0;
z1=ones(3,100);
z1(1:3,1)=x0;
z1(1:3,2)=x1;
k=1;
while ((norm(z1(1:3,k+1)-z1(1:3,k))>=10^-4)&(k<=maxiter))
A=subs(f,[x y z],z1(1:3,k+1));
B=subs(jb,[x y z],z1(1:3,k+1));
w1(1:3,k+1)=z1(1:3,k+1)-inv(B)*A;
C=subs(jb,[x y z],(z1(1:3,k+1)+2*w1(1:3,k+1))/3);
z1(1:3,k+2)=z1(1:3,k+1)-4*inv(B+3*C)*A;
k=k+1;
end
if k>maxiter
disp('too many iterations')
else
disp('Number of iteration k=')
disp(k)
68
iter=k;
disp('value of X(j) at iteration j where j=0,1,2,...,k')
disp(z1(1:3,1:k+1))
disp('stopping condition ||X(j)-X(j-1)|| for j=1,...,k')
for j=1:k
disp(norm(z1(1:3,j+1)-z1(1:3,j)))
end
disp('Solution='),disp(z1(1:3,k+1))
soln=subs(f,[x y z],z1(1:3,k+1));
disp('check for solution')
disp('f[X(k)]=')
disp(soln)
end
Appendix-C
MATLAB code for PCM-2
File name: pcm-2.m
%predictor corrector method-2 to solve nonlinear system of equations f(X)=0 with three
equations and three unknowns
%input f and x0
%x0-initial guess a column vector in R^3
%e.g. x0=[0.1 0.1 -0.1]';
%f-a column vector valued function from R^3 into R^3
%e.g.
f=[3*x-cos(y*z)-0.5;x^2-81*(y+0.1)^2+sin(z)+1.06;exp(-x*y)+20*z+((10*sym(pi)3)/3)];
function pcm_2(f,x0)
syms x y z
maxiter=100;
jb=jacobian(f,[x y z]);
fx0=subs(f,[x y z],x0);
jbx0=subs(jb,[x y z],x0);
y0=x0-inv(jbx0)*fx0;
jbxy0=subs(jb,[x y z],(2*x0+y0)/3);
x1=x0-4*inv(jbx0+3*jbxy0)*fx0;
w1=ones(3,100);
w1(1:3,1)=y0;
z1=ones(3,100);
z1(1:3,1)=x0;
z1(1:3,2)=x1;
k=1;
while ((norm(z1(1:3,k+1)-z1(1:3,k))>=10^-4)&(k<=maxiter))
A=subs(f,[x y z],z1(1:3,k+1));
B=subs(jb,[x y z],z1(1:3,k+1));
w1(1:3,k+1)=z1(1:3,k+1)-inv(B)*A;
B1=subs(jb,[x y z],w1(1:3,k+1));
69
C=subs(jb,[x y z],(2*z1(1:3,k+1)+w1(1:3,k+1))/3);
z1(1:3,k+2)=z1(1:3,k+1)-4*inv(3*C+B1)*A;
k=k+1;
end
if k>maxiter
disp('too many iterations')
else
disp('Number of iteration k=')
disp(k)
iter=k;
disp('value of X(j) at iteration j where j=0,1,2,...,k')
disp(z1(1:3,1:k+1))
disp('stopping condition ||X(j)-X(j-1)|| for j=1,...,k')
for j=1:k
disp(norm(z1(1:3,j+1)-z1(1:3,j)))
end
disp('Solution='),disp(z1(1:3,k+1))
soln=subs(f,[x y z],z1(1:3,k+1));
disp('check for solution')
disp('f[X(k)]=')
disp(soln)
end
Appendix-D
MATLAB code for Broyden’s method
File name: broyden3x3.m
The code is given by:
%Broyden method to solve nonlinear system of equations f(X)=0 with three equations and
three unknowns
%input f and x0
%x0-initial guess a column vector in R^3
%e.g. x0=[0.1 0.1 -0.1]';
%f-a column vector valued function from R^3 into R^3
%e.g.
f=[3*x-cos(y*z)-0.5;x^2-81*(y+0.1)^2+sin(z)+1.06;exp(-x*y)+20*z+((10*sym(pi)3)/3)];
function broyden3x3(f,x0)
syms x y z
maxiter=100;
jb=jacobian(f,[x y z]);
B0=subs(jb,[x y z],x0);
d0=-inv(B0)*subs(f,[x y z],x0);
x1=x0+d0;
z1=ones(3,100);
z1(1:3,1)=x0;
z1(1:3,2)=x1;
70
Bk=B0;
dk=d0;
k=1;
while ((norm(z1(1:3,k+1)-z1(1:3,k))>=10^-4)&(k<=maxiter))
A=subs(f,[x y z],z1(1:3,k+1))*dk';
c=dk'*dk;
Bk=Bk+(A/c);
dk=-inv(Bk)*subs(f,[x y z],z1(1:3,k+1));
z1(1:3,k+2)=z1(1:3,k+1)+dk;
k=k+1;
end
if k>maxiter
disp('too many iterations')
else
disp('Number of iteration k=')
disp(k)
iter=k;
disp('value of X(j) at iteration j where j=0,1,2,...,k')
disp(z1(1:3,1:k+1))
disp('stopping condition ||X(j)-X(j-1)|| for j=1,...,k')
for j=1:k
disp(norm(z1(1:3,j+1)-z1(1:3,j)))
end
disp('Solution='),disp(z1(1:3,k+1))
soln=subs(f,[x y z],z1(1:3,k+1));
disp('check for solution')
disp('f[X(k)]=')
disp(soln)
end
Appendix- E
MATLAB Codes
Broyden2x2.m
%Broyden method to solve nonlinear system of equations f(X)=0 with two equations and two
unknowns
%input f and x0
%x0-initial guess a column vector in R^2
%e.g. x0=[0.1 0.1]';
%f-a column vector valued function from R^2 into R^2
%e.g. f=[x^2-81*(y+0.1)^2+sin(z)+1.06;exp(-x*y)+20*z+((10*sym(pi)-3)/3)];
function broyden2x2(f,x0)
syms x y
maxiter=100;
jb=jacobian(f,[x y]);
B0=subs(jb,[x y],x0);
d0=-inv(B0)*subs(f,[x y],x0);
71
x1=x0+d0;
z1=ones(2,100);
z1(1:2,1)=x0;
z1(1:2,2)=x1;
Bk=B0;
dk=d0;
k=1;
while ((norm(z1(1:2,k+1)-z1(1:2,k))>=10^-4)&(k<=maxiter))
A=subs(f,[x y],z1(1:2,k+1))*dk';
c=dk'*dk;
Bk=Bk+(A/c);
dk=-inv(Bk)*subs(f,[x y],z1(1:2,k+1));
z1(1:2,k+2)=z1(1:2,k+1)+dk;
k=k+1;
end
if k>maxiter
disp('too many iterations')
else
disp('Number of iteration k=')
disp(k)
iter=k;
disp('value of X(j) at iteration j where j=0,1,2,...,k')
disp(z1(1:2,1:k+1))
disp('stopping condition ||X(j)-X(j-1)|| for j=1,...,k')
for j=1:k
disp(norm(z1(1:2,j+1)-z1(1:2,j)))
end
disp('Solution='),disp(z1(1:2,k+1))
soln=subs(f,[x y],z1(1:2,k+1));
disp('check for solution')
disp('f[X(k)]=')
disp(soln)
end
Appendix- F
MATLAB Codes
nm2x2.m
%Newton method to solve nonlinear system of equations f(X)=0 with two equations and two
unknowns
%input f and x0
%x0-initial guess a column vector in R^2
%e.g. x0=[0.1 0.1]';
%f-a column vector valued function from R^2 into R^2
%e.g. f=[3*x-cos(y)-0.5;x^2-81*(y+0.1)^2+1.06];
function nm2x2(f,x0)
syms x y
72
maxiter=100;
jb=jacobian(f,[x y]);
A=subs(f,[x y],x0);
B=subs(jb,[x y],x0);%the jacobian matrix of f evaluated at initial guess x0
x1=x0-inv(B)*A;
z1=ones(2,100);
z1(1:2,1)=x0;
z1(1:2,2)=x1;
k=1;
while ((norm(z1(1:2,k+1)-z1(1:2,k))>=10^-4)&(k<=maxiter))
A=subs(f,[x y],z1(1:2,k+1));
B=subs(jb,[x y],z1(1:2,k+1));
z1(1:2,k+2)=z1(1:2,k+1)-inv(B)*A;
k=k+1;
end
if k>maxiter
disp('too many iterations')
else
disp('Number of iteration k=')
disp(k)
iter=k;
disp('value of X(j) at iteration j where j=0,1,2,...,k')
disp(z1(1:2,1:k+1))
disp('stopping condition ||X(j)-X(j-1)|| for j=1,...,k')
for j=1:k
disp(norm(z1(1:2,j+1)-z1(1:2,j)))
end
disp('Solution='),disp(z1(1:2,k+1))
soln=subs(f,[x y],z1(1:2,k+1));
disp('check for solution')
disp('f[X(k)]=')
disp(soln)
end
Appendix- G
MATLAB Codes
Pcml.m
%predictor corrector method-1 to solve nonlinear system of equations f(X)=0 with two
equations and two unknowns
%input f and x0
%x0-initial guess a column vector in R^2
%e.g. x0=[0.1 0.1]';
%f-a column vector valued function from R^2 into R^2
%e.g. f=[x^2-81*(y+0.1)^2+sin(z)+1.06;exp(-x*y)+20*z+((10*sym(pi)-3)/3)];
function pcm1(f,x0)
syms x y
73
maxiter=100;
jb=jacobian(f,[x y]);
fx0=subs(f,[x y],x0);
jbx0=subs(jb,[x y],x0);
y0=x0-inv(jbx0)*fx0;
jbxy0=subs(jb,[x y],(x0+2*y0)/3);
x1=x0-4*inv(jbx0+3*jbxy0)*fx0;
w1=ones(2,100);
w1(1:2,1)=y0;
z1=ones(2,100);
z1(1:2,1)=x0;
z1(1:2,2)=x1;
k=1;
while ((norm(z1(1:2,k+1)-z1(1:2,k))>=10^-4)&(k<=maxiter))
A=subs(f,[x y],z1(1:2,k+1));
B=subs(jb,[x y],z1(1:2,k+1));
w1(1:2,k+1)=z1(1:2,k+1)-inv(B)*A;
C=subs(jb,[x y],(z1(1:2,k+1)+2*w1(1:2,k+1))/3);
z1(1:2,k+2)=z1(1:2,k+1)-4*inv(B+3*C)*A;
k=k+1;
end
if k>maxiter
disp('too many iterations')
else
disp('Number of iteration k=')
disp(k)
iter=k;
disp('value of X(j) at iteration j where j=0,1,2,...,k')
disp(z1(1:2,1:k+1))
disp('stopping condition ||X(j)-X(j-1)|| for j=1,...,k')
for j=1:k
disp(norm(z1(1:2,j+1)-z1(1:2,j)))
end
disp('Solution='),disp(z1(1:2,k+1))
soln=subs(f,[x y],z1(1:2,k+1));
disp('check for solution')
disp('f[X(k)]=')
disp(soln)
end
Appendix- H
MATLAB Codes
Pcm2.m
%predictor corrector method-2 to solve nonlinear system of
equations f(X)=0 with two equations and two unknowns
%input f and x0
74
%x0-initial guess a column vector in R^2
%e.g. x0=[0.1 0.1]';
%f-a column vector valued function from R^2 into R^2
%e.g. f=[x^2-81*(y+0.1)^2+sin(z)+1.06;exp(x*y)+20*z+((10*sym(pi)-3)/3)];
function pcm2(f,x0)
syms x y
maxiter=100;
jb=jacobian(f,[x y]);
fx0=subs(f,[x y],x0);
jbx0=subs(jb,[x y],x0);
y0=x0-inv(jbx0)*fx0;
jbxy0=subs(jb,[x y],(2*x0+y0)/3);
x1=x0-4*inv(jbx0+3*jbxy0)*fx0;
w1=ones(2,100);
w1(1:2,1)=y0;
z1=ones(2,100);
z1(1:2,1)=x0;
z1(1:2,2)=x1;
k=1;
while ((norm(z1(1:2,k+1)-z1(1:2,k))>=10^-4)&(k<=maxiter))
A=subs(f,[x y],z1(1:2,k+1));
B=subs(jb,[x y],z1(1:2,k+1));
w1(1:2,k+1)=z1(1:2,k+1)-inv(B)*A;
B1=subs(jb,[x y],w1(1:2,k+1));
C=subs(jb,[x y],(2*z1(1:2,k+1)+w1(1:2,k+1))/3);
z1(1:2,k+2)=z1(1:2,k+1)-4*inv(3*C+B1)*A;
k=k+1;
end
if k>maxiter
disp('too many iterations')
else
disp('Number of iteration k=')
disp(k)
iter=k;
disp('value of X(j) at iteration j where j=0,1,2,...,k')
disp(z1(1:2,1:k+1))
disp('stopping condition ||X(j)-X(j-1)|| for j=1,...,k')
for j=1:k
disp(norm(z1(1:2,j+1)-z1(1:2,j)))
end
disp('Solution='),disp(z1(1:2,k+1))
soln=subs(f,[x y],z1(1:2,k+1));
disp('check for solution')
disp('f[X(k)]=')
disp(soln)
end