Convergence of Gauss Newtons method

Applied Mathematics and Computation 170 (2005) 686–705 www.elsevier.com/locate/amc Convergence of Gauss–NewtonÕs method and uniqueness of the solution Jinhai Chen *, Weiguo Li School of Mathematics and Computational Sciences, University of Petroleum, Dongying 257061, Shandong Province, PR China Abstract In this paper, we study the convergence of Gauss–NewtonÕs method for nonlinear least squares problems Under the hypothesis that derivative satisfies some kinds of weak Lipschitz condition, we obtain the exact estimates of the radii of convergence ball of Gauss–NewtonÕs method and the uniqueness ball of the solution New results can be used to determinate approximation zero of Gauss–NewtonÕs method Ó 2005 Published by Elsevier Inc Keywords: Gauss–NewtonÕs method; Lipschitz condition; Convergence ball; Uniqueness ball Introduction We consider the nonlinear least squares problems: F ðxÞ :¼ f ðxÞT f ðxÞ * Corresponding author E-mail address: cjh_maths@yahoo.com.cn (J Chen) 0096-3003/$ - see front matter Ó 2005 Published by Elsevier Inc doi:10.1016/j.amc.2004.12.055 ð1:1Þ J Chen, W Li / Appl Math Comput 170 (2005) 686–705 687 where f(x): Rn ! Rm is Frechet differentiable, m P n And in all cases k Æ k refers to the 2-norm Gauss–NewtonÕs method, defined by h iÀ1 T T xnþ1 ¼ xn À f ðxn Þ f ðxn Þ f ðxn Þ f ðxn Þ; n P ð1:2Þ is one of the best-known methods for the problem (1.1) For the Gauss–NewtonÕs method, analyzes for local and rate of convergence properties are mostly restricted in quality [1,6,15,16], only considering the existence neighborhood of the convergence, but it cannot make us clearly see how big the radius of the convergence ball Let x* denote the solution of (1.1), Bðx; rÞ denote an open ball with radius r and center x, and let Bðx; rÞ denote its closure Traub and Wozniakowski [5] and Wang [12] independently gave an exact estimate for the convergence ball of NewtonÕs method Under the hypothesis that f (x) satisfies the some kind Lipschitz condition Z qðxÞ À1 LðuÞdu; 8x Bðxà ; rÞ; ð1:3Þ kf ðxÃ Þ ðf ðxÞ À f ðxs ÞÞk sqðxÞ where q(x) = kx À x*k, xs = x* + s(x À x*), and L is monotonic function, Wang [13,14] studied the convergence of the NewtonÕs method In this paper, we consider the convergence of the Gauss–NewtonÕs method Under more general conditions, we obtain the convergence domain of the Gauss–NewtonÕs method and the uniqueness domain of the solution Further, we can prove the optimality of the estimation of the radii New results can be used to determinate approximation zero of Gauss–NewtonÕs method Special and generalized Lipschitz condition The condition on the function f(x) kf ðxÞ À f ðxs Þk Lkx À xs k; 8x Bðxà ; rÞ; ð2:1Þ s where x = x* + s(x À x*),0 s 1, is usually called radius Lipschitz condition in the ball B(x*, r) with constant L Sometimes if it is only required to satisfy kf ðxÞ À f ðxà Þk Lkx À xà k; 8x Bðxà ; rÞ; ð2:2Þ we call it the center Lipschitz condition in the ball B(x*, r) with constant L Furthermore, the L in the Lipschitz condition need not be a constant, but a positive integrable function, If this is the case, then (2.1) or (2.2) is replaced by 688 J Chen, W Li / Appl Math Comput 170 (2005) 686–705 s kf ðxÞ À f ðx Þk Z qðxÞ LðuÞdu; 8x Bðxà ; rÞ; s 1; ð2:3Þ LðuÞdu; 8x Bðxà ; rÞ; ð2:4Þ sqðxÞ or kf ðxÞ À f ðxà Þk Z qðxÞ where q(x) = kx À x*k At the same time, the corresponding ÔLipschitz conditionÕ is referred to as having the L average Let Rm·n denote the set of all m · n matrix A, A  denote the Moore–Penrose inverse of matrix A, and if A has full rank (namely: rank (A) = min(m, n) = n) then A  = (ATA)À1AT Now, we give some Lemmas Lemma 2.1 (See [2,7]) Suppose that A,E Rm·n, B = A + E, kA kkEk < 1, rank(A) = rank(B), then kBy k kAy k ; À kAy kkEk ð2:5Þ and if rank(B) = rank(A) = min(m,n), we can obtain pffiffiffi y 2kA k kEk y y kB À A k À kAy kkEk ð2:6Þ Lemma 2.2 Suppose that A,E Rm·n(mPn), B = A + E, kEA k < 1, rank(A) = n, then rank(B) = n Proof In fact, B = A + E = (I + EA )A, from the condition kEA k < 1, we know I + EA  is invertible So rank(B) = rank(A) = n Lemma 2.3 Let Z t LðuÞuaÀ1 du; hðtÞ ¼ a t a P 1; t r; ð2:7Þ where L(u) is a positive integrable function and nondecreasing monotonically in [0,r] Then h(t) is nondecreasing with respect to t Proof In fact, by the monotonicity of L, a P 1, we obtain J Chen, W Li / Appl Math Comput 170 (2005) 686–705  689 Z  t2 hðt2 Þ À hðt1 Þ ¼ À a LðuÞuaÀ1 du t  Z t2   Z t1  1 ¼ a þ aÀ a LðuÞuaÀ1 du t t1 t2 t1  Z t2   Z t1  1 P Lðt1 Þ a þ aÀ a uaÀ1 du t t1 t1 t2  Z t2 Z t1  1 ¼ Lðt1 Þ a þ a uaÀ1 du ¼ t t0 t1 Rt for < t1 < t2 Thus hðtÞ ¼ t1a LðuÞuaÀ1 du is nondecreasing with respect to t ta2 Z t2 Lemma 2.4 Suppose that Z t gðtÞ ¼ LðuÞðt À uÞdu; t ð2:8Þ where L(u) is a positive integrable function in [0,r] Then g(t) is increasing monotonically with respect to t Proof In fact, by the positivity of L, we have Z Z t2 t1 gðt2 Þ À gðt1 Þ ¼ LðuÞðt2 À uÞdu À LðuÞðt1 À uÞdu t2 t1  Z t2   Z t1  Z t2 1 ¼ LðuÞdu À þ À LðuÞudu t2 t1 t2 t1 t1   Z t1 Z t2 Z t2 1 P LðuÞdu À LðuÞdu À À LðuÞudu t2 t1 t1 t1   Z t1 1 ¼ À LðuÞudu > t1 t2 for < t1 < t2 Thus g(t) is increasing with respect to t h Convergence ball of Gauss–NewtonÕs method Theorem 3.1 Suppose x* satisfies (1.1), f has a continuous derivative in B(x*,r) f (x*) has full rank, and f satisfies the radius Lipschitz condition with L average 690 J Chen, W Li / Appl Math Comput 170 (2005) 686–705 kðf ðxÞ À f ðxs ÞÞk Z qðxÞ LðuÞdu; 8x Bðxà ; rÞ; s 1; ð3:1Þ sqðxÞ where xs = x* + s(x À x*), q(x) = k x À x*k, and L is nondecreasing Let r > satisfy pffiffiffi R r Rr b LðuÞudu 2cb LðuÞdu Rr Rr þ ð3:2Þ rð1 À b LðuÞduÞ rð1 À b LðuÞduÞ Then Gauss–NewtonÕs method is convergent for all x0 B(x*,r) and R qðx Þ b 0 LðuÞudu à   kxn À xà k2 kxnþ1 À x k R qðx0 Þ qðx0 Þ À b LðuÞdu pffiffiffi R qðx0 Þ 2cb LðuÞdu kxn À xà k; þ R qðx0 Þ qðx0 Þð1 À b LðuÞduÞ ð3:3Þ where b ¼ k½f ðxà ÞT f ðxà ފÀ1 f ðxà ÞT k c ¼ kf ðxà Þk; ð3:4Þ and pffiffiffi R qðx0 Þ R qðx Þ b 0 LðuÞudu 2cb LðuÞdu   q¼ þ R qðx Þ R qðx0 Þ qðx0 Þð1 À b 0 LðuÞduÞ qðx0 Þ À b LðuÞdu ð3:5Þ is less than Moreover, if c = 0, then kxn À xà k q2 n À1 kx0 À xà k; n ¼ 1; 2; ð3:6Þ Proof Arbitrarily choosing x0 B(x*, r), where r satisfies (3.2), then q determined by (3.5) is less than In fact, by the monotonicity of L and Lemma 2.3, we have pffiffiffi R qðx0 Þ R qðx Þ b 0 LðuÞudu 2cb LðuÞdu   qðx0 Þ þ   qðx0 Þ q¼ R qðx0 Þ R qðx Þ 2 qðx0 Þ À b LðuÞdu qðx0 Þ À b 0 LðuÞdu pffiffiffi R r Rr b LðuÞudu 2cb LðuÞdu Á qðx0 Þ þ Rr Rr < 2À qðx0 Þ r ð1 À b LðuÞduÞ r À b LðuÞdu kx0 À xà k satisfy r Z r LðuÞðr À uÞdu ð7:7Þ Then Eq (1.1) has a unique solution x* in B(x*,r) Proof Suppose x0 B(x*,r), x05x* is also a solution of (1.1) Then we have h iÀ1 T T f ðxÃ Þ f ðxÃ Þ f ðx0 Þ f ðx0 Þ ¼ 0: ð7:8Þ Hence x0 À xà ¼ x0 À xà À ½f ðxà ÞT f ðxà ފÀ1 f ðx0 ÞT f ðx0 Þ ¼ ½f ðxà ÞT f ðxà ފÀ1 f ðxà ÞT ½f ðxà Þðx0 À xÃ Þ À f ðx0 Þ þ f ðxà ފ Z ½f ðxÃ Þ ¼ ½f ðxà ÞT f ðxà ފÀ1 f ðxà ÞT À f ðxà þ sðx0 À xà Þފðx0 À xà Þds; where s By the condition (7.6), we obtain Z à kx0 À x k k½f ðxà ÞT f ðxà ފÀ1 f ðxà ÞT ½f ðxÃ Þ À f ðxà þ sðx0 À xà Þފkkx0 À xà kds Z Z sqðx0 Þ Z qðx0 Þ LðuÞduqðx0 Þds ¼ LðuÞðqðx0 Þ À uÞdu 0 Rt From L(u) > and Lemma (2.4), we have LðuÞðt À uÞdu is increasing monotonically with respect to t Hence, by (7.7) we obtain Z qðx0 Þ Ã LðuÞðqðx0 Þ À uÞdu kx0 À x k t qðx0 Þ qðx0 Þ qðx0 Þ < r Z Z qðx0 Þ LðuÞðqðx0 Þ À uÞdu r LðuÞðr À uÞdu kx0 À xà k This is in contradiction with assumption Thus, it follows that x0 = x* Analogous to Section 5, we can obtain the following results h 700 J Chen, W Li / Appl Math Comput 170 (2005) 686–705 Theorem 7.3 Suppose that the equality sign holds in the inequality (7.3) in the Theorem 7.1 Then the given value r of the convergence ball is the best possible Furthermore, r only depends on L, but is independent of f Proof We notice that when r is determined by equality Rr LðuÞudu ¼1 ð7:9Þ r there exits f satisfying (7.1) and (7.2) in B(x*,r) and x0 on the boundary of the closed ball such that Gauss–NewtonÕs method fails In fact, the following is an example on the scaled case: ( À 32 x þ L4 x2 ; x r; f ðxÞ ¼ ð7:10Þ À 32 x À L4 x2 ; Àr x < 0; where L is a positive constant and x0 = r, xn = (À1)nr Theorem 7.4 Suppose that the equality sign holds in the inequality (7.7) in the Theorem (7.2) Then the given value r of the convergence ball is the best possible Furthermore, r only depends on L, but is independent of f Proof We notice that when r is determined by equality Z r LðuÞðr À uÞdu ¼ r ð7:11Þ there exists f satisfying (7.6) in B(x*, r) and x on the boundary of the closed T ball such that F ðxÃ Þ ¼ 12 f ðxÞ f ðxÞ An example of this is (5.2) in which b = and x = x* + r h Combining Theorems (7.1) and (7.2) with Theorems (7.3) and (7.4), and taking L as a constant, the following Corollaries are obtained directly Corollary 7.5 Suppose x* satisfies (1.1), f(x*) = 0, f has a continuous derivative in B(x*, r) f (x*) has full rank, and f satisfies (7.2) and y s kf ðxÞ ðf ðxÞ À f ðxÞ ÞÞk ð1 À sÞLkx À xà k; s   T 8x Bðxà ; rÞ; ð7:12Þ T where x = x* + s(x À x*), f (x) = [f (x) f (x)] , L is positive number and r = 2/L Then Gauss–NewtonÕs method (1.2), is convergent for all x0 B(x*, r) and for q¼ Lkx0 À xà k the inequality (7.4) holds Moreover, the given r is the best possible ð7:13Þ J Chen, W Li / Appl Math Comput 170 (2005) 686–705 701 Corollary 7.6 Suppose x* satisfies (1.1), f(x*) = 0, f has a continuous derivative 0 in B(x*,r) f (x*) has full rank, and f satisfies the center Lipschitz condition: y kf ðxÃ Þ ðf ðxÞ À f ðxà ÞÞk Lkx À xà k; 0   T À1 8x Bðxà ; rÞ; ð7:14Þ T where f (x*) = [f (x*) f (x*)] f (x*) , L is positive number and r = 2/L Then Eq (1.1) has a unique solution x* in B(x*,r) Moreover, the given r is the best possible and is independent of the f Taking LðuÞ ¼ 2c ð1 À cuÞ ð7:15Þ ; we obtain the following corollaries Corollary 7.7 Suppose x* satisfies (1.1), f(x*) = 0, has a continuous derivative in B(x*, r), f (x*) has full rank, and f satisfies (7.2) and kf ðxÞy ðf ðxÞ À f ðxs ÞÞk 1 À ; 2 ð1 À ckx À xà kÞ ð1 À sckx À xà kÞ 8x Bðxà ; rÞ; ð7:16Þ where xs = x*p+ffiffiffi s(x À x*), f (x)  = [f (x)T f (x)]À1f (x)T, c is positive number and r ¼ ð3 À 5Þ=ð2cÞ Then Gauss–NewtonÕs method (1.2), is convergent for all x0 B(x*, r) and for ckx0 À xà k q¼ ð7:17Þ ð1 À ckx0 À xà kÞ the inequality (7.4) holds Moreover, the given r is the best possible and is independent of the f Corollary 7.8 Suppose x* satisfies (1.1), f(x*) = 0, f has a continuous derivative in B(x*,r), f (x*) has full rank, and f satisfies the center Lipschitz condition: y kf ðxÃ Þ ðf ðxÞ À f ðxà ÞÞk ð1 À ckx À xà kÞ À 1; 8x Bðxà ; rÞ ð7:18Þ where f (x*) =[f (x*)T f (x*)]À1f (x*)T, c is positive number and r = 1/2c Then Eq (1.1) has a unique solution x* in B(x*, r) Moreover, the given r is the best possible and is independent of the f For the conditions (7.6), (7.14) and (7.18), if f(x): Rn ! Rn (namely: m = n), taking f (x*)  = f (x*)À1, Theorems 7.2 and 7.4 with Corollaries (7.6) and (7.8) are obtained by Wang ([14], Theorems (4.1) and (5.2) with Corollaries (6.2) and (6.4), so Theorems (7.2) and (7.4) with Corollaries (7.6) and (7.8) are more general 702 J Chen, W Li / Appl Math Comput 170 (2005) 686–705 Remark In this section, we used the f(x*) = and the properties of pseudoinverse of f (x) in an essential way We expect analogous results to hold for the situation of f(x*)50 under weaker Lipschitz conditions of f (x) [3,4,11,13,14], but cannot prove those Using Theorems (7.1) and (7.2), we also derive some new properties in essence about the convergence of NewtonÕs method and the uniqueness of the solutions of the equation In the following example, let c be a positive number Example Taking À3 LðuÞ ¼ 2ccð1 À cuÞ ; ð7:19Þ we obtain that if the right-hand side in (7.16) is replaced by c ð1 À ckx0 À xà kÞ À c ð1 À sckx0 À xà kÞ ; ð7:20Þ we have pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi þ c À 4c þ c2 r¼ 2c ð7:21Þ and q¼ cckx0 À xà k ð1 À ckx0 À xà kÞ ð7:22Þ If the right-hand side in (7.18) is replaced by c ð1 À ckx0 À xà kÞ À c; ð7:23Þ then r¼ ðc þ 1Þc ð7:24Þ Applications to determination of an approximation zero To study the property of quadratic convergence of NewtonÕs method and computational complexity of zeros, Smale [8–10] proposed the definitions of approximation zeros of NewtonÕs method With the SmaleÕs studies, we can propose a new definition of the approximation zeros for the Gauss–NewtonÕs method This definition is as follows J Chen, W Li / Appl Math Comput 170 (2005) 686–705 703 Definition 8.1 If x02Rn such that Gauss–NewtonÕs method (1.2) for f(x):Rn ! Rm is well defined and (3.6) is satisfied with q ¼ 12, then x0 is called an approximation zero of the adjoint zeros x* of f By Example of Section 6, solving d from the equation q¼ bd À 2ðb þ 1Þd þ ðb þ 1Þd2 ð8:1Þ yields d¼ ð2q þ 1Þðb þ 1Þ À À qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi bðð2q þ 1Þ2 ðb þ 1Þ À 1Þ 2qðb þ 1Þ ð8:2Þ : Thus we have Theorem 8.2 Suppose f(x*) = 0, f has a continuous derivative in B(x*, d/c) f(x*, d/c) f (x*) has full rank, and f satisfies the Lipschitz condition: kf ðxÞ À f ðxs Þk ð1 À ckx0 À xà kÞ2 À ð1 À sckx0 À xà kÞ2 ; 8x Bðxà ; d=cÞ; s 1; ð8:3Þ where c and q are positive numbers with < q < 1, d is determined by (8.2) If x0 satisfies ckx0 À xà k d; ð8:4Þ then Gauss–NewtonÕs method (1.2) is well defined and (3.6) is satisfied In particular, if pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2b þ À bð4b þ ; ð8:5Þ ckx0 À xà k bþ1 x0 is an approximation zero of the adjoint x* When f is analytic in B(x*,d/c), condition (8.3) is satisfied by f such that    ðnÞ Ã   f ðx Þ cnÀ1 ; n P ð8:6Þ n!  In fact, using (8.6) we have X nÀ2 kf 00 ðxÞk nðn À 1ÞcnÀ1 kx À xà k ¼ n¼2 2c ð1 À ckx À xà kÞ ð8:7Þ Hence (8.3) holds, and we have Corollary 8.3 Suppose f(x*) = 0, f has a continuous derivative in B(x*, d/c) f (x*) has full rank, and for some q (0,1), d is determined by (8.2), and there 704 J Chen, W Li / Appl Math Comput 170 (2005) 686–705 holds (8.6) If x0 satisfies ckx0 À x*k < d, then Gauss–NewtonÕs method (1.2) is well defined and (3.6) is satisfied In particular, if (8.5) is satisfied, x0 is an approximation zero of the adjoint x* Example of Section also give other results which can be applied to study the computational complexity By Example of Section 7, solving d from the equation q¼ cd ð8:8Þ ð1 À cdÞ2 yields pffiffiffiffiffiffiffiffiffiffiffiffiffiffi þ 2q À þ 4q d¼ 2cq ð8:9Þ Thus we have Theorem 8.4 Suppose f(x*) = 0, f has a continuous derivative in B(x*,d/c) f (x*) has full rank, and f satisfies the Lipschitz condition: kf ðxÞy ðf ðxÞ À f ðxs ÞÞk c ð1 À ckx0 À xà kÞ2 c À ; ð1 À sckx0 À xà kÞ 8x Bðxà ; d=cÞ; ð8:10Þ and (7.2), where xs = x* + s(x À x*), f (x)  = [f (x)T f (x)]À1 f (x)T, c, c and q are positive numbers with < q < 1, d is determined by (8.9) If x0 satisfies ckx0 À xà k < d; ð8:11Þ then Gauss–NewtonÕs method (1.2) is well defined and (7.4) is satisfied, In particular, if pffiffiffi 2À à ckx0 À x k < ; ð8:12Þ c x0 is an approximation zero of the adjoint x* Remark In fact, the results in this paper can be generalized in real or complex infinite dimensional Hilbert space, we will continue to further study for the convergence of Gauss–NewtonÕs method in real or complex infinite dimensional Hilbert space J Chen, W Li / Appl Math Comput 170 (2005) 686–705 705 Acknowledgement Supported by the Financially- Aiding Program for the Backbone Teachers of Ministry of Education of China and Natural Science Foundation of University of Petroleum References [1] T Yamamoto, Historical developments in convergence analysis for NewtonÕs and Newton-like methods, J Comput Appl Math 124 (2000) 1–23 [2] G.W Stewart, On the continuity of the generalized inverse, SIAM J Appl Math 17 (1960) 33–45 [3] I.K Argyros, The Newton-Kantorovich method under mild differentiability conditions and the Ptak estimates, Monatsch Math 101 (1990) 175–193 [4] I.k Argyros, On NewtonÕs method under mild differentiability conditions and applications, Appl Math Comput 102 (1999) 177–183 [5] J.F Traub, H Wozniakowski, Covergence and complexity of Newton iteration, J Assoc Comput Math 29 (1979) 250–258 [6] J.M Ortega, W.C Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, New York, 1970 [7] P.A Wedin, Perturbation theory for pseudo-inverse, BIT 13 (1973) 217–232 [8] S Smale, The fundamental theorem of algebra and complexity theory, Bull Amer Math Soc (1981) 1–36 [9] S Smale, NewtonÕs method estimates from data at one point, in: R Ewing, K Gross, C Martin (Eds.), The Merging of Disciplines: New Directions in Pure, Applied and Computational Mathematics, Springer, New York, 1986, pp 185–196 [10] S Smale, Complexity theroy and numerical analysis, Acta Numer (1997) 523–551 [11] T.J Ypina, Affine invariant convergence results for NewtonÕs method, BIT 22 (1982) 108–118 [12] X Wang, The covergence on NewtonÕs method, Kexue Tongbao (A Special Issue of Mathematics, Physics and Chemistry) 25 (1980) 36–37 [13] X Wang, Covergence of NewtonÕs method and inverse function in Banach space, Math Comput 68 (1999) 169–186 [14] X Wang, Covergence of NewtonÕs method and uniqueness of the solution of equations in Banach space, IMA J Numer Anal 20 (2000) 123–134 [15] W.M Hau¨ßler, A Kantorrovich-type convergence analysis for the Guass-Newton method, Numer Math 48 (1986) 119–125 [16] M.Z Nashed, X Chen, Convergence of Newton-like methods for singular operator equations using outer inverses, Numer Math 66 (1993) 235–257