Nonlinear equations and their solution by iteration

5 Nonlinear Equations and Their Solution by Iteration Nonlinear functional analysis is the study of operators lacking the property of linearity. In this chapter, we consider nonlinear operator equations and their numerical solution. We begin the consideration of operator equations which take the form u = T (u),u∈ K. (5.0.1) Here, V is a Banach space, K is a subset of V ,andT : K → V .The solutions of this equation are called fixed points of the operator T ,asthey are left unchanged by T. The most important method for analyzing the solvability theory for such equations is the Banach fixed-point theorem.We present the Banach fixed-point theorem in Section 5.1 and then discuss its application to the study of various iterative methods in numerical analysis. We then consider an extension of the well-known Newton method to the more general setting of Banach spaces. For this purpose, we introduce the differential calculus for nonlinear operators on normed spaces. We conclude the chapter with a brief introduction to another means of studying (5.0.1), using the concept of the rotation of a completely continu- ous vector field. We also generalize to function spaces the conjugate gradient iteration method for solving linear equations. There are many generaliza- tions of the ideas of this chapter, and we intend this material as only a brief introduction. 202 5. Nonlinear Equations and Their Solution by Iteration 5.1 The Banach fixed-point theorem Let V be a Banach space with the norm · V ,andletK be a subset of V . Consider an operator T : K → V defined on K. We are interested in the existence of a solution of the operator equation (5.0.1) and the possibility of approximating the solution u by the following iterative method. Pick an initial guess u 0 ∈ K, and define a sequence {u n } by the iteration formula u n+1 = T (u n ),n=0, 1, (5.1.1) To have this make sense, we identify another requirement that must be imposed upon T : T (v) ∈ K ∀v ∈ K. (5.1.2) The problem of solving an equation f(u) = 0 (5.1.3) for some operator f : K ⊂ V → V can be reduced to an equivalent fixed- point problem of the form (5.0.1) by setting T (v)=v − c 0 f(v)forsome constant scalar c 0 = 0, or more generally, T (v)=v − F (f (v)) with an operator F : V → V satisfying F (w)=0 ⇐⇒ w =0. Thus any result on the fixed-point problem (5.0.1) can be translated into a result for an equation (5.1.3). In addition, the iterative method (5.1.1) then provides a possible approximation procedure for solving the equation (5.1.3). In the following Section 5.2, we look at such applications for solving equations in a variety of settings. For the iterative method to work, we must assume something more than (5.1.2). To build some insight as to what further assumptions are needed on the operator T, consider the following simple example. Example 5.1.1 Take V to be the real line R,andT an affine operator, Tx = ax+ b, x ∈ R for some constants a and b. Now define the iterative method induced by the operator T.Letx 0 ∈ R, and for n =0, 1, , define x n+1 = ax n + b. It is easy to see that x n = x 0 + nb if a =1, and x n = a n x 0 + 1 − a n 1 − a b if a =1. 5.1 The Banach fixed-point theorem 203 Thus in the non-trivial case a = 1, the iterative method is convergent if and only if |a| < 1. Notice that the number |a| occurs in the property |Tx− Ty|≤|a||x − y|∀x, y ∈ R.  Definition 5.1.2 We say an operator T : K ⊂ V → V is contractive with contractivity constant α ∈ [0, 1) if T (u) − T (v) V ≤ α u − v V ∀u, v ∈ K. The operator T is called non-expansive if T (u) − T (v) V ≤u − v V ∀u, v ∈ K, and Lipschitz continuous if there exists a constant L ≥ 0 such that T (u) − T (v) V ≤ L u − v V ∀u, v ∈ K. We see the following implications: contractivity =⇒ non-expansiveness =⇒ Lipschitz continuity =⇒ continuity. Theorem 5.1.3 (Banach fixed-point theorem) Assume that K is a nonempty closed set in a Banach space V , and further, that T : K → K is a contractive mapping with contractivity constant α, 0 ≤ α<1. Then the following results hold. (1) Existence and uniqueness: There exists a unique u ∈ K such that u = T (u). (2) Convergence and error estimates of the iteration: For any u 0 ∈ K,the sequence {u n }⊂K defined by u n+1 = T (u n ), n =0, 1, , converges to u: u n − u V → 0 as n →∞. For the error, the following bounds are valid. u n − u V ≤ α n 1 − α u 0 − u 1  V , (5.1.4) u n − u V ≤ α 1 − α u n−1 − u n  V , (5.1.5) u n − u V ≤ α u n−1 − u V . (5.1.6) 204 5. Nonlinear Equations and Their Solution by Iteration Proof. Since T : K → K, the sequence {u n } is well-defined. Let us first prove that {u n } is a Cauchy sequence. Using the contractivity of the map- ping T ,wehave u n+1 − u n  V ≤ α u n − u n−1  V ≤···≤α n u 1 − u 0  V . Then for any m ≥ n ≥ 1, u m − u n  V ≤ m−n−1  j=0 u n+j+1 − u n+j  V ≤ m−n−1  j=0 α n+j u 1 − u 0  V ≤ α n 1 − α u 1 − u 0  V . (5.1.7) Since α ∈ [0, 1), u m − u n  V → 0asm, n →∞.Thus{u n } is a Cauchy sequence; and since K is a closed set in the Banach space V , {u n } has a limit u ∈ K. We take the limit n →∞in u n+1 = T(u n ) to see that u = T (u) by the continuity of T , i.e., u is a fixed-point of T . Suppose u 1 ,u 2 ∈ K are both fixed-points of T .Thenfromu 1 = T (u 1 ) and u 2 = T (u 2 ), we obtain u 1 − u 2 = T (u 1 ) − T (u 2 ). Hence u 1 − u 2  V = T (u 1 ) − T (u 2 ) V ≤ α u 1 − u 2  V which implies u 1 − u 2  V =0sinceα ∈ [0, 1). So a fixed-point of a con- tractive mapping is unique. Now we prove the error estimates. Letting m →∞in (5.1.7), we get the estimate (5.1.4). From u n − u V = T (u n−1 ) − T (u) V ≤ α u n−1 − u V we obtain the estimate (5.1.6). This estimate together with u n−1 − u V ≤u n−1 − u n  V + u n − u V implies the estimate (5.1.5).  This theorem is called by a variety of names in the literature, with the contractive mapping theorem another popular choice. It is also called Picard iteration in settings related to differential equations. As an application of the Banach fixed-point theorem, we consider the unique solvability of a nonlinear equation in a Hilbert space. 5.1 The Banach fixed-point theorem 205 Theorem 5.1.4 Let V be a Hilbert space. Assume T : V → V is strongly monotone and Lipschitz continuous, i.e., there exist two constants c 1 ,c 2 > 0 such that for any v 1 ,v 2 ∈ V , (T (v 1 ) − T (v 2 ),v 1 − v 2 ) ≥ c 1 v 1 − v 2  2 , (5.1.8) T (v 1 ) − T (v 2 )≤c 2 v 1 − v 2 . (5.1.9) Then for any b ∈ V , there is a unique u ∈ V such that T (u)=b. (5.1.10) Moreover, the solution u depends Lipschitz continuously on b:IfT (u 1 )=b 1 and T (u 2 )=b 2 , then u 1 − u 2 ≤ 1 c 1 b 1 − b 2 . (5.1.11) Proof. The equation T (u)=b is equivalent to u = u − θ (T(u) −b) for any θ = 0. Define an operator T θ : V → V by the formula T θ (v)=v −θ (T (v) −b) . Let us show that for θ>0 sufficiently small, the operator T θ is contractive. Write T θ (v 1 ) − T θ (v 2 )=(v 1 − v 2 ) − θ (T θ (v 1 ) − T θ (v 2 )) . Then, T θ (v 1 ) − T θ (v 2 ) 2 = v 1 − v 2  2 − 2θ (T θ (v 1 ) − T θ (v 2 ),v 1 − v 2 ) + θ 2 T θ (v 1 ) − T θ (v 2 ) 2 . Use the assumptions (5.1.8) and (5.1.9) to obtain T θ (v 1 ) − T θ (v 2 ) 2 ≤  1 − 2c 2 θ + c 2 1 θ 2  v 1 − v 2  2 . For θ ∈ (0, 2c 2 /c 2 1 ), 1 − 2c 2 θ + c 2 1 θ 2 < 1 and T θ is a contraction. Then by the Banach fixed-point theorem, T θ has a unique fixed-point u ∈ V . Hence, the equation (5.1.10) has a unique solution. Now we prove the Lipschitz continuity of the solution with respect to the right hand side. From T (u 1 )=b 1 and T (u 2 )=b 2 , we obtain T (u 1 ) − T (u 2 )=b 1 − b 2 . 206 5. Nonlinear Equations and Their Solution by Iteration Then (T (u 1 ) − T (u 2 ),u 1 − u 2 )=(b 1 − b 2 ,u 1 − u 2 ) . Apply the assumption (5.1.8) and the Cauchy-Schwarz inequality, c 1 u 1 − u 2  2 ≤b 1 − b 2 u 1 − u 2 , which implies (5.1.11).  The proof technique of Theorem 5.1.4 will be employed in Chapter 11 in proving existence and uniqueness of solutions to some variational inequal- ities. The condition (5.1.8) relates to the degree of monotonicity of T (v) as v varies. For a real-valued function T (v) of a single real variable v,the constant c 1 can be chosen as the infimum of T  (v) over the domain of T , assuming this infimum is positive. Exercise 5.1.1 In the Banach fixed-point theorem, we assume (1) V is a com- plete space, (2) K is a nonempty closed set in V ,(3)T : K → K, and (4) T is contractive. Find examples to show that each of these assumptions is necessary for the result of the theorem; in particular, the result fails to hold if all the other assumptions are kept except that T is only assumed to satisfy the inequality T (u) − T (v) V < u −v V ∀u, v ∈ V, u = v. Exercise 5.1.2 Assume K is a nonempty closed set in a Banach space V , and that T : K → K is continuous. Suppose T m is a contraction for some positive integer m. Prove that T has a unique fixed-point in K. Moreover, prove that the iteration method u n+1 = T (u n ),n=0, 1, 2, converges. Exercise 5.1.3 Let T be a contractive mapping on V to V . By Theorem 5.1.3, for every y ∈ V , the equation v = T (v)+y has a unique solution, call it u(y). Show that u(y) is a continuous function of y. Exercise 5.1.4 Let V be a Banach space, and let T be a contractive mapping on K ⊂ V to K,withK = {v ∈ V |v≤r} for some r>0. Assume T (0) = 0. Show that v = T (v)+y has a unique solution in K for all sufficiently small choices of y ∈ V . 5.2 Applications to iterative methods The Banach fixed-point theorem presented in the preceding section con- tains most of the desirable properties of a numerical method. Under the stated conditions, the approximation sequence is well-defined, and it is con- vergent to the unique solution of the problem. Furthermore, we know the 5.2 Applications to iterative methods 207 convergence rate is linear (see (5.1.6)), we have an a priori error estimate (5.1.4) which can be used to determine the number of iterations needed to achieve a prescribed solution accuracy before actual computations take place,andwealsohaveana posteriori error estimate (5.1.5) which gives a computable error bound once some numerical solutions are calculated. In this section, we apply the Banach fixed-point theorem to the analysis of numerical approximations of several problems. 5.2.1 Nonlinear equations Given a real-valued function of a real variable, f : R → R , we are interested in computing its real roots, i.e., we are interested in solving the equation f(x)=0,x∈ R . (5.2.1) There are a variety of ways to reformulate this equation as an equivalent fixed-point problem of the form x = T (x),x∈ R . (5.2.2) Some examples are T (x) ≡ x −f(x) or more generally T (x) ≡ x − c 0 f(x) for some constant c 0 = 0. A more sophisticated example is T(x)=x − f(x)/f  (x), in which case the iterative method becomes the celebrated New- ton’s method. For this last example, we generally use Newton’s method only for finding simple roots of f(x), which means we need to assume f  (x) =0 when f(x) = 0. We return to a study of the Newton’s method later in Section 5.4. Specializing the Banach fixed-point theorem to the problem (5.2.2), we have the following well-known result. Theorem 5.2.1 Let −∞ <a<b<∞ and T :[a, b] → [a, b] be a con- tractive function with contractivity constant α ∈ [0, 1). Then the following results hold. (1) Existence and uniqueness: There exists a unique solution x ∈ [a, b] to the equation x = T (x). (2) Convergence and error estimates of the iteration: For any x 0 ∈ [a, b], the sequence {x n }⊂[a, b] defined by x n+1 = T (x n ), n =0, 1, , converges to x: x n → x as n →∞. For the error, there hold the bounds |x n − x|≤ α n 1 − α |x 0 − x 1 |, |x n − x|≤ α 1 − α |x n−1 − x n |, |x n − x|≤α |x n−1 − x|. 208 5. Nonlinear Equations and Their Solution by Iteration The contractiveness of the function T is guaranteed from the assumption that sup a≤x≤b |T  (x)| < 1. Indeed, using the Mean Value Theorem, we then see that T is contractive with the contractivity constant α =sup a≤x≤b |T  (x)|. 5.2.2 Linear systems Let A ∈ R m×m be an m by m matrix, and let us consider the linear system Ax = b, x ∈ R m (5.2.3) where b ∈ R m is given. It is well-known that (5.2.3) has a unique solution x for any given b if and only if A is non-singular, det(A) =0. Let us reformulate (5.2.3) as a fixed point problem and introduce the corresponding iteration methods. A common practice for devising iterative methods of solving (5.2.3) is by using a matrix splitting A = N − M with N chosen in such a way that the system Nx = k is easily and uniquely solvable for any right side k. Then the linear system (5.2.3) is rewritten as Nx = Mx + b. This leads naturally to an iterative method for solving (5.2.3): Nx n = Mx n−1 + b,n=1, 2, (5.2.4) with x 0 a given initial guess of the solution x. To more easily analyze the iteration, we rewrite these last two equations as x = N −1 Mx + N −1 b, x n = N −1 Mx n−1 + N −1 b. The matrix N −1 M is called the iteration matrix. Subtracting the two equa- tions, we obtain the error equation x − x n = N −1 M (x − x n−1 ) . Inductively, x − x n =  N −1 M  n (x − x 0 ) ,n=0, 1, 2, (5.2.5) 5.2 Applications to iterative methods 209 We see that the iterative method converges if N −1 M < 1, where ·is some matrix operator norm, i.e., it is a norm induced by some vector norm ·: A =sup x=0 Ax x . (5.2.6) Recall that for a square matrix A, a necessary and sufficient condition for A n → 0asn →∞is r σ (A) < 1. This follows from the Jordan canonical form of a square matrix. Here, r σ (A)isthespectral radius of A: r σ (A)=max i |λ i (A)|, with {λ i (A)} the set of all the eigenvalues of A. Note the from the error relation (5.2.5), we have convergence x n → x as n →∞for any initial guess x 0 , if and only if (N −1 M) n → 0asn →∞. Therefore, for the itera- tive method (5.2.4), a necessary and sufficient condition for convergence is r σ (N −1 M) < 1. The spectral radius of a matrix is an intrinsic quantity of the matrix, whereas a matrix norm is not. It is thus not surprising that a necessary and sufficient condition for convergence of the iterative method is described in terms of the spectral radius of the iteration matrix. We would also expect something of this kind since in finite dimensional spaces, convergence of {x n } in one norm is equivalent to convergence in every other norm (see Theorem 1.2.13 from Chapter 1). We have the following relations between the spectral radius and norms of a matrix A ∈ R m×m . 1. r σ (A) ≤A for any matrix operator norm ·. This result follows immediately from the definition of r σ (A), the defining relation of an eigenvalue, and the fact that the matrix norm ·is generated by a vector norm. 2. For any ε>0, there exists a matrix operator norm · A,ε such that r σ (A) ≤A A,ε ≤ r σ (A)+ε. For a proof, see [107, p. 12]. Thus, r σ (A)=inf{A|·is a matrix operator norm}. 3. r σ (A) = lim n→∞ A n  1/n for any matrix norm ·. Note that here the norm can be any matrix norm, not necessarily the ones generated by vector norms as in (5.2.6). This can be proven by using the Jordan canonical form; see [13, p. 490]. For applications to the solution of discretizations of Laplace’s equation and some other elliptic partial differential equations, it is useful to write A = D + L + U, 210 5. Nonlinear Equations and Their Solution by Iteration where D is the diagonal part of A, L and U are the strict lower and upper triangular parts. If we take N = D, then (5.2.4) reduces to Dx n = b − (L + U) x n−1 , which is the vector representation of the Jacobi method; the corresponding componentwise representation is x n,i = 1 a ii ⎛ ⎝ b i −  j=i a ij x n−1,j ⎞ ⎠ , 1 ≤ i ≤ m. If we take N = D + L, then we obtain the Gauss-Seidel method (D + L)x n = b − U x n−1 or equivalently, x n,i = 1 a ii ⎛ ⎝ b i − i−1  j=1 a ij x n,j − m  j=i+1 a ij x n−1,j ⎞ ⎠ , 1 ≤ i ≤ m. A more sophisticated splitting is obtained by setting N = 1 ω D + L, M = 1 − ω ω D −U, where ω = 0 is an acceleration parameter. The corresponding iterative method with the (approximate) optimal choice of ω is called the SOR (successive overrelaxation) method. The componentwise representation of the SOR method is x n,i = x n−1,i + ω ⎛ ⎝ b i − i−1  j=1 a ij x n,j − m  j=i+1 a ij x n−1,j ⎞ ⎠ , 1 ≤ i ≤ m. In vector form, we write it in the somewhat more intuitive form z n = D −1 [b − Lx n − Ux n−1 ] , x n = ωz n +(1− ω) x n−1 . For linear systems arising in difference solutions of some model partial differential equation problems, there is a well-understood theory for the choice of an optimal value of ω; and with that optimal value, the iteration converges much more rapidly than does the original Gauss-Seidel method on which it is based. Additional discussion of the framework (5.2.4) for iteration methods is given in [13, Section 8.6]. [...]... Fr´chet derivative f (v) and calculate e it 230 5 Nonlinear Equations and Their Solution by Iteration Exercise 5.3.16 Let V be a real Hilbert space, a(·, ·) : V × V → R a symmetric, continuous bilinear form, and ∈ V Compute the Fr´chet derivative of the e functional 1 f (v) = a(v, v) − (v), v ∈ V 2 Exercise 5.3.17 (a) Find the derivative of the nonlinear operator given in the right hand side of (5.2.14)... u − v ∀ (t, u), (t, v) ∈ Qb , where L is a constant independent of t Let M = max (t,u)∈Qb f (t, u) 216 5 Nonlinear Equations and Their Solution by Iteration and c0 = min c, b M Then the initial value problem (5.2.18) has a unique continuously differentiable solution u(·) on [t0 − c0 , t0 + c0 ]; and the iterative method (5.2.20) converges for any initial value u0 for which z − u0 < b, max |t−t0 |≤c0... (u), f2 (u)) is differentiable at u0 , and B (u0 )h = b(f1 (u)h, f2 (u)) + b(f1 (u), f2 (u)h) h ∈ V 222 5 Nonlinear Equations and Their Solution by Iteration Proposition 5.3.7 (Chain rule) Let U , V and W be normed spaces Let f : K ⊂ U → V , g : L ⊂ V → W be given with f (K) ⊂ L Assume u0 is an interior point of K, f (u0 ) is an interior point of L If f (u0 ) and g (f (u0 )) exist as Fr´chet derivatives,... is continuously Fr´chet differentiable e in a neighborhood of (u0 , v0 ) if and only if fu (u, v) and fv (u, v) are continuous in a neighborhood of (u0 , v0 ) The above discussion can be extended straightforward to maps of several variables 226 5 Nonlinear Equations and Their Solution by Iteration 5.3.4 The Gˆteaux derivative and convex minimization a Let us first use the notion of Gˆteaux derivative... solvability theory for them We discuss here some commonly seen nonlinear integral equations of the second kind The integral equation b k(x, y, u(y)) dy + f (x), u(x) = µ a≤x≤b (5.2.10) a is called a Urysohn integral equation Here we assume that f ∈ C[a, b] and k ∈ C([a, b] × [a, b] × R) (5.2.11) 212 5 Nonlinear Equations and Their Solution by Iteration Moreover, we assume k satisfies a uniform Lipschitz... [F (u + t (u∗ − u)) − F (u)] dt (u∗ − u) 0 and by taking the norm, 1 T (u) − T (u∗ ) ≤ [F (u)]−1 F (u + t (u∗ − u)) − F (u) dt u∗ − u 0 1 ≤ [F (u)]−1 L t u∗ − u dt u∗ − u 0 Hence, T (u) − T (u∗ ) ≤ c0 L u − u∗ 2 2 (5.4.5) Choose δ < 2/(c0 L) with the property B(u∗ , δ) ⊂ N (u∗ ); and note that α ≡ c0 Lδ/2 < 1 232 5 Nonlinear Equations and Their Solution by Iteration Then (5.4.5) implies T (u) − u∗... convergent, provided u0 is chosen sufficiently close to u∗ 236 5 Nonlinear Equations and Their Solution by Iteration 5.5 Completely continuous vector fields There are other means of asserting the existence of a solution to an equation For example, if f ∈ C[a, b], and if f (a) f (b) < 0, then the intermediate value theorem asserts the existence of a solution in [a, b] to the equation f (x) = 0 We convert this... (5.3.10) by an argument similar to the one used in the proof of Theorem 5.3.17 for the part “(a) =⇒ (b)” Now assume u satisfies (5.3.10) Then since f is convex, f (v) ≥ f (u) + f (u), v − u ≥ f (u) 228 5 Nonlinear Equations and Their Solution by Iteration When K is a subspace, we can take v in (5.3.10) to be v + u for any v ∈ K to obtain f (u), v ≥ 0 ∀ v ∈ K Since K is a subspace, −v ∈ K and f (u),... − k(t, s, v(s))] ds a Then t |u(s) − v(s)| ds |T u(t) − T v(t)| ≤ M a and |T u(t) − T v(t)| ≤ M u − v ∞ (t − a) Since t T 2 u(t) − T 2 v(t) = [k(t, s, T u(s)) − k(t, s, T v(s))] ds, a we get t T 2 u(t) − T 2 v(t) ≤ M |T u(s) − T v(s)| ds a ≤ [M (t − a)]2 u−v 2! ∞ (5.2.17) 214 5 Nonlinear Equations and Their Solution by Iteration By a mathematical induction, we obtain |T m u(t) − T m v(t)| ≤ Thus T... = T [F (tu + (1 − t) w) (u − w)] 224 5 Nonlinear Equations and Their Solution by Iteration Applying the ordinary mean-value theorem, we have a θ ∈ [0, 1] for which F (u) − F (w) V = g(1) − g(0) = g (θ) = T [F (θu + (1 − θ) w) (u − w)] ≤ T F (θu + (1 − θ) w) (u − w) ≤ F (θu + (1 − θ) w) u−w U W The formula (5.3.7) follows immediately Corollary 5.3.12 Let U and V be normed spaces, K a connected open . differential equations, it is useful to write A = D + L + U, 210 5. Nonlinear Equations and Their Solution by Iteration where D is the diagonal part of A, L and U are the strict lower and upper triangular. 5 Nonlinear Equations and Their Solution by Iteration Nonlinear functional analysis is the study of operators lacking the property of linearity. In this chapter, we consider nonlinear. =max (t,u)∈Q b f(t, u) 216 5. Nonlinear Equations and Their Solution by Iteration and c 0 =min  c, b M  . Then the initial value problem (5.2.18) has a unique continuously differen- tiable solution u(·) on