Tài liệu Digital Signal Processing Handbook P21 docx

Ali H. Sayed, et. Al. “Recursive Least-Squares Adaptive Filters.” 2000 CRC Press LLC. <http://www.engnetbase.com>. RecursiveLeast-SquaresAdaptive Filters AliH.Sayed UniversityofCalifornia, LosAngeles ThomasKailath StanfordUniversity 21.1ArrayAlgorithms ElementaryCircularRotations • ElementaryHyperbolicRo- tations • Square-Root-FreeandHouseholderTransformations • ANumericalExample 21.2TheLeast-SquaresProblem GeometricInterpretation • StatisticalInterpretation 21.3TheRegularizedLeast-SquaresProblem GeometricInterpretation • StatisticalInterpretation 21.4TheRecursiveLeast-SquaresProblem ReducingtotheRegularizedForm • TimeUpdates 21.5TheRLSAlgorithm EstimationErrorsandtheConversionFactor • Updateofthe MinimumCost 21.6RLSAlgorithmsinArrayForms Motivation • AVeryUsefulLemma • TheInverseQRAlgo- rithm • TheQRAlgorithm 21.7FastTransversalAlgorithms ThePrewindowedCase • Low-RankProperty • AFastArray Algorithm • TheFastTransversalFilter 21.8Order-RecursiveFilters JointProcessEstimation • TheBackwardPredictionErrorVec- tors • TheForwardPredictionErrorVectors • ANonunity ForgettingFactor • TheQRDLeast-SquaresLatticeFilter • TheFilteringorJointProcessArray 21.9ConcludingRemarks References Thecentralprobleminestimationistorecover,togoodaccuracy,asetofunobservableparameters fromcorrupteddata.Severaloptimizationcriteriahavebeenusedforestimationpurposesoverthe years,butthemostimportant,atleastinthesenseofhavinghadthemostapplications,arecriteria thatarebasedonquadraticcostfunctions.Themoststrikingamongtheseisthelinearleast-squares criterion,whichwasperhapsfirstdevelopedbyGauss(ca.1795)inhisworkoncelestialmechanics. Sincethen,ithasenjoyedwidespreadpopularityinmanydiverseareasasaresultofitsattractive computationalandstatisticalproperties.Amongtheseattractiveproperties,themostnotablearethe factsthatleast-squaressolutions: •canbeexplicitlyevaluatedinclosedforms; •canberecursivelyupdatedasmoreinputdataismadeavailable,and c  1999byCRCPressLLC • are maximum likelihood estimators in the presence of Gaussian measurement noise. The aim of this chapter is to provide an overview of adaptive filtering algorithms that result when the least-squares criterion is adopted. Over the last several years, a wide variety of algorithms in this class has been derived. They all basically fall into the following main groups (or variations thereof): recursive least-squares (RLS) algorithms and the corresponding fast versions (known as FTF and FAEST), QR and inverse QR algorithms, least-squares lattice (LSL), and QR decomposition-based least-squares lattice (QRD-LSL) algorithms. Table 21.1 lists these different variants and classifies them into order-recursive and fixed-order algorithms. The acronyms and terminology are not important at this stage and will be explained as the discussion proceeds. Also, the notation O(M) is used to indicate that each iteration of an algorithm requires of the order of M floating point operations (additions and multiplications). In this sense, some algorithms are fast (requiring only O(M)), while others are slow (requiring O(M 2 )). The value of M is the filter order that will be introduced in due time. TABLE 21.1 Most Common RLS Adaptive Schemes Adaptive Order Fixed Cost per Algorithm Recursive Order Iteration RLS x O(M 2 ) QR and Inverse QR x O(M 2 ) FTF, FAEST x O(M) LSL x O(M) QRD-LSL x O(M) It is practically impossible to list here all the relevant references and all the major contributors to the rich field of adaptive RLS filtering. The reader is referred to some of the textbooks listed at the end of this chapter for more comprehensive treatments and bibliographies. Here we wish to stress that, apart from introducing the reader to the fundamentals of RLS filtering, one of our goals in this exposition is to present the different versions of the RLS algorithm in computationally convenient so-called array forms. In these forms, an algorithm is described as a sequence of elementary operations on arrays of numbers. Usually, a prearray of numbers has to be triangularized by a rotation, or a sequence of elementary rotations, in order to yield a postarray of numbers. The quantities needed to form the next prearray can then be read off from the entries of the postarray, and the procedure can be repeated. The explicit forms of the rotation matrices are not needed in most cases. Such array descriptions are more truly algorithms in the sense that they operate on sets of numbers and provide other sets of numbers, with no explicit equations involved. The rotations themselves can be implemented in a variety of well-known ways: as a sequence of elementary circular or hyperbolic rotations, in square-root- and/or division-free forms, as Householder transformations, etc. These may differ in computational complexity, numerical behavior, and ease of hardware (VLSI) imple- mentation. But, if preferred, explicit expressions for the rotation matrices can also be written down, thus leading to explicit sets of equations in contrast to the array forms. For this reason, and although the different RLS algorithms that we consider here have already been derived in many different ways in earlier places in the literature, the derivation and presentation in this chapter are intended to provide an alternative unifying exposition that we hope will help a reader get a deeper appreciation of this class of adaptive algorithms. c  1999 by CRC Press LLC Notation We use small boldface letters to denote column vectors (e.g., w) and capital boldface letters to denote matrices (e.g., A). The symbol I n denotes the identity matrix of size n × n, while 0 denotes a zero column. The symbol T denotes transposition. This chapter deals with real-valued data. The case of complex-valued data is essentially identical and is treated in many of the references at the end of this chapter. Square-Root Factors A symmetric positive-definite matrix A is one that satisfies A = A T and x T Ax > 0 for all nonzero column vectors x. Any such matrix admits a factorization (also known as eigen- decomposition) of the form A = UU T , where U is an orthogonal matrix, namely a square matrix that satisfies UU T = U T U = I, and  is a diagonal matrix with real positive entries. In particular, note that AU = U, which shows that the columns of U are the right eigenvectors of A and the entries of  are the corresponding eigenvalues. Note also that we can write A = U 1/2 ( 1/2 ) T U T , where  1/2 is a diagonal matrix whose entries are(positive)square-roots ofthediagonal entries of . Since  1/2 isdiagonal, ( 1/2 ) T =  1/2 .Ifwe introduce the matrix notation A 1/2 = U 1/2 , then we can alternatively write A = (A 1/2 )(A 1/2 ) T . This can be regarded as a square-root factorization of the positive-definite matrix A. Here, the notation A 1/2 is used to denote one such square-root factor, namely the one constructed from the eigen-decomposition of A. Note, however, that square-root factors are not unique. For example, we may multiply the diagonal entries of  1/2 by ±1  s and obtain a new square-root factor for  and, consequently, a new square- root factor for A. Also, given any square-root factor A 1/2 , and any orthogonal matrix  (satisfying  T = I)we can define a new square-root factor for A as A 1/2  since (A 1/2 )(A 1/2 ) T = A 1/2 ( T )(A 1/2 ) T = A . Hence, square factors are highly nonunique. We shall employ the notation A 1/2 to denote any such square-root factor. They can be made unique, e.g., by insisting that the factors be symmetric or that they be triangular (with positive diagonal elements). In most applications, the triangular form is preferred. For convenience, we also write  A 1/2  T = A T/2 ,  A 1/2  −1 = A −1/2 ,  A −1/2  T = A −T/2 . Thus, note the expressions A = A 1/2 A T/2 and A −1 = A −T/2 A −1/2 . 21.1 Array Algorithms The array form is so important that it will be worthwhile to explain its generic form here. An array algorithm is described via rotation operations on a prearray of numbers, chosen to obtain a certain zero pattern in a postarray. Schematically, we write     xxxx xxxx xxxx xxxx      =     x 000 xx00 xxx0 xxxx     , where  is any rotation matrix that triangularizes the prearray. In general,  is required to be a J−orthogonal matrix in the sense that it should satisfy the normalization J T = J, where J c  1999 by CRC Press LLC is a given signature matrix with ±1  s on the diagonal and zeros elsewhere. The orthogonal case corresponds to J = I since then  T = I. A rotation  that transforms a prearray to triangular form can be achieved in a variety of ways: by using a sequence of elementary Givens and hyperbolic rotations, Householder transformations, or square-root-free versions of such rotations. Here we only explain the elementary forms. The other choices are discussed in some of the references at the end of this chapter. 21.1.1 Elementary Circular Rotations An elementary 2 × 2 orthogonal rotation  (also known as Givens or circular rotation) takes a row vector  ab  and rotates it to lie along the basis vector  10  . More precisely, it performs the transformation  ab   =  ±  |a| 2 +|b| 2 0  . (21.1) The quantity ±  |a| 2 +|b| 2 that appears on the right-hand side is consistent with the fact that the prearray,  ab  , and the postarray,  ±  |a| 2 +|b| 2 0  , must have equal Euclidean norms (since an orthogonal transformation preserves the Euclidean norm of a vector). An expression for  is given by  = 1  1 + ρ 2  1 −ρ ρ 1  ,ρ= b a ,a= 0. (21.2) In the trivial case a = 0 we simply choose  as the permutation matrix,  =  01 10  . The orthogonal rotation (21.2) can also be expressed in the alternative form:  =  c −s sc  , where the so-called cosine and sine parameters, c and s, respectively, are defined by c = 1  1 + ρ 2 ,s= ρ  1 + ρ 2 . The name circular rotation for  is justified by its effect on a vector; it rotates the vector along the circle of equation x 2 + y 2 =|a| 2 +|b| 2 , by an angle θ that is determined by the inverse of the above cosine and/or sine parameters, θ = tan −1 ρ,in order to align it with the basis vector  10  .The trivial case a = 0 corresponds to a 90 degrees rotation in an appropriate clockwise (if b ≥ 0)or counterclockwise (if b<0) direction. 21.1.2 Elementary Hyperbolic Rotations An elementary 2 × 2 hyperbolic rotation  takes a row vector  ab  and rotates it to lie either along the basis vector  10  (if |a| > |b|) or along the basis vector  01  (if |a| < |b|). More precisely, it performs either of the transformations  ab   =  ±  |a| 2 −|b| 2 0  if |a| > |b| , (21.3) c  1999 by CRC Press LLC  ab   =  0 ±  |b| 2 −|a| 2  if |a| < |b|. (21.4) The quantity  ±(|a| 2 −|b| 2 ) that appears on the right-hand side of the above expressions is con- sistent with the fact that the prearray,  ab  , and the postarrays must have equal hyperbolic “norms.” By the hyperbolic “norm” of a row vector x T we mean the indefinite quantity x T Jx, which can be positive or negative. Here, J =  10 0 −1  = (1 ⊕−1). An expression for a hyperbolic rotation  that achieves (21.3)or(21.4)isgivenby  = 1  1 − ρ 2  1 −ρ −ρ 1  , (21.5) where ρ =    b a when a = 0 and |a| > |b| a b when b = 0 and |b| > |a| The hyperbolic rotation (21.5) can also be expressed in the alternative form:  =  ch −sh −sh ch  , where the so-called hyperbolic cosine and sine parameters, ch and sh, respectively, are defined by ch = 1  1 − ρ 2 ,sh= ρ  1 − ρ 2 . The name hyperbolic rotation for  is again justified by its effect on a vector; it rotates the original vector along the hyperbola of equation x 2 − y 2 =|a| 2 −|b| 2 , by an angle θ determined by the inverse of the above hyperbolic cosine and/or sine parameters, θ = tanh −1 [ρ], in order to align it with the appropriate basis vector. Note also that the special case |a|=|b| corresponds to a row vector  ab  with zero hyperbolic norm since |a| 2 −|b| 2 = 0. It is then easy to see that there does not exist a hyperbolic rotation that will rotate the vector to lie along the direction of one basis vector or the other. 21.1.3 Square-Root-Free and Householder Transformations We remark that the above expressions for the circular and hyperbolic rotations involve square-root operations. In many situations, it may be desirable to avoid the computation of square-roots because it is usually expensive. For this and other reasons, square-root- and division-free versions of the above elementary rotations have been developed and constitute an attractive alternative. Therefore one could use orthogonal or J−orthogonal Householder reflections (for given J)to simultaneously annihilate several entries in a row, e.g., to transform  xxxx  directly to the form  x  000  . Combinations of rotations and reflections can also be used. We omit the details here but the idea is clear. There are many different ways in which a prearray of numbers can be rotated into a postarray of numbers. c  1999 by CRC Press LLC 21.1.4 A Numerical Example Assume we are given a 2 × 3 prearray A, A =  0.875 0.15 1.0 0.675 0.35 0.5  , (21.6) and wish to triangularize it via a sequence of elementary circular rotations, i.e., reduce A to the form A =  x 00 xx0  . (21.7) This can be obtained, among several different possibilities, as follows. We start by annihilating the (1, 3) entry of the prearray (21.6) by pivoting with its (1, 1) entry. According to expression (21.2), the orthogonal transformation  1 that achieves this result is given by  1 = 1  1 + ρ 2 1  1 −ρ 1 ρ 1 1  =  0.6585 −0.7526 0.7526 0.6585  ,ρ 1 = 1 0.875 . Applying  1 to the prearray (21.6) leads to (recall that we are only operating on the first and third columns, leaving the second column unchanged):  0.875 0.15 1 0.675 0.35 0.5    0.6585 0 −0.7526 01 0 0.7526 0 0.6585   =  1.3288 0.1500 0.0000 0.8208 0.3500 −0.1788  . (21.8) We now annihilate the (1, 2) entry of the resulting matrix in the above equation by pivoting with its (1, 1) entry. This requires that we choose  2 = 1  1 + ρ 2 2  1 −ρ 2 ρ 2 1  =  0.9937 −0.1122 0.1122 0.9937  ,ρ 2 = 0.1500 1.3288 . (21.9) Applying  2 to the matrix on the right-hand side of (21.8) leads to (now we leave the third column unchanged)  1.3288 0.1500 0.0000 0.8208 0.3500 0.1788    0.9937 −0.1122 0 0.1122 0.9937 0 001   =  1.3373 0.0000 0.0000 0.8549 0.2557 0.1788  . (21.10) We finally annihilate the (2, 3) entry of the resulting matrix in (21.10) by pivoting with its (2, 2) entry. In principle this requires that we choose  3 = 1  1 + ρ 2 3  1 −ρ 3 ρ 3 1  =  0.8195 0.5731 −0.5731 0.8195  ,ρ 3 = 0.1788 −0.2557 , (21.11) and apply it to the matrix on the right-hand side of (21.10), which would then lead to  1.3373 0.0000 0.0000 0.8549 −0.2557 0.1788    100 00.8195 0.5731 0 −0.5731 0.8195   =  1.3373 0.0000 0.0000 0.8549 −0.3120 0.0000  . (21.12) c  1999 by CRC Press LLC Alternatively, this last step could have been implemented without explicitly forming  3 . We simply replace the row vector  −0.2557 0.1788  , which contains the (2, 2) and (2, 3) entries of the prearrayin(21.12), by the row vector  ±  (−0.2557) 2 + (0.1788) 2 0.0000  ,whichisequalto  ±0.3120 0.0000  . We choose the positive sign in order to conform with our earlier convention that the diagonal entries of triangular square-root factors are taken to be positive. The resulting postarray is therefore  1.3373 0.0000 0.0000 0.8549 0.3120 0.0000  . (21.13) We have exhibited a sequence of elementary orthogonal transformations that triangularizes the prearray of numbers (21.6). The combined effect of the sequence of transformations { 1 , 2 , 3 } corresponds to the orthogonal rotation  required in (21.7). However, note that we do not need to knowortoform =  1  2  3 . It will become clear throughout our discussion that the different adaptive RLS schemes can be de- scribed in array forms, where the necessary operations are elementary rotations as described above. Such array descriptions lend themselves rather directly to parallelizable and modular implementa- tions. Indeed, once a rotation matrix is chosen, then all the rows of the prearray undergo the same rotationtransformation and can thus be processed in parallel. Returningto the above example, where we started with the prearray A, we see that once the first rotation is determined, both rows of A are then transformed by it, and can thus be processed in parallel, and by the same functional (rotation) block, to obtain the desired postarray. The same remark holds for prearrays with multiple rows. 21.2 The Least-Squares Problem Now that we have explained the generic form of an array algorithm, we return to the main topic of this chapter and formulate the least-squares problem and its regularized version. Once this is done, we shall then proceed to describe the different variants of the recursive least-squares solution in compact array forms. Let w denote a column vector of n unknown parameters that we wish to estimate, and consider a set of (N +1) noisy measurements{d(i)} that are assumed to be linearly related to w via the additive noise model d(j) = u T j w + v(j) , where the {u j } are given column vectors. The (N + 1) measurements can be grouped together into a single matrix expression:      d(0) d(1) . . . d(N)         d =      u T 0 u T 1 . . . u T N         A w +      v(0) v(1) . . . v(N)         v , or, more compactly, d = Aw + v. Because of the noise component v, the observed vector d does not lie in the column space of the matrix A. The objective of the least-squares problem is to determine the vector in the column space of A that is closest to d in the least-squares sense. More specifically, any vector in the range space of A can be expressed as a linear combination of its columns, say A ˆw for some ˆw. It is therefore desired to determine the particular ˆw that minimizes the distance between d and A ˆw, min w d − Aw 2 . (21.14) c  1999 by CRC Press LLC The resulting ˆw is called the least-squares solution and it provides an estimate for the unknown w. The term A ˆw is called the linear least-squares estimate (l.l.s.e.) of d. The solution to (21.14) always exists and it follows from a simple geometric argument. The orthogonal projection of d onto the column span of A yieldsavector ˆ d that is the closest to d in the least-squares sense. This is because the resulting error vector (d − ˆ d) will be orthogonal to the column span of A. In other words, the closest element ˆ d to d must satisfy the orthogonality condition A T (d − ˆ d) = 0. That is, and replacing ˆ d by A ˆw, the corresponding ˆw must satisfy A T Aˆw = A T d . These equations always have a solution ˆw. But while a solution ˆw may or may not be unique (depending on whether A is or is not full rank), the resulting estimate ˆ d = A ˆw is always unique no matter which solution ˆw we pick. This is obvious from the geometric argument because the orthogonal projection of d onto the span of A is unique. If A is assumed to be a full rank matrix then A T A is invertible and we can write ˆw = (A T A) −1 A T d . (21.15) 21.2.1 Geometric Interpretation The quantity A ˆw provides an estimate for d; it corresponds to the vector in the column span of A that is closest in Euclidean norm to the given d. In other words, ˆ d = A  A T A  −1 A T · d  = P A · d , where P A denotes the projector onto the range space of A. Figure 21.1 is a schematic representation of this geometric construction, where R(A) denotes the column span of A. FIGURE 21.1: Geometric interpretation of the least-squares solution. 21.2.2 Statistical Interpretation The least-squares solution also admits an important statistical interpretation. For this purpose, assume that the noise vector v is a realization of a vector-valued random variable that is normally distributed with zero mean and identity covariance matrix, written v ∼ N[0, I]. In this case, the observation vector d will be a realization of a vector-valued random variable that is also normally c  1999 by CRC Press LLC distributed with mean Aw and covariance matrix equal to the identity I. This is because the random vectors are related via the additive model d = Aw + v. The probability density function of the observation process d is then given by 1  (2π) (N+1) · exp  − 1 2 ( d − Aw ) T (d − Aw )  . (21.16) It follows, in this case, that the least-squares estimator ˆw is also the maximum likelihood (ML) estimator because it maximizes the probability density function over w, given an observation vector d. 21.3 The Regularized Least-Squares Problem A more general optimization criterion that is often used instead of (21.14) is the following min w  ( w −¯w ) T  −1 0 (w −¯w) +d − Aw 2  . (21.17) This is still a quadratic cost function in the unknown vector w, but it includes the additional term ( w −¯w ) T  −1 0 (w −¯w), where  0 is a given positive-definite (weighting) matrix and ¯w is also a given vector. Choosing  0 =∞·I leads us back to the original expression (21.14). A motivation for (21.17) is that the freedom in choosing  0 allows us to incorporate additional a priori knowledge into the statement of the problem. Indeed, different choices for  0 would indicate how confident we are about the closeness of the unknown w to the given vector ¯w. Assume, for example, that we set  0 =  · I,where is a very small positive number. Then the first term in the cost function (21.17) becomes dominant. It is then not hard to see that, in this case, the cost will be minimized if we choose the estimate ˆw close enough to ¯w in order to annihilate the effect of the first term. In simple words, a “small”  0 reflects a high confidence that ¯w is a good and close enough guess for w. On the other hand, a “large”  0 indicates a high degree of uncertainty in the initial guess ¯w. Onewayofsolvingthe regularizedoptimizationproblem(21.17)istoreduceittothe standardleast- squares problem (21.14). This can be achieved by introducing the change of variables w  = w −¯w and d  = d − A ¯w.Then(21.17) becomes min w   (w  ) T  −1 0 w  +   d  − Aw    2  , which can be further rewritten in the equivalent form min w       0 d   −   −1/2 0 A  w      2 . This is now of the same form as our earlier minimization problem (21.14), with the observation vector d in (21.14)replacedby  0 d   , and the matrix A in (21.14)replacedby   −1/2 0 A  . c  1999 by CRC Press LLC [...]... least-squares problem as it arises in the context of adaptive filtering Consider a sequence of (N + 1) scalar data points, {d(j )}N=0 , also known as reference or desired j T signals, and a sequence of (N + 1) row vectors {uj }N=0 , also known as input signals Each input j T vector uj is a 1 × M row vector whose individual entries we denote by {uk (j )}M , viz., k=1 T uj = u1 (j ) u2 (j ) uM (j ) (21.26) The... description of the problem is shown in Fig 21.2 At each time instant j , the inputs of the M channels are linearly combined via the coefficients of the weight vector and the resulting signal is T compared with the desired signal d(j ) This results in a residual error e(j ) = d(j ) − uj w, for every j , and the objective is to find a weight vector w in order to minimize the (exponentially weighted and regularized)... section deal with this kind of minimization Now suppose that our interest in solving (21.60) is not to explicitly determine the weight estimate wM,N , but rather to determine estimates for the reference signals {d(·)}, say T dM (N) = uM,N wM,N = estimate of d(N ) of order M Likewise, for the higher-order problem, T dM+1 (N) = uM+1,N wM+1,N = estimate of d(N ) of order M + 1 The resulting estimation errors... d(2) d(N )         −     0 u(1) u(2) u(N ) 0 0 u(1) u(N − 1) 0 0 0 u(N − 2)    w (1)   3 w3 (2)    w3 (3)  (21.61) w3 A3,N dN where dN denotes the vector of desired signals up to time N , and A3,N denotes a three-column matrix of input data {u(·)}, also up to time N The optimal solution is denoted by w3,N The subscript N indicates that it is an estimate based on . also known as reference or desired signals, and a sequence of (N + 1) row vectors {u T j } N j=0 , also known as input signals. Each input vector u T j is. via the coefficients of the weight vector and the resulting signal is compared with the desired signal d(j). This results in a residual error e(j) = d(j)−