Adaptive algorithms are used to iteratively update a weight vector, an imation of the optimal solution also called Wiener solution, as input data is presented approx-in a streamapprox-in
MSE and MSD AnalysisforLMS/Newton
Dynamics of the weight error power vector
In section 3.3.1 we used the interpretation of the LMS/Newton algorithm as an LMS-like adaptation so we could use the results from section 3.2.1 for LMS to obtain expressions for the weight error power vector for the LMS/Newton algorithm under stationarity conditions.
In this section we will use the same strategy for the nonstationary case, considering the underlying Markov model for the Wiener solution as described in section 6.1.
In the beginning of section 3.3, Eq (3.57) shows that the LMS/Newton algorithm in
Eq (2.34) is equivalent to the LMS algorithm in Eq (3.56) applied on the transformed input and weight vector defined in Eq (3.54) In our model of nonstationarity, only the crosscorrelation between the input and the desired response changes with time, keeping the input statistics stationary This implies that the transformations in Eq (3.54) are still meaningful in our nonstationarity model, so we can rely our analysis of the LMS/Newton algorithm on its LMS-like representation of Eq (3.56) The first step is to realize that since the Wiener solution is changing with time, we need to introduce a time index for it in Eqs (3.59) and (3.60), re-expressing the transformed Wiener solution as
0? = A2QT ui, (6.29) and redefining the transformed weight error vector as
Considering Eq.s (6.4), (6.29), and (6.30), Eq (3.61) is still valid, which, in combination with Eqs (3.62) and (3.64), make Eqs (3.63) and (3.65) valid as well for our nonstationary case In lieu of (3.65), to obtain an expression for the LMS/Newton weight-error power vector, we can just find F, for its associated LMS-like form of Eq (3.56) To do this, we first need to find the equation that governs the changes of 7; multiplying Eq (6.1) on the left by A2Q7, and using Eq (6.29), we get
15} ¡ = awe + Ấy, (6.31) where ^¿ is defined as
= A2, (6.33) and has an associated autocorrelation matrix
From Eqs (3.54), (3.57) and (6.32), it is easy to verify that the assumption of +„ being zero | mean and independent of {a}, ¢;} implies that 4 is also zero mean and independent of {a,, €;}; furthermore, the assumptions described in section 3.2.1 on the random processes {ax, d,} also hold on the random processes {%;,d,} Hence, we can argue in a similar manner as in section 6.2.1 to obtain the equation that is analogous to Eq (6.9), correspond- ing to Eqs (3.56) and (3.54) Thus, in lieu of Eqs (3.56), (3.58) and (6.31), we can replace
C;, with C,, R with 1, R, with R,, and yz with pAavg, to translate Eq (6.9) into
Applying the diagonal operator on both sides of Eq (6.36), and neglecting second order terms in jz we obtain
Substituting Eq (3.65) in Eq (6.37), multiplying on the left by A~+, and using Eq (6.39) we finally get
Fru (1— 4pAave) Fe + Aw rE A + YT (6.40)
Observe the strong resemblance between the expression for #¿ in Eq (6.40) for our non- stationary case and the one we obtained for the stationary analysis in Eq (3.76) We can see that the only difference is that in the nonstationary case there is an extra driving term
Y which introduces the effect of the changes of the Wiener solution Since the dynamics remain unchanged, we can conclude that, unlike with LMS, with LMS/Newton the adap- tation capabilities of the weight vector to changes of the Wiener solution along all the eigendirections are the same This has a particular effect on the steady-state behavior of the weight vector which we analyze next.
The value of ¿ required for Eq (6.40) implies that is small enough so that |1 — 4uÀzvg| < 1, therefore the limit in Eq (6.15) exists Hence, we can take the limit on both sides of Eq (6.40) and solve for F,,, obtaining the steady-state weight error power vector for LMS/Newton to be
This expression shows that the asymptotic weight-deviation power from the Wiener solu- tion for the LMS/Newton algorithm is determined by two factors as in the LMS case The first term in Eq: (6.16) is due to “gradient noise”, proportional to ju; and the second term is due to “lag”, inversely proportional to yz Therefore, there is a trade-off between reducing the error due to gradient noise and the error due to lag, regulated by ứ In the next sections we will find the values of ys that minimize the asymptotic MSE and MSD of LMS/Newton for our nonstationary model of the Wiener solution, so we can compare them with the ones of LMS in section 6.4, arriving at the concept of Nonstationary Efficiency.
MSE Steady state performance under Nonstationary conditions 111 6.3.3 MSD Steady state performance under Nonstationary conditions 112
To obtain an expression for the MSE in terms of the error power vector, we will use once again the interpretation of the LMS/Newton algorithm in terms of the LMS adaptation of
Eq (3.56) Specifically, Eq (3.57) implies that the error obtained using LMS/Newton is the same as the one obtained when using the LMS algorithm of Eq (3.56); furthermore, ^ is zero mean and independent of {a;,, cÿ}, and the assumptions described in section 3.2.1 on the random processes {#, d,} also hold with the random processes {#,, d„} Hence, we can argue in a similar manner as in section 6.2.2, and write the equation analogous to
Eq (6.17) associated with Eq (3.56), by replacing F., with F,, and with 1 in lieu of
Eq (3.58) obtaining | boo HET +1, (6.42) using Eq (3.65), Eq (6.42) translates into boo BET + AT E2, (6.43) which is exactly the same as Eq (6.17) Substituting Eq (6.41) in Eq (6.43), we get
Using Eqs (2.22), (6.2), (6.11) and (6.14) we obtain the following identity
MY =Tr(RRy) =ElyRwl (6.45) and recalling Eq (3.47), we can rewrite Eq (6.44) as the following expression for the asymptotic excess MSE wa 1co HÀzvy€*b Tr(RR,) (6.46)
Equation (6.46) was derived in [132] for the EW-RLS algorithm, replacing /Àavg with (1— 8)/2; this corroborates our assertion in section 2.4 on the similarity between LMS/Newton and EW-RLS asymptotically, when their parameters are related by Eq (2.63) The value of that minimizes the above expression for the asymptotic excess MSE is the one such that both terms in the right hand side of Eq (3.53) are equal, so we have
MSE ~ opt Xavg ~ in agreenment with the results in [132] for the EW-RLS algorithm Substituting Eq (6.47) into Eq (6.46) we obtain the minimum excess MSE for LMS/Newton under our nonsta- tionarity model to be
6.3.3 MSD Steady state performance under Nonstationary conditions
To obtain an expression for the MSD for LMS/Newton we substitute Eq (6.41) into
Noo HÀzg€*1TA~!+ Tet © (6.49) avg which can be rewritten as
Noo © UAavgé* Tr Ro + TA Tr(R}) (6.50)1 avg and using Eq (6.45) as
Noo HÀzy€” TY Ro + Tr(R,) (6.51)
As in the MSE case, Equation (6.51) was derived in [126, 132, 171] for the EW-RLS algorithm, but with ¿ replaced with (1— 8) /2Aayg; this further corroborates our assertion on the similarity between LMS/Newton and EW-RLS asymptotically when their parameters are related by Eq (2.63) The value of that minimizes the asymptotic MSD is the one that causes both terms of the right hand side of Eq (6.51) to be equal, so we have
Msp, | I(R,) in agreenment with the results in [132] for the EW-RLS algorithm Substituting Eq (6.52) into Eq (6.51) we obtain the minimum excess MSD for LMS under our nonstationarity model to be ma = 1//§* Tr(R-) Tr(R,) (6.53).
LMS Nonstationary Efficiency 2 ee eee eee 113
LMS MSE Nonstationary Efficlency
As an indicator of the steady-state MSE performance of the LMS algorithm with respect to LMS/Newton under our nonstationary conditions, we define the LMS MSE Nonstationary Efficiency as the ratio between the asymptotic excess MSE for the LMS and LMS/Newton algorithms as follows
LMS MSE Transient Efficiency = (| (6.54) oo,min
Since the steady-state excess MSE is positive’, LMS will be better than LMS/Newton in the steady-state MSE sense if the MSE Nonstationary Efficiency is more than one, and worse, if it is less than one.
Substituting Eqs (6.22) and (6.48) into Eq (6.54) we get v 2 aus) ~ TY) THR)
| which can be rewritten using Eqs (4.21), (6.19) and (6.45) as šLMS/Newton \ 2 T so mi XY
We would like to emphasize the remarkable similarity between the expressions just ob- tained for the LMS MSE Nonstationary Efficiency and the ones for the LMS MSE Tran- sient Efficiency Specifically, notice that we can go between Eqs (4.24) and (6.55) by simply interchanging Co with R,; and between Eq (4.17) and (6.56) by interchanging
Fo with Y From the definitions of Co, R,, Fo and Y, in Eqs (3.3), (6.2), (3.6) and (6.14) respectively, we realize that the role the initial weight vector vo plays in the LMS MSE Transient Efficiency is exactly the same role that the Wiener solution steps +¿ play in the LMS MSE Nonstationary Efficiency Therefore, all of our discussion of the LMS MSE Transient Efficiency in section 4.2.1 can be essentially repeated here for the LMS 3The steady state excess MSE can be negative only under very special circumstances [22, 99, 100, 209,
210, 214, 215], which we are not considering here.
MSE Nonstationary Efficiency by simply replacing MSE Transient performance with MSE Nonstationary performance, and replacing the initial weight deviation vector 0ọ, and its as- sociated power Fo, with the Wiener solution steps -y;, and its power Y We proceed in this fashion to make such analogy between the Stationary and Nonstationary Efficiencies more evident.
The ratios in Eq (6.55) and (6.56) are invariant to scaling of the matrices R and R., (or equivalently, scaling of the eigenvalues and Y), hence, without loss of generality, we can scale them so that the denominators are one In this way, Eq (6.56) simplifies to
The components of YT are the powers of the Wiener solution steps along each of the eigen- vectors of R Thus, Eq (6.57) succinctly shows how the MSE Nonstationary performance of LMS depends on the statistics of the input signal and Wiener solution steps According to Eq (6.58), the sum of the components of YT is one, and since they are nonnegative, the right hand side of Eq (6.57) can be interpreted as the weighted average of the eigenvalues, where the weights are the components of YT Since the mean of the eigenvalues is one, the LMS MSE Nonstationary Efficiency can be larger, equal or smaller than one, depending on the eigenvalue distribution and Y For example, consider the case when all the compo- nents of Y have the same power, say ơ?; we will refer to this as uniform nonstationarity, compactly expressed as uniform nonstationarity: Y = 071 (6.59)
Under this condition, it is clear that the LMS MSE Nonstationary Efficiency is one, regard- less of the eigenvalue distribution The condition in Eq (6.59) is trivially attained when the components of -y, are uncorrelated with each other and all have the same power This is precisely the case analyzed by Widrow et al in [265, 267], arriving to the same results we have shown here under that particular model of nonstationarity, namely, that the op- timal steady-state MSE performance of LMS and LMS/Newton are the same under such conditions of nonstationarity.
Eq (6.57) also holds if the scaling of the eigenvalues and Y is done such that
Soi -1 1 _ 1, (6.60)L T i=l L so we can have the dual interpretation of the LMS MSE Transient Efficiency as the eigenvalue- weighted average of the components of Y, whose mean is one With this interpretation, it is obvious then that if all the eigenvalues are identical, the LMS MSE Nonstationary Effi- ciency is one, regardless of the statistics of ~y,; this is consistent with the fact that LMS and LMS/Newton are essentially equivalent in such circumstances Another fact easy to derive from Eq (6.57) and either Eq (6.58) or Eq (6.60) is that the LMS MSE Nonstationary
Efficiency cannot exceed L, i.e the minimum steady-state excess MSE for LMS/Newton cannot be more than L times larger than the one for LMS.
In general, we can infer from Eqs (6.57) and (6.58), that if most of the power of the Wiener solution steps is along the eigenvectors with eigenvalues larger/smaller than Xavg, then the LMS MSE Nonstationary Efficiency will be larger/smaller than one, implying that LMS is better/worse than LMS/Newton in the steady-state MSE sense Thus, Eq (6.57) provides a simple measure to quantify the fact that when the Wiener solution changes are mostly along the fast/slow modes, LMS tracks the Wiener solution better/worse than LMS/Newton in the MSE sense.
Using Eqs (6.19) and (6.45) we can rewrite Eqs (6.57) and Eq (6.58) as
=LMS/Newton ca = Ely, Ry] (6.61) with ave = Elll%|]= 1, (6.62) and, as an alternative to the conditions in Eq (6.62), Eq (6.60) can be also rewritten as
Equation (6.61) is analogous to Eq (4.25) for the Stationary analysis, and it shows that the LMS MSE Nonstationary Efficiency can be interpreted as the expected value of theR-norm of +¿, hence, as the reader might expect, we will be able to express it in terms of the Fourier spectra in the case R is Toeplitz as we it was done the LMS MSD TransientEfficiency in section 5.3.3.
LMS MSD Nonstationary Effclency
In the stationary case studied in previous chapters, the analysis of the MSD performance was parallel to the one for MSE In our nonstationary analysis, the same pattern occurs.
As an indicator of the steady-state MSD performance of the LMS algorithm with respect to LMS/Newton under our nonstationary conditions, we define the LMS MSD Nonstation- ary Efficiency as the ratio between the asymptotic MSD for the LMS and LMS/Newton algorithms as follows
LMS MSD Nonstationary Efficiency 2 [oes | (6.64)
Notice that the ratio is taken in the opposite way as in the MSE case‘; thus, if the LMS MSD Nonstationary Efficiency is larger/smaller than one, LMS is better/worse than LMS/Newton in the steady-state MSD sense.
Substituting Eqs (6.28) and (6.53) into Eq (6.64) we get
T]52 nản ° ~ LTr(R*R,) (6 65) nỊLMS/Newton ~ Tr(R-1) Tr( R,) oo,min which can be rewritten using Eqs (4.35), (6.19) and (6.25) as dồn AY
*The reason for taking the ratio in the opposite way to the MSE is to make the analogy between the MSD and MSE analysis more evident
We would like to emphasize the remarkable similarity between the expressions just obtained for the LMS MSD Nonstationary Efficiency and the ones for the LMS MSD Transient Efficiency Notice that we can go between Eqs (4.37) and (6.65) by simply in- terchanging Co with R,; and between Eq (4.31) and (6.66) by interchanging Fo with TY. From the definitions of Co, R,, Fo and Y, in Eqs (3.3), (6.2), (3.6) and (6.14) respec- tively, we realize that the role the initial weight vector vp plays in the LMS MSD Transient Efficiency is exactly the same role that the Wiener solution steps +¿ play in the LMS MSD Nonstationary Efficiency Therefore, all of our discussion on the LMS MSD Transient Efficiency in section 4.2.2 can be essentially repeated here for the LMS MSD Nonstation- ary Efficiency by simply replacing MSD Transient performance with MSD Nonstationary performance, and replacing the initial weight deviation vector vo, and its associated power
Fo, with the Wiener solution steps +¿ and its power Y We proceed in this fashion to make this analogy between the Stationary and Nonstationary Efficiencies more evident.
The ratios in Eq (6.65) and (6.66) are invariant to scaling of the matrices R~! and Ry
(or equivalently, scaling of the eigenvalues and Y), hence, without loss of generality, we can scale them so that the denominators are one In this way, Eq (6.66) simplifies to nLMS/Newton ? 7 œ,min —1
Comparing Eqs (6.57) and (6.67), we can easily realize a direct analogy between the LMS MSE Nonstationary Efficiency and the LMS MSD Nonstationary Efficiency This analogy is in the sense that the arguments in section 6.4.1 for the LMS MSE Nonstationary Efficiency can be repeated here, but replacing the eigenvalues with their reciprocals, and recalling that LMS is better/worse than LMS/Newton when the LMS MSD Nonstationary Efficiency is smaller/larger than one.
Eq (6.67) succinctly shows how the MSE Nonstationary performance of LMS depends on the statistics of the input signal and Wiener solution steps According to Eq (6.68), the sum of the components of YT is one, and since they are nonnegative, the right hand side of
Eq (6.67) can be interpreted as the weighted average of the eigenvalue reciprocals, where the weights are the components of Y Since the mean of the eigenvalue reciprocals is one,
‘the LMS MSE Nonstationary Efficiency can be larger, equal or smaller than one, depending on the eigenvalue distribution and Y For example, under the uniform nonstationarity condition defined in Eq (3.90), the LMS MSD Transient Efficiency is one regardless of the eigenvalue distribution
Eq (6.67) also holds if the scaling of the eigenvalues and YT is done such that
—=—_—=] 6.69 so we can have the dual interpretation of the LMS MSD Transient Efficiency as the eigenvalue- reciprocal weighted average of the components of Y, whose mean is one With this inter- pretation, it is obvious then that if all the eigenvalues are identical, the LMS MSE Nonsta- tionary Efficiency is one, regardless of the statistics of +„; this is consistent with the fact that LMS and LMS/Newton are essentially equivalent in such circumstances Another fact easy to derive from Eq (6.67) and either Eq (6.68) or Eq (6.69) is that the LMS MSE Nonstationary Efficiency cannot exceed J, i.e the minimum steady-state MSD for LMS cannot be more than L times larger than the one for LMS/Newton
In general, Eqs (6.67) and (6.68) imply that if most of the power of the Wiener solution steps is along the eigenvectors with eigenvalues whose reciprocal is smaller/larger than
(X) are? then the LMS MSE Nonstationary Efficiency will be smaller/larger than one, im- plying that LMS is better/worse than LMS/Newton in the asymptotic MSD sense Thus,
Eq (6.67) quantitatively corroborates the intuitive idea that when the Wiener solution changes are mostly along the fast/slow modes, LMS tracks the Wiener solution better/worse than LMS/Newton in the MSE sense This result is in agreement with the fact that when using LMS, the tracking of the Wiener solution changes along modes highly excited by the input is better than for the changes along the poorly excited modes That is, if the
Wiener solution steps have most of their power along highly-excited modes, and little power in the poorly-excited ones, then the tracking capabilities of LMS will be better thanLMS/Newton’s.
Using Eqs (6.19) and (6.25) we can rewrite Eqs (6.67) and Eq (6.68) as
; Tr(R-1 with TẾT” = gllwlP]=1 671) and, as an alternative to the conditions in Eq (6.71), Eq (6.69) can be also rewritten as
Equation (6.70) is analogous to Eq (4.38) for the Stationary analysis, and it shows that the LMS MSD Nonstationary Efficiency can be interpreted as the expected value of the
R-'-norm of +¿, hence, as the reader might expect, we will be able to express it in terms of the Fourier spectra in the case FR is Toeplitz and 7 is large as it was done for the LMS MSD Transient Efficiency in section 5.3.3.
Comparing Eqs (6.61) and Eq (6.70), we observe that the analogy between the LMSMSE and MSD Nonstationary Efficiency reduces to the exchange of R with its inverse,and reversing the order of the LMS and LMS/Newton superscripts in the ratio defining theLMS MSD Nonstationary Efficiency.
Discussion 2 cu Q Q Q HQ HQ kg V na 120
As it was mentioned at the end of section 2.4, the EW-RLS algorithm can approximate LMS/Newton for large values of k when the input signal is stationary given that a close to unity forgetting factor ỉ is set according to Eq (2.63) Since the input is assumed to be stationary in our model of nonstationarity, our expressions for Nonstationary Efficiency perfectly agree with the results in [76, 132], where they compared the LMS and EW-RLS algorithms under the same model of nonstationarity The translation of such results into the frequency domain was not previously done however, and is the subject of the next section.
We should also mention that previous work on the performance of LMS vs EW-RLS in the nonstationary case has also been studied in specific applications; for example, when tracking a chirped signal[15, 25, 31, 177, 178, 181, 255], a first order Markov communica- tion channel [26, 153], or a wide sense stationary uncorrelated scattering channel [94, 162].
A sinusoidal cancellation task was analyzed in [201], multipath channel equalization in
CDMA systems was considered in [252], and an empirical analysis of the equalization of | nonstationary wireless channels in mobile ad-hoc networks was done in [254] Our results corroborate and extend most of such previous results.
LMS Nonstationary Efficiency in terms of Spectra
Fourier Spectrum of the Wiener solution steps
The definition of the Fourier spectrum for the Wiener solution steps is based on the defi- nition of the power spectrum of L-dimensional random vectors in section 5.2, and as the reader might expect, we have two types for it, the continuous frequency and the discrete frequency versions.
The continuous Fourier spectrum of the Wiener solution steps is defined as
T(e“) £E notice that since we are assuming the random process to be stationary, the expectation above is independent of time, we could drop the time subscript in notation for T(e?“) The discrete Fourier spectrum of the Wiener solution steps is defined as
Notice that exchanging +; with vp in the above definitions, F'(e7“) and I'[zn] are analogous to E||Vo(e%)|7] and E[|V[m]|?] respectively.
Using Eqs (6.73) and (6.74) we can verify verify that Z[||+;|?] = 1 implies
SO we are now ready to translate the expressions we have so far for the LMS NonstationaryEfficiency into the frequency domain.
LMS MSE Nonstationary Efficiency in terms of spectra
The expressions we found for the LMS Nonstationary Efficiency are analogous to the ones for the LMS Transient Efficiency by replacing the initial weight error vector vp with the Wiener solution steps +„ Therefore, all of our results for the LMS MSE Transient Effi- ciency in terms of Fourier Spectra in section 5.3.2, can be directly translated into results for the LMS MSE Nonstationary Efficiency by simply replacing MSE Transient perfor- mance with MSE Nonstationary performance, and replacing ứọ and its power spectrum with +¿ and its power spectrum We proceed in this fashion to make this analogy between the Stationary and Nonstationary Efficiency in terms of spectra more evident.
In section 4.2.1 we showed that the LMS MSE Transient Efficiency is determined by the expected value of the R-norm of +¿, and it was shown in section 5.3.1 we could trans- late that norm into the Fourier Domain So we are now ready to obtain the following expressions for the LMS MSE Transient Efficiency in terms of the continuous Fourier spectra Replacing œ with “+¿ in (5.13), using Eq (6.73), and substituting into Eq (6.61) we obtain
: an om ju — — Jue = 1, 6.77 with al ®„„(e?“)du x, \T'(e”) dw | (6.77) where Eq (6.77) is derived from Eq (6.62) using Eqs (5.5) and (6.75).
In a similar way, combining Eq (6.61) with (5.22) and Eq (6.74), we obtain the fol- lowing expressions for the LMS MSE Transient Efficiency in terms of the discrete Fourier spectra
JEMSIN ewton M-1 jis * 3, ®z„[m]T[m] (6.78) =0 with 57d) Seal] = D7 Tim] =1, (6.79) m=0 m=0 where Eq (6.79) is derived from Eq (6.62) using Eq (6.75).
Thus, we have transformed the original expressions for the LMS MSE Nonstationary Efficiency from equations (6.57) and Eq (6.62) into the frequency domain by replacing the eigenvalues with the input power spectrum, and Y with the spectrum of the Wiener solution steps Furthermore, the normalization conditions in Eqs (6.77) and (6.79) imply that the right hand sides of Eqs (6.76) and (6.78) can be interpreted as the weighted av- erage of the input power spectrum with the spectrum of the Wiener solution steps as the weighting function Since the “uniform” average of the input spectrum is one, the LMS MSE Nonstationary Efficiency will be smaller, greater or equal to one depending on the spectral distribution of the input and the Wiener solution steps For example, under the uniform nonstationarity condition defined in Eq (6.59), the spectrum of the Wiener solu- tion steps is white’, so the LMS MSE Transient Efficiency is one, regardless of the input spectrum, as expected from the results in section 6.4.1 A similar result was found in [162] based on a frequency domain analysis, where the authors showed the tracking performance equivalence between the LMS and EW-RLS algorithms under the assumption that all of the components of the Wiener solution changes are uncorrelated with each other and have the same distribution.
As an alternative to the normalization of Eq (6.79), the equation analogous to Eq (6.63)
5To verify this, define e(w) Ê [1 e2 ef e(E-D], so we can write V(e”) = Ellyn (ei) |?] = e(w) E lyn} Je* (w) = e(w)Rye*(w); and applying Eq (6.59) we get V(e) = 07 L for all w. in the discrete frequency domain can be obtained by dividing ®„„[rm] by M and multiply- ing T[m] by Ä⁄ (so the right hand side of Eq (6.78) remains unchanged) to get
> ®„„[m] = 7 on Tịm (6.80) so we have the dual interpretation of the LMS MSE Nonstationary Efficiency as the input- spectrum weighted average of the spectrum of the Wiener solution steps, whose mean is one® With this interpretation, it is obvious then that if the input spectrum is white, the LMS
MSE Nonstationary Efficiency is one, regardless of the spectrum of the Wiener solution steps; this is consistent with the fact that LMS and LMS/Newton are essentially equivalent in such circumstances.
| In general, for a colored input spectrum, we can infer from Eqs (6.76) and (6.78), and their companion equations (6.77) and (6.79) that: if most of the power spectrum of the Wiener solution steps is at frequencies where the input power spectrum is larger/smaller than its average, then the LMS MSE Nonstationary Efficiency will be larger/smaller than one, i.e., LMS will be better/worse than LMS/Newton in the steady-state MSE sense. Considering the normalization of Eq.(6.80) instead of Eq (6.79) in the previous para- graph, we obtain its dual by simply exchanging the roles of the input power spectrum and the power spectrum of the Wiener solution steps.
From the observations above, we can make some statements about the MSE Nonsta- tionary Efficiency of LMS based on approximate spectral information For example, if the power spectrum of both the input and the Wiener solution steps are “low-pass”, LMS will have better tracking capabilities than LMS/Newton On the other hand, if the input power spectrum is “low-pass” or “high-pass”, but the power spectrum of the Wiener so- lution steps is “high-pass” or “low-pass” respectively, in a way that their spectra don’t overlap much, then LMS will have worse tracking capabilities than LMS/Newton This is in agreement with the intuitive idea that when using LMS, the tracking of the Wiener solu- tion changes along modes highly excited by the input would be better than for the changes along the poorly excited modes That is, if the Wiener solution steps have most of theirNotice that the dual interpretation of the LMS MSE Nonstationary efficiency in the continuous frequency domain case is immediate from Eq (6.77) without further manipulations power along highly-excited modes, and little power in the poorly-excited ones, then the tracking capabilities of LMS will better than LMS/Newton’s.
These results are of practical relevance, since in certain applications, certain approxi- mate knowledge of the spectral content of the input and changes of the Wiener solution is known a-priori, which, together with the above results, allows for the estimation of the per- formance of the LMS algorithm before its implementation Examples of such applications are explained in detail in section 6.6.
To conclude this section, we observe that the LMS MSE Nonstationary Effiency ex- pression analogous to Eq (6.55) in the continuous frequency domain can be found to be fo"®n0(e)T (e) du
( oom Jy" Baa(e)dw f"T (e*)dw and similarly, the expression for the discrete frequency domain results in oe m=o Pal] 3= TỊm] oo,min šLMS/Newton \ 2 M-1 ,
( ©o,min xMS m=o Pa „im Ihr (6.82)
LMS MSD Nonstationary Efficiency in terms of spectra
In section 6.4.2 we showed that the LMS MSD Nonstationary Efficiency is determined by the expected value of the R~+-norm of +, and in section 5.3.1 we showed we could trans- late that norm into the Fourier Domain only in the limit as Ù goes to infinity Therefore, we can only obtain approximate expressions for the LMS MSD Transient Efficiency in terms of Fourier Spectra for large values of L The derivations and assertions for the LMS MSD
Transient Efficiency are parallel to the ones we did for the MSE case.
Replacing v with + in (5.27), using Eq (6.73), and substituting in Eq (6.70), we have for sufficiently large L, nh 2 L2 nein ~ al Dre (e*) P(e) dw (6.83) with — af ®-'(e“) n1) T(e)dw = 1, (6.84) 1 2z where Eq (6.84) is derived from Eq (6.71) using Eq.(6.75) for the normalization of T(e?“) and Eqs (5.23) and (5.25) for the normalization of ®=1(e7“), Hence, the normalization of ®= (3) in Eq (6.84) is equivalent to the normalization of R~! in Eq (6.71) only 1 in the limit as L goes to infinity.
In a similar way, combining Eq (6.70) with (5.22) and Eq (6.74), we obtain that, for sufficiently large L, the LMS MSD Transient Efficiency can also be approximated in terms of the discrete Fourier spectra as follows
1 M-1 M-1 Ũ —1 _ _ with 2 ®=1[m] = " T[m] = 1, (6.86) where Eq (6.79) is derived from Eq (6.71) using Eq.(6.75) for the normalization of T[m|], and Eqs (5.23) and (5.25) are used for the normalization of ®z*[m] Hence, the normal- ization of ®Z1[m] in Eq (6.86) is equivalent to the normalization of R~! in Eq (6.71) only in the limit as Ủ goes to infinity.
Thus, for large values of L, we have transformed the original expressions for the LMS
MSD Nonstationary Efficiency from equations (6.67) and Eq (6.68) into the frequency domain by replacing the reciprocal of the eigenvalues with the reciprocal of the input power spectrum, and Y with the spectrum of the Wiener solution steps Furthermore, the normalization conditions in Eqs (6.84) and (6.86) imply that the right hand sides of Egs (6.83) and (6.85) can be interpreted as the weighted average of the reciprocal of the input power spectrum with the spectrum of the Wiener solution steps as the weighting function Since the “uniform” average of the reciprocal of the input spectrum is one, the LMS MSD Nonstationary Efficiency will be smaller, greater or equal to one depending on the spectral distribution of the input and Wiener solution steps For example, under the uniform nonstationarity condition defined in Eq (6.59), the spectrum of the Wiener solution steps is white, so the LMS MSD Transient Efficiency is one, regardless of the input spectrum, as expected from the results in section 6.4.1.
As an alternative to the normalization of Eq (6.86), the equation analogous to Eq (6.72)
1n the discrete frequency domain can be obtained by dividing 6-1 [(m] by M and multiply- ing I'[m] by Ä (so the right hand side of Eq (6.85) remains unchanged) to get
3 Sel) = 55 Tim) = 1, (6.87) m=0 m=0 so we have the dual interpretation of the LMS MSD Nonstationary Efficiency as the reciprocal-input-spectrum weighted average of the spectrum of the Wiener solution steps, whose mean is one’ With this interpretation, it is obvious then that if the input spectrum is white, the LMS MSD Nonstationary Efficiency is one, regardless of the spectrum of the Wiener solution steps; this is consistent with the fact that LMS and LMS/Newton are essentially equivalent in such circumstances.
In general, for a colored input spectrum and large values of L, we can infer from Eqs (6.83) and (6.85), and their companion equations (6.84) and (6.86), that: if most of the power spectrum of the Wiener solution steps is at frequencies where the reciprocal of the input power spectrum is smaller/larger than its average, then the LMS MSD Nonstationary Efficiency will be smallerNarger than one, i.e., LMS will be better/worse than LMS/Newton in the steady-state MSD sense.
Considering the normalization of Eq (6.80) instead of (6.79) in the previous paragraph, we have its dual by simply exchanging the roles of the reciprocal of the input power spec- trum and the spectrum of the Wiener solution steps.
Therefore, considering large values of L, we can make statements about the MSD Nonstationary Efficiency of LMS based on rough knowledge of the spectra as we did for the MSE case For example, if the input power spectrum is “low-pass”, its reciprocal is
“high-pass”, and if the power spectrum of the Wiener solution steps is also “low-pass”,then the LMS MSD Nonstationary Efficiency will be likely smaller than one, so LMS will have better tracking capabilities than LMS/Newton On the other hand, if the input power spectrum is “low-pass” or “high-pass”, but the initial weight deviation spectrum is “high- pass” or “low-pass” respectively so that it substantially overlaps with the reciprocal of the input spectrum, then the LMS MSD Nonstationary Efficiency will be likely bigger thanTNotice that the dual interpretation of the LMS MSD Transient efficiency in the continuous frequency domain case is immediate from Eq (6.84) without further manipulations one, so LMS will have worse tracking capabilities than LMS/Newton.
The relevance of the expressions above for applications in practice is the same as in the MSE case, and even though their approximations are valid only for large values of L, they can be still useful with a moderate size of L as illustrated by some example applications in section 6.6.
To conclude this section, we observe that the LMS MSD Nonstationary Effiency ex- pression analogous to Eq (6.65) in the continuous frequency domain can be found to be nis, \" 81 (i) (eo) deo 688)
(+ Sâu Jo" Ba (ei) dw fo" P(e) dw and similarly, the expression for the discrete frequency domain results in
Noomin Ì ico Pad mE fm] 2
LMS/Newton > M M—1 x~—1 M-1 (6.89) oo,min mao Dix [m] 2 m=0 Tịm)]
Discussion 2 0 AT a 128
Under our model of nonstationarity, the statistics of the input are constant and the changes of the Wiener solution are modeled as a random walk With this kind of nonstationarity, we showed that the performance of the LMS algorithm is subject to its gradient descent nature in a similar manner as its transient performance does Namely, the fact that the dynamics of the weight error vector for the highly excited modes are “faster” than for the poorly excited ones has the effect that the LMS response to changes in the Wiener solution along the fast modes is better than along the slow modes.
The striking similarity between our expressions for the LMS Transient Efficiency and LMS Nonstationary Efficiency reflects the fact that the transient and nonstationary behav- ior of the weight vector are strongly linked The fact that we can obtain one expression from the other by exchanging the roles of vp and -y, is in accordance with the intuition that we can regard changes of the Wiener solution as a continuous resetting of the initial con- ditions Since we know that LMS adapts “fast” with initial conditions set along the highly excited modes, it makes sense then that it is able to effectively track changes in the Wiener solution in those directions Of course, this is reflected in the MSE and MSD only after the transients associated with “big” initial deviations for the Wiener solution have died out.
For example, consider a situation where we initialize the weight vector along the poorly excited modes far enough from the Wiener solution, and that the changes of the Wiener solution are mostly along the highly excited modes In this case, the weight error vector will initially converge slowly to its asymptotic value due to its high initial power along the poorly excited modes; but once those components have reached their steady state, they will be mostly driven by the changes of the Wiener solution, which are along highly excited modes, so the weight vector should track those changes better than LMS/Newton would do.
From our results, we can rewrite the previous paragraph replacing modes with fre- quencies, and it can be extended to a comparison between LMS and EW-RLS replac- ing LMS/Newton with EW-RLS Previous work on the frequency domain tracking perfor- mance of LMS versus EW-RLS relied on large values of Z [122, 171] Our nonstationary analysis on the MSE performance of LMS does not assume L to be large, and it provides a succinct qualitative and quantitative performance evaluation of LMS tracking performance in the frequency domain.
Last, we would like to remark about the analogy between our expressions for the LMS
Transient Efficiency and LMS Nonstationary Efficiency Notice that both of these can be written as the expected value of the R-norm of up and “y¿ respectively for the MSE performance case, and that replacing R with RTM! leads to the Efficiency expressions for the MSD.
Applications 2 ee kia 129
We will illustrate the results obtained for the random walk model of nonstationarity with several simulation examples The desired response was obtained according to the system identification diagram of Fig 5.2; the FIR filter and the adaptive filter both had 16 taps,therefore, the Wiener solution was equal to the FIR filter impulse response, and obeys the random walk model of Eq (6.1) with a = 1 The weight vector was initialized in all cases to the Wiener solution at time zero, i.e., wo = wi This was done since were only interested in the steady state behavior.
The input signal was generated in both cases by passing unit power stationary and uniformly distributed white noise through an FIR “coloring” filter composed of 32 taps. The additive noise was independent of the input signal and had power o? = 0.01 Since the number of taps of the adaptive filter matched the length of the FIR filter in the unknown plant block, the noise power was equal to the minimum MSE, le €* = ơ2 For the calculation of the MSD Transient Efficiency according to Eq (5.32), we used N = 32 and
M = 200 to compute ®„„|m] In all of the simulations, the values of / were set to their optimum values in order to minimize either the steady-state MSE or MSD Whenever we mention the calculation of the LMS MSE Transient efficiency according to the formula, we refer to the value obtained when using Eq (6.61), or equivalently, (6.76) or (6.78) For the LMS MSD Transient Efficiency case, the calculation is primarily done using Eq (6.70), which is not the same as using Eqs (6.76) or (6.78) (only in the limit as LZ — oo); in fact, we also do the calculation using (6.78) to illustrate that even though large values of L are assumed for that approximation, it turns out to work well given that N is large enough so that ®„„[m] > 0 for all m, this was ensured in all cases by letting N = 32 and M = 200 to compute ®z„„|mm] In all of the simulations, ju was set to minimize either the steady-state MSE or MSD as it corresponded.
The learning curves were generated by averaging over an ensemble of 10000 runs of the adaptive filter The number of iterations used was large enough so that “steady-state” was essentially reached for both LMS and LMS/Newton To estimate the steady-state excess MSE and MSD from the simulations, the mean of the ensemble average of the last
The Fourier spectra for the input and Wiener solution steps +„ were chosen in order to illustrate how approximate knowledge on such spectra can gives us a qualitative assessment of the steady-tracking performance of LMS relative to that of LMS/Newton before its implementation under the nonstationarity model considered.
We start by considering a case where the power spectrum of the input and of the Wiener solution steps have both a low-pass characteristic? with approximately the same cut-off frequency as shown in Fig 6.1.
> changes in the oosL Wiener solution
Figure 6.1: Nonstationary example 1: Power spectrum of input and Wiener solution changes.
The LMS MSE Nonstationary Efficiency is therefore expected to be larger than one, which is indeed the case From the formula, it was calculated to be 1.723, very close to the value of 1.774 obtained from the simulation results, shown in Fig 6.2 This means that, for these statistics of the input and Wiener solution changes, the steady-state excess MSE for LMS/Newton is approximately /1.723 ~ 1.3 times larger than the one of LMS if their values of yz are set according to Eqs (6.47) and (6.21) respectively, so that their asymptotic MSE are minimized.
Regarding the MSD transient performance, Fig 6.3 shows the MSD learning curves for LMS and LMS/Newton with their ”s set to their optimum values according to Eqs (6.27) ®The initial Wiener solution w% had also a low-pass spectrum, although this is irrelevant for the steady- state behavior The LMS MSE
0.109 E Ị Dee eee ee L AAAù.>“>._aạnnẴnớ J Dee rr | penne T Seepene ne Tree.
0.107 ee vo : ee ear at : ee ee ed h ơ | oiosL SUE bu iu