Adaptive Filtering and Change Detection Fredrik Gustafsson Copyright © 2000 John Wiley & Sons, Ltd ISBNs: 0-471-49287-6 (Hardback); 0-470-84161-3 (Electronic) Part Ill: Parameter estimation Adaptive Filtering and Change Detection Fredrik Gustafsson Copyright © 2000 John Wiley & Sons, Ltd ISBNs: 0-471-49287-6 (Hardback); 0-470-84161-3 (Electronic) Adaptive filtering 5.1 Basics 5.2 Signal models 5.2.1 Linear regression models 5.2.2 Pseudo-linear regression models 5.3.System identification 5.3.1 Stochasticanddeterministic least squares 5.3.2 Model structure selection 5.3.3 Steepest descent minimization 5.3.4 Newton-Raphson minimization 5.3.5 Gauss-Newton minimization 5.4.Adaptivealgorithms 5.4.1 LMS 5.4.2 RLS 5.4.3 Kalman filter 5.4.4 Connections andoptimal simulation 5.5.Performanceanalysis 5.5.1 LMS 5.5.2 RLS 5.5.3 Algorithm optimization 5.6.Whitenessbasedchangedetection 5.7 A simulationexample 5.7.1 Time-invariant AR model 5.7.2 Abruptly changing AR model 5.7.3 Time-varying AR model 5.8.Adaptive filters incommunication 5.8.1 Linear equalization 5.8.2 Decision feedback equalization 5.8.3 Equalizationusing theViterbialgorithm 5.8.4 Channelestimationin equalization 5.8.5 Blind equalization 5.9.Noisecancelation 5.9.1 Feed-forward dynamics 5.9.2 Feedback dynamics 114 115 115 119 121 121 124 126 127 128 133 134 138 142 143 144 145 147 147 148 149 150 150 151 153 155 158 160 163 165 167 167 171 AdaDtive 114 5.10.Applications 173 5.10.1.Human EEG 173 173 5.10.2.DCmotor 5.10.3 Frictionestimation 175 5.11 Speech coding in GSM 185 5.A Square root implementation 189 190 5.B Derivations 5.B.1 Derivation of LS algorithms 191 5.B.2 Comparingon-line and off-lineexpressions 193 199 5.B.3 Asymptotic expressions 5.B.4 Derivation of marginalization 200 5.1 Basics The signal model in this chapter is, in its most general form, The noise is here assumed white with variance X, and will sometimes be restricted to be Gaussian The last expression is in a polynomial form, whereas G, H are filters Time-variability is modeled by time-varying parameters Bt The adaptive filtering problem is to estimate these parametersby an adaptive filter, &+l = Qt + Kt% where E t is an application dependent error from the model We point out particular cases of (5.1) of special interest, but first, three archetypical applications will be presented: 0 Consider first Figure 5.l(a) The main approach to system identification is to run a model in parallel with the true system, and thegoal is to get F(B) M G See Section 5.3 The radio channel in a digital communication system is well described by a filter G(q;B) An important problem is to find an inverse filter, and this problem is depicted in Figure 5.l(b) This is also known as the inverse system identijication problem It is often necessary to include an overall delay The goal is to get F(B)G M 4-O In equalization, a feed-forward signal qPDut,called training sequence, is available during 5.2 Sianal models 115 a learning phase In blind equalization, no training signal is available The delay, as well as the order of the equalizer are design parameters Both equalization and blind equalization are treated in Section 5.8 The noise cancelation, or Acoustic Echo Cancelation (AEC), problem in Figure 5.l(c) is to remove the noise component in y = s+v by making use of an external sensor measuring the disturbance U in v = Gu The goal is to get F ( ) M G, so that d M S This problem is identical to system identification, which can be realized by redrawing the block diagram However, there are some particular twists unique for noise cancelation See Section 5.9 Literature There are many books covering the area of adaptive filtering Among those most cited, we mention Alexander (1986), Bellanger (1988), Benveniste et al (1987b), Cowan and Grant (1985),Goodwin and Sin (1984), Hayes (1996), Haykin (1996), C.R Johnson (1988), Ljung and Soderstrom (1983), Mulgrew and Cowan (1988), Treichler et al.(1987), Widrow and Stearns (1985) and Young (1984) Survey papers of general interest are Glentis et al.(1999), Sayed and Kailath (1994) and Shynk (1989) Concerning the applications, system identification is described in Johansson (1993), Ljung (1999) and Soderstrom and Stoica (1989), equalization in the books Gardner (1993), Haykin (1994), Proakis (1995) and survey paper Treichler et al (1996), and finally acoustic echo cancelation in the survey papers Breining et al (1999), Elliottand Nelson (1993) and Youhong and Morris (1999) 5.2 Signal models In Chapter 2, we have seen a number of applications that can be recast to estimating the parameters in linear regression models This section summarizes more systematically the different special cases of linear regressions and possible extensions 5.2.1 linear regressionmodels We here point out some common special cases of the general filter structure (5.1) that can be modeled as linear regression models, characterized by a regression vector V t and a parameter vector 19 The linear regression is defined AdaDtive 116 (a) System identification The goal is to get a perfect system model F ( ) = G Training (b) Equalization The goal is to get a perfectchannel inverse F ( ) = GP', in which case the transmitted information is perfectly recovered (c) Noise cancelation The goal is to get a perfect model F ( ) = G of the acoustic path from disturbance to listener Figure 5.1 Adaptive filtering applications 5.2 Sianal models 117 The measurement yt is assumed to be scalar, and a possible extension to the multi-variable case is given at the end of this section The most common model in communication applications is the Finite Impulse Response ( F I R ) model: To explicitely include the model order, FIR(n) is a standard shorthand notation It is natural to use this model for communication channels,where echoes give rise to the dynamics It is also the dominating model structure in realtime signal processing applications, such as equalization and noise cancelling Example 5.7 Multi-pathfading In mobile communications, multi-path fading is caused by reflections, or echoes, in the environment, This specular multi-path is illustrated in Figure 5.2 Depending upon where the reflections occur, we get different phenomena: 0 Local scattering occurs near the receiver Here the difference in arrival time of the different rays is less than the symbol period, which means that no dynamic model can describe the phenomenon in discrete time Instead, the envelope of the received signal is modeled as a stochastic variable with Rayleigh distribution or Rice distribution The former distribution arises when the receiver is completely shielded from the transmitter, while the latter includes the effect of a stronger direct ray The dynamical effects of this 'channel' are much faster than the symbol frequency and imply a distortion of the waveform This phenomonen is called frequency selective fading Near-field scattering occurs at intermediate distance between the transmitter and receiver Here the difference in arrival time of the different rays is larger than the symbol period, so a discrete time echo model can be used to model the dynamic behaviour First, in continuous time the scattering can be modeled as AdaDtive 118 Figure 5.2 Multi-path fading is caused by reflections in environment where q are real-valued time delays rather than multiples of a sample interval This becomes a FIR model after sampling to discrete time Far-field scattering occurs close to the transmitter Thereceived rays can be treated as just one ray Good surveys of multi-path fading are Ahlin and Zander (1998) and Sklar (1997), while identification of such a radio channel is described in Newson and Mulgrew (1994) For modeling time series, an Auto-Regressive ( A R ) model is often used: (5.9) (Pt = (-%l, l9 ,-Yt-n) = (a1, a2, , a J T T (5.10) (5.11) AR(n) is a shorthand notation This is a flexible structure for many real-world signals like speech signals (Section 2.4.4), seismical data (Section 2.4.3) and biological data (Sections 2.4.1 and 2.4.2) One particular application is to use the model for spectral analysis as an alternative to transform based methods Example 5.2 Speechmodeling Speech is generated in three different ways Voiced sound, like all vowels and ‘m’, is originating in the vocal chord In signal processing terms, thevocal cord generates pulses which are modulatedin the throat and mouth.Unvoiced 5.2 Sianal models 119 sound, like ’S’ and ’v’, is a modulated air stream, where the air pressure from the lungs can be modeled as white noise Implosive sound, like ‘k’ and ‘b’, is generated by building up an air pressure which is suddenly released In all three cases, the human vocal system can be modeled as a series of cylinders and an excitation source (the ‘noise’ et) which is either a pulse train, white noise or a pulse Each cylinder can be representedby a second order AR model, which leads to a physical motivation of why AR models are suitable for speech analysis and modeling Time-variability in the parameters is explained by the fact that thespeaker is continuously changingthe geometryof the vocal tract Incontrol andadaptivecontrol, where there is a known controlsignal U available and where e represents measurement noise, the Auto-Regressive model with eXogenous input (ARX) is common: (5.12) (5.13) ARX(n,,nb,n,) is acompactshorthandnotation.Thisstructure doesnot follow in a straightforward way from physical modeling, but is rather a rich structure whose main advantage is that there aresimple estimation algorithms for it 5.2.2 Pseudo-linearregressionmodels In system modeling, physical arguments often lead to the deterministic signal part of the measurements being expressed as a linear filter, The main difference of commonly used model structures is how and where the noise enters the system Possible model structures, that not exactly fit the linear regression framework, are ARMA, OE and ARMAX models These can be expressed as a pseudo-linear regression, where the regressor p t ( ) depends on the parameter AdaDtive 120 The AR model has certain shortcomings for some other real world signals that are less resonant Then the Auto-RegressiveMovingAverage ( A R M A ) model might be better suited, (5.16) The Output Error ( O E ) model, which is of the Infinite Impulse Response(IIR) type, is defined as additive noise to the signal part (5.19) H ( q ; )= Pt(6) = (-Yt-1 (5.20) + et-l, , = ( f l ,f , , fn,, - yt-,, + et-,,, h , b2, ,b , y ut-1, , ~ t - , ~ ) ~ (5.21) (5.22) Note that the regressor contains the noise-free output, which can be written yt - et That is, the noise never enters the dynamics The OE models follow naturally fromphysical modeling of systems, assumingonly measurement noise as stochastic disturbance For modeling systems where the measurement noise is not white but still more correlated than that described by an ARX model, an Auto-Regressive Moving Average model with eXogenous input ( A R M A X ) model is often used: (5.23) (5.24) This model has found a standard application in adaptive control The common theme in ARMA, ARMAX and OE models is that they can be written as a pseudo-linear regression Yt + = 'Pt(Q)Q et, (5.27) where the regressor depends on the true parameters The parameter dependence comes from the fact that the regressor is a function of the noise For an 5.3 Svstem identification 121 ARMA model, the regressor in (5.17) contains et, which can be computed as and similarly for ARMAX and OE models The natural approximation is to just plug in the latest possible estimate of the noise That is, replace et with the residuals E t , et = E t = 4 ; 8) C(q;8) yt, ~ This is the approach in the extended least squares algorithm described in the next section The adaptive algorithms and change detectors developed in the sequel are mainly discussed with respect to linear regression models However, they can be applied to OE, ARMA and ARMAX as well, with the approximation that the noise et is replaced by the residuals Multi-InputMulti-Output (MIMO) models are usually considered to be built up as ng X nu independent models, where ny = dim(y) and nu = dim(u), one from each inputto each output MIMO adaptivefilterscanthusbe considered as a two-dimensional array of Single-Input Single-Output (SISO) adaptive filters 5.3 Systemidentification This section overviews and gives some examples of optimization algorithms used in system identification in general As it turns out, these algorithms are fundamental for the understanding and derivation of adaptive algorithms as well 5.3.1 Stochastic and deterministic least squares The algorithms will be derived from a minimization problem Let ) = Yt - bt = Yt - pp (5.28) Least squares optimization aimsat minimizing a quadratic loss function V(e), = argminV(0) e The (generally unsolvable) adaptive filtering problem canbe stated asminimizing the loss function v(e)= &;(e) (5.29) .. .Adaptive Filtering and Change Detection Fredrik Gustafsson Copyright © 2000 John Wiley & Sons, Ltd ISBNs: 0-471-49287-6 (Hardback); 0-470-84161-3 (Electronic) Adaptive filtering... filters Time-variability is modeled by time-varying parameters Bt The adaptive filtering problem is to estimate these parametersby an adaptive filter, &+l = Qt + Kt% where E t is an application dependent... dim(u), one from each inputto each output MIMO adaptivefilterscanthusbe considered as a two-dimensional array of Single-Input Single-Output (SISO) adaptive filters 5.3 Systemidentification This