An introduction to statistical signal processing last edition (robert gray)

An Introduction to Statistical Signal Processing f −1 (F ) f F Pr(f ∈ F ) = P ({ω : ω ∈ F }) = P (f −1 (F )) ✲ September 21, 2003 ii An Introduction to Statistical Signal Processing Robert M Gray and Lee D Davisson Information Systems Laboratory Department of Electrical Engineering Stanford University and Department of Electrical Engineering and Computer Science University of Maryland iv c 1999–2003 by the authors v to our Families vi Contents Preface xi Glossary xv Introduction Probability 2.1 Introduction 2.2 Spinning Pointers and Flipping Coins 2.3 Probability Spaces 2.3.1 Sample Spaces 2.3.2 Event Spaces 2.3.3 Probability Measures 2.4 Discrete Probability Spaces 2.5 Continuous Probability Spaces 2.6 Independence 2.7 Elementary Conditional Probability 2.8 Problems 11 11 15 23 28 31 42 45 55 69 70 74 Random Objects 3.1 Introduction 3.1.1 Random Variables 3.1.2 Random Vectors 3.1.3 Random Processes 3.2 Random Variables 3.3 Distributions of Random Variables 3.3.1 Distributions 3.3.2 Mixture Distributions 3.3.3 Derived Distributions 3.4 Random Vectors and Random Processes 3.5 Distributions of Random Vectors 85 85 85 89 93 95 104 104 108 111 115 117 vii viii CONTENTS 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.5.1 Multidimensional Events 3.5.2 Multidimensional Probability Functions 3.5.3 Consistency of Joint and Marginal Distributions Independent Random Variables Conditional Distributions 3.7.1 Discrete Conditional Distributions 3.7.2 Continuous Conditional Distributions Statistical Detection and Classification Additive Noise Binary Detection in Gaussian Noise Statistical Estimation Characteristic Functions Gaussian Random Vectors Simple Random Processes Directly Given Random Processes 3.15.1 The Kolmogorov Extension Theorem 3.15.2 IID Random Processes 3.15.3 Gaussian Random Processes Discrete Time Markov Processes 3.16.1 A Binary Markov Process 3.16.2 The Binomial Counting Process 3.16.3 Discrete Random Walk 3.16.4 The Discrete Time Wiener Process 3.16.5 Hidden Markov Models Nonelementary Conditional Probability Problems Expectation and Averages 4.1 Averages 4.2 Expectation 4.3 Functions of Random Variables 4.4 Functions of Several Random Variables 4.5 Properties of Expectation 4.6 Examples 4.6.1 Correlation 4.6.2 Covariance 4.6.3 Covariance Matrices 4.6.4 Multivariable Characteristic Functions 4.6.5 Differential Entropy of a Gaussian Vector 4.7 Conditional Expectation 4.8 Jointly Gaussian Vectors 4.9 Expectation as Estimation 118 119 120 127 129 130 132 134 137 144 146 147 152 154 157 157 158 158 159 159 162 165 166 167 168 170 189 189 192 195 202 202 204 204 207 208 209 211 213 216 219 CONTENTS 4.10 4.11 4.12 4.13 4.14 4.15 4.16 4.17 4.18 4.19 4.20 Implications for Linear Estimation Correlation and Linear Estimation Correlation and Covariance Functions The Central Limit Theorem Sample Averages Convergence of Random Variables Weak Law of Large Numbers Strong Law of Large Numbers Stationarity Asymptotically Uncorrelated Processes Problems ix Second-Order Theory 5.1 Linear Filtering of Random Processes 5.2 Linear Systems I/O Relations 5.3 Power Spectral Densities 5.4 Linearly Filtered Uncorrelated Processes 5.5 Linear Modulation 5.6 White Noise 5.6.1 Low Pass and Narrow Band Noise 5.7 Time-Averages 5.8 Mean Square Calculus 5.8.1 Mean Square Convergence Revisited 5.8.2 Integrating Random Processes 5.8.3 Linear Filtering 5.8.4 Differentiating Random Processes 5.8.5 Fourier Series 5.8.6 Sampling 5.8.7 Karhunen-Loueve Expansion 5.9 Linear Estimation and Filtering 5.9.1 Discrete Time 5.9.2 Continuous Time 5.10 Problems A Menagerie of Processes 6.1 Discrete Time Linear Models 6.2 Sums of IID Random Variables 6.3 Independent Stationary Increment Processes 6.4 Second-Order Moments of ISI Processes 225 227 234 238 240 242 249 251 255 261 263 285 286 288 294 296 302 305 309 309 313 314 320 323 325 328 332 336 342 342 355 359 377 378 382 384 387 x CONTENTS 6.5 Specification of Continuous Time ISI Processes 6.6 Moving-Average and Autoregressive Processes 6.7 The Discrete Time Gauss-Markov Process 6.8 Gaussian Random Processes 6.9 The Poisson Counting Process 6.10 Compound Processes 6.11 Exponential Modulation 6.12 Thermal Noise 6.13 Ergodicity 6.14 Problems 389 391 393 394 395 398 400 405 407 411 A Preliminaries A.1 Set Theory A.2 Examples of Proofs A.3 Mappings and Functions A.4 Linear Algebra A.5 Linear System Fundamentals A.6 Problems 423 423 431 435 436 439 444 B Sums and Integrals B.1 Summation B.2 Double Sums B.3 Integration B.4 The Lebesgue Integral 451 451 454 455 457 C Common Univariate Distributions 461 D Supplementary Reading 463 Bibliography 468 Index 473 Bibliography [1] R B Ash Real Analysis and Probability Academic Press, New York, 1972 [2] E Asplund and L Bungart A First Course in Integration Holt,Rinehart and Winston, New York, 1966 [3] D.T Bertsekas and J.N Tsitsiklis Introduction to Probability Athena Scientific, Boston, 2002 [4] P Billingsley Ergodic Theory and Information Wiley, New York, 1965 [5] A G Bose and K N Stevens Introductory Network Theory Harper & Row, New York, 1965 [6] R Bracewell The Fourier Transform and Its Applications McGrawHill, New York, 1965 [7] L Breiman Probability Addison-Wesley, Menlo Park, CA, 1968 [8] C T Chen Introduction to Linear System Theory Holt, Rinehart and Winston, New York, 1970 [9] K L Chung Markov Chains with Stationary Transition Probabilities Springer-Verlag, New York, 1967 [10] K L Chung A Course in Probability Theory Academic Press, New York, 1974 [11] T M Cover, P Gacs, and R M Gray Kolmogorov’s contributions to information theory and algorithmic complexity Ann Probab., 17:840– 865, 1989 [12] T.M Cover and J A Thomas Elements of Information Theory Wiley, New York, 1991 469 470 BIBLIOGRAPHY [13] H Cram´er and M R Leadbetter Stationary and Related Stochastic Processes Wiley, New York, 1967 [14] W B Davenport and W L Root An Introduction to the Theory of Random Signals and Noise McGraw-Hill, New York, 1958 [15] J L Doob Stochastic Processes Wiley, New York, 1953 [16] A W Drake Fundamentals of Applied Probability Theory McGrawHill, San Francisco, 1967 [17] L E Dubins and L J Savage Inequalities for Stochastic Processes: How to Gamble If You Must Dover, New York, 1976 [18] E B Dynkin Markov Processes Springer-Verlag, New York, 1965 [19] W Feller An Introduction to Probability Theory and its Applications, volume Wiley, New York, 1960 3rd ed [20] T Fine Properties of an optimal digital system and applications IEEE Trans Inform Theory, 10:287–296, Oct 1964 [21] R Gagliardi Introduction to Communications Engineering Wiley, New York, 1978 [22] I I Gikhman and A V Skorokhod Introduction to the Theory of Random Processes Saunders, Philadelphia, 1965 [23] B V Gnedenko The Theory of Probability Chelsea, New York, 1963 Translated from the Russian by B D Seckler [24] B V Gnedenko and A Ya Khinchine An Elementary Introduction to the Theory of Probability Dover, New York, 1962 Translated from the 5th Russian edition by L F Boron [25] R M Gray Toeplitz and circulent matrices: II ISL technical report no 6504–1, Stanford University Information Systems Laboratory, April 1977 Revised and updated many times The most recent version is available at http:/ee.stanford.edu/~gray/toeplitz.html [26] R M Gray Probability, Random godic Properties Springer-Verlag, http://ee.stanford.edu/arp.html Processes, and ErNew York, 1988 [27] R M Gray and J G Goodman Fourier Transforms Kluwer Academic Publishers, Boston, Mass., 1995 BIBLIOGRAPHY 471 [28] R M Gray and J C Kieffer Asymptotically mean stationary measures Ann Probab., 8:962–973, 1980 [29] U Grenander and M Rosenblatt Statistical Analysis of Stationary Time Series Wiley, New York, 1957 [30] U Grenander and G Szego Toeplitz Forms and Their Applications University of California Press, Berkeley and Los Angeles, 1958 [31] P R Halmos Measure Theory Van Nostrand Reinhold, New York, 1950 [32] P R Halmos Lectures on Ergodic Theory Chelsea, New York, 1956 [33] T Hida Stationary Stochastic Processes Princeton University Press, Princeton, NJ, 1970 [34] D Huff and I Geis How to Take a Chance W W Norton, New York, 1959 [35] T Kailath Linear Systems Prentice-Hall, Englewood Cliffs, NJ, 1980 [36] T Kailath Lectures on Wiener and Kalman Filtering CISM Courses and Lectures No 140 Springer-Verlag, New York, 1981 [37] J G Kemeny and J L Snell Finite Markov Chains D Van Nostrand, Princeton, NJ, 1960 [38] A N Kolmogorov Foundations of the Theory of Probability Chelsea, New York, 1950 [39] J Lamperti Stochastic Processes: A Survey of the Mathematical Theory Springer-Verlag, New York, 1977 [40] R S Liptser and A N Shiryayev Statistics of Random Processes Springer-Verlag, New York, 1977 Translated by A B Aries [41] M Loeve Probability Theory D Van Nostrand, Princeton, NJ, 1963 Third Edition [42] E Lukacs Stochastic Convergence Heath, Lexington, MA, 1968 [43] L E Maistrov Probability Theory: A Historical Sketch Academic Press, New York, 1974 Translated by S Kotz [44] H P McKean, Jr Stochastic Integrals Academic Press, New York, 1969 472 BIBLIOGRAPHY [45] J Neveu Discrete-Parameter Martingales North-Holland, New York, 1975 Translated by T P Speed [46] J R Newman The World of Mathematics, volume Simon & Schuster, New York, 1956 [47] A Papoulis The Fourier Integral and Its Applications McGraw-Hill, New York, 1962 [48] A Papoulis Signal Analysis McGraw-Hill, New York, 1977 [49] K R Parthasarathy Probability Measures on Metric Spaces Academic Press, New York, 1967 [50] E Parzen Stochastic Processes Holden Day, San Francisco, 1962 [51] M Rosenblatt Markov Processes: Structure and Asymptotic Behavior Springer-Verlag, New York, 1971 [52] H L Royden Real Analysis Macmillan, London, 1968 [53] Yu A Rozanov Stationary Random Processes Holden Day, San Francisco, 1967 Translated by A Feinstein [54] W Rudin Principles of Mathematical Analysis McGraw-Hill, New York, 1964 [55] G F Simmons Introduction to Topology and Modern Analysis McGraw-Hill, New York, 1963 [56] K Steiglitz An Introduction to Discrete Systems Wiley, New York, 1974 [57] A A Sveshnikov Problems in Probability Theory, Mathematical Statistics,and Theory of Random Functions Dover, New York, 1968 [58] Jr W.B Davenport Mean-square convergence M.I.T Department of Electrical Engineering 6.573 mimeographed notes, February 1965 [59] P Whittle Probability Penguin Books, Middlesex,England, 1970 [60] N Wiener The Fourier Integral and Certain of Its Applications Cambridge University Press, New York, 1933 [61] N Wiener Time Series: Extrapolation,Interpolation,and Smoothing of Stationary Time Series with Engineering Applications M I T Press, Cambridge, MA, 1966 BIBLIOGRAPHY 473 [62] N Wiener and R.E.A.C Paley Fourier Transforms in the Complex Domain Am Math Soc Coll Pub., Providence, RI, 1934 [63] E Wong Introduction to Random Processes Springer-Verlag, New York, 1983 [64] A M Yaglom An Introduction to the Theory of Stationary Random Functions Prentice-Hall, Englewood Cliffs, NJ, 1962 Translated by R A Silverman Index L2 , 314 Φ function, 63 δ-response, 102, 286 area, 11 arithmetic average, 101 ARMA, 381 ARMA random process, 382 asymptotic equipartiton property, 275 asymptotically mean stationary, 465 asymptotically uncorrelated, 261, 262 autocorrelation function, 235 autocorrelation matrix, 230 autoregressive, 299, 391 binary, 160 autoregressive filter, 380 autoregressive random process, 382 average, 189 probabilistic, 47 statistical, 47 average power, 294 axioms, 18 axioms of probability, 25 C´esaro mean, 101 a.m.s., 465 absolute moment, 56 centralized, 196 absolute moments, 196 abstract space, 423 addition, 315 modulo 2, 135 additive finite, 43 additive Gaussian noise, 140 additive noise, 137 continuous, 139 discrete, 137 additivity, 18, 43 countable, 43 finite, 18 affine, 210, 223, 228, 229 algebra, 24 almost everywhere, 110 alphabet, 104 continuous, 116 discrete, 116 mixed, 116 amplitude continuous, 116 discrete, 116 amplitude modulation (AM), 305 Balakrishnan, A.B., 308 Banach space, 317 Banach space, 317 baseband noise, 309 Bayes risk, 136 Bayes’ rule, 131, 133, 136 Bernoulli process, 94, 158 binary autoregressive filter, 365 binary detection, 144 binary filters, 364 474 INDEX binary Markov process, 159, 301 binary pmf, 48 binary random variable, 200 binomial, 461 binomial coefficient, 461 binomial counting process, 162, 163, 383 binomial pmf, 48 binomial theorem, 165 bit, 187 Bochner’s theorem, 295 Bonferoni inequality, 77 Borel field, xv, 36, 37 Borel sets, 37 Borel space, 55 Borel-Cantelli lemma, 252 Bose-Einstein distribution, 413 Brownian motion, 240 calculus of probability, 105 Cartesian product, 29 categorical, 424 Cauchy sequence, 317 Cauchy-Schwartz inequality, 248, 268, 316, 319 Cauchy-Schwarz inequality, 208 causal, 444 cdf, 81, 107, 119 central limit theorem, 201, 238, 240 chain rule, 131, 162 channel noisy, 135 channel with additive noise, 413 chaotic, 184 characteristic function, xvi, 137, 147, 150, 199 Gaussian random variable, 151 Gaussian random vector, 153 multivariable, 209 Chernoff inequality, 254 chi-squared, 113 475 classification statistical, 136 closed interval, 28 coin flip, 19 coin flipping, 49, 86, 124 collectively exhaustive, 435 complement, 428 complementary Phi function, xvi complementation, 428 complete, 76 complete metric space, 317 complete the square, 126, 140 completecomplete the square the square, 151 completeness of L2 , 318 completing the square, 141 completion, 76 complex valued random vector, 115 complex-valued random variables, 314 composite function, 100 compound process, 398 conditional differential entropy, 275 conditional distribution conditional, 130 continuous, 132 conditional distributions, 129 conditional expectation, 213 conditional mean, 142 conditional pmf, 130 conditional probability, 70 nonelementary, 168, 170 conditional variance, 142 consistency, 16, 91, 120, 121 consistent, 91, 157 continuity, 44 continuity from above, 45 continuity from below, 45 continuous, 436 continuous probability spaces, 55 continuous space, 28 continuous time, 116 476 converge in mean square, 244 convergence almost everywhere, 243 almost surely, 243 mean square, 244 pointwise, 243 w.p 1, 243 with probability one, 243 convergence in distribution, 239, 243 convergence in mean square, 244 convergence in probability, 244 convergence of correlation, 318 convergence of expectation, 318 convergence of random variables, 242 convergence with probability 1, 251 convolution, 440 discrete, 138 modulo 2, 138 sum, 138 coordinate function, 100 correlation, 205 correlation coefficient, 133, 277 cosine identity, 304 cost, 136 countable, 434 counting process, 163 binomial, 162 covariance, 207 covariance function, 235 covariance matrix, 209 cross correlation, 268 cross-correlation, 346 cross-covariance, 217 cross-spectral density, 346 cumalitive distribitution function, 119 cumulative distribution function, 81, 107 dc response, 289 INDEX decision rule, 135 decreasing sets, 33 delta Dirac, 65 DeMorgan’s law, 31, 433 density mass, 11 dependent, 92 derived distribution, 21, 88, 111 detection, 135 determinant, 211 dice, 23, 125 difference symmetric, 430 differential entropy, 212 differentiating random processes, 325 Dirac delta, 65, 293 directly given, 21 discrete spaces, 28 discrete time, 93, 116 discrete time integrator, 383 discrete time signals, 29 discrete time Wiener process, 166 disjoint, 17, 428, 434 distance, 77, 315 distribution, 65, 87, 104, 105, 117 convergence in, 239 infinitely divisible, 240 joint, 122 marginal, 122 stable, 278 distributions conditional, 129 domain, 435 domain of definition, 21 dominated convergence theorem, 204 dot product, 437 double convolution, 292 doubly exponential, 462 doubly stochastic, 409 INDEX eigenfunction, 338 eigenvalue, 338, 438 eigenvector, 438 element, 423 elementary events, 23 elementary outcomes, 23 empty set, xv, 19, 427 equivalent random variables, 89 ergodic decomposition, 410 ergodic theorem, 192, 261, 263 ergodic theorems, 189 ergodicity, 407 error, mean squared, 219 estimate minimum mean squared error, 222 estimation, 146, 219 linear, 342 maximum a posteriori, 146 recursive, 350 statistical, 146 estimator optimal, 221 Euclidean space, 29 even, 195 event, 24 event space, 12, 24, 31 trivial, 27 events elementary, 23 exclusive or, 135 expectation, 7, 46, 56, 189, 191, 193 conditional, 213 fundamental theorem of, 197 iterated, 214 nested, 214 expected value, 192 experiment, 6, 12, 18, 26 exponential, 462 exponential modulation, 400 exponential pdf, 62 477 exponential random variable, 194 field, 24 filter, 159 autoregressive, 380 finite impulse response, 287, 378 infinite impulse response, 378 linear, 440 moving average, 378 transversal, 378 filtered Poisson process, 421 filters binary, 364 finest-grain, 28 finite additivity, 18 finite impulse response (FIR), 287 finite impulse response filter, 378 FIR, 287, 378 Fourier Series, 328 Fourier transform, 148, 150, 200, 292, 312, 379 frequency domain, 293 frequency modulation (FM), 400, 401 function, 435 identity, 47 measurable, 98 functions of random variables, 195 functions or random variables, 202 fundamental theorem of expectation, 197 Galois field, 134 Gamma, 462 Gauss-Markov process, 393 Gaussian, 462 jointly, 216 multidimensional, 67 Gaussian pdf, 62, 200 Gaussian process, 324 Gaussian random process, 158, 236, 394 478 Gaussian random vector, 153, 216 Gaussian random vectors, 152 generating event spaces, 37 geometric, 461 geometric random variable, 194 Gram-Schmidt, 339 half-open interval, 28 Hamming weight, 55, 160 hard limiter, 99 Hermitian, 226, 236, 437–439 hidden Markov model, 167 Hilbert space, 317 identically distributed, 89 identity function, 47 identity mapping, 88 iid, 94, 129, 158 IIR, 378 image, 435 impulse response, 440 increasing sets, 33 increments, 385 independent, 385 stationary, 385 independence, 69, 127 linear, 205 independent, 127, 206 independent and stationary increments, 384 independent identically distributed, 94, 129 independent increment process, independent increments, 385 independent random variables, 127 indexed family, 116 indicator function, xvi, 16, 46, 198 induction, 50 inequality Tchebychev, 245 infinite countably, 434 INDEX infinite impulse response filter, 378 infinitely divisible, 240 inner product, 317, 437 inner product space, 317 input event, 96 input signal, xi, 2, 86 integer sets, xvi integral Lebesgue, 57, 457 Riemann, 320 intersection, 428 interval, xv, 28, 427 closed, xv, 427 half-closed, 427 half-open, 427 open, xv, 427 intervals, 36 inverse image, 87, 435 inverse image formula, 88 isi, 385 iterated expectation, 214, 215 Jacobian, 114 joint distribution, 122 jointly Gaussian, 152 Kalman-Bucy filtering, 350 Karhunen-Loeve expansion, 337 Kolmogorov extension theorem, 94, 157 Kolmogorov, A.N., 23, 43, 94 Kronecker delta response, 102 Kronecker delta response, 443 Laplace transform, 150, 200 Laplace transfrom, 150 Laplacian, 462 Laplacian pdf, 61 law of large numbers, 189, 192 strong, 255 laws of large numbers, Lebesgue integral, 9, 59, 457 limit in the mean, 244 INDEX limits of probabilities, 43 limits of sets, 33 linear, 228, 440 linear estimation, 225, 342 linear functional equation, 388 linear least squares error, 227 linear models, 378, 382 linear modulation, 302 linearity of expectation, 203 linearly filtered uncorrelated processes, 296 linearly independent, 339 Lloyd-Max quantizer, 281 logistic, 462 low band noise, 309 MAP, 136 MAP detector, 144 MAP estimation, 146 mapping, 435 marginal pmf, 91 Markov chain, 162 Markov inequality, 246 Markov process, 159, 162, 165, 167 continuous time, 391 martingale, 280 mass, 11 matrix, 437 Hermitian, 437 max function, 175 maximum a posteriori, 136 maximum a posteriori estimation, 146 maximum likelihood estimation, 147 mean, 47, 56, 101, 192 conditional, 142 mean ergodic theorem, 249, 261, 263 mean function, 158 mean square, 244 mean square convergence, 314 mean square integral, 321 479 mean square integration, 320 mean square sampling theorem, 334 mean squared error, 219, 221 mean value theorem, 106 mean vector, 208 measurable, 98 measurable space, 25 measure probability, 42 measure theory, 11 memoryless, 159 Mercer’s theorem, 340 metric, 77, 314, 315 metric space, 315 function, 175 minimum distance, 145 minimum mean square estimate, 223 mixture, 69, 108, 409 mixture probability, 68 ML estimator, 147 MMSE, 220 modulation nonlinear, 278 modulo addition, 135 modulo arithmetic, 134, 160 moment kth, 47, 56 centralized, 196 first, 47, 56 second, 56, 208 moment generating function, 200 moments, 196, 199 absolute, 196 centralized, 56 monotone convergence theorem, 204 monotonically decreasing, 114 monotonically increasing, 114 moving average, 280, 299, 378 moving average filter, 378 moving averate, 391 480 moving-average random process, 382 MSE, 219 multidimensional pdf’s, 66 multidimensional pmf’s, 54 multiplication, 315 mutually exclusive, 17, 428 mutually independent, 94 narrow band noise, 309 nested, 33 nested expectation, 214 noise white, 296, 305 noisy channel, 135 nonempty, 424 nonlinear modulation, 278 nonnegative definite, 153, 439 normed linear space, 315, 316 numeric, 424 odd, 195 olutput event, 96 one-sided, 29, 93, 440 one-step prediction, 223 one-to-one, 436 onto, 436 open interval, 28 operation, 146 optimal, 356 orhtogonal, 330 orthogonal, 233, 338, 344 orthogonality condition, 357 orthogonality principle, 230, 233, 234, 344 outer product, 437 output signal, xi, 2, 86 output space, 21 Paley-Wiener criteria, 308 Parceval’s theorem, 332 partition, 435 Pascal’s distribution, 264 INDEX pdf, 17, 60 k-dimensional, 67 chi-squared, 113 doubly exponential, 61, 462 elementary conditional, 73 exponential, 61, 62, 462 Gamma, 462 Gaussian, 61, 62, 462 Laplacian, 61, 462 logistic, 462 multidimensional, 66 Rayleigh, 462 uniform, 17, 61, 62, 462 Weibull, 462 phase modulation (PM), 400, 401 Phi function, xvi, 63 pmf, 20, 48 binary, 48, 461 binomial, 48, 461 conditional, 72, 130 Poisson, 462 product, 124 uniform, 48, 461 pmf:geometric, 461 point, 423 pointwise convergence, 243 Poisson, 462 Poisson counting process, 384, 391, 395, 396, 421 Poisson pmf, 53 positive definite, 68, 126, 153, 226, 439 power set, 35 power spectral density, 293–295 pre-Hilbert space, 317 prediction one-step, 223 predictor one-step, 222 optimal, 221 preimage, 435 probabilistic average, 192 INDEX probability a posteriori, 71 a priori, 71 conditional, 71 unconditional, 71 probability density function, 17, 60 probability mass function, 20, 48 probability measure, xvi, 12, 18, 25, 42 probability of error, 135 probability space, 6, 11, 12, 18, 23, 26 complete, 76 trivial, 27 probability theory, 11 product pmf, 124 product space, 29, 426 product spaces, 39 projection, 100 pseudo-random number, 22 pulse amplitude modulation (PAM), 366 quadratic mean, 244 quantization, 156, 182 quantizer, 21, 99, 181, 447, 458 random object, 85 random phase process, 156 random proces Markov, 159 random process, 1, 93, 115, 116 ARMA, 382 autoregressive, 382 Bernoulli, 158 counting, 163 Gaussian, 158 iid, 158 isi, 385 Markov, 167 moving-average, 382 481 random rotations, 155 random telegraph wave, 400 random variable, 21, 46, 56, 85, 87, 95 complex-valued, 115 continuous, 110 discrete, 110 Gaussian, 98 mixture, 110 random variable, 97 random variables, complex valued, 314 equivalent, 89 independent, 127 random vector, 89, 90, 115 Gaussian, 152, 216 iid, 129 random walk, 165, 280 range, 21, 435 range space, 436 Rayleigh, 462 rectangles, 41 recursive estimation, 350 regression, 146 regression coefficients, 380 relative frequency, 23, 191 Riemann integral, 59, 320 Riemann integration, 264 sample autocorrelation, 297 sample average, 101, 240 sample points, 23 sample space, 12, 23, 28 sample value, 86 sampling expansion, 333 sampling function, 100 sampling interval, 333 sampling theorem mean square, 334 scalar, 315 second order moments, 387 482 second-order input/output relations, 285 sequence, 39 Cauchy, 317 sequence space, 30 set, 426 empty, 427 one-point, 427 singleton, 427 universal, 423 set difference, 430 set theory, 423 sets decreasing, 33 increasing, 33 Shannon, Claude, 275 shift, 256 sigma-algebra, 24 sigma-field, 12, 24, 31 sigma-field generated, 37 signal, 14, 439 continuous time, 30 discrete time, 29 signal processing, xi, 2, 14, 20, 21, 46 simple function, 458 single-sided, 440 singular, 153, 211 singular Gaussian distribution, 211 sliding average, 411 space, 423 continuous, 28 empty, 424 Hilbert, 317 inner product, 317 metric, 315 normed linear, 315 pre-Hilbert, 317 product, 29, 426 trivial, 424 spectrum, 312 INDEX spinning pointer, 15 square root, 226 square-integrable, 314 stable, 287, 441, 444 stable distribution, 278 standard deviation, 208 stationarity properties, 255 stationarity property, 256 stationary, 258–260 first order, 256 strict sense, 258 strictly, 258 weakly, 225, 256 stationary increments, 385 statistical classification, 134, 136 statistical detection, 134 statistical estimation, 146 Stieltjes integral, 108, 198 stochastic process, 1, 116 strict stationarity, 259 strong law of large numbers, 251, 255 subset, 426 sum of independent random variables, 151 sums of iid random variables, 382 superposition, 440 symmetric, 437 symmetric difference, 430 system, 439 continuous time, 440 discrete time, 440 linear, 440 tapped delay line, 378 Taylor series, 53, 200 Tchebychev inequality, 245 telescoping sum, 44 thermal noise, 22, 405 Thevinin’s theorem, 407 threshold detector, 145 time INDEX continuous, 116 discrete, 116 time average, 101, 309 time series, 93, 116 time-invariant, 440 Toeplitz, 257 Toeplitz matrix, 257, 454 transform Laplace, 150 transpose, 116 transversal filter, 378 trivial probability space, 27 trivial sigma-field, 103 two-sided, 29, 440 Tyche, 15 uncorrelated, 205, 206, 208, 235, 242 asymptotically, 261, 262 uncountable, 434 uniform, 461, 462 uniform pdf, 17, 48, 62 uniform random variable, 194 union, 428 union bound, 77 unit δ response, 102 unit impulse, 65 universal set, 423 variance, 47, 56, 196, 208 conditional, 142 vector, 39, 436 random, 115 volume, 11 waveform, 39 weak law of large numbers, 249 weakly stationary, 225, 256, 289 Weibull, 462 weight, 11 white noise, 296, 305, 341 Wiener process, 301, 383, 384 continuous time, 391 483 discrete time, 166 Wiener, Norbert, Wiener-Hopf equation, 342, 358 ... in statistical signal processing, communications, control, image and video processing, speech and audio processing, medical signal processing, geophysical signal processing, and classical statistical. .. taught for many years at Stanford University and at the University of Maryland using the book: An Introduction to Statistical Signal Processing Much of the basic content of this course and of the... intact and together Comments, corrections, and suggestions should be sent to rmgray@stanford.edu Every effort will be made to fix typos and take suggestions into an account on at least an annual