ADAPTIVE FILTER THEORY

Transcription

1 ADAPTIVE FILTER THEORY Fifth Edition Simon Haykin Communications Research Laboratory McMaster University Hamilton, Ontario, Canada International Edition contributions by Telagarapu Prabhakar Department of Electronics and Communication Engineering GMR Institute of Technology Rajam, Andhra Pradesh, India Upper Saddle Ri' Indianapolis Lone Dubai Madrid Hong Ki JLB Darmstadt llllllllllllllllllllll :isco New York Tokyo Montreal Amsterdam Cape Town

2 Contents Preface 10 Acknowledgments 16 Background and Preview The Filtering Problem Linear Optimum Filters Adaptive Filters Linear Filter Structures Approaches to the Development of Linear Adaptive Filters Adaptive Beamforming Four Classes of Applications Historical Notes 38 Chapter 1 Stochastic Processes and Models Partial Characterization of a Discrete-Time Stochastic Process Mean Ergodic Theorem Correlation Matrix Correlation Matrix of Sine Wave Plus Noise Stochastic Models Wold Decomposition Asymptotic Stationarity of an Autoregressive Process Yule-Walker Equations Computer Experiment: Autoregressive Process of Order Two Selecting the Model Order Complex Gaussian Processes Power Spectral Density Properties of Power Spectral Density Transmission of a Stationary Process Through a Linear Filter Cramer Spectral Representation for a Stationary Process Power Spectrum Estimation Other Statistical Characteristics of a Stochastic Process Polyspectra Spectral-Correlation Density Summary and Discussion 102 Problems 103 Chapter 2 Wiener Filters Linear Optimum Filtering: Statement of the Problem Principle of Orthogonality 110 4

3 Contents Minimum Mean-Square Error Wiener-Hopf Equations Error-Performance Surface Multiple Linear Regression Model Example Linearly Constrained Minimum-Variance Filter Generalized Sidelobe Cancellers Summary and Discussion 140 Problems 142 Chapter 3 Linear Prediction Forward Linear Prediction Backward Linear Prediction Levinson-Durbin Algorithm Properties of Prediction-Error Filters Schur-Cohn Test Autoregressive Modeling of a Stationary Stochastic Process Cholesky Factorization Lattice Predictors All-Pole, All-Pass Lattice Filter Joint-Process Estimation Predictive Modeling of Speech Summary and Discussion 207 Problems 209 Chapter 4 Method of Steepest Descent Basic Idea of the Steepest :Descent Algorithm The Steepest-Descent Algorithm Applied to the Wiener Filter Stability of the Steepest-Descent Algorithm Example The Steepest-Descent Algorithm Viewed as a Deterministic Search Method Virtue and Limitation of the Steepest-Descent Algorithm Summary and Discussion 241 Problems 242 Chapter 5 Method of Stochastic Gradient Descent Principles of Stochastic Gradient Descent Application 1: Least-Mean-Square (LMS) Algorithm Application 2: Gradient-Adaptive Lattice Filtering Algorithm Other Applications of Stochastic Gradient Descent Summary and Discussion 263 Problems 264 Chapter 6 The Least-Mean-Square (LMS) Algorithm Signal-Flow Graph Optimality Considerations Applications Statistical Learning Theory Transient Behavior and Convergence Considerations Efficiency Computer Experiment on Adaptive Prediction Computer Experiment on Adaptive Equalization 311

4 6 Contents 6.9 Computer Experiment on a Minimum-Variance Distortionless-Response Beamformer Summary and Discussion 324 Problems 326 Chapter 7 Normalized Least-Mean-Square (LMS) Algorithm and Its Generalization Normalized LMS Algorithm: The Solution to a Constrained Optimization Problem 7.2 Stability of the Normalized LMS Algorithm Step-Size Control for Acoustic Echo Cancellation Geometric Considerations Pertaining to the Convergence Process for Real-Valued Data Affine Projection Adaptive Filters Summary and Discussion 352 Problems 353 Chapter 8 Block-Adaptive Filters Block-Adaptive Filters: Basic Ideas Fast Block LMS Algorithm Unconstrained Frequency-Domain Adaptive Filters Self-Orthogonalizing Adaptive Filters Computer Experiment on Adaptive Equalization Subband Adaptive Filters Summary and Discussion 393 Problems 394 Chapter 9 Method of Least-Squares Statement of the Linear Least-Squares Estimation Problem Data Windowing Principle of Orthogonality Revisited Minimum Sum of Error Squares Normal Equations and Linear Least-Squares Filters Time-Average Correlation Matrix $ Reformulation of the Normal Equations in Terms of Data Matrices Properties of Least-Squares Estimates Minimum-Variance Distortionless Response (MVDR) Spectrum Estimation Regularized MVDR Beamforming Singular-Value Decomposition Pseudoinverse Interpretation of Singular Values and Singular Vectors Minimum-Norm Solution to the Linear Least-Squares Problem Normalized LMS Algorithm Viewed as the Minimum-Norm Solution to an Underdetermined Least-Squares Estimation Problem Summary and Discussion 442 Problems 443 Chapter 10 The Recursive Least-Squares (RLS) Algorithm Some Preliminaries The Matrix Inversion Lemma The Exponentially Weighted RLS Algorithm Selection of the Regularization Parameter Updated Recursion for the Sum of Weighted Error Squares Example: Single-Weight Adaptive Noise Canceller Statistical Learning Theory 462

5 Contents Efficiency Computer Experiment on Adaptive Equalization Summary and Discussion 471 Problems 472 Chapter 11 Robustness Robustness, Adaptation, and Disturbances Robustness: Preliminary Considerations Rooted in H" Optimization Robustness of the LMS Algorithm Robustness of the RLS Algorithm Comparative Evaluations of the LMS and RLS Algorithms from the Perspective of Robustness Risk-Sensitive Optimality Trade-Offs Between Robustness and Efficiency Summary and Discussion 492 Problems 492 Chapter 12 Finite-Precision Effects Quantization Errors Least-Mean-Square (LMS) Algorithm Recursive Least-Squares (RLS) Algorithm Summary and Discussion 515 Problems 516 Chapter 13 Adaptation in Nonstationary Environments Causes and Consequences of Nonstationarity The System Identification Problem Degree of Nonstationarity Criteria for Tracking Assessment Tracking Performance of the LMS Algorithm Tracking Performance of the RLS Algorithm Comparison of the Tracking Performance of LMS and RLS Algorithms Tuning of Adaptation Parameters Incremental Delta-Bar-Delta (IDBD) Algorithm Autostep Method Computer Experiment: Mixture of Stationary and Nonstationary Environmental Data Summary and Discussion 552 > Problems 553 Chapter 14 Kalman Filters Recursive Minimum Mean-Square Estimation for Scalar Random Variables Statement of the Kalman Filtering Problem The Innovations Process Estimation of the State Using the Innovations Process Filtering Initial Conditions Summary of the Kalman Filter Optimality Criteria for Kalman Filtering Kalman Filter as the Unifying Basis for RLS Algorithms Covariance Filtering Algorithm Information Filtering Algorithm Summary and Discussion 589 Problems 590

6 8 Contents Chapter 15 Square-Root Adaptive Filtering Algorithms Square-Root Kalman Filters Building Square-Root Adaptive Filters on the Two Kalman Filter Variants QRD-RLS Algorithm Adaptive Beamforming Inverse QRD-RLS Algorithm Finite-Precision Effects Summary and Discussion 620 Problems 621 Chapter 16 Order-Recursive Adaptive Filtering Algorithm Order-Recursive Adaptive Filters Using Least-Squares Estimation: An Overview Adaptive Forward Linear Prediction Adaptive Backward Linear Prediction Conversion Factor Least-Squares Lattice (LSL) Predictor Angle-Normalized Estimation Errors First-Order State-Space Models for Lattice Filtering QR-Decomposition-Based Least-Squares Lattice (QRD-LSL) Filters Fundamental Properties of the QRD-LSL Filter Computer Experiment on Adaptive Equalization Recursive (LSL) Filters Using A Posteriori Estimation Errors Recursive LSL Filters Using A Priori Estimation Errors with Error Feedback Relation Between Recursive LSL and RLS Algorithms Finite-Precision Effects Summary and Discussion 685 Problems 687 Chapter 17 Blind Deconvolution Overview of Blind Deconvolution Channel Identifiability Using Cyclostationary Statistics Subspace Decomposition for Fractionally Spaced Blind Identification Bussgang Algorithm for Blind Equalization Extension of the Bussgang Algorithm to Complex Baseband Channels Special Cases of the Bussgang Algorithm Fractionally Spaced Bussgang Equalizers Estimation of Unknown Probability Distribution Function of Signal Source Summary and Discussion 745 Problems 746 Epilogue Robustness, Efficiency, and Complexity Kernel-Based Nonlinear Adaptive Filtering 753 Appendix A Theory of Complex Variables 770 A.l Cauchy-Riemann Equations 770 A.2 Cauchy's Integral Formula 772 A.3 Laurent's Series 774 A.4 Singularities and Residues 776 A.5 Cauchy's Residue Theorem 777 A.6 Principle of the Argument 778 A.7 Inversion Integral for the z-transform 781 A.8 Parseval's Theorem 783 /

7 Appendix B Wirtinger Calculus for Computing Complex Gradients 785 B.l Wirtinger Calculus: Scalar Gradients 785 B.2 Generalized Wirtinger Calculus: Gradient Vectors 788 B.3 Another Approach to Compute Gradient Vectors 790 B.4 Expressions for the Partial Derivatives df/dz and df/dz* 791 Appendix C Method of Lagrange Multipliers 792 C.l Optimization Involving a Single Equality Constraint 792 C.2 Optimization Involving Multiple Equality Constraints 793 C.3 Optimum Beamformer 794 Appendix D Estimation Theory 795 D.l Likelihood Function 795 D.2 Cramer-Rao Inequality 796 D.3 Properties of Maximum-Likelihood Estimators 797 D.4 Conditional Mean Estimator 798 Appendix E Eigenanalysis 800 E.l The Eigenvalue Problem 800 E.2 Properties of Eigenvalues and Eigenvectors 802 E.3 Low-Rank Modeling 816 E.4 Eigenfilters 820 E.5 Eigenvalue Computations 822 Appendix F Langevin Equation of Nonequilibrium Thermodynamics 825 F.l Brownian Motion 825 F.2 Langevin Equation 825 Appendix G Rotations and Reflections 827 G.l Plane Rotations 827 G.2 Two-Sided Jacobi Algorithm 829 G.3 Cyclic Jacobi Algorithm 835 G.4 Householder Transformation 838 G.5 The QR Algorithm 841 Appendix H Complex Wishart Distribution 848 H.l Definition 848 H.2 The Chi-Square Distribution as a Special Case 849 H.3 Properties of the Complex Wishart Distribution 850 H.4 Expectation of the Inverse Correlation Matrix fl> _1 («) 851 Glossary 852 Text Conventions 852 Abbreviations 855 Principal Symbols 858 Bibliography 864 Suggested Reading 879 Index 897 Contents 9