Lecture 3: Linear FIR Adaptive Filtering Gradient based adaptation: Steepest Descent Method
|
|
- Crystal Skinner
- 6 years ago
- Views:
Transcription
1 1 Lecture 3: Linear FIR Adaptive Filtering Gradient based adaptation: Steepest Descent Method Adaptive filtering: Problem statement Consider the family of variable parameter FIR filters, computing their output according to y(n) = w 0 (n)u(n)+w 1 (n)u(n 1) + w M 1 (n)u(n M +1) = M 1 k=0 w k (n)u(n k) =w(n) T u(n), n =0, 1, 2,, where parameters w(n) are allowed to change at every time step, u(t) is the input signal, d(t) is the desired signal and {u(t) and d(t)} are jointly stationary Given the parameters w(n) =[w 0 (n),w 1 (n),w 2 (n),,w M 1 (n)] T, find an adaptation mechanism w(n + 1) = w(n) + δw(n), or written componentwise w 0 (n +1) = w 0 (n)+δw 0 (n) w 1 (n +1) = w 1 (n)+δw 1 (n) w M 1 (n +1) = w M 1 (n)+δw M 1 (n) such as the adaptation process converges to the parameters of the optimal Wiener filter, w(n +1) w o, no matter where the iterations are initialized (ie w(0))
2 Lecture 3 2 The key of adaptation process is the existence of an error between the output of the filter y(n) and the desired signal d(n): e(n) =d(n) y(n) =d(n) w(n) T u(n) If the parameter vector w(n) will be used at all time instants, the performance criterion will be (see last Lecture) J(n) =J w(n) = E[e(n)e(n)] = E[(d(n) w T (n)u(n))(d(n) u T (n)w(n))] = E[d 2 (n)] 2E[d(n)u T (n)]w(n)+w T (n)e[u(n)u T (n)]w(n) = σd 2 2p T w(n)+w T (n)rw(n) = σd 2 2 M 1 i=0 p( i)w i + M 1 l=0 M 1 i=0 w l w i R i,l (1)
3 Lecture 3 3 Expressions of the gradient vector
4 Lecture 3 4 The gradient of the criterion J(n) with respect to the parameters w(n) is w(n) J(n) = J(n) w 0 (n) J(n) w 1 (n) J(n) w M 1 (n) = 2p(0)+2 M 1 i=0 R 0,i w i (n) 2p( 1) + 2 M 1 i=0 R 1,i w i (n) 2p( M +1)+2 M 1 i=0 R 1,M 1 w i (n) = 2p +2Rw(n) (2) w(n) J(n) = 2Ed(n)u(n)+2 M 1 i=0 Eu(n)u(n i)w i (n) 2Ed(n)u(n 1) + 2 M 1 i=0 Eu(n 1)u(n i)w i (n) 2Ed(n)u(n M +1)+2 M 1 i=0 Eu(n M +1)u(n i)w i (n) = 2Eu(n) ( d(n) M 1 i=0 u(n i)w i (n) ) 2Eu(n 1) ( d(n) M 1 i=0 u(n i)w i (n) ) 2Eu(n M +1) ( d(n) M 1 i=0 u(n i)w i (n) )
5 Lecture 3 5 = 2E (u(n)e(n)) 2E (u(n 1)e(n)) 2E (u(n M +1)e(n)) = 2E u(n) u(n 1) u(n M +1) e(n) = 2Eu(n)e(n) (3) ((2) and (3) give the solution of exercise 7, page 229 [Haykin 2002]) Steepest descent algorithm w(n +1) = w(n) 1 2 µ w(n)j(n) = w(n)+µeu(n)e(n)
6 Lecture 3 6 SD ALGORITHM Steepest descent search algorithm for finding the Wiener FIR optimal filter
7 Lecture 3 7 Given the autocorrelation matrix R = Eu(n)u T (n) the cross-correlation vector p(n) = Eu(n)d(n) Initialize the algorithm with an arbitrary parameter vector w(0) Iterate for n =0, 1, 2, 3,,n max w(n +1)=w(n)+µ[p Rw(n)] Stop iterations if p Rw(n) < ε Designer degrees of freedom: µ, ε, n max
8 Lecture 3 8 Equivalent forms of the adaptation equations We have the following equivalent forms of adaptive equations, each showing one facet (or one interpretation) of the adaptation process 1 Solving p = Rw o for w o using iterative schemes: w(n +1)=w(n)+µ[p Rw(n)] 2 or componentwise w 0 (n +1)=w 0 (n)+µ(p(0) M 1 i=0 R 0,i w i (n)) w 1 (n +1)=w 1 (n)+µ(p( 1) M 1 i=0 R 1,i w i (n)) w M 1 (n +1)=w M 1 (n)+µ(p( M +1) M 1 i=0 R M 1,i w i (n)) 3 Error driven adaptation process (See Figure at page 6): w(n +1)=w(n)+µ[Ee(n)u(n)] 4 or componentwise w 0 (n +1)=w 0 (n)+µ(ee(n)u 0 (n)) w 1 (n +1)=w 1 (n)+µ(ee(n)u 1 (n)) w M 1 (n +1)=w M 1 (n)+µ(ee(n)u M 1 (n))
9 Lecture 3 9 Example: Running SD algorithm to solve for the Wiener filter derived in the example of Lecture 2 Rw = p; r u(0) r u (1) r u (1) r u (0) w 0 w 1 = Ed(n)u(n) Ed(n)u(n 1) with the solution We will use the Matlab code w 0 w 1 w 0 w 1 = = R_u=[11 05 ; % Define autocovariance matrix ]; p= [05271 ; ] % Define cross-correlation vector W=[-1;-1]; mu= 001; % Initial values for the adaptation algorithm Wt=W; % Wt will record the evolution of vector W for k=1:1000 % Iterate 1000 times the adaptation step W=W +mu*(p-r_u*w); % Adaptation Equation! Quite simple! Wt=[Wt W]; % Wt records the evolution of vector W end % Is W the Wiener Filter?
10 Lecture W_1 W_2 Criterion Surface and Adaptation process 10 1 W(2) Parameter error w(n) w_ W(1) n (iteration) Figure 1: Adaptation Step µ =001 Left: Adaptation process starts from an arbitary point w(0) = [ 1, 1] T markedwith a cross and converges to Wiener filter parameters w(0) = [ ] T ( x point, in the center of ellipses) Right The convergence rate is exponential (note the logarithmic scale for the parametric error) In 1000 iterations the parameter error reaches 10 3
11 Lecture W_1 W_2 Criterion Surface and Adaptation process W(2) Parameter error w(n) w_ W(1) n (iteration) Figure 2: Adaptation Step µ =1 Left: Stable oscillatory adaptation process starting from an arbitary point w(0) = [ 1, 1] T marked with a cross and convergence to Wiener filter parameters w(0) = [ ] T ( x point, in the center of ellipses) Right The convergence rate is exponential In about 70 iterations the parameter error reaches 10 16
12 Lecture 3 12 W_1 W_2 Criterion Surface and Adaptation process W(2) Parameter error w(n) w_ W(1) n (iteration) Figure 3: Adaptation Step µ =4 Left: Unstable oscillatory adaptation process starting from an arbitary point w(0) = [ 1, 1] T marked with a cross and divergence from Wiener filter parameters w(0) = [ ] T ( x point, in the center of ellipses) Right The divergence rate is exponential In about 400 iterations the parameter error reaches 10 30
13 Lecture W_1 W_2 Criterion Surface and Adaptation process W(2) Parameter error w(n) w_ W(1) n (iteration) Figure 4: Adaptation Step µ =0001 Left: Very slow adaptation process, starting from an arbitary point w(0) = [ 1, 1] T marked with a cross It will eventually converge to Wiener filter parameters w(0) = [ ] T ( x, point in the center of ellipses) Right The convergence rate is exponential (note the logarithmic scale for the parametric error) In 1000 iterations the parameter error reaches 07
14 Lecture 3 14 Stability of Steepest Descent algorithm Write the adaptation algorithm as w(n +1)=w(n)+µ[p Rw(n)] (4) But from Wiener-Hopf equation p = Rw o and therefore w(n +1)=w(n)+µ[Rw o Rw(n)] = w(n)+µr[w o w(n)] (5) Now subtract Wiener optimal parameters,w o, from both members w(n +1) w o = w(n) w o + µr[w o w(n)] = (I µr)[w(n) w o ] and introducing the vector c(n) =w(n) w o,wehave c(n +1)=(I µr)c(n) Let λ 1,λ 2,,λ M be the (real and positive) eigenvalues and let q 1,q 2,,q M be the (generally complex) eigenvectors of the matrix R, thus satisfying Rq i = λ i q i (6) Then the matrix Q =[q 1 q 2 q M ] can transform R to diagonal form Λ = diag(λ 1,λ 2,,λ M ) where the superscript H means complex conjugation and transposition R = QΛQ H (7) c(n +1)=(I µr)c(n) =(I µqλq H )c(n)
15 Lecture 3 15 Left- multiplying by Q H Q H c(n +1)=Q H (I µqλq H )c(n) =(Q H µq H QΛQ H )c(n) =(I µλ)q H c(n) We notate ν(n) the rotated and translated version of w(n) ν(n) =Q H c(n) =Q H (w(n) w o ) (8) and now we have ν(n +1)=(I µλ)ν(n) where the initial value ν(0) = Q H (w(0) w o ) We can write componentwise the recursions for ν(n) ν k (n +1)=(1 µλ k )ν k (n) k =1, 2,,M which can be solved easily to give ν k (n) =(1 µλ k ) n ν k (0) k =1, 2,,M For the stability of the algorithm 1 < 1 µλ k < 1 k =1, 2,,M or 0 <µ< 2 λ k k =1, 2,,M or 0 <µ< 2 λ max STABILITY CONDITION!
16 Lecture 3 16 where λ max is the maximum eigenvalue of autocovariance matrix R If the stability condition is respected, this will result in ν k (n) 0, ie w k (n) w ok when n, k =1, 2,,M Notating τ k = 1 log( 1 µλ k ),wehave 1 µλ k = exp( 1 τ k ) and therefore ν k (n) = 1 µλ k n ν k (0) = exp( n τ k ) ν k (0) which is a decreasing exponential, which needs approximately 4τ k steps to reduce ν k (0) to 2% of its value Thus, we have obtained * a condition of stability, * information about the transient behavior and speed of convergence of parameters ν k (n) We can transfer back to the original parameters the results of analysis: The matrix Q =[q 1 q 2 q M ] obeys Q H Q = QQ H = I and multiplying by Q the extreme terms of the equalities ν(n) =Q H c(n) =Q H (w(n) w o ) Qν(n) =w(n) w o
17 Lecture 3 17 w(n) =w o + Qν(n) =w o +[q 1 q 2 Writing componentwise the equality, we have q M ]ν(n) =w o + M q k ν k (n) k=1 w i (n) =w o,i + M q k,i ν k (0)(1 µλ k ) n k=1 The fastest rate of convergence is obtained for the eigenvalue λ k which gives the least term 1 µλ k, while the slowest rate of convergence results for the eigenvalue λ k which gives the largest term 1 µλ k If we limit µ such that 0 < 1 µλ k < 1, then 1 log(1 µλ max ) τ a Transient behavior of the Mean -squared error 1 log(1 µλ min ) In Lecture 2 we obtained the canonical form of the quadratic form which expresses the mean square error: where ν was defined as Taking into account that J w(n) = J wo + M λ i ν i 2 i=1 ν(n) =Q H c(n) =Q H (w(n) w o ) (9) ν k (n) =(1 µλ k ) n ν k (0) k =1, 2,,M
18 Lecture 3 18 we have finally The convergence J w(n) = J wo + M λ i (1 µλ i ) 2n ν i (0) 2 i=1 J w(n) J wo (10) takes place under the same conditions as parameter convergence, and the rate of convergence is bounded, similarly, by 1 2log(1 µλ max ) τ a 1 2log(1 µλ min ) (11)
SGN Advanced Signal Processing: Lecture 4 Gradient based adaptation: Steepest Descent Method
SGN 21006 Advanced Signal Processing: Lecture 4 Gradient based adaptation: Steepest Descent Method Ioan Tabus Department of Signal Processing Tampere University of Technology Finland 1 / 20 Adaptive filtering:
More informationCh4: Method of Steepest Descent
Ch4: Method of Steepest Descent The method of steepest descent is recursive in the sense that starting from some initial (arbitrary) value for the tap-weight vector, it improves with the increased number
More informationLecture 7: Linear Prediction
1 Lecture 7: Linear Prediction Overview Dealing with three notions: PREDICTION, PREDICTOR, PREDICTION ERROR; FORWARD versus BACKWARD: Predicting the future versus (improper terminology) predicting the
More informationComputer exercise 1: Steepest descent
1 Computer exercise 1: Steepest descent In this computer exercise you will investigate the method of steepest descent using Matlab. The topics covered in this computer exercise are coupled with the material
More information2.6 The optimum filtering solution is defined by the Wiener-Hopf equation
.6 The optimum filtering solution is defined by the Wiener-opf equation w o p for which the minimum mean-square error equals J min σ d p w o () Combine Eqs. and () into a single relation: σ d p p 1 w o
More informationLecture 6: Block Adaptive Filters and Frequency Domain Adaptive Filters
1 Lecture 6: Block Adaptive Filters and Frequency Domain Adaptive Filters Overview Block Adaptive Filters Iterating LMS under the assumption of small variations in w(n) Approximating the gradient by time
More informationAdaptive Filtering Part II
Adaptive Filtering Part II In previous Lecture we saw that: Setting the gradient of cost function equal to zero, we obtain the optimum values of filter coefficients: (Wiener-Hopf equation) Adaptive Filtering,
More informationLMS and eigenvalue spread 2. Lecture 3 1. LMS and eigenvalue spread 3. LMS and eigenvalue spread 4. χ(r) = λ max λ min. » 1 a. » b0 +b. b 0 a+b 1.
Lecture Lecture includes the following: Eigenvalue spread of R and its influence on the convergence speed for the LMS. Variants of the LMS: The Normalized LMS The Leaky LMS The Sign LMS The Echo Canceller
More informationLecture Notes in Adaptive Filters
Lecture Notes in Adaptive Filters Second Edition Jesper Kjær Nielsen jkn@es.aau.dk Aalborg University Søren Holdt Jensen shj@es.aau.dk Aalborg University Last revised: September 19, 2012 Nielsen, Jesper
More informationELEG-636: Statistical Signal Processing
ELEG-636: Statistical Signal Processing Gonzalo R. Arce Department of Electrical and Computer Engineering University of Delaware Spring 2010 Gonzalo R. Arce (ECE, Univ. of Delaware) ELEG-636: Statistical
More informationLeast Mean Square Filtering
Least Mean Square Filtering U. B. Desai Slides tex-ed by Bhushan Least Mean Square(LMS) Algorithm Proposed by Widrow (1963) Advantage: Very Robust Only Disadvantage: It takes longer to converge where X(n)
More informationAdaptive Filters. un [ ] yn [ ] w. yn n wun k. - Adaptive filter (FIR): yn n n w nun k. (1) Identification. Unknown System + (2) Inverse modeling
Adaptive Filters - Statistical digital signal processing: in many problems of interest, the signals exhibit some inherent variability plus additive noise we use probabilistic laws to model the statistical
More informationCHAPTER 4 ADAPTIVE FILTERS: LMS, NLMS AND RLS. 4.1 Adaptive Filter
CHAPTER 4 ADAPTIVE FILTERS: LMS, NLMS AND RLS 4.1 Adaptive Filter Generally in most of the live applications and in the environment information of related incoming information statistic is not available
More informationELEG-636: Statistical Signal Processing
ELEG-636: Statistical Signal Processing Gonzalo R. Arce Department of Electrical and Computer Engineering University of Delaware Spring 2010 Gonzalo R. Arce (ECE, Univ. of Delaware) ELEG-636: Statistical
More informationAdaptive Filter Theory
0 Adaptive Filter heory Sung Ho Cho Hanyang University Seoul, Korea (Office) +8--0-0390 (Mobile) +8-10-541-5178 dragon@hanyang.ac.kr able of Contents 1 Wiener Filters Gradient Search by Steepest Descent
More informationCh5: Least Mean-Square Adaptive Filtering
Ch5: Least Mean-Square Adaptive Filtering Introduction - approximating steepest-descent algorithm Least-mean-square algorithm Stability and performance of the LMS algorithm Robustness of the LMS algorithm
More informationParametric Signal Modeling and Linear Prediction Theory 1. Discrete-time Stochastic Processes
Parametric Signal Modeling and Linear Prediction Theory 1. Discrete-time Stochastic Processes Electrical & Computer Engineering North Carolina State University Acknowledgment: ECE792-41 slides were adapted
More informationV. Adaptive filtering Widrow-Hopf Learning Rule LMS and Adaline
V. Adaptive filtering Widrow-Hopf Learning Rule LMS and Adaline Goals Introduce Wiener-Hopf (WH) equations Introduce application of the steepest descent method to the WH problem Approximation to the Least
More informationNeural Network Training
Neural Network Training Sargur Srihari Topics in Network Training 0. Neural network parameters Probabilistic problem formulation Specifying the activation and error functions for Regression Binary classification
More informationEE482: Digital Signal Processing Applications
Professor Brendan Morris, SEB 3216, brendan.morris@unlv.edu EE482: Digital Signal Processing Applications Spring 2014 TTh 14:30-15:45 CBC C222 Lecture 11 Adaptive Filtering 14/03/04 http://www.ee.unlv.edu/~b1morris/ee482/
More informationEE482: Digital Signal Processing Applications
Professor Brendan Morris, SEB 3216, brendan.morris@unlv.edu EE482: Digital Signal Processing Applications Spring 2014 TTh 14:30-15:45 CBC C222 Lecture 11 Adaptive Filtering 14/03/04 http://www.ee.unlv.edu/~b1morris/ee482/
More informationLecture 4 1. Block-based LMS 3. Block-based LMS 4. Standard LMS 2. bw(n + 1) = bw(n) + µu(n)e (n), bw(k + 1) = bw(k) + µ u(kl + i)e (kl + i),
Standard LMS 2 Lecture 4 1 Lecture 4 contains descriptions of Block-based LMS (7.1) (edition 3: 1.1) u(n) u(n) y(n) = ŵ T (n)u(n) Datavektor 1 1 y(n) M 1 ŵ(n) M 1 ŵ(n + 1) µ Frequency Domain LMS (FDAF,
More information26. Filtering. ECE 830, Spring 2014
26. Filtering ECE 830, Spring 2014 1 / 26 Wiener Filtering Wiener filtering is the application of LMMSE estimation to recovery of a signal in additive noise under wide sense sationarity assumptions. Problem
More informationChapter 2 Wiener Filtering
Chapter 2 Wiener Filtering Abstract Before moving to the actual adaptive filtering problem, we need to solve the optimum linear filtering problem (particularly, in the mean-square-error sense). We start
More informationAdaptive Filtering. Squares. Alexander D. Poularikas. Fundamentals of. Least Mean. with MATLABR. University of Alabama, Huntsville, AL.
Adaptive Filtering Fundamentals of Least Mean Squares with MATLABR Alexander D. Poularikas University of Alabama, Huntsville, AL CRC Press Taylor & Francis Croup Boca Raton London New York CRC Press is
More informationAn Adaptive Sensor Array Using an Affine Combination of Two Filters
An Adaptive Sensor Array Using an Affine Combination of Two Filters Tõnu Trump Tallinn University of Technology Department of Radio and Telecommunication Engineering Ehitajate tee 5, 19086 Tallinn Estonia
More information3.4 Linear Least-Squares Filter
X(n) = [x(1), x(2),..., x(n)] T 1 3.4 Linear Least-Squares Filter Two characteristics of linear least-squares filter: 1. The filter is built around a single linear neuron. 2. The cost function is the sum
More informationAdaptive Systems Homework Assignment 1
Signal Processing and Speech Communication Lab. Graz University of Technology Adaptive Systems Homework Assignment 1 Name(s) Matr.No(s). The analytical part of your homework (your calculation sheets) as
More informationAdaptive SP & Machine Intelligence Linear Adaptive Filters and Applications
Adaptive SP & Machine Intelligence Linear Adaptive Filters and Applications Danilo Mandic room 813, ext: 46271 Department of Electrical and Electronic Engineering Imperial College London, UK d.mandic@imperial.ac.uk,
More informationECE580 Partial Solution to Problem Set 3
ECE580 Fall 2015 Solution to Problem Set 3 October 23, 2015 1 ECE580 Partial Solution to Problem Set 3 These problems are from the textbook by Chong and Zak, 4th edition, which is the textbook for the
More informationSome definitions. Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization. A-inner product. Important facts
Some definitions Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 A matrix A is SPD (Symmetric
More informationCh6-Normalized Least Mean-Square Adaptive Filtering
Ch6-Normalized Least Mean-Square Adaptive Filtering LMS Filtering The update equation for the LMS algorithm is wˆ wˆ u ( n 1) ( n) ( n) e ( n) Step size Filter input which is derived from SD as an approximation
More informationVasil Khalidov & Miles Hansard. C.M. Bishop s PRML: Chapter 5; Neural Networks
C.M. Bishop s PRML: Chapter 5; Neural Networks Introduction The aim is, as before, to find useful decompositions of the target variable; t(x) = y(x, w) + ɛ(x) (3.7) t(x n ) and x n are the observations,
More informationAdvanced Digital Signal Processing -Introduction
Advanced Digital Signal Processing -Introduction LECTURE-2 1 AP9211- ADVANCED DIGITAL SIGNAL PROCESSING UNIT I DISCRETE RANDOM SIGNAL PROCESSING Discrete Random Processes- Ensemble Averages, Stationary
More informationProject: Vibration of Diatomic Molecules
Project: Vibration of Diatomic Molecules Objective: Find the vibration energies and the corresponding vibrational wavefunctions of diatomic molecules H 2 and I 2 using the Morse potential. equired Numerical
More informationMachine Learning and Adaptive Systems. Lectures 5 & 6
ECE656- Lectures 5 & 6, Professor Department of Electrical and Computer Engineering Colorado State University Fall 2015 c. Performance Learning-LMS Algorithm (Widrow 1960) The iterative procedure in steepest
More information4. Multilayer Perceptrons
4. Multilayer Perceptrons This is a supervised error-correction learning algorithm. 1 4.1 Introduction A multilayer feedforward network consists of an input layer, one or more hidden layers, and an output
More informationLinear Optimum Filtering: Statement
Ch2: Wiener Filters Optimal filters for stationary stochastic models are reviewed and derived in this presentation. Contents: Linear optimal filtering Principle of orthogonality Minimum mean squared error
More informationRegular paper. Adaptive System Identification Using LMS Algorithm Integrated with Evolutionary Computation
Ibraheem Kasim Ibraheem 1,* Regular paper Adaptive System Identification Using LMS Algorithm Integrated with Evolutionary Computation System identification is an exceptionally expansive topic and of remarkable
More informationDeep Learning. Authors: I. Goodfellow, Y. Bengio, A. Courville. Chapter 4: Numerical Computation. Lecture slides edited by C. Yim. C.
Chapter 4: Numerical Computation Deep Learning Authors: I. Goodfellow, Y. Bengio, A. Courville Lecture slides edited by 1 Chapter 4: Numerical Computation 4.1 Overflow and Underflow 4.2 Poor Conditioning
More informationNon-linear least squares
Non-linear least squares Concept of non-linear least squares We have extensively studied linear least squares or linear regression. We see that there is a unique regression line that can be determined
More informationIn the Name of God. Lecture 11: Single Layer Perceptrons
1 In the Name of God Lecture 11: Single Layer Perceptrons Perceptron: architecture We consider the architecture: feed-forward NN with one layer It is sufficient to study single layer perceptrons with just
More informationInstructor: Dr. Benjamin Thompson Lecture 8: 3 February 2009
Instructor: Dr. Benjamin Thompson Lecture 8: 3 February 2009 Announcement Homework 3 due one week from today. Not so long ago in a classroom very very closeby Unconstrained Optimization The Method of Steepest
More informationUnconstrained minimization of smooth functions
Unconstrained minimization of smooth functions We want to solve min x R N f(x), where f is convex. In this section, we will assume that f is differentiable (so its gradient exists at every point), and
More informationAdaptive Beamforming Algorithms
S. R. Zinka srinivasa_zinka@daiict.ac.in October 29, 2014 Outline 1 Least Mean Squares 2 Sample Matrix Inversion 3 Recursive Least Squares 4 Accelerated Gradient Approach 5 Conjugate Gradient Method Outline
More informationA METHOD OF ADAPTATION BETWEEN STEEPEST- DESCENT AND NEWTON S ALGORITHM FOR MULTI- CHANNEL ACTIVE CONTROL OF TONAL NOISE AND VIBRATION
A METHOD OF ADAPTATION BETWEEN STEEPEST- DESCENT AND NEWTON S ALGORITHM FOR MULTI- CHANNEL ACTIVE CONTROL OF TONAL NOISE AND VIBRATION Jordan Cheer and Stephen Daley Institute of Sound and Vibration Research,
More informationNumerical optimization
Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal
More informationLecture 19 IIR Filters
Lecture 19 IIR Filters Fundamentals of Digital Signal Processing Spring, 2012 Wei-Ta Chu 2012/5/10 1 General IIR Difference Equation IIR system: infinite-impulse response system The most general class
More informationMath 312 Lecture Notes Linear Two-dimensional Systems of Differential Equations
Math 2 Lecture Notes Linear Two-dimensional Systems of Differential Equations Warren Weckesser Department of Mathematics Colgate University February 2005 In these notes, we consider the linear system of
More informationTMA4180 Solutions to recommended exercises in Chapter 3 of N&W
TMA480 Solutions to recommended exercises in Chapter 3 of N&W Exercise 3. The steepest descent and Newtons method with the bactracing algorithm is implemented in rosenbroc_newton.m. With initial point
More informationLecture: Adaptive Filtering
ECE 830 Spring 2013 Statistical Signal Processing instructors: K. Jamieson and R. Nowak Lecture: Adaptive Filtering Adaptive filters are commonly used for online filtering of signals. The goal is to estimate
More informationAdvanced Signal Processing Adaptive Estimation and Filtering
Advanced Signal Processing Adaptive Estimation and Filtering Danilo Mandic room 813, ext: 46271 Department of Electrical and Electronic Engineering Imperial College London, UK d.mandic@imperial.ac.uk,
More informationNumerical methods part 2
Numerical methods part 2 Alain Hébert alain.hebert@polymtl.ca Institut de génie nucléaire École Polytechnique de Montréal ENE6103: Week 6 Numerical methods part 2 1/33 Content (week 6) 1 Solution of an
More informationPETROV-GALERKIN METHODS
Chapter 7 PETROV-GALERKIN METHODS 7.1 Energy Norm Minimization 7.2 Residual Norm Minimization 7.3 General Projection Methods 7.1 Energy Norm Minimization Saad, Sections 5.3.1, 5.2.1a. 7.1.1 Methods based
More informationChapter 1.6. Perform Operations with Complex Numbers
Chapter 1.6 Perform Operations with Complex Numbers EXAMPLE Warm-Up 1 Exercises Solve a quadratic equation Solve 2x 2 + 11 = 37. 2x 2 + 11 = 37 2x 2 = 48 Write original equation. Subtract 11 from each
More informationConjugate Gradient (CG) Method
Conjugate Gradient (CG) Method by K. Ozawa 1 Introduction In the series of this lecture, I will introduce the conjugate gradient method, which solves efficiently large scale sparse linear simultaneous
More informationAssesment of the efficiency of the LMS algorithm based on spectral information
Assesment of the efficiency of the algorithm based on spectral information (Invited Paper) Aaron Flores and Bernard Widrow ISL, Department of Electrical Engineering, Stanford University, Stanford CA, USA
More informationMachine Learning. A Bayesian and Optimization Perspective. Academic Press, Sergios Theodoridis 1. of Athens, Athens, Greece.
Machine Learning A Bayesian and Optimization Perspective Academic Press, 2015 Sergios Theodoridis 1 1 Dept. of Informatics and Telecommunications, National and Kapodistrian University of Athens, Athens,
More informationMultimedia Communications. Differential Coding
Multimedia Communications Differential Coding Differential Coding In many sources, the source output does not change a great deal from one sample to the next. This means that both the dynamic range and
More informationWe are IntechOpen, the world s leading publisher of Open Access books Built by scientists, for scientists. International authors and editors
We are IntechOpen, the world s leading publisher of Open Access books Built by scientists, for scientists 4, 6, M Open access books available International authors and editors Downloads Our authors are
More informationMassachusetts Institute of Technology
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.011: Introduction to Communication, Control and Signal Processing QUIZ, April 1, 010 QUESTION BOOKLET Your
More informationNumerical optimization. Numerical optimization. Longest Shortest where Maximal Minimal. Fastest. Largest. Optimization problems
1 Numerical optimization Alexander & Michael Bronstein, 2006-2009 Michael Bronstein, 2010 tosca.cs.technion.ac.il/book Numerical optimization 048921 Advanced topics in vision Processing and Analysis of
More informationMMSE System Identification, Gradient Descent, and the Least Mean Squares Algorithm
MMSE System Identification, Gradient Descent, and the Least Mean Squares Algorithm D.R. Brown III WPI WPI D.R. Brown III 1 / 19 Problem Statement and Assumptions known input x[n] unknown system (assumed
More informationSparse Least Mean Square Algorithm for Estimation of Truncated Volterra Kernels
Sparse Least Mean Square Algorithm for Estimation of Truncated Volterra Kernels Bijit Kumar Das 1, Mrityunjoy Chakraborty 2 Department of Electronics and Electrical Communication Engineering Indian Institute
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Lecture 5, Continuous Optimisation Oxford University Computing Laboratory, HT 2006 Notes by Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The notion of complexity (per iteration)
More informationEECS 275 Matrix Computation
EECS 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 20 1 / 20 Overview
More information, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are
Quadratic forms We consider the quadratic function f : R 2 R defined by f(x) = 2 xt Ax b T x with x = (x, x 2 ) T, () where A R 2 2 is symmetric and b R 2. We will see that, depending on the eigenvalues
More informationIS NEGATIVE STEP SIZE LMS ALGORITHM STABLE OPERATION POSSIBLE?
IS NEGATIVE STEP SIZE LMS ALGORITHM STABLE OPERATION POSSIBLE? Dariusz Bismor Institute of Automatic Control, Silesian University of Technology, ul. Akademicka 16, 44-100 Gliwice, Poland, e-mail: Dariusz.Bismor@polsl.pl
More informationFALL 2018 MATH 4211/6211 Optimization Homework 4
FALL 2018 MATH 4211/6211 Optimization Homework 4 This homework assignment is open to textbook, reference books, slides, and online resources, excluding any direct solution to the problem (such as solution
More informationMATH 205C: STATIONARY PHASE LEMMA
MATH 205C: STATIONARY PHASE LEMMA For ω, consider an integral of the form I(ω) = e iωf(x) u(x) dx, where u Cc (R n ) complex valued, with support in a compact set K, and f C (R n ) real valued. Thus, I(ω)
More informationODE Final exam - Solutions
ODE Final exam - Solutions May 3, 018 1 Computational questions (30 For all the following ODE s with given initial condition, find the expression of the solution as a function of the time variable t You
More informationLesson 1. Optimal signalbehandling LTH. September Statistical Digital Signal Processing and Modeling, Hayes, M:
Lesson 1 Optimal Signal Processing Optimal signalbehandling LTH September 2013 Statistical Digital Signal Processing and Modeling, Hayes, M: John Wiley & Sons, 1996. ISBN 0471594318 Nedelko Grbic Mtrl
More information5 Kalman filters. 5.1 Scalar Kalman filter. Unit delay Signal model. System model
5 Kalman filters 5.1 Scalar Kalman filter 5.1.1 Signal model System model {Y (n)} is an unobservable sequence which is described by the following state or system equation: Y (n) = h(n)y (n 1) + Z(n), n
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More information4.6 Iterative Solvers for Linear Systems
4.6 Iterative Solvers for Linear Systems Why use iterative methods? Virtually all direct methods for solving Ax = b require O(n 3 ) floating point operations. In practical applications the matrix A often
More informationECE 636: Systems identification
ECE 636: Systems identification Lectures 3 4 Random variables/signals (continued) Random/stochastic vectors Random signals and linear systems Random signals in the frequency domain υ ε x S z + y Experimental
More informationNumerical Optimization
Numerical Optimization Unit 2: Multivariable optimization problems Che-Rung Lee Scribe: February 28, 2011 (UNIT 2) Numerical Optimization February 28, 2011 1 / 17 Partial derivative of a two variable function
More informationGradient Descent. Sargur Srihari
Gradient Descent Sargur srihari@cedar.buffalo.edu 1 Topics Simple Gradient Descent/Ascent Difficulties with Simple Gradient Descent Line Search Brent s Method Conjugate Gradient Descent Weight vectors
More informationEE 225a Digital Signal Processing Supplementary Material
EE 225A DIGITAL SIGAL PROCESSIG SUPPLEMETARY MATERIAL EE 225a Digital Signal Processing Supplementary Material. Allpass Sequences A sequence h a ( n) ote that this is true iff which in turn is true iff
More informationMulti-Frame Factorization Techniques
Multi-Frame Factorization Techniques Suppose { x j,n } J,N j=1,n=1 is a set of corresponding image coordinates, where the index n = 1,...,N refers to the n th scene point and j = 1,..., J refers to the
More informationLet H(z) = P(z)/Q(z) be the system function of a rational form. Let us represent both P(z) and Q(z) as polynomials of z (not z -1 )
Review: Poles and Zeros of Fractional Form Let H() = P()/Q() be the system function of a rational form. Let us represent both P() and Q() as polynomials of (not - ) Then Poles: the roots of Q()=0 Zeros:
More informationIE 5531: Engineering Optimization I
IE 5531: Engineering Optimization I Lecture 15: Nonlinear optimization Prof. John Gunnar Carlsson November 1, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I November 1, 2010 1 / 24
More informationPostulates of Quantum Mechanics
EXERCISES OF QUANTUM MECHANICS LECTURE Departamento de Física Teórica y del Cosmos 018/019 Exercise 1: Stern-Gerlach experiment Postulates of Quantum Mechanics AStern-Gerlach(SG)deviceisabletoseparateparticlesaccordingtotheirspinalonga
More informationR-Linear Convergence of Limited Memory Steepest Descent
R-Linear Convergence of Limited Memory Steepest Descent Fran E. Curtis and Wei Guo Department of Industrial and Systems Engineering, Lehigh University, USA COR@L Technical Report 16T-010 R-Linear Convergence
More informationConvergence Evaluation of a Random Step-Size NLMS Adaptive Algorithm in System Identification and Channel Equalization
Convergence Evaluation of a Random Step-Size NLMS Adaptive Algorithm in System Identification and Channel Equalization 1 Shihab Jimaa Khalifa University of Science, Technology and Research (KUSTAR) Faculty
More informationHirschman Optimal Transform Block LMS Adaptive
Provisional chapter Chapter 1 Hirschman Optimal Transform Block LMS Adaptive Hirschman Optimal Transform Block LMS Adaptive Filter Filter Osama Alkhouli, Osama Alkhouli, Victor DeBrunner and Victor DeBrunner
More informationPractice Final Exam Solutions for Calculus II, Math 1502, December 5, 2013
Practice Final Exam Solutions for Calculus II, Math 5, December 5, 3 Name: Section: Name of TA: This test is to be taken without calculators and notes of any sorts. The allowed time is hours and 5 minutes.
More informationPMR5406 Redes Neurais e Lógica Fuzzy Aula 3 Single Layer Percetron
PMR5406 Redes Neurais e Aula 3 Single Layer Percetron Baseado em: Neural Networks, Simon Haykin, Prentice-Hall, 2 nd edition Slides do curso por Elena Marchiori, Vrije Unviersity Architecture We consider
More informationNonlinear Optimization for Optimal Control
Nonlinear Optimization for Optimal Control Pieter Abbeel UC Berkeley EECS Many slides and figures adapted from Stephen Boyd [optional] Boyd and Vandenberghe, Convex Optimization, Chapters 9 11 [optional]
More informationSuppose that the approximate solutions of Eq. (1) satisfy the condition (3). Then (1) if η = 0 in the algorithm Trust Region, then lim inf.
Maria Cameron 1. Trust Region Methods At every iteration the trust region methods generate a model m k (p), choose a trust region, and solve the constraint optimization problem of finding the minimum of
More informationOptimal and Adaptive Filtering
Optimal and Adaptive Filtering Murat Üney M.Uney@ed.ac.uk Institute for Digital Communications (IDCOM) 26/06/2017 Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/2017 1 / 69 Table of Contents 1
More informationAdap>ve Filters Part 2 (LMS variants and analysis) ECE 5/639 Sta>s>cal Signal Processing II: Linear Es>ma>on
Adap>ve Filters Part 2 (LMS variants and analysis) Sta>s>cal Signal Processing II: Linear Es>ma>on Eric Wan, Ph.D. Fall 2015 1 LMS Variants and Analysis LMS variants Normalized LMS Leaky LMS Filtered-X
More informationJUST THE MATHS UNIT NUMBER 1.6. ALGEBRA 6 (Formulae and algebraic equations) A.J.Hobson
JUST THE MATHS UNIT NUMBER 1.6 ALGEBRA 6 (Formulae and algebraic equations) by A.J.Hobson 1.6.1 Transposition of formulae 1.6. of linear equations 1.6.3 of quadratic equations 1.6. Exercises 1.6.5 Answers
More informationAN ITERATION. In part as motivation, we consider an iteration method for solving a system of linear equations which has the form x Ax = b
AN ITERATION In part as motivation, we consider an iteration method for solving a system of linear equations which has the form x Ax = b In this, A is an n n matrix and b R n.systemsof this form arise
More informationDecision Weighted Adaptive Algorithms with Applications to Wireless Channel Estimation
Decision Weighted Adaptive Algorithms with Applications to Wireless Channel Estimation Shane Martin Haas April 12, 1999 Thesis Defense for the Degree of Master of Science in Electrical Engineering Department
More informationConjugate gradient algorithm for training neural networks
. Introduction Recall that in the steepest-descent neural network training algorithm, consecutive line-search directions are orthogonal, such that, where, gwt [ ( + ) ] denotes E[ w( t + ) ], the gradient
More informationIntroduction to Signals and Systems Lecture #4 - Input-output Representation of LTI Systems Guillaume Drion Academic year
Introduction to Signals and Systems Lecture #4 - Input-output Representation of LTI Systems Guillaume Drion Academic year 2017-2018 1 Outline Systems modeling: input/output approach of LTI systems. Convolution
More informationGaussian, Markov and stationary processes
Gaussian, Markov and stationary processes Gonzalo Mateos Dept. of ECE and Goergen Institute for Data Science University of Rochester gmateosb@ece.rochester.edu http://www.ece.rochester.edu/~gmateosb/ November
More informationExploring the energy landscape
Exploring the energy landscape ChE210D Today's lecture: what are general features of the potential energy surface and how can we locate and characterize minima on it Derivatives of the potential energy
More informationChapter 10 Conjugate Direction Methods
Chapter 10 Conjugate Direction Methods An Introduction to Optimization Spring, 2012 1 Wei-Ta Chu 2012/4/13 Introduction Conjugate direction methods can be viewed as being intermediate between the method
More information