Order Recursion Introduction Order versus Tie Updates Matrix Inversion by Partitioning Lea Levinson Algorith Interpretations Exaples Introduction Rc d There are any ways to solve the noral equations Solutions are atheatically equivalent May not be nuerically equivalent May differ in Nueric stability Order of coputation Usefulness of interediate quantities: partial autocorrelation coefficients Studying properties of solution: iniu phase, J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 2 Order Updates R c d Suppose we wish to solve the noral equations for a range of M values May help select the final odel order If we have solved for the optial c C,canweore efficiently solve for c +? This is called an order update In this set of slides I have dropped the o subscript to siplify notation We will only discuss the optial paraeter vector Tie Updates R(n)c(n) d(n) Suppose we are estiating r x (l) and r yx (l) fro a finite data segent, x(n) for n {0,,...,N } Let s say we have solved the noral equations for this data set When a new observation is obtained, can we solve the updated noral equations ore efficiently by using the solution for the first N points? Called tie updates The dual proble of order updates These techniques are essentially adaptive filters (chapter 0) J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 3 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 4
Scope of Order Updates R c d The text covers all of the cases we have discussed Any linear estiation proble Optiu FIR filters Optiu FIR filters for stationary processes While the solutions are insightful for the first two cases, they are no ore efficient coputationally We will only discuss the last case Stationary Case Rc d When the process is stationary, R is Toeplitz This perits ore efficient algoriths to be used In general the inversion of an M M atrix requires O(M 3 ) operations When R is Toeplitz we can Invert the atrix in O(M 2 ) operations Perfor an LDL decoposition in O(M 2 ) operations See Section 7.7 for details We will consider both the stationary FIR filter and nonstationary linear cobiner cases J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 5 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 6 Linear Cobiner: Notation In this set of notes and subscript is used to indicate the size of atrices and vectors [[ x [x R + E x x + [ R r b + rb ρ b [[ [ d + E[x + y x E y d x + d + r b E [ x x + ρ b E [ x + 2 Inversion of Partitioned eritian Matrices Suppose we know R and we wish to copute R+ efficiently so that we can solve the noral equations R + c + d + The inverse of a eritian atrix is also eritian We are not assuing R is Toeplitz yet [ Q + R+ Q q Q + q q [ [ R r R + Q + b Q q r b ρ b q q [ I 0 0 R Q + r b q I r bq + ρ b q 0 R q + r b q 0 r bq + ρ b q J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 7 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 8
Inversion of Partitioned eritian Matrices Continued R Q + r b q I r bq + ρ b q 0 R q + r b q 0 r bq + ρ b q We only have three unknowns, so only three of these equations are needed to obtain the solutions. q R r b q q ρ b rb R r b q R r b ρ b rb R r b Q R R r b q R + R r b ( R r b ) ρ b rb R r b Inversion of Partitioned eritian Matrices Continued Now if we define q q b R r b Q R ρ b rb R r b R r b ρ b rb R r b + R r b ( R r b ) ρ b rb R r b α b ρ b r br r b ρ b + r bb The we can siplify the expressions for the coponents of Q + as q α b q b α b Q R + b b α b J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 9 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 0 q α b Matrix Inversion by Partitioning Lea q b α b We can then express R+ as ( [ R+ Q q q q [ R 0 0 0 R + α b [ b Q R ) + b b α b b + b b α b b α b α b [b This is a rank-one odification Called the atrix inversion by partitioning lea α b Matrix Inversion Counterpart Following siilar steps, we can show [ R+ ρf rf r f R f a R f r f [ 0 0 0 R f + [ α f a α f ρ f + r fr f r f [ a J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 2
and Interpretations e b x + ˆx + x + + b [ T x x 2... x e f x ˆx x + a [ T x2 x 3... x + P b ρ b r br r b α b P f ρ f r fr f r f α f The vector b is the MMSE estiator of the last eleent of x + fro the first eleents Siilarly, a is the MMSE estiator of the first eleent of x + fro the last eleents If x + [ x(n) x(n )... x(n ), then these are the forward (FLP) and backward (BLP) linear predictors eritian Inversion Suary b R r b α b ρ b + r bb R 0 [ R + 0 0 + α b [ b [b We have shown that we can obtain the inverse of R + fro the inverse of R with a rank one update Requires the R and α b be invertible Applies to any eritian atrix The update requires O( 2 ) operations Is ore efficient than solving the noral equations repeatedly for every of interest Is not ore efficient if only c M is of interest But how is c + is related to c? J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 3 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 4 Order Updates of Paraeter Vectors c + R+ d + [ R 0 0 0 [ R d + 0 [ [ c b + 0 [ d d + [ b k c + α b b d + d + α b [ b [b k c β c α b β c b d + d + [ d d + ere the c subscript is presuably an indicator that these coefficients are for updating the paraeter vector c Order Updates of Paraeter Vectors Discussion b R r b β c b d + d + [ [ c b c + + k 0 c α b ρ b + r bb k c β c α [ b R + 0 0 R 0 + [ b [b α b The update equation for c + is called a Levinson recursion If we know b, we can deterine c + owever, solving for b requires a atrix-vector product which requires O( 2 ) operations for the update to c + Contrary to the text, this approach is ore efficient than solving for c + R+ d + directly, which would require O( 3 ) operations WecandoevenbetterifR + is Toeplitz J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 5 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 6
Stationary Case: Notation When x(n) and y(n) are jointly stationary and x + (n) [ x(n) x(n )... x(n ) r [ r() r(2)... r() T and R + is Toeplitz r(0)... r( ) r(). R +..... [ r(0) r r ( )... r(0) r() T r R r ()... r () r(0) [ [ [ ρf r f R Jr R r r f R rj b r(0) rb ρ b [ d d + d + Stationary Case: Autocorrelation Matrix Inversion Fro the atrix inversion by partitioning lea, we have [ R+ R 0 0 + [ b [b 0 P b b R r b R Jr P b α b ρ b + r bb r(0) + r Jb P f J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 7 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 8 Stationary Case: Paraeter Updates [ [ c b c + + k 0 c k c β c P b b R Jr β c b d + d + ( R Jr ) d + d + r JR d + d + r Jc + d + The text has the wrong expression for this last equality So far we haven t gained anything The ost expensive operation coputationally is still solving for b and requires O( 2 ) operations Stationary Case: The Trick Recall that in the stationary case the FLP and BLP are related b Ja P P b P f ence, we can trivially deterine b fro a and vice versa Suppose now that we solve for the FLP coefficients. In this case y(n) x(n +) c a d [ r()... r( ) T r The recursive expression for the optial FLP coefficients is then [ [ [ [ a b a Ja a + + k 0 + k 0 k β β b P r + r ( +)a T Jr + r ( +) Cool! J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 9 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 20
Stationary Case: BLP/FLP MMSE P r(0) + r a r(0) + a T r P + r(0) + r +a + r(0) + [ r r ( +) ([ a 0 [ ) Ja + k r(0) + r a + ( r b + r ( +) ) k P + β k P + β k Siilarly, Stationary Case: FIR Filter MMSE k c β c P P c(+) P c β ck c P c β c 2 P P c Thus the MMSE of the FIR filter can also only decrease as increases! P +( k P ) k P ( k 2 ) The BLP MMSE therefore is only decreased as we increase in the stationary case J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 2 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 22 Levinson Algorith Given {r(l)} M 0, {d } M 0, P y, initialize the algorith with P r(0) β 0 k 0 a 0 For 0,,...,M r [ r() r(2)... r() T k c β c P c + β a T Jr + r ( +) P P + β k k β P [ [ a Ja a + + k 0 β c c Jr + d + [ [ c Ja + 0 k c Levinson and Levinson-Durbin Algoriths The Levinson algorith consists of two parts. Recursions for the optiu FLP/BLP 2. Recursions for the optiu filter If the application is only FLP or BLP, the algorith can be siplified This siplified algorith is called the Levinson-Durbin algorith (see text for details) There are other algoriths for calculating the optial lattice filter coefficients under the sae conditions P c(+) P c + β c k c See Table 7.2 in text for ore orderly presentation of the algorith. J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 23 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 24
Partial Correlation pc(y, x )E [ (x(n ) ˆx(n )) (y(n) ŷ(n)) E[e b (n)e (n) b d + d + β c Recall that the partial correlation is defined as the correlation between x(n ) and y(n) after the influence of the interediate variables {x(n),x(n ),...,x(n +)} has been reoved Equivalent Solutions for Optiu Linear Prediction There are three equivalent representations of optiu linear predictors of a stationary process Direct-for filter structure: {P M,a,a 2,...,a M } Lattice filter structure: {P M,k 0,k,...,k M } Autocorrelation sequence: {r(0),r(),...,r(m)} It is possible to convert fro any set of these paraeters to any other set Covered in Section 7.6, but we won t discuss in lecture That is we have built estiators for x(n ) and y(n) and calculate the correlation of the residuals The residual for x(n ) is the backward prediction error J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 25 J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 26 Reaining Relationship to partial correlation Exaple J. McNaes Portland State University ECE 539/639 Order Recursion Ver..0 27