6. Kalman filter implementation for linear algebraic equations. Karhunen-Loeve decomposition

Size: px

Start display at page:

Download "6. Kalman filter implementation for linear algebraic equations. Karhunen-Loeve decomposition"

Asher Brooks
6 years ago
Views:

1 6. Kalma filter implemetatio for liear algebraic equatios. Karhue-Loeve decompositio 6.1. Solvable liear algebraic systems. Probabilistic iterpretatio. Let A be a quadratic matrix (ot obligatory osigular. Cosider a system of solvable liear algebraic equatio Ax = y, (6.1 that is uder sigular matrix A system (6.1 possesses a solutio. It is well kow that all solutios of (6.1 uder the sigular matrix A are described as follows: x = A y + ( I A A h, (6.2 where A is the Moore-Perose pseudoiverse matrix of A ad I is the uite matrix. The vectors A y ad ( I A A h are orthogoal i a sese their ier product is equal to zero. I fact, sice A A is symmetric matrix ad A AA = A (see Sectio 5.2 i Lect. 5 we have A y, ( I A Ah = h T (I A A T A y = h T (I A AA y = h T (A A AA y = 0. Hece, owig to a arbitrariess of h, the solutio A y has the miimal Euclidea orm. Without of a prior iformatio, it is impossible to fid a solutio correspodig to a specific h (uless A is osigular. We give ow a probabilistic descriptio of the solutio x = A y. Let X be a radom vector with EX = 0 ad Cov(X, X = I (here I is the uite matrix ad Y = AX. Assume the radom vector Y is observable (measurable, the matrix A is kow ad it is required to fid the best estimate of X. Clearly that as such estimate X = Ê(X Y ca be take. Notice that Cov(X, Y = Cov(X, XA T = A T ad Cov(Y, Y = ACov(X, XA T = AA T. Therefore, by Theorem 5.1 (Lect. 5, X = A T (AA T. O the other had, sice A T (AA T = A (see, Sectio 5.2, (6, we have Cosider ow the solutio x = A y. Set The X = A Y. (6.3 z = X T x. (6.4 E Xz = EA Y z = EA AXX T x = A ACov(X, Xx. 1

2 Further, sice Cov(X, X = I ad x = A y ad A AA = A, we have E Xz = x. ( Kalma filter for X. For a defiiteess, assume that A = A ad deote by a k, k = 1,..., rows of A. By Y k, k = 1,..., deote etries of Y. For otatioal coveiece, itroduce also a sequece X k, k = 0, 1,..., with X 0 = X ad X k = X k 1. Obviously, X k X. Nevertheless, these otios allow us to preset the relatio Y = AX i a form compatible with oiseless versio of (5.8 (Lect. 5 X k = X k 1 Y k = a k X k 1. (6.6 Sice the Kalma filter is also valid for the oiseless model, required oly the use of pseudoiverse matrices, for X k = Ê(X k M k, where M k is the liear space geerated by 1, Y 1,..., Y k, ad P k = E(X k X k (X k X k T the followig Kalma filter occurs X k = X k 1 + P k 1 a T ak P k 1 a T k Yk a k Xk 1 P k = P k 1 P k 1 a T k ak P k 1 a T ak (6.7 k P k 1 subject to the iitial coditios X 0 = 0 ad P 0 = I. Notice that, whereas X k X, we have X k = Ê(X M k. Hece X = X. Deote by A k the sub-matrix of A cotaiig first k rows of A ad by Y k the sub-vector of Y cotaiig first k etries of Y. Owig to Y k = A k X, we have X k = A k Y k. So, the Kalma filter geerates a sequece of projectios At the same time, sice we fid A 1 Y 1, A 2 Y 2,..., A Y (= A Y. X k X k = X A k y k = X A k A kx = (I A k A kx, P k = (I A k A k; (6.8 recall that Cov(X, X = I, (I A k A k is symmetric matrix, ad (A k A k 2 = A k A k (see, Sectio 5.2, Lect. 5. 2

3 6.3. Kalma filter for x. Deote by y k, k = 1,..., y k etries of y = Ax ad by y k the sub-vector of y cotaiig first k etries of y. Similar to (6.5, with z defied i (6.4, we fid E X k z = E X k X T x = EA k yk X T x Notice that ad = EA k A kxx T x = A k A kcov(x, Xx x = x. = A k A kx = A k yk := x k. (6.9 EY k z = EA k Cov(X, Xx = A k x = y k, EY k z = Ea k Cov(X, Xx = a t x = y k. (6.10 Now, takig ito the cosideratio (6.9 ad (6.10 ad multiplyig the left- ad right-had sides of the first equatio i (6.7 by z ad the takig the expectatio of the expressios obtaied, we trasform (6.7 ito the Kalma filter geeratig x k, k = 1,..., : x k = x k 1 + P k 1 a( T ak P k 1 a T k yk a k Xk 1 P k = P k 1 P k 1 a T k ak P k 1 a T ak (6.11 k P k 1, where x 0 = 0, P 0 = I. A atural questio arises: why it makes sese to use (6.11 for a computatio of A y? Our argumets are the followig. The algorithm give i (6.11 applies the simplest pseudoiverse operatio for ak P k 1 a T k for the scalar ak P k 1 a T k, so that { 1 ak P k 1 a T k = a k P k 1, if a a T k P k 1 a T k > 0 k 0, otherwise. (6.12 It is clear that (6.12, as ay pseudoiverse operatio, is ill-posed, sice a k P k 1 a T k decreases with icreasig k ad beig positive might be too 1 close to zero that the fractal a k P k 1 ca ot be computed correctly. a T k Nevertheless, a structure of (6.12 allows to cotrol the correctess of such type computatio. A verificatio of the computatio correctess heavily uses the fact that a k P k 1 a T k is the trace of the matrix P k 1 a T k a kp k 1 (i fact, by (6.8, trace(p k 1 a T k a kp k 1 = a k Pk 1 2 at k = a k P k 1 a T k. Cosequetly, we have { tracep k 1 a T k ak P k 1 a T ak 1, if a k P k 1 a T k k P k 1 = > 0 0, otherwise ad ay essetial deviatio from 0 or 1 fixes the icorrectess. The ext questio is what idicates a k P k 1 a T k = 0? 3

4 Lemma 6.1. a k P k 1 a T k = mi a k k 1 c j a j 2. Particularly, if c 1,...,c k 1 a k P k 1 a T k = 0, the the row a k is a liear combiatio of a 1,..., a k. Proof. Set = a k k 1 j=1 c ja j, where umbers a j s are chose such that to miimize the Euclidea orm 2. Let c be the vectorrow with etries c 1,..., c k ad otice that = a k ca k 1 ad so j=1 2 = a k 2 2a k A T k 1c T + ca k 1 A T k 1c T. Sice 2 is the miimal i c, the vector c solves the liear equatio c = 0, that is c(a k 1 A T k 1 = a ka T k 1 ad so c = a k A T k 1(A k 1 A T k 1 = a k A k 1. Hece, = a k (I A k 1 A k 1 ad therefore 2 = a k (I 2A k 1 A k 1 + (A k 1 A k 1 2 a k = a k (I A k 1 A k 1a T k = a k P k 1 a T k (for more details see Sectio 5.2 Lect. 5. Karhue-Loeve decompositio Let ξ k, k = 1,..., be zero mea sequece of radom variables with the correlatio fuctio R(k, l. From poit of view of a theoretical cosideratio ad simulatio as well, it is useful if ξ k are geerated with a help of liear trasformatio of a white oise similar to differet models of statioary radom sequeces i the wide sese give i Lect Orthoormal eigevectors vectors of correlatio matrix. A matrix R = R with etries R(k, l is the correlatio matrix of ξ = ξ 1 ξ 2., that is R = EξξT ad ξ so R is oegative defiite matrix (i fact, x T Rx = Ex ξ ξ T x = E x T ξ 2 0. It is well kow from Theory of matrices that eigevectors of a oegative defiite matrix ca be trasformed i a system of orthoormal vectors. I other words, eigevectors of R ϕ 1 = (ϕ(1, 1,..., ϕ(1, ϕ 2 = (ϕ(2, 1,..., ϕ(2,... =... ϕ N = (ϕ(n, 1,..., ϕ(, 4

5 ca be chose such that for ay k, l ϕk, ϕ l = ϕ(k, jϕ(l, j = j=1 { 1, k = l 0, otherwise. (6.13 Deote S the quadratic matrix with rows ϕ 1, ϕ 2,..., ϕ ad otice that (6.13 provides S T S = I, where I is the uite matrix. Hece, S T = S 1 ad S is called orthogoal matrix. Deote by λ j the right eigevector of R correspodig to ϕ j : Rϕ j = λ j ϕ j. Owig to properties of S, we have λ 1 ϕ 1 λ 2 ϕ 2 S T RS = S T R. λ ϕ = diag(λ 1,..., λ, (6.14 where diag(λ 1,..., λ N is the diagoal matrix with eigevalues of R o the diagoal. Sice R is the oegative matrix, its eigevalues are oegative as well Karhue-Loeve theorem. Theorem 6.2. There exists zero mea radom vector with orthogoal etries ε 1, ε 2,..., ε with Eε 2 j 1 such that for every k = 1, 2,..., ξ k = λj ϕ(k, jε j. (6.15 j=1 Proof. Assume first that R is osigular matrix, that is all λ j s are positive. The diag 1 (λ 1, λ 2,..., λ = diag(λ 1 1, λ 1 2,..., λ 1. Set ε = diag(λ 1/2 1, λ 1/2 S T ξ. (6.16 Notice that Eξ = 0 ad Eξξ T = diag(λ 1/2 1, λ 1/2 S T RSdiag(λ 1/2 1, λ 1/2 = diag(λ 1/2 1, λ 1/2 diag(λ1, λ 2,..., λ diag(λ 1/2 1, λ 1/2 = I, that is etries of radom vector ε are zero mea ad orthogoal. Further, sice S T = S 1 ad diag 1( λ 1/2 1, λ 1/2 ( 1/2 = diag λ, (6.15 is implied by (6.16 uder osigular R, that is Sε. (6.17 ξ = diag ( λ 1/2 If R is sigular, the, istead of (6.16, let us itroduce ε = diag (λ 1/2 5 S T ξ, (6.18

6 here diag is defied similarly to diag 1 with zeros located o same place where λ s are zeros. Now, itroduce zero mea radom vector (colum of ξ size η with orthogoal etries of the uite variace. Also assume that η ad ξ are orthogoal ad itroduce ε = ( I diag ( λ 1/2 diag (λ 1/2 (6.19 Heceforth, omittig argumets i diag(λ 1/2 2,..., λ 1/2, write. ε = ε + ε = diag S T ξ + ( I diag diag η. (6.20 Takig ito cosideratio of the pseudoiverse matrix properties (see, Sectio 5.2 i Lect. 5 it is readily to check that Eε = 0 ad Eεε T = I. Uder sigular R, the formula, give i (6.17, remais with ε from (6.20. Ideed, deote by ξ the right had of (6.17 with that ε ad show that ξ = ξ with probability oe. For this purpose, it suffices to prove that E ξ ξ 2 = 0. Write E ξ ξ 2 = E(ξ ξ(ξ ξ T = ( I Sdiag( diag ( S T R ( I Sdiag( diag ( S T T = ( I Sdiag( diag ( S T Sdiag 2 ( S T ( I Sdiag( diag ( S T = 0. Remark 6.3. The Karhue-Loeve decompositio holds true for a radom vector ξ with etries from a coutable set ξ 1, ξ 2,..., ξ,..., provided that R(k, k <. k=1 6

Linear regression. Daniel Hsu (COMS 4771) (y i x T i β)2 2πσ. 2 2σ 2. 1 n. (x T i β y i ) 2. 1 ˆβ arg min. β R n d

Linear regression. Daniel Hsu (COMS 4771) (y i x T i β)2 2πσ. 2 2σ 2. 1 n. (x T i β y i ) 2. 1 ˆβ arg min. β R n d Liear regressio Daiel Hsu (COMS 477) Maximum likelihood estimatio Oe of the simplest liear regressio models is the followig: (X, Y ),..., (X, Y ), (X, Y ) are iid radom pairs takig values i R d R, ad Y