Kernels for Dynamic Textures

Transcription

1 Kernels for Dynamic Textures S.V.N. Vishwanathan National ICT Australia and Australian National University Joint work with Alex Smola and René Vidal S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 1

2 Roadmap Introduction to Kernel Methods Why kernels? Kernels on Dynamical Systems Trajectories, Noise Models Computation Dynamical Textures ARMA Models Approximate Solutions Kernel Computation Experiments Outlook and Conclusion S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 2

3 Classification Data: Task: Pairs of observations (x i, y i ) Underlying distribution P(x, y) Examples (blood status, cancer), (transactions, fraud) Find a function f(x) which predicts y given x The function f(x) must generalize well S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 3

4 Optimal Separating Hyperplane Minimize 1 2 w 2 subject to y i ( w, x i + b) 1 for all i S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 4

5 Kernels and Nonlinearity Problem: Linear functions are often too simple to provide good estimators Idea 1: Map to a higher dimensional feature space via Φ : x Φ(x) and solve the problem there Replace every x, x by Φ(x), Φ(x ) Idea 2: Instead of computing Φ(x) explicitly use a kernel function k(x, x ) := Φ(x), Φ(x ) A large class of functions are admissible as kernels Non-vectorial data can be handled if we can compute meaningful k(x, x ) S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 5

6 Roadmap Introduction to Kernel Methods Why kernels? Kernels on Dynamical Systems Trajectories, Noise Models Computation Dynamical Textures ARMA Models Approximate Solutions Kernel Computation Experiments Outlook and Conclusion S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 6

7 The Basic Idea Key Observation: Trajectories are easily observable Similar trajectories similar systems Restrict attention to interesting cases Average over noise models Kernels Using Dynamical Systems: Simulate system for both inputs Similar time evolution similar inputs Kernels on Dynamical Systems: Restrict to interesting initial conditions Simulate both the systems Similar time evolution similar systems S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 7

8 Notation X - state space (Hilbert space) A - time evolution operators T - time of measurement µ - nice probability measure on T Discounting Factors: For some λ > 0 µ(t) = λ 1 e λt for T = R + 0 Time Evolution: We study µ(t) = e λt 1 e λ for T = N 0 x A (t) := A(t)x for A A S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 8

9 Trajectories and Kernels Comparing Trajectories: Using the dot product on X we define a dot product on X T θ, θ := E µ [ θ(t), θ (t) ] for θ, θ X T Extending to Dynamical Systems: Identify a dynamical system with its trajectory and define ] k((x, A), ( x, Ã)) := E µ [ A(t)x, Ã(t) x Other Ideas: A nicely decaying measure required for convergence Modify the dot product in X Covariance matrices? Rational kernels and transducers S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 9

10 Special Cases Kernels on Dynamical Systems: Restrict attention to x = x Compare trajectory for identical initial conditions Take expectation if interested in a range of x ] k(a, Ã) := E x [k((x, A), (x, Ã)) More generally k(a, Ã) := E A EÃ E x [ k((x, A), (x, Ã)) ] Kernels Using Dynamical Systems: Restrict attention to a particular dynamical system As before we can take expectations over A k(x, x) := E x E x E A [k((x, A), ( x, A))] S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 10

11 Discrete Linear Systems Linear Systems: We assume time propagation occurs as x A (t + 1) = A x A (t) + a t + ξ t In closed form t x A (t) = A t x 0 + A t i ξ i + A t i a t i=0 To avoid messy math assume a t = 0 and hence t x A (t) = A t x 0 + A t i ξ i i=0 Contribution to kernel due to A as well as noise S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 11

12 Continuous Linear Systems Linear Systems: Sytem dynamics here are described by d dt x A(t) = A x A (t) + a(t) + ξ(t) Here ξ(t) with E[ξ(t)] = 0 is a stochastic process and x A (t) = exp(a t)x 0 + t 0 exp(a(t τ))(a(τ) + ξ(τ))dτ As before we assume a(t) = 0 We even assume ξ(τ) = 0 (avoids messy math again!) x A (t) = exp(a t)x 0 Kernel contribution only due to A S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 12

13 Convergence Criterion Discrete Case: Let A and B and W be linear operators The matrix norms obey 0 A, B Λ For suitable λ with e λ > Λ 2 and W 0 M := e λt A t W B t t=0 Sylvester equation e λ AMB + W = M Continuous Case: We define M := 0 e λt exp(at) W exp(bt) dt Sylvester equation (A + λ 2 1)M + M(B + λ 2 1) = W S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 13

14 Gory Details Contribution due to A: [ ] p e λt A t x, Ãt x := p x e λt (A t ) W Ãt t=0 t=0 = p x M x x Contribution due to noise: p E ξ t=0 t=0 j,j =0 t e λt A t j ξ j, ξ Ãt j j ( [ ]) = p tr C ξ e λt (A t ) M Ãt := p tr(c ξ M) In above equations p is a normalizing term S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 14

15 Delving Deeper More on M and M: The matrix M and M look like [ ] M := e λt (A t ) W Ãt t=0 and [ ] M := e λt (A t ) M Ãt t=0 Sylvester Equation: Both M and M satisfy the Sylvester equation e λ A M Ã +W = M and e λ A M Ã +M = M Can be solved for in cubic time S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 15

16 Discrete Kernel Discrete Case: Putting it all together k((a, x), (Ã, x)) = p [ x M x + tr(c ξ M) ] Note that C ξ is the covariance matrix of ξ t Can assume different noise models per time step Initial Conditions: C be the covariance matrix of the initial conditions If we set x = x then k((a, x), (Ã, x)) = p [ tr(cm) + tr(c ξ M) ] S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 16

17 Continuous Kernel Contribution due to A: Since we assumed a(t) = ξ(t) = 0 we get k((x, A), ( x, Ã)) = λ 1 e λt exp(a t)x, exp(ã t) x dt The Final Form: The kernel can be expressed as k((x, A), ( x, Ã)) = λ 1 x M x where 0 (A + λ 2 1)M + M (Ã +λ 2 1) = W Solution in cubic time by solving Sylvester equation S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 17

18 Special Cases Snapshot: If we consider only the snapshot at time instance T k((x, A), ( x, Ã)) = λ 1 x exp(a t)w exp(ã t) x Initial Conditions: Fix A = Ã Now we just solve Dynamical Systems: M = 1 2 (A +λ 2 1) 1 W Fix x = x to get k(a, Ã) = λ 1 tr(mc) Here C is the covariance matrix of initial conditions S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 18

19 Graph Kernels Graph Laplacian: Let E be the adjacency matrix and D := diag(e 1) L := E D and L := D 1 2 LD 1 2 Diffusion Process: We can define a diffusion process by d x(t) = Lx(t) dt Diffusion Kernel (Kondor and Lafferty, 2002): If we measure overlap at time instance T we get K = exp(lt ) exp(lt ) K ij is the probability that state l reached from i and j S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 19

20 Graph Kernels Undirected Graphs (Kondor and Lafferty, 2002): Here L is symmetric and hence yields K = exp(2lt ) Labeled Graphs (Gärtner, 2002): If W acts as an indicator for node labels Say W ij = 1 if two nodes have same label For other fancy weights see (Kashima et al, 2003) Averaged Graph Laplacian: If we average over a range of T values K = 1 ( L + λ2 ) S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 20

21 Roadmap Introduction to Kernel Methods Why kernels? Kernels on Dynamical Systems Trajectories, Noise Models Computation Dynamical Textures ARMA Models Approximate Solutions Kernel Computation Experiments Outlook and Conclusion S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 21

22 ARMA Models ARMA Model: An auto-regressive moving average model is x(t + 1) = A x(t) + B v(t) y(t) = φ(x(t)) + w(t) x(t) is a hidden variable v(t) and w(t) are IID random noise Linear Gaussian Model: If φ is linear and the noise is white Gaussian: x(t + 1) = A x(t) + v(t) v(t) N(0, Q) y(t) = C x(t) + w(t) w(t) N(0, R) Fix scaling by demanding that C C = 1 S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 22

23 Dynamic Textures Image Model: y(t) R m are the observed noisy images x(t) R n (n < m) are hidden variables Modeling: A sequence of images {y(1),..., y(τ)} is observed Ideally we want to solve A(τ), C(τ), Q(τ), R(τ) = arg max p(y(1),..., y(τ)) A,C,Q,R Exact Solution: n4sid in MATLAB solves above problem Does not scale well if m is large Impractical for images where m 10 5 S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 23

24 Approximate Solution Problem To Solve: For any variable z(t) define Z τ i := [z(i),..., z(τ)] We are solving Solving By SVD: Y τ 1 = C X τ 1 + W τ 1 with C C = 1 Solving for arg min C,X τ 1 W yields C(τ) = U and X(τ) = ΣV where Y τ 1 = UΣV Solving for arg min A X τ 2 A X τ 1 yields A(τ) = ΣV D 1 V (V D 2 V ) 1 Σ 1 [ ] [ ] 0 0 1(τ 1) 0 Here D 1 = and D 1 (τ 1) 0 2 = 0 0 S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 24

25 Dynamic Texture Kernel Kernel Definition: Estimate model and compute kernels between models If we average out the noise then for some W 0 [ ] k((x 0, A, C), (x 0, A, C )) := E e λt yt W y t v,w Kernel Computation: t=1 The kernel can be computed as k = x 0 Mx 0 + ( e λ 1 ) 1 tr [ Q M + W R ] The matrices M and M satisfy M = e λ A C W C A +e λ A M A M = C W C +e λ A M A S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 25

26 Experimental Setup Typical Textures: Some sample textures A long clip was cut to shorter clips of 120 frames each Freak Textures: We also collected some freak textures S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 26

27 Results Kernel Induced Metric: Clips closer on a axis are from the same master clip We plot the kernel induced metric for λ = 0.9 and 0.1 Results fairly independent of the cholice of λ Notice the block diagonal structure of the metric matrix S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 27

28 Roadmap Introduction to Kernel Methods Why kernels? Kernels on Dynamical Systems Trajectories, Noise Models Computation Dynamical Textures ARMA Models Approximate Solutions Kernel Computation Experiments Outlook and Conclusion S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 28

29 Conclusion A new method to embed dynamical systems Analytical solutions for linear systems Many graph kernels are special cases Analytical solutions require cubic time Are better solutions possible for special cases? Extensions to nonlinear systems? Application to dynamical textures Works with approximate model parameters Picks out clips from the same master clip Close relations to rational kernels of Cortes et. al. More information at S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 29

30 Questions? S.V.N. Vishwanathan: Kernels for Dynamic Textures, Page 30