Review: Learning Bimodal Structures in Audio-Visual Data

Transcription

1 Review: Learning Bimodal Structures in Audio-Visual Data CSE 704 : Readings in Joint Visual, Lingual and Physical Models and Inference Algorithms Suren Kumar Vision and Perceptual Machines Lab 106 Davis Hall UB North Campus surenkum@buffalo.edu January 28, 2014 Suren Kumar January 28, /19

2 Summary What? Learn bimodally informative structures from audio-visual signals Why? Understand complex relationship between the inputs to different sensory modalities How? Represent audio-video signals as sparse sum of kernels consisting of audio-waveform and a spatio-temporal visual basis. Motivation: Biological Evidence, Evolution Suren Kumar January 28, 2014 Introduction 2/19

3 Literature Detect synchronous co-occurrences of transient structures in different modalities. 1. Extract fixed and predefined unimodal features in audio and video stream separately 2. Analyze correlation between the resulting feature representation What is the problem with it? Suren Kumar January 28, 2014 Introduction 3/19

4 Literature Detect synchronous co-occurrences of transient structures in different modalities. 1. Extract fixed and predefined unimodal features in audio and video stream separately 2. Analyze correlation between the resulting feature representation What is the problem with it? Suren Kumar January 28, 2014 Introduction 3/19

5 Basic Steps Capture bimodal signal structure by shift-invariance sparse generative model. Unsupervised learning for forming an overcomplete dictionary Suren Kumar January 28, 2014 Introduction 4/19

6 p th norm of a vector x in Euclidean space is defined as x p = ( n i=1 x i p ) 1 p Suren Kumar January 28, 2014 Background 5/19

7 Matching Pursuit Consider a dictionary D = (g) g D in a Hilbert space H. The dictionary is countable, normalized ( g = 1), complete, but not orthogonal and possibly redundant. Sparse Approximation Problem: Given N > 0 and f H, construct an N-term combination f N = k=1,2,..n c kg k with g k D which approximates f at best, and study how fast f N converges to f. Sparse Recovery Problem: if f has an unknown representation f = g c g g with (c g ) g D a (possibly) sparse sequence, recover this sequence exactly or approximately from the data of f. Building an optimal sparse representation of arbitrary signals is NP-hard problem. Suren Kumar January 28, 2014 Background:Matching Pursuit 6/19

8 Matching Pursuit Matching pursuit is a greedy approximation to the problem. Initialization f 0 = 0 Projection Step: At step k 1, projection step is the approximation of f f k 1 = Span{g 1,..., g k 1 } Selection Step: Choice of next element based on residual r k 1 = f f k 1 g k = arg max g D < r k 1, g > Suren Kumar January 28, 2014 Background:Matching Pursuit 7/19

9 Convolutional Generative Model Audio visual data s = (a, v), a(t), v(x, y, t) Dictionary {φ k }, φ k = (φ (a) k (t), φ(v) k (x, y, t)) Each atom can be translated to any point in space and time using operator T (p,q,r) T (p,q,r) = (φ (a) k (t r), φ(v) k (x p, y q, t r)) Thus an audio-visual signal can be represented as s K nk k=1 i=1 c k i T (p,q,r)ki φ k, c ki = (c (a), c (v) k i ) Coding Find the coefficients (sparse) Learning Learning the dictionary k i Suren Kumar January 28, 2014 Audio-Visual Representation 8/19

10 Audio-Visual Matching Pursuit Transient substructures that co-occur simultaneously are indicative of common underlying physical cause. Discrete audio-visual translation τ Signal Approximation. Start with R 0 s = s Suren Kumar January 28, 2014 Audio-Visual Representation 9/19

11 Audio-Visual Matching Pursuit The function φ 0 and its spatio-temporal translation are chosen maximizing similarity measures C(R 0 s, φ) When to Stop? Suren Kumar January 28, 2014 Audio-Visual Representation 10/19

12 Audio-Visual Matching Pursuit The function φ 0 and its spatio-temporal translation are chosen maximizing similarity measures C(R 0 s, φ) When to Stop? Number of iterations N or the maximum value of C between residual and dictionary elements falls below a threshold. Similarity measure should reflect properties of human perception Unaffected by small relative time-shifts Suren Kumar January 28, 2014 Audio-Visual Representation 10/19

13 Choosing the norm Suren Kumar January 28, 2014 Audio-Visual Representation 11/19

14 Learning Find the kernel functions from a given set of audio-visual data. Assuming the noise to be gaussian Suren Kumar January 28, 2014 Learning:Gradient Ascent Learning 12/19

15 Learning Find the kernel functions from a given set of audio-visual data. Assuming the noise to be gaussian Suren Kumar January 28, 2014 Learning:Gradient Ascent Learning 12/19

16 Learning Update: φ k [j] = φ k [j 1] + ηδφ k Suren Kumar January 28, 2014 Learning:Gradient Ascent Learning 13/19

17 Learning Key Idea : Update only one atom φ k Update each function with principal component of the residual errors. In general has faster convergence compared to GA. Suren Kumar January 28, 2014 Learning:K-SVD algorithm 14/19

18 Synthetic Data Audio - 3 sine waves and video has four black shapes Suren Kumar January 28, 2014 Experiments 15/19

19 Audio-Visual Speech Speaker uttering the digits from zero to nine in english Study convergence for 1 and 2 norm with GA and K-SVD C 1 encodes joint audio-visual structures better. K-SVD is faster compared to GA. Suren Kumar January 28, 2014 Experiments 16/19

20 Suren Kumar January 28, 2014 Experiments 17/19

21 Sound Source Localization CUAVE - Visual Distractor and Acoustin Distractor Filter Audio Corresponding audio filter audio function and store the maximum projection Filter with corresponding video function and store maximum projection. Cluster video position Suren Kumar January 28, 2014 Experiments 18/19

22 Summary Audio - Visual Matching Pursuit Similarity measures and learning algorithms Testing for speaker localization with distractors. Suren Kumar January 28, 2014 Experiments 19/19

23 Reference popov/learning/cohen.pdf Suren Kumar January 28, 2014 Experiments 20/19