Signal Modeling: From Convolutional Sparse Coding to Convolutional Neural Networks

Transcription

1 Signal Modeling: From Convolutional Spare Coding to Convolutional Neural Network Vardan Papyan The Computer Science Department Technion Irael Intitute of technology Haifa 32000, Irael Joint work with Jeremia Sulam Yaniv Romano Prof. Michael Elad The reearch leading to thee reult ha been received funding from the European union' Seventh Framework Program (FP/ ) ERC grant Agreement ERC-SPARSE

2 Part I Motivation and Background 2

3 Our Starting Point: Image Denoiing Original Image X White Gauian Noie E Noiy Image Y Many image denoiing algorithm can be cat a the minimization of an energy function of the form 1 2 X Y G(X) Relation to meaurement Prior or regularization 3

4 Leading Image Denoiing Method are built upon powerful patch-baed local model: K-SVD: pare repreentation modeling of image patche [Elad & Aharon, 06] BM3D: combine parity and elf-imilarity [Dabov, Foi, Katkovnik & Egiazarian 07] EPLL: ue GMM of the image patche [Zoran & Wei 11] CSR: clutering and parity on patche [Dong, Li, Lei & Shi 11] MLP: multi-layer perceptron [Burger, Schuler & Harmeling 12] NCSR: non-local parity with centralized coefficient [Dong, Zhang, Shi & Li 13] WNNM: weighted nuclear norm of image patche [Gu, Zhang, Zuo & Feng 14] SSC GSM: nonlocal parity with a GSM coefficient model [Dong, Shi, Ma & Li 15] 4

5 The Spare-Land Model Aume every patch a linear combination of a few column, called atom, from a matrix that i termed a dictionary. The operator R i extract the i-th n-dimenional patch from X R N. Spare coding: P 0 : min γ i γ i 0. t. R i X = Ωγ i N n m > n Ω = n R i X R i = i-th location n γ i * R i for 1D ignal 5

6 Patch Denoiing Given a noiy patch R i Y, olve P ϵ 0 : γ i = argmin γ i 0. t. R i Y Ωγ i 2 ϵ γ i Clean patch: Ω γ i P 0 and (P 0 ϵ ) are hard to olve Greedy method uch a Orthogonal Matching Puruit (OMP) or Threholding Convex relaxation uch a Bai Puruit (BP) P ϵ 2 1 : min γ i 1 + ξ R i Y Ωγ i 2 γ i 6

7 Conider thi Algorithm [Elad & Aharon, 06] Noiy Image Initial Dictionary Uing K-SVD Update the dictionary Recontructed Image Denoie each patch Uing OMP X = argmin X, γ i i,ω 1 2 X Y λ Relation to meaurement i γ i 0 + μ R i X Ωγ i 2 2 Prior or regularization 7

8 What i Miing? Over the year, many reearcher kept reviiting thi algorithm and the line of thinking behind it, with a clear feeling that the final word ha not been aid, and that key feature are till lacking. What i miing? Here i what our group thought of A multi-cale treatment [Ophir, Lutig & Elad 11] [Sulam, Ophir & Elad 14] [Papyan & Elad 15] Exploiting elf-imilaritie [Ram & Elad 13] [Romano, Protter & Elad 14] Puhing to better agreement on the overlap [Romano & Elad 13] [Romano & Elad 15] Enforcing the local model on the final patche (EPLL) [Sulam & Elad 15] Beyond all Beyond thee, a all key thee, part a that key i part miing that i a miing theoretical i a backbone for the local theoretical model a a backbone way to characterize for the local the model unknown a a image. way to characterize the global unknown image 8

9 Miing Theoretical Backbone? The core global-local model aumption on X: i R i X = Ωγ i where γ i 0 k Every patch in the unknown ignal i expected to have a pare repreentation w.r.t. the ame dictionary Ω Quetion to conider: I. Who are thoe ignal belonging to thi model? Do they exit? II. Under which condition on Ω would thi model be feaible? III. How doe one ample from thi model? IV. How hould we perform puruit properly (and locally) under thi model? V. How hould we learn Ω if thi i indeed the model? 9

10 In thi Talk Limitation of patch averaging Convolutional Spare Coding (CSC) model Multi-Layer Convolutional Spare Coding (ML-CSC) Theoretical tudy of CSC Convolutional neural network (CNN) Freh view of CNN through the eye of parity 10

11 Part II Convolutional Spare Coding Working Locally Thinking Globally-Part I: Theoretical Guarantee for Convolutional Spare Coding Working Locally Thinking Globally-Part II: Stability and Algorithm for Convolutional Spare Coding Vardan Papyan, Jeremia Sulam and Michael Elad 11

12 Convolutional Sparity Aume X =

13 Convolutional Spare Repreentation Formally, conider the following global parity-baed model X = i=1 C i R N N i a banded and Circulant matrix containing a ingle atom with all of it hift. Γ i R N are it correponding coefficient. m C i Γ i = DΓ n C i = N 13

14 Two Interpretation C 1 C 2 C 3 = m D = n D L 14

15 Thi model ha been ued in the pat [Lewicki & Sejnowki 99] [Hahimoto & Kurata, 00] Mot work have focued on olving efficiently it aociated puruit, called convolutional pare coding, uing the BP algorithm. P ϵ 2 1 : min Γ 1 + ξ Y DΓ 2 Convolutional Γ dictionary Several application were demontrated: Inpainting [Heide, Heidrich & Wetztein 15] Super-reolution [Gu, Zuo, Xie, Meng, Feng & Zhang 15] Pattern detection in image and the analyi of intrument in muic ignal [Mørup, Schmidt & Hanen 08] However, little i known regrading it theoretical apect. 15

16 Claical Spare Theory (Noiele) Mutual Coherence: μ D = max i j d i T d j [Donoho & Elad 03] For a ignal X = DΓ, if Γ 0 < thi olution i necearily the paret. μ D then [Donoho & Elad 03] The OMP and BP are guaranteed to recover the true pare code auming that Γ 0 < μ D. [Tropp 04] [Donoho & Elad 03] 16

17 The Need for a Theoretical Study Auming that m = 2 and n = 64we have that μ(d) A a reult, uniquene and ucce of puruit i guaranteed a long a Γ 0 < μ(d) Thi i a very peimitic reult! Le than 8 non-zero GLOBALLY!!! 17

18 The Local Repreentation R i X = (2n 1)m n γ i X = DΓ R i X = Ωγ i tripe-dictionary tripe vector Adjacent repreentation overlap, a they kip by m item a we weep through the patche of X 18

19 The Local Repreentation R i+1 X = (2n 1)m n γ i+1 X = DΓ R i X = Ωγ i tripe-dictionary tripe vector Adjacent repreentation overlap, a they kip by m item a we weep through the patche of X 19

20 Inherent Poitive Propertie A clear global model with a hift invariant local prior. Every patch ha a pare repreentation w.r.t. to a local dictionary Ω. No diagreement on the patch overlap. Related to the current common practice of patch averaging. The ignal can be written a X = DΓ = 1 n i R i T Ωγ i R i T put the patch Ωγ i in the i-th location in the N-dimenional vector. The patch averaging cheme olve the pare coding problem independently for every patch while convolutional pare coding eek for the repreentation of all the patche together. 20

21 The l 0, Norm and the P 0, Problem Γ 0, = max i γ i 0 m = 2 γ i P 0, : min Γ 0, Γ. t. X = DΓ A global pare vector i likely if it can repreent every patch in the ignal parely. The Main Quetion We Aim to Addre: I. Uniquene of the olution to thi problem? II. Guaranteed recovery of the olution via global OMP/BP? 21

22 Uniquene via Mutual Coherence P 0, : min Γ 0, Γ. t. X = DΓ Theorem: If a olution Γ i found for (P 0, ) uch that: Γ 0, < μ D Then thi i necearily the unique globally optimal olution to thi problem. We hould be excited about thi reult and later one becaue they poe a local contraint for a global guarantee, and a uch, they are far more optimitic compared to the global guarantee 8 non-zero per tripe can reult in 0.06 N non-zero globally 22

23 Recovery Guarantee P 0, : min Γ 0,. t. X = DΓ Γ Let olve thi problem via OMP or BP, applied globally Theorem: If a olution Γ of (P 0, ) atifie: Γ 0, < μ D Then global OMP and BP are guaranteed to find it. Both OMP and BP do not aume local parity but till ucceed. One could propoe algorithm that rely on thi aumption 23

24 From Ideal to Noiy Signal So far, we have aumed an ideal ignal X = DΓ. However, in practice we uually have Y = DΓ + E where E i due to noie or model deviation. To handle thi, we redefine our problem a: ϵ P 0, : min Γ 0,. t. Y DΓ 2 ϵ Γ The Main Quetion We Aim to Addre: I. Stability of the olution to thi problem? II. Stability of the olution obtained via global OMP/BP? III. The ame recovery done via local operation? 24

25 Stability of Pϵ 0, via Stripe-RIP [Cande & Tao 05] ϵ P 0, : min Γ 0,. t. Y DΓ 2 ϵ Γ Γ Definition: D i aid to atify Stripe-RIP with contant δ k if: 1 δ k Δ 2 2 DΔ δ k Δ 2 for any vector Δ with Δ 0, = k. Theorem: If the true repreentation Γ atifie Γ 0, = k < μ D ϵ Then a olution Γ for (P 0, ) mut be cloe to it 2 4ϵ 2 4ϵ 2 Γ Γ 2 1 δ 2k 1 2k 1 μ D δ k (k 1)μ(D) 25

26 Local Noie Aumption Thu far, our analyi relied on the local parity of the underlying olution Γ, which wa enforced through the l 0, norm. In what follow, we preent tability guarantee for both OMP and BP that will alo depend on the local energy in the noie vector E. Thi will be enforced via the l 2, norm, defined a: p E 2, = max i R i E 2 26

27 Stability of OMP Theorem: If Y = DΓ + E where Γ 0, < μ D 1 μ D E 2, Then OMP run for Γ 0 iteration will 1. Find the correct upport 2. Γ OMP Γ 2 2 E 2 1 Γ 0, 1 μ D p Γ min 27

28 Stability of Lagrangian BP P 1 ϵ : Γ BP = min Γ 1 2 Y DΓ ξ Γ 1 p Theorem: For Y = DΓ + E, if ξ = 4 E 2, Γ 0, < μ D Then we are guaranteed that and 1. The upport of Γ BP i contained in that of Γ 2. Γ BP Γ 7.5 E 2, p 3. Every entry greater than 7.5 E 2, will be found 4. Γ BP i unique. p Proof relie on the work of [Tropp 06] 28

29 Stability of Lagrangian BP P 1 ϵ : Γ BP = min Γ 1 2 Y DΓ ξ Γ 1 p Theorem: For Y = DΓ + E, if ξ = 4 E 2, and Γ 0, < Theoretical foundation for recent work tackling the μ D convolutional pare coding Then we are guaranteed that problem via BP 1. The upport of Γ BP i contained in that of Γ 2. Γ BP Γ 7.5 E 2, 3. Every entry greater than 7.5 E 2, 4. Γ BP i unique. p [Britow, Erikon & Lucey 13] [Wohlberg 14] [Kong & Fowlke 14] p [Britow & Lucey 14] will be found [Heide, Heidrich & Wetztein 15] [Šorel & Šroubek 16] Proof relie on the work of [Tropp 06] 29 28

30 Global Puruit via Local Proceing P ϵ 1 : Γ BP = min 2 Y DΓ ξ Γ 1 Thu far, we have een that while the CSC i a global model, it theoretical guarantee rely on local propertie. Yet thi global-local relation can alo be exploited for practical purpoe. Next, we how how one can olve the global BP problem uing only local operation. Iterative Soft Threholding [Blumenath & Davie 08]: Projection onto L 1 ball Γ t = S ξ/c Thi can be equally written a: i α i t = S ξ/c α i t 1 + D L T R i (Y DΓ t 1 ) local pare code Γ 1 Γ t c DT Y DΓ t 1 Gradient tep local dictionary global aggregation local reidual * c > 0.5 λ max (D T D) 30 Γ α i

31 Simulation Detail: Signal length: N = 300 Patch ize: n = 25 Unique atom: p = 5 Global parity: k = 40 Number of iteration: 400 Lagrangian: ξ = 4 E 2, p 31

32 Partial Summary of CSC What we have een o far i a new way to analyze the global CSC model uing local parity contrain. We proved: Uniquene of the olution to the noiele problem. Stability of the olution to the noiy problem. Guarantee of ucce and tability of both OMP and BP. We obtained guarantee and algorithm that operate locally while claiming global optimality. 32

33 Part III Going Deeper Convolutional Neural Network Analyzed via Convolutional Spare Coding Vardan Papyan, Yaniv Romano and Michael Elad 33

34 CSC and CNN There eem to be a relation between CSC and CNN: Both have a convolutional tructure. Both ue a data driven approach for training their model. The mot popular non-linearity employed in CNN, called ReLU, i known to be connected to parity. In thi part, we aim to make thi connection clear in order to provide a theoretical undertanding of CNN through the eye of parity. But firt, a hort review of CNN 34

35 CNN X ReLU ReLU [LeCun, Bottou, Bengio and Haffner 98] [Krizhevky, Sutkever & Hinton 12] [Simonyan & Zierman 14] [He, Zhang, Ren & Sun 15] ReLU z = max 0, z 35

36 CNN N X m 1 N N m 2 N N n 1 n 1 W 2 N n 0 n 0 W 1 No pooling tage: Can be replaced by a convolutional layer with increaed tride without lo in performance [Springenberg, Doovitkiy, Brox & Riedmiller 14] The current tate-of-the-art in image recognition doe not ue it [He, Zhang, Ren & Sun 15] 36

37 Mathematically... f X, W i, b i = ReLU b 2 + W T 2 ReLU b 1 + W T 1 X Z 2 R Nm 2 b W T 2 R Nm 2 Nm 1 2 R Nm 2 n 1 m 1 b 1 R Nm 1 W 1 T R Nm 1 N m 2 m 1 n 0 X R N ReLU ReLU m 1 37

38 Training Stage of CNN Conider the tak of claification, for example. Given a et of ignal X j and their correponding label h X j j, j the CNN learn an end-to-end mapping. min W i, b i,u j l h X j, U, f X, W i, b i True label Claifier Output of lat layer 38

39 Back to CSC X R N D 1 R N Nm 1 m 1 n 0 Γ 1 R Nm 1 Similarly, the repreentation themelve could alo be aumed to have uch a tructure. Γ 1 R Nm 1 D 2 R Nm 1 Nm 2 Γ 2 R Nm 2 n 1 m 1 m 2 Convolutional parity aume an inherent tructure i preent in natural ignal. m 1 Multi-Layer CSC (ML-CSC) 39

40 Deep Coding and Learning Problem DCP λ : Find a et of repreentation atifying X = D 1 Γ 1 Γ 1 0, λ 1 Γ 1 = D 2 Γ 2 Γ 2 0, λ 2 Γ K 1 = D K Γ K Γ K 0, λ K DLP λ : min l h X j, U, DCP X j, D i D i i=1 K,U j True label Tak driven dictionary learning [Mairal, Bach & Ponce 12] Claifier Deepet repreentation obtained by olving the DCP 40

41 Deep Coding and Learning Problem DCP E λ : Find a et of repreentation atifying Y D 1 Γ 1 2 E 0 Γ 1 0, λ 1 Γ 1 D 2 Γ 2 2 E 1 Γ 2 0, λ 2 Γ K 1 D K Γ K 2 E K 1 Γ K 0, λ K DLP λ E : min l h X j, U, DCP Y j, D i D i i=1 K,U j True label Claifier Deepet repreentation obtained by olving the DCP 41

42 The implet puruit in the pare repreentation i the threholding algorithm. Given an input ignal X, thi operate by: D T X P β D T X Retricting the coefficient to be nonnegative doe not retrict the expreivene of the model ReLU = Soft Nonnegative Threholding [Fawzi, Davie & Froard 15] 42

43 Conider thi for Solving the DCP Layered threholding (LT): Etimate Γ 1 via the threholding algorithm Γ 2 = P β2 D T 2 P β1 D T 1 X Etimate Γ 2 via the threholding algorithm DCP λ : Find a et of repreentation atifying X = D 1 Γ 1 Γ 1 0, λ 1 Γ 1 = D 2 Γ 2 Γ 2 0, λ 2 Γ K 1 = D K Γ K Γ K 0, λ K Forward pa of CNN: f X = ReLU b 2 + W 2 T ReLU b 1 + W 1 T X The layered (oft nonnegative) threholding and the forward pa algorithm are equal!!! 43

44 Conider thi for Solving the DLP DLP: min l h X j, U, DCP X j, D i K,U j D i i=1 CNN training: Etimate via the layered threholding algorithm min W i, b i,u j l h X j, U, f X, W i, b i The problem olved by the training tage of CNN and the DLP are equal auming that the DCP i approximated via the layered threholding algorithm 44

45 Theoretical Quetion M A X = D 1 Γ 1 Γ 1 = D 2 Γ 2 Γ K 1 = D K Γ K X Y DCP λ E Layered Threholding (Forward Pa) K Γ i i=1 Γ i i L 0, pare Other? 45

46 Uniquene of DCP λ DCP λ : Find a et of repreentation atifying X = D 1 Γ 1 Γ 1 0, λ 1 Γ 1 = D 2 Γ 2 Γ 2 0, λ 2 Γ K 1 = D K Γ K Γ K 0, λ K Theorem: If a et of olution Γ K i i=1 i found for (DCP λ ) uch that: Γ i 0, λ i < μ D i Then thee are necearily the unique olution to thi problem. The feature map CNN aim to recover are unique I thi et unique? [Papyan, Sulam & Elad 16] 47

47 Stability of DCP λ E Theorem: If the true repreentation Γ K i i=1 Γ i 0, λ i < μ D i And the error threhold for (DCP E λ ) are E 0 2 = E 2 2, E i 2 = 2 4E i Γ i 0, atify 1 μ D i Then the et of olution Γ i i=1 obtained by olving thi problem mut be cloe to the true one Γ i Γ i 2 The problem CNN aim to olve i table under certain condition K 2 Ei 2 [Papyan, Sulam & Elad 16] 49

48 Stability of LT Theorem: If Γ i 0, < μ D i Then the layered hard oft threholding will* 1. Find the correct upport p 2. Γ LT i i Γ i ε 2, L We have defined ε 0 p L = E 2, and Γ min i Γ max 1 i μ D i ε L i 1 Γ i max i p ε L = Γ i 0, ε i 1 L + μ D i Γ i 0, 1 Γ max i + β i The tability of the forward pa i guaranteed if the underlying repreentation are locally pare and the noie i locally bounded * For correctly choen threhold 51

49 Limitation of the Forward Pa The tability analyi reveal everal inherent limitation of the forward pa algorithm: Even in the noiele cae, it i incapable of recovering the olution of the DCP problem. It ucce depend on the ratio Γ i min / Γ i max. Thi i a direct conequence of the forward pa algorithm relying on the imple threholding operator. The ditance between the true pare vector and the etimated one increae exponentially a function of the layer depth. In the next and final part we propoe a new algorithm attempting to olve ome of thee problem. 52

50 Part IV What Next? Convolutional Neural Network Analyzed via Convolutional Spare Coding Vardan Papyan, Yaniv Romano and Michael Elad 54

51 Layered Bai Puruit (Noiele) Our Goal: DCP λ : Find a et of repreentation atifying X = D 1 Γ 1 Γ 1 0, λ 1 Layered threholding: Γ 2 = P β2 D 2 T P β1 D 1 T X Threholding i the implet puruit known in the field of parity. Layered BP: Γ 1 = D 2 Γ 2 Γ 2 0, λ 2 Γ K 1 = D K Γ K Γ K 0, λ K Γ 1 LBP = min Γ 1 Γ 1 1. t. X = D 1 Γ 1 Γ 2 LBP = min Γ 2 Γ 2 1. t. Γ 1 LBP = D 2 Γ 2 Deconvolutional network [Zeiler, Krihnan, Taylor & Fergu 10] 55

52 Guarantee for Succe of Layered BP Our Goal: DCP λ : Find a et of repreentation atifying X = D 1 Γ 1 Γ 1 0, λ 1 Γ 1 = D 2 Γ 2 Γ 2 0, λ 2 Γ K 1 = D K Γ K Γ K 0, λ K Theorem: If a et of olution Γ i i=1 of (DCP λ ) atify Γ i 0, λ i < μ D i Then the layered BP i guaranteed to find them. The layered BP can retrieve the underlying repreentation in the noiele cae, a tak in which the forward pa fail. It ucce doe not depend on the ratio Γ i min / Γ i max. [Papyan, Sulam & Elad 16] 56

53 Stability of Layered BP Theorem: Auming that Γ i 0, < μ D i Then we are guaranteed that* 1. The upport of Γ i LBP i contained in that of Γ i 2. Γ i LBP Γ i 2, ε L i 3. Every entry in Γ i greater than ε i p L / Γ i 0, will be found ε L i = 7.5 i E 2, i j=1 p Γ j 0, K * For correctly choen ξ i i=1 [Papyan, Sulam & Elad 16] 58

54 Layered Iterative Threholding Layered BP Γ LBP 1 1 = min Γ 1 2 Y D 1Γ ξ 1 Γ 1 1 Γ LBP 1 2 = min Γ 2 2 Γ 1 LBP 2 D 2 Γ ξ2 Γ 2 1 Can be een a a recurrent neural network [Gregor & LeCun 10] Layered IT Γ 1 t = S ξ1 /c 1 Γ 1 t c 1 D 1 T Y D 1 Γ 1 t 1 Γ 2 t = S ξ2 /c 2 Γ 2 t c 2 D 2 T Γ 1 D 2 Γ 2 t 1 * c i > 0.5 λ max (D i T D i ) 59

55 Concluion We decribed the limitation of patch baed proceing a a motivation for the CSC model. We then preented a theoretical tudy of thi model both in a noiele and a noiy etting. A multi-layer extenion for it, tightly connected to CNN, wa propoed and imilarly analyzed. Finally, an alternative to the forward pa algorithm wa preented. Future Work: leveraging theoretical inight into practical implication 60

56 Quetion? 61