Fast Convolutional Sparse Coding (FCSC)

Transcription

1 Fat Convolutional Spare Coding (FCSC) Bailey ong Department of Computer Science Univerity of California, Irvine Charle C. Fowlke Department of Computer Science Univerity of California, Irvine Introduction Convolutional pare coding remedie the hortcoming of the traditional (patch-baed) pare coding by modeling hift invariance directly. Thi yield more parimoniou coding cheme but increae complexity by coupling the coding problem for neighboring image patche where they overlap. Here we decribe an optimization approach which exploit the eparability of convolution acro band in the frequency domain in order to make dictionary learning efficient. Our preentation follow the notation in the paper of [Britow et al. IEEE CVPR ] but include additional implementation detail and correct ome inaccuracie and typo. Convolutional coding in the frequency domain The objective for convolutional pare coding i arg min d,z x d k z k + z k.t. d k k =,...,, where d k R M i the k-th filter, z k R D i the correponding pare feature map, and x R D M+ i an image. (Note that M D.) For implicity, we dicu uing -dimenional image ignal, but thi generalize to -dimenional image ignal a well. It i well known that convolution correpond to element-wie multiplication in the frequency domain, o it i more efficient to rewrite thi objective in the frequency domain. Unfortunately thi cannot be done due to the l penalty term, which i not rotation invariant. One way to work around thi problem i by introducing two auxiliary variable and t o that we can optimize part of the problem in the frequency domain and another part in the patial domain. The objective can be rewritten a arg min ˆd,,ẑ,t D ˆx ˆd k ẑ k + t k.t. = F ˆdk k =,..., k =,..., t k = F ẑ k k =,..., = C k =,...,, where ˆx, ˆd k, ẑ k C D,, t k R D, and F i the invere dicrete Fourier tranform matrix. The original variable d and z are moved to the frequency domain, and are now denoted a ˆd and ẑ, while the auxiliary variable and t take their place in the patial domain and contraint are added to couple them together. The careful oberver may alo noticed that although originally d k wa M dimenional, i now D dimenional. The mapping matrix C zero out any entrie of which are outide the deired patial upport o the final contraint aure the learned dictionary element have mall patial extent. Notation: F i the dicrete Fourier tranform matrix and F i the invere DFT matrix defined a F = D F. Since the tranform matrix F i orthogonal, the quared l -norm in the patial and frequency domain are equivalent up to a cale factor (i.e., D = ŝ ). The matrix C i the mapping matrix that truncate a vector R D R M and pad the truncation with zero back to R D. Thi i can be defined a the product of two matrice C = PT, where P i our padding matrix (a D M identity matrix) and T i our truncation matrix (a M D identity matrix). For the vector d,, z, and t where the ubcript Thi work wa upported by NSF DBI-4 and NSF DBI- () ()

2 i dropped, we refer to a uper vector with component tacked together (e.g. = [ T,..., T ]T ). The ame applie for the vector in the frequency domain, ŝ = [(F ),..., (F ). Optimization by ADMM To olve thi problem with variable in both the frequency and patial domain, the Alternating Direction Method of Multiplier (ADMM) approach i ued. We begin by writing the augmented Lagrangian that incorporate the equality contraint coupling the auxiliary variable, t to the original variable d, z. L(ˆd,, ẑ, t, λ, λ t ) = D ˆx ˆd k ẑ k + t k + + F ˆdk + F ẑ k t k +.t. k =,..., = C k =,...,, λ T k(f ˆdk ) λ T tk(f ẑ k t k ) () where λ k,, λ tk, and are the Lagrange multiplier aociated with and t repectively. Subproblem z: ẑ ẑ ẑ ẑ L(ẑ; ˆd,, t, λ, λ t ) D ˆx ˆd k ẑ k + F ẑ k t k + D ˆx ˆDẑ + D ẑ ˆt + D ˆλ t(ẑ ˆt) = ( ˆD ˆD + µt I) ( ˆD ˆx + ˆt ˆλ t ), λ T tk(f ẑ k t k ) (4) where ˆD = [diag(ˆd ),..., diag(ˆd )] i D D. Although the matrix ˆD ˆD i quite large, D D, it i pare banded and ha an efficient variable reordering uch that ẑ can be found by olving D independent linear ytem. Subproblem t: t k t k L(t k ; ˆd k,, ẑ k, λ k, λ tk ) t k t k t k + F ẑ k t k + λ T tk(f ẑ k t k ) t k + z k t k + λ T tk(z k t k ) () Given z k and λ tk in the patial domain, it turn out that each coefficient in t k i independent of all other coefficient, t ki t ki (z ki t ki ) + λ tki (z ki t ki ) + t ki. () Which can be efficiently compute in cloed form uing the hrinkage (oft-threholding) function, ( t ki = ign z ki + λ ) { tki max z ki + λ tki },. ()

3 Subproblem d: ˆd ˆd ˆd ˆd L(ˆd;, ẑ, t, λ, λ t ) D ˆx ˆd k ẑ k + F ˆdk + D ˆx Ẑˆd + D ˆd ŝ + D ˆλ (ˆd ŝ) = (Ẑ Ẑ + I) (Ẑ ˆx + ŝ ˆλ ) λ T k(f ˆdk ) where Ẑ = [diag(ẑ ),..., diag(ẑ )] i D D. Like ubproblem z, there i an efficient variable reordering uch that ˆd can be found by olving D independent linear ytem. Subproblem : k L( ; ˆd k, ẑ k, t k, λ k, λ tk ).t. = C F ˆdk + λ T k(f ˆdk ) d k + λ T k(d k ) An equivalent formulation i to drop the patial upport contraint on and intead preerve only the M coefficient of d k and λ k aociated with the mall patial upport. k.t.. Cd k + (Cλ k ) T (Cd k ) The coefficient of outide the patial upport (where Cd k i ) are automatically driven to by the l norm (ee appendix for detail). We can olve the uncontrained verion of thi problem by pulling the linear term into the quadratic, completing the quare k Cd k + (Cλ k ) T (Cd k ) d k + λ T k( d k ) where d k = Cd k, λ k = Cλ k ( dk + ) λk µ d k + λk + d kt dk + λ T d k k ( dk + ) λk µ The optimal olution to the uncontrained problem (denoted by ) i thu given in cloed form by (8) (9) () = Cd k + Cλ k, () Since thi objective i iotropic, we can imply enforce the norm contraint by projecting the uncontrained olution back on to the contraint et: { k k = if. () otherwie Lagrange Multiplier Update: λ (i+) λ (i+) t = λ (i) + (d (i+) (i+) ) () = λ (i) t + (z (i+) t (i+) ) (4)

4 Penalty Update: µ (i+) = { τµ (i) if µ (i) < µ max µ (i) otherwie () Algorithm Convolutional Spare Coding uing Fourier Subproblem and ADMM : Initialize (), z (), t (), λ (), λ () t, µ (), µ () t : Perform FFT (), z (), t (), λ (), λ () : i = 4: repeat : dictionary update: t ŝ (), ẑ (), ˆt (), : Solve for ˆd (i+) given ŝ (i), ẑ (i) (i), ˆλ uing Eq. 8 : Perform invere FFT ˆd (i+) d (i+) 8: Solve for (i+) given d (i+) uing Eq. 9: Solve for (i+) by projecting (i+) uing Eq. : code update: () () ˆλ, ˆλ t : Solve for ẑ (i+) given ˆt (i) (i), ˆλ t, ˆd (i+) uing Eq. 4 : Perform invere FFT ẑ (i+) z (i+) : Solve for t (i+) given z (i+), λ (i) t uing Eq. 4: Update Lagrange multiplier vector uing Eq. and Eq. 4 : Update penaltie uing Eq. : Perform FFT, t, λ, λ t ŝ, ˆt, ˆλ, ˆλ t : i = i + 8: until ˆd,, ẑ, t ha converged 4 Effect of ADMM parameter on optimization Thi ection aim to provide ome intuition regarding the hyper-parameter of FCSC optimization, pecifically we want to undertand how the hyper-parameter interact with one another in term of the convergence rate and the objective value. In all our experiment, we ue image that are contrat normalized. The convergence criteria i that we ve reach a feaible olution (i.e., = F ˆd and t = F ẑ). The FCSC algorithm ha three hyper-parameter,,, and. can be thought of a the parameter that control the convergence rate of the dictionary learning, a it i the tep-ize at which we move toward a feaible olution. Likewie, i the ame for the coding. The hyper-parameter in our original objective control the parity of the code found. (a) (b) (c) Figure : (a) Example A. odoratum gra pollen grain image (SEM, x). (b) Viualization of pare coding of pollen texture. Peudo-color indicate dictionary element with the larget magnitude coefficient (c) Correponding dictionary learned from image of pollen texture uing convolutional pare coding with = 4

5 4. Effect of tep-ize parameter on coding Firt we look at the coding hyper-parameter; uing a pre-trained dictionary, we vary and. Since control the parity of the code, we expect (and the reult how) that a it increae the recontruction error would alo increae. Perhap not urpriingly, the ame behavior i oberved w.r.t. the objective value. A increae the code become more pare to the point where they re all zero. At that point the objective value i jut the recontruction error with a vector of zero. We ee that both the recontruction error and the objective value at convergence increae lowly a increae. In another experiment (not hown) we have een the objective value be roughly the ame for all value of and =, when a minimum number of iteration i enforced (in thoe experiment it wa ). Thee iteration are neceary to overcome the memory of the initial value of t in the ADMM iteration. In term of the convergence rate, for all value of we generally ee that the number of iteration required before convergence decreae a increae. Thi i alo to be expected a we previouly decribed a the tep-ize toward a feaible olution. Ignoring =, we generally oberve that the number of iteration goe down a increae for any particular value of. Thi i a bit urpriing, but can poibly be explained by conidering how parity i induced. We know that the hrinkage function i ued to olve the t-ubproblem, and that at each iteration the code are hrunk by toward zero. If i large enough, then doe not grow enough at the early iteration uch that additional coefficient of z are not zeroed out. Thi in turn magnifie the coefficient of λ t to make z more like t, thu leading to fater convergence. 4. Effect of tep-ize and parity parameter on dictionary learning In the econd experiment, we fix the tep-ize and imultaneouly learn the dictionary and the code. Here the reult are quite different from before. We ee that mut be trictly greater than for FCSC to learn anything meaningful. When = both the recontruction error and the objective value are largely unchanged w.r.t.. Finally in term of convergence, we continue to oberve that a increae the number of iteration to convergence decreae for all value of, conitent with our firt experiment. 4. Effect of tep-ize on runtime and accuracy In the third experiment, we fix the hyper-parameter equal to and look at how and interact. With fixed, we ee that the objective value increae a and increae in the domain > and >. Thi eem to be a reult of the optimization enforcing contraint too quickly. For mall value of and (not hown) we alo ee erratic behavior. If we ignore the region where i le than, we ee that both the objective and the recontruction error are mooth w.r.t. the hyper-parameter. Thi i quite nice, a we can chooe a computational trade-off between lowering the objective value and the number of iteration to convergence. We may alo ak whether the optimization produce qualitatively good filter. A good learned filter houldn t reemble the initialization and in general we expect to ee Gabor-like oriented edge filter. After examining ome learned filter that were ubjectively good/bad, we defined goodne a having a tandard deviation greater than.. Much higher than the flat initialization which ha a tandard deviation of. and other bad looking filter that had tandard deviation le than. in the cae we examined. While not a perfect meaure by any mean, we can ee in the next et of figure that the recontruction error i indeed correlated with the goodne meaure. x 4... Original objective x 4 4 Recontruction error 8 4 Iter. to convergence Figure : Effect of, on coding with fixed dictionary.

6 x 4. Original objective x 4 Recontruction error Iter. to convergence Figure : Effect of, on joint coding and dictionary learning ( = ). Appendix A - Leat-quare filter with contrained patial upport To ee why Eq. i equivalent to the previou argmin (Eq. 9), conider the following problem = d + λ T (d ).t. = C [ ] d + d + λ T (d ) + λ T (d ),.t. =, where we plit each vector into two ub-vector: the part that correpond to mall patial upport (denoted with ) and the part that doe not (denoted with ). By plitting into and, we can think about each a eparate problem a there are no term that contain both. d + λ T (d ) Td + Tλ T (Td ) d + λ T (d ).t. = So the problem amount to an uncontrained linear leat quare; the problem can be tranformed into an equivalent uncontrained problem with a trivial olution when certain condition hold. Thee condition are that d and λ are null vector. The matrix C doe exactly thi, a it preerve the coefficient of any vector correponding to the mall patial upport and zero out the ret. Original objective Iter. to convergence Good Figure 4: Joint effect of, on olution quality and run time ( = )

7 Appendix B - Implementation detail During our dicuion of ubproblem z and d, we mentioned an efficient variable reordering to find the optimal olution by olving D independent linear ytem. Firt we define the following matrice Ẑ = [ẑ,..., ẑ, ˆT = [ˆt,..., ˆt, ˆD = [ˆd,..., ˆd, Ŝ = [ŝ,..., ŝ ˆΛ t = [ˆλt,..., ˆλ t, ˆΛ = [ˆλ,..., ˆλ ˆ Dx = [ˆd ˆx,..., ˆd ˆx, ˆ Zx = [ẑ ˆx,..., ẑ ˆx and denote Ẑi, ˆT i, ˆD i, Ŝi, ˆΛ ti, ˆΛ i, Dxi ˆ, and Zx ˆ i to be the i-th column of their repective matrice. We can then olve the ubproblem z with and ubproblem d with Ẑ i = ( ˆD i ˆD i + I) ( Dx ˆ i + ˆTi + ˆΛ ti ), ˆD i = (ẐiẐ i + I) ( ˆ Zx i + Ŝ i + ˆΛ i ). In Eq. 8, we olved for ˆd k which the invere Fourier tranform need to be applied to get d k. However, ince only a ub-vector of d k i needed computing the invere of the entire vector i wateful. If we look at the definition of C, we ee that we can define a new matrix Φ = TF that will invert jut the part of the vector we need. And the maller M i with repect to D, the greater the aving we gain. We can redefine Eq. and Eq. a = Φˆd k + Φˆλ k. () { k k = P if otherwie () Our entire dicuion thu far ha revolved around a ingle image, to handle multiple image i quite traightforward and only require u to change Eq. 8, N N ˆd = ( Ẑ xẑx + I) ( Ẑ xˆx x + ŝ ˆλ ). (8) x= x= Note that ˆT i unrelated to th truncation matrix T