Technical Report: Multidimensional, Downsampled Convolution for Autoencoders

Transcription

1 Techncal Report: Multdmensonal, Downsampled Convoluton for Autoencoders Ian Goodfellow August 9, 2010 Abstract Ths techncal report descrbes dscrete convoluton wth a multdmensonal kernel. Convoluton mplements matrx multplcaton by a sparse matrx wth several elements constraned to be equal to each other. To mplement a convolutonal autoencoder, the gradents of ths operaton, the transpose of ths operaton, and the gradents of the transpose are all needed. When usng standard convoluton, each of these supplementary operatons can be descrbed as a convoluton on slghtly moded arguments. When the output s mplctly downsampled by movng the kernel n more than one pxel at each step, we must dene two new operatons n order to compute all of the necessary values. 1 Dentons Let L be our loss functon, W our weghts denng the kernel, d a vector of strdes, H our hdden unts, and V our vsble unts. H cj ndexes poston c (an N-dmensonal ndex) wthn feature map for example j. V s of the same format as H. W cj ndexes the weght at poston c wthn the kernel, connectng vsble channel to hdden channel j. Convoluton wth downsamplng s performed (assumng W s pre-pped) by H cj = k,m W km V d c+k,m,j (Where s elementwse product) In the followng sectons, I derve all of the necessary operatons to use ths operaton n an autoencoder. You may want to skp drectly to the summary of results, secton 7. 2 Basc gradent The gradent of the loss functon wth respect to the weghts s gven by 1

2 = = W cj k,m,n = k,n k,m,n H kmn H kmn W cj p,q W pqmv d k+p,q,n H kmn W cj p,q W pqjv d k+p,q,n H kjn W cj = k,n H kjn V d k+c,,n W cj = k,m H kjm V d k+c,,m Wth a few dmshues, the gradent can be computed as a convoluton provded that d s all 1s. However, f d s not 1 for any element, then we have a problem because durng forward prop, the ndex nto the output s multpled by the strde, whle durng computaton of the gradent, the ndex nto the kernel s multpled by the strde. 3 Transpose We can thnk of strded convoluton as multplcaton by a matrx M. Let h be H reshaped nto a vector and v be V reshaped nto a vector. Then h = Mv Let hr(c,, j) be a reshapng functon that maps ndces n H to ndces n h. Let vr(c,, j) be the same for V and v. Then h hr(c,,j) = k,m W km v vr(d c+k,m,j) Thus M hr(c,,j),vr(d c+k,m,j) = W km where all the relevant ndces take on approprate values, and 0 elsewhere. Suppose we want to calculate R, a tensor n the same shape as V, such that R cj = r vr(c,,j) and r = M T h. r a = M ab h b 2

3 r a = c,,j,k,m vr(d c+k,m,j)=a R q,m,j = W km H cj W km H cj To sum over the correct set of values for c and k, the we wll need a modulus operator or saved nformaton from a prevous teraton of a for loop, unless d = 1. So ths s not a convoluton n the large strde case. In the case where d = 1, we have R q,m,j = W w p,m, H q w+p,,j p where w = W.shape 1. Changng some varable names, we get R c,,j = k,m W w k,,m H c w+k,m,j Recall that a strde 1 convoluton s gven by: H cj = k,m W km V c+k,m,j So our transpose may be calculated by paddng d 1 zeros to H (each dmenson of the vector gves the number of zeros to pad to each dmenson of the hdden unt tensor), ppng all the spatal dmensons of the kernel, and exchangng the nput channel and output channel dmensons of the kernel. 4 New notaton I'm gong to make up some new notaton now, snce our operaton sn't really convoluton (downsamplng s bult nto the operaton, we don't p the kernel, etc). From here on out, I wll wrte H cj = k,m W km V c d+k,m,j as and H = d V 3

4 as R q,m,j = W km H cj R = T d H and (H = d V ) W cj = k,m H kjm V d k+c,,m as W L(H = d V ) = ( H L)# d V 5 Autoencoder gradents To make an autoencoder, we'll need to be able to compute R = g v (b v + W T d g h (b h + W d V )) Ths means we're also gong to need to be able to take the gradent of T H) wth respect to both W (so we know how to update the encodng weghts) and H, so we'll be able to propagate gradents back to the encodng layer. Fnally, when we stack the autoencoders nto a convoluton MLP, we'll need to be able to propagate gradents back from one layer to another, so we must also nd the gradent of ) wth respect to V. 5.1 Gradents of the loss appled to the transpose Wth respect to the weghts so R qmj = = W xyz q,m,j W km H cj R qmj R qmj W xyz = q,m,j W kmh cj R qmj W xyz 4

5 = q,j c d c+x=q W xyzh czj R qyj W xyz = q,j H czj R qyj c d c+x=q = c,j Changng some varable names, we get W cj = k,m R d c+x,y,j H czj H kjm R d k+c,,m Recall that the gradent of ) wth respect to the kernel s: W cj = k,m H kjm V d k+c,,m Ths has the same form as the gradent we just derved, e both use the new #operaton. Thus we can wrte that the gradent of T H) wth respect to the kernel s gven by Wth respect to the nputs W L(R = T d H) = H# d R L so R qmj = = H xyz q,m,j W km H cj R qmj R qmj H xyz = q,m,j = q,m W kmh cj R qmj H xyz k d x+k=q W kmyh xyz R qmz H xyz 5

6 = q,m = k,m Changng some varable names, we get H cj = k,m W kmy R qmz k d x+k=q R d x+k,m,z W kmy W km R d c+k,m,j Remember that so we can wrte H cj = k,m W km V d c+k,m,j H L(R = T d H) = d R L 5.2 Gradent of the loss appled wth respect to the nputs The above s sucent to make a sngle layer autoencoder. To stack on top of t, we also need to compute: (H = d V ) V xyz = cj = c = cj H cj H cj V xyz k,m W kmv d c+k,m,j H cj V xyz k d c+k=x W kyv xyz H cz V xyz = c W ky H cz k d c+k=x = c,k d c+k=x Changng varable names around, we get (H = d V ) = V qmj H cz W ky V L(H = d V ) = T d H L W km H cj 6

7 6 The rest of the gradents We now know enough to make a stacked autoencoder. However, there are stll some gradents that may be taken and t would be nce f our ops supported all of them. op's gradent can be expessed n terms T and #, and T op's gradent can be expressed n terms and #. Thus f we add a gradent method to the # op, our ops wll be nntely derentable for all combnatons of varables. 6.1 # wth respect to kernel Let A = B# d C. Then = c,,j (A) B xyz = c,,j (A) A cj A cj B xyz (A) k,m B kjmc d k+c,,m A cj B xyz = c, (A) A cy B xyz C d x+c,,z B xyz = c, (A) A cy C d x+c,,z Renamng varables, we get So (A) B cj = k,m 6.2 # wth respect to nput Let A = B# d C. Then (A) A km C d c+k,m,j B L(A = B# d C) = ( A L)@ d C = c,,j (A) C xyz = c,,j (A) A cj A cj C xyz (A) k,m B kjmc d k+c,,m A cj C xyz 7

8 = c,k d k+c=x = Renamng varables, we get (A) C qmj = j c,k d k+c=x (A) A cyj B kjz C xyz C xyz j (A) A cyj B kjz (A) A km B cj C L(A = B# d C) = ( A L)@ T d B 7 Summary We have dened these operatons: Strded convoluton: H = d V H cj = k,m W km V c d+k,m,j Transpose of strded convoluton: R = T d H R qmj = W km H cj Gradent of strded convoluton wth respect to weghts: A = B# d C A cj = k,m B kjm C d k+c,,m We have observed that, f and only f f and only f d = T and # may both be expressed n terms by modfyng ther arguments wth the operatons of zero paddng, exchangng tensor dmensons, or mrror magng tensor dmensons. More 1 T and #T may be expressed n ths way 1 n terms 1. We have shown that the followng denttes are sucent to compute any dervatve any of the three operatons: W L(H = d V ) = ( H L)# d V V L(H = d V ) = T d H L 8

9 W L(R = T d H) = H# d R L H L(R = T d H) = d R L B L(A = B# d C) = ( A L)@ d C C L(A = B# d C) = ( A L)@ T d B 9