TENSOR LAYERS FOR COMPRESSION OF DEEP LEARNING NETWORKS. Cris Cecka Senior Research Scientist, NVIDIA GTC 2018

Transcription

1 TENSOR LAYERS FOR COMPRESSION OF DEEP LEARNING NETWORKS Cris Cecka Senior Research Scientist, NVIDIA GTC 2018

2 Tensors Computations and the GPU AGENDA Tensor Networks and Decompositions Tensor Layers in Deep Learning 2

3 TENSOR COMPUTATIONS AND THE GPU Modern data is inherently multi-dimensional. 3

4 TENSOR CONTRACTIONS Core primitive of multilinear algebra BLAS level 3 unbounded compute intensity. X

5 TENSOR LIBRARIES Explicit permutation dominates. Y. Shi, U. N. Niranjan, A. Anandkumar and C. Cecka, "Tensor Contractions with Extended BLAS Kernels on CPU and GPU," 2016 IEEE 23rd International Conference on High Performance Computing (HiPC), Hyderabad, 2016, pp X

6 CONTRACTIONS : Single GEMM (Provided compact layout) X

7 BATCHED MATRIX-MATRIX MULTIPLY cublas<t>gemmstridedbatched X

8 CONTRACTIONS : Single SB-GEMM (Any layout) X

9 APPLICATION: FFT : Tensor/FFT vendor optimized cublas<t>gemmstridedbatched : Custom kernel : FMM Communication StridedBatchedGEMM: 75%+ of the runtime 1.5x over cufft on 2xV x over cufft on 8xV100 Cris Cecka. Low communication FMM-accelerated FFT on GPUs." In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '17). ACM, New York, NY, USA. X

10 WHY TENSORS? 10

11 DENSITY AND SPARSITY H. Anzt, S. Tomov, J. Dongarra, Energy Efficiency and Performance Frontiers for Sparse Computations on GPU Supercomputers," PMAM

12 DENSITY AND SPARSITY In general, need < 5% sparsity for a computational win. Solutions Block-sparse Locally dense and globally sparse 12

13 TENSOR DECOMPOSITIONS Decompositions for data sparse representations. 13

14 <latexit sha1_base64="yfuqph02d2wzf4wnsgc+egamvqw=">aaab83icbvbns8naej3ur1q/qh69lbbbu0leua9c1yvhcsyw2lam2227dloju5tccf0dxjyoepxpeppfug1z0oqdgcd7m8zmcxpbtxhdl6ewtlyyulzcl21sbm3vlhf3hnscksp8gotynuputhdjfmonym1emyxcwrrh8gbqn0zmar7leznowbbhx/iep2isflrrjamkl+tkvqdccavudoqv8xjsgrz1tvmz3y1pgjfpqectw56bmcbdztgvbfjqp5olsifyzy1ljuzmb9ns6ak5skqx9gjlsxoyu39ozbhppy5c2xmhgehfbyr+57vs0zspmi6t1dbj54t6qsamjtmesjcrro0yw4jucxsroqnusi3nqwrd8bzf/kv8k+pf1bs7rdsu8zskcachcaweneenbqeoplb4hcd4gvdn5dw7b877vlxg5dp78avoxzfrs5bt</latexit> <latexit sha1_base64="yfuqph02d2wzf4wnsgc+egamvqw=">aaab83icbvbns8naej3ur1q/qh69lbbbu0leua9c1yvhcsyw2lam2227dloju5tccf0dxjyoepxpeppfug1z0oqdgcd7m8zmcxpbtxhdl6ewtlyyulzcl21sbm3vlhf3hnscksp8gotynuputhdjfmonym1emyxcwrrh8gbqn0zmar7leznowbbhx/iep2isflrrjamkl+tkvqdccavudoqv8xjsgrz1tvmz3y1pgjfpqectw56bmcbdztgvbfjqp5olsifyzy1ljuzmb9ns6ak5skqx9gjlsxoyu39ozbhppy5c2xmhgehfbyr+57vs0zspmi6t1dbj54t6qsamjtmesjcrro0yw4jucxsroqnusi3nqwrd8bzf/kv8k+pf1bs7rdsu8zskcachcaweneenbqeoplb4hcd4gvdn5dw7b877vlxg5dp78avoxzfrs5bt</latexit> <latexit sha1_base64="yfuqph02d2wzf4wnsgc+egamvqw=">aaab83icbvbns8naej3ur1q/qh69lbbbu0leua9c1yvhcsyw2lam2227dloju5tccf0dxjyoepxpeppfug1z0oqdgcd7m8zmcxpbtxhdl6ewtlyyulzcl21sbm3vlhf3hnscksp8gotynuputhdjfmonym1emyxcwrrh8gbqn0zmar7leznowbbhx/iep2isflrrjamkl+tkvqdccavudoqv8xjsgrz1tvmz3y1pgjfpqectw56bmcbdztgvbfjqp5olsifyzy1ljuzmb9ns6ak5skqx9gjlsxoyu39ozbhppy5c2xmhgehfbyr+57vs0zspmi6t1dbj54t6qsamjtmesjcrro0yw4jucxsroqnusi3nqwrd8bzf/kv8k+pf1bs7rdsu8zskcachcaweneenbqeoplb4hcd4gvdn5dw7b877vlxg5dp78avoxzfrs5bt</latexit> <latexit sha1_base64="yfuqph02d2wzf4wnsgc+egamvqw=">aaab83icbvbns8naej3ur1q/qh69lbbbu0leua9c1yvhcsyw2lam2227dloju5tccf0dxjyoepxpeppfug1z0oqdgcd7m8zmcxpbtxhdl6ewtlyyulzcl21sbm3vlhf3hnscksp8gotynuputhdjfmonym1emyxcwrrh8gbqn0zmar7leznowbbhx/iep2isflrrjamkl+tkvqdccavudoqv8xjsgrz1tvmz3y1pgjfpqectw56bmcbdztgvbfjqp5olsifyzy1ljuzmb9ns6ak5skqx9gjlsxoyu39ozbhppy5c2xmhgehfbyr+57vs0zspmi6t1dbj54t6qsamjtmesjcrro0yw4jucxsroqnusi3nqwrd8bzf/kv8k+pf1bs7rdsu8zskcachcaweneenbqeoplb4hcd4gvdn5dw7b877vlxg5dp78avoxzfrs5bt</latexit> <latexit sha1_base64="ri3kwg/mnrn2znxkfmv//ojvv6i=">aaacmxicbvdlssnafj34rpvvdelmsaiuslke6kjodeoygrgfjotj9lydo3kwmxfqykf4f/6bw/0axym48yectf3y1gmd5557l2fu8wpopdlnd2npewv1bb2wudzc2t7zle3t38koerrsgvfith0igbmqbmuuh3ysgaq+h5y/vmr7rqcqkkxhrrrf4aakh7ieo0rpysvvnicoasu8bwreyu6hdnce4qvcmkmcgywjhbqkqv4wi16pbfbmmfaisaakjkzoeqvvpxvrjibquu6k7fhmrnyucmuoh6zojbk0xzd0oanpsakqbjo+mmphwunixit0cxueq383uhjioqp8pzmfi+d7ufhfr5oo3pmbsjbofir0ytrlofyrztpcxsaakj7shfdb9f8xhrbbqnkzzrgonnzm8lis+rawiv2tnfesm9ny/xkatwedoin0gixuq3v0jzrirhq9orf0it6mz+pd+ds+jqnlxntnam3a+pkfd4eprq==</latexit> <latexit sha1_base64="ri3kwg/mnrn2znxkfmv//ojvv6i=">aaacmxicbvdlssnafj34rpvvdelmsaiuslke6kjodeoygrgfjotj9lydo3kwmxfqykf4f/6bw/0axym48yectf3y1gmd5557l2fu8wpopdlnd2npewv1bb2wudzc2t7zle3t38koerrsgvfith0igbmqbmuuh3ysgaq+h5y/vmr7rqcqkkxhrrrf4aakh7ieo0rpysvvnicoasu8bwreyu6hdnce4qvcmkmcgywjhbqkqv4wi16pbfbmmfaisaakjkzoeqvvpxvrjibquu6k7fhmrnyucmuoh6zojbk0xzd0oanpsakqbjo+mmphwunixit0cxueq383uhjioqp8pzmfi+d7ufhfr5oo3pmbsjbofir0ytrlofyrztpcxsaakj7shfdb9f8xhrbbqnkzzrgonnzm8lis+rawiv2tnfesm9ny/xkatwedoin0gixuq3v0jzrirhq9orf0it6mz+pd+ds+jqnlxntnam3a+pkfd4eprq==</latexit> <latexit sha1_base64="ri3kwg/mnrn2znxkfmv//ojvv6i=">aaacmxicbvdlssnafj34rpvvdelmsaiuslke6kjodeoygrgfjotj9lydo3kwmxfqykf4f/6bw/0axym48yectf3y1gmd5557l2fu8wpopdlnd2npewv1bb2wudzc2t7zle3t38koerrsgvfith0igbmqbmuuh3ysgaq+h5y/vmr7rqcqkkxhrrrf4aakh7ieo0rpysvvnicoasu8bwreyu6hdnce4qvcmkmcgywjhbqkqv4wi16pbfbmmfaisaakjkzoeqvvpxvrjibquu6k7fhmrnyucmuoh6zojbk0xzd0oanpsakqbjo+mmphwunixit0cxueq383uhjioqp8pzmfi+d7ufhfr5oo3pmbsjbofir0ytrlofyrztpcxsaakj7shfdb9f8xhrbbqnkzzrgonnzm8lis+rawiv2tnfesm9ny/xkatwedoin0gixuq3v0jzrirhq9orf0it6mz+pd+ds+jqnlxntnam3a+pkfd4eprq==</latexit> <latexit sha1_base64="ri3kwg/mnrn2znxkfmv//ojvv6i=">aaacmxicbvdlssnafj34rpvvdelmsaiuslke6kjodeoygrgfjotj9lydo3kwmxfqykf4f/6bw/0axym48yectf3y1gmd5557l2fu8wpopdlnd2npewv1bb2wudzc2t7zle3t38koerrsgvfith0igbmqbmuuh3ysgaq+h5y/vmr7rqcqkkxhrrrf4aakh7ieo0rpysvvnicoasu8bwreyu6hdnce4qvcmkmcgywjhbqkqv4wi16pbfbmmfaisaakjkzoeqvvpxvrjibquu6k7fhmrnyucmuoh6zojbk0xzd0oanpsakqbjo+mmphwunixit0cxueq383uhjioqp8pzmfi+d7ufhfr5oo3pmbsjbofir0ytrlofyrztpcxsaakj7shfdb9f8xhrbbqnkzzrgonnzm8lis+rawiv2tnfesm9ny/xkatwedoin0gixuq3v0jzrirhq9orf0it6mz+pd+ds+jqnlxntnam3a+pkfd4eprq==</latexit> <latexit sha1_base64="/q3t93gl7ycydt4l5wu1z0hwdha=">aaacnxicdvdlsgmxfm3uv62vqks3wsk4kjmiqitc1y0boyk1hc4wzdk3bwgmmyqzsqz9kjd+h7tuxki49rdmh6c2eihhcm653htpkhcmtg0prdzc4tlysn61sla+sblv3n65u3eqkdrpzgpzdigczgtundmcmokeegucgkhvcqq37keqfotb3u/ai0hhsdajrbvkl14tn+ekph//bgadjrigilqyhwf2arhr7lr/axdht72c/wljltvjwvpamyismlbnlz67yuztcismncjvcuxeexmrmleog4kbkkgi7zeotawujallzeozb/jamcfux9i8ofgy/dmrkuipfhqyp1mxq2a1efmx1kp1+9tlmehsdyjobrvtjnwmrxnikemgmvcnifqysyumxsij1sbpggnbmt15htspymdl5+a4vl2yppfhe2gfhsihnaaquki1vecupaihekvv1pp1yr1bhxnrzpr27kjfzx1+axlcqjw=</latexit> <latexit sha1_base64="/q3t93gl7ycydt4l5wu1z0hwdha=">aaacnxicdvdlsgmxfm3uv62vqks3wsk4kjmiqitc1y0boyk1hc4wzdk3bwgmmyqzsqz9kjd+h7tuxki49rdmh6c2eihhcm653htpkhcmtg0prdzc4tlysn61sla+sblv3n65u3eqkdrpzgpzdigczgtundmcmokeegucgkhvcqq37keqfotb3u/ai0hhsdajrbvkl14tn+ekph//bgadjrigilqyhwf2arhr7lr/axdht72c/wljltvjwvpamyismlbnlz67yuztcismncjvcuxeexmrmleog4kbkkgi7zeotawujallzeozb/jamcfux9i8ofgy/dmrkuipfhqyp1mxq2a1efmx1kp1+9tlmehsdyjobrvtjnwmrxnikemgmvcnifqysyumxsij1sbpggnbmt15htspymdl5+a4vl2yppfhe2gfhsihnaaquki1vecupaihekvv1pp1yr1bhxnrzpr27kjfzx1+axlcqjw=</latexit> <latexit sha1_base64="/q3t93gl7ycydt4l5wu1z0hwdha=">aaacnxicdvdlsgmxfm3uv62vqks3wsk4kjmiqitc1y0boyk1hc4wzdk3bwgmmyqzsqz9kjd+h7tuxki49rdmh6c2eihhcm653htpkhcmtg0prdzc4tlysn61sla+sblv3n65u3eqkdrpzgpzdigczgtundmcmokeegucgkhvcqq37keqfotb3u/ai0hhsdajrbvkl14tn+ekph//bgadjrigilqyhwf2arhr7lr/axdht72c/wljltvjwvpamyismlbnlz67yuztcismncjvcuxeexmrmleog4kbkkgi7zeotawujallzeozb/jamcfux9i8ofgy/dmrkuipfhqyp1mxq2a1efmx1kp1+9tlmehsdyjobrvtjnwmrxnikemgmvcnifqysyumxsij1sbpggnbmt15htspymdl5+a4vl2yppfhe2gfhsihnaaquki1vecupaihekvv1pp1yr1bhxnrzpr27kjfzx1+axlcqjw=</latexit> <latexit sha1_base64="/q3t93gl7ycydt4l5wu1z0hwdha=">aaacnxicdvdlsgmxfm3uv62vqks3wsk4kjmiqitc1y0boyk1hc4wzdk3bwgmmyqzsqz9kjd+h7tuxki49rdmh6c2eihhcm653htpkhcmtg0prdzc4tlysn61sla+sblv3n65u3eqkdrpzgpzdigczgtundmcmokeegucgkhvcqq37keqfotb3u/ai0hhsdajrbvkl14tn+ekph//bgadjrigilqyhwf2arhr7lr/axdht72c/wljltvjwvpamyismlbnlz67yuztcismncjvcuxeexmrmleog4kbkkgi7zeotawujallzeozb/jamcfux9i8ofgy/dmrkuipfhqyp1mxq2a1efmx1kp1+9tlmehsdyjobrvtjnwmrxnikemgmvcnifqysyumxsij1sbpggnbmt15htspymdl5+a4vl2yppfhe2gfhsihnaaquki1vecupaihekvv1pp1yr1bhxnrzpr27kjfzx1+axlcqjw=</latexit> <latexit sha1_base64="aqfzgufox1r0sgh/gadmhi+mxiw=">aaagyxic7vrpa9swhfu71+u8p027yy9iyzbbaurg2eyzlo6yywflwohnkgu5ezfli8lbm+opua/qe7/brhtmdl02ozvumb8ynt97ehlv8ityzprg6gzn94g1zz/cf+q8fvl02chg8oizygpj6jrkpjnxevaum0gnmmlor3jjcrpxehmt3tf65rcqfcvej73oazjihwaji1gban5oju4q0qutpwarbzkjupc0mimcox2h3enobelmasksbteqcviiseu6y79ysghheev8fy7q2ihj/xu8oyejv/05deafr02he0ej9x3+zgbazpc7gaif0bzkpqnj6tpqs/yclrnqexc6clbfbivzfvn/inhmgyfyutnwe3gtgij2luad2ydosjfsoqnhss08louwxfizwk1kuciay7lcczozuocuqrbsfmifxxomhqzt8wkng/bveyvolvqnkxgmwc9vx6vjf2mzqidvwpkjvnbuklulkojdncf6u8kysuo0xxuaiwtmrzasscremx3cuaxup6pr8folbilpxh3reh8nw7pztp99caxegbhwwgtwbj6aczafxppu/bb+wr/2bm3hhthhd9bdnfbmc9az+/g3hyi3rw==</latexit> <latexit sha1_base64="aqfzgufox1r0sgh/gadmhi+mxiw=">aaagyxic7vrpa9swhfu71+u8p027yy9iyzbbaurg2eyzlo6yywflwohnkgu5ezfli8lbm+opua/qe7/brhtmdl02ozvumb8ynt97ehlv8ityzprg6gzn94g1zz/cf+q8fvl02chg8oizygpj6jrkpjnxevaum0gnmmlor3jjcrpxehmt3tf65rcqfcvej73oazjihwaji1gban5oju4q0qutpwarbzkjupc0mimcox2h3enobelmasksbteqcviiseu6y79ysghheev8fy7q2ihj/xu8oyejv/05deafr02he0ej9x3+zgbazpc7gaif0bzkpqnj6tpqs/yclrnqexc6clbfbivzfvn/inhmgyfyutnwe3gtgij2luad2ydosjfsoqnhss08louwxfizwk1kuciay7lcczozuocuqrbsfmifxxomhqzt8wkng/bveyvolvqnkxgmwc9vx6vjf2mzqidvwpkjvnbuklulkojdncf6u8kysuo0xxuaiwtmrzasscremx3cuaxup6pr8folbilpxh3reh8nw7pztp99caxegbhwwgtwbj6aczafxppu/bb+wr/2bm3hhthhd9bdnfbmc9az+/g3hyi3rw==</latexit> <latexit sha1_base64="aqfzgufox1r0sgh/gadmhi+mxiw=">aaagyxic7vrpa9swhfu71+u8p027yy9iyzbbaurg2eyzlo6yywflwohnkgu5ezfli8lbm+opua/qe7/brhtmdl02ozvumb8ynt97ehlv8ityzprg6gzn94g1zz/cf+q8fvl02chg8oizygpj6jrkpjnxevaum0gnmmlor3jjcrpxehmt3tf65rcqfcvej73oazjihwaji1gban5oju4q0qutpwarbzkjupc0mimcox2h3enobelmasksbteqcviiseu6y79ysghheev8fy7q2ihj/xu8oyejv/05deafr02he0ej9x3+zgbazpc7gaif0bzkpqnj6tpqs/yclrnqexc6clbfbivzfvn/inhmgyfyutnwe3gtgij2luad2ydosjfsoqnhss08louwxfizwk1kuciay7lcczozuocuqrbsfmifxxomhqzt8wkng/bveyvolvqnkxgmwc9vx6vjf2mzqidvwpkjvnbuklulkojdncf6u8kysuo0xxuaiwtmrzasscremx3cuaxup6pr8folbilpxh3reh8nw7pztp99caxegbhwwgtwbj6aczafxppu/bb+wr/2bm3hhthhd9bdnfbmc9az+/g3hyi3rw==</latexit> <latexit sha1_base64="aqfzgufox1r0sgh/gadmhi+mxiw=">aaagyxic7vrpa9swhfu71+u8p027yy9iyzbbaurg2eyzlo6yywflwohnkgu5ezfli8lbm+opua/qe7/brhtmdl02ozvumb8ynt97ehlv8ityzprg6gzn94g1zz/cf+q8fvl02chg8oizygpj6jrkpjnxevaum0gnmmlor3jjcrpxehmt3tf65rcqfcvej73oazjihwaji1gban5oju4q0qutpwarbzkjupc0mimcox2h3enobelmasksbteqcviiseu6y79ysghheev8fy7q2ihj/xu8oyejv/05deafr02he0ej9x3+zgbazpc7gaif0bzkpqnj6tpqs/yclrnqexc6clbfbivzfvn/inhmgyfyutnwe3gtgij2luad2ydosjfsoqnhss08louwxfizwk1kuciay7lcczozuocuqrbsfmifxxomhqzt8wkng/bveyvolvqnkxgmwc9vx6vjf2mzqidvwpkjvnbuklulkojdncf6u8kysuo0xxuaiwtmrzasscremx3cuaxup6pr8folbilpxh3reh8nw7pztp99caxegbhwwgtwbj6aczafxppu/bb+wr/2bm3hhthhd9bdnfbmc9az+/g3hyi3rw==</latexit> <latexit sha1_base64="wsz5rmw8cwo8oy0jbztjxbo84hs=">aaacwxiclvfds8mwfe2rbnn+tffos3aopo1wbpvhmpxfxwnwddyy0vrui6zptvjxlp1jx0twrwhmh6buvngg5hdovcnnszhyprtjvfv2yupaovhal29sbm3vvhb37lwssqoetxgioyfrwjkatzpnoznkihhior0+xk/89jnixrjxp0cpbdezcnznlggj9sppl72cpyxxa/shdjjiw5hoyv7g2kdrovhrwu77/zzarn+nnncvunpqzhr4mbhzuknzthqvvz9kabad0jqtpbquk+ogj1izymfc9jmfkagpzabdqwwjqqx5njoxpjrkhpujnetopfv/duqkvmouh6bsjdhui95e/mvrzrp/hurmpjkgqwcx9toodyinoeoisacajwwhvdizk6zdignv5jfkjgr38cnlxdupx9td29na82qergntown0jfx0hprobrwqhyh6q59wwspah7ztl+zyrns25j1v9at29quzkldw</latexit> <latexit sha1_base64="wsz5rmw8cwo8oy0jbztjxbo84hs=">aaacwxiclvfds8mwfe2rbnn+tffos3aopo1wbpvhmpxfxwnwddyy0vrui6zptvjxlp1jx0twrwhmh6buvngg5hdovcnnszhyprtjvfv2yupaovhal29sbm3vvhb37lwssqoetxgioyfrwjkatzpnoznkihhior0+xk/89jnixrjxp0cpbdezcnznlggj9sppl72cpyxxa/shdjjiw5hoyv7g2kdrovhrwu77/zzarn+nnncvunpqzhr4mbhzuknzthqvvz9kabad0jqtpbquk+ogj1izymfc9jmfkagpzabdqwwjqqx5njoxpjrkhpujnetopfv/duqkvmouh6bsjdhui95e/mvrzrp/hurmpjkgqwcx9toodyinoeoisacajwwhvdizk6zdignv5jfkjgr38cnlxdupx9td29na82qergntown0jfx0hprobrwqhyh6q59wwspah7ztl+zyrns25j1v9at29quzkldw</latexit> <latexit sha1_base64="wsz5rmw8cwo8oy0jbztjxbo84hs=">aaacwxiclvfds8mwfe2rbnn+tffos3aopo1wbpvhmpxfxwnwddyy0vrui6zptvjxlp1jx0twrwhmh6buvngg5hdovcnnszhyprtjvfv2yupaovhal29sbm3vvhb37lwssqoetxgioyfrwjkatzpnoznkihhior0+xk/89jnixrjxp0cpbdezcnznlggj9sppl72cpyxxa/shdjjiw5hoyv7g2kdrovhrwu77/zzarn+nnncvunpqzhr4mbhzuknzthqvvz9kabad0jqtpbquk+ogj1izymfc9jmfkagpzabdqwwjqqx5njoxpjrkhpujnetopfv/duqkvmouh6bsjdhui95e/mvrzrp/hurmpjkgqwcx9toodyinoeoisacajwwhvdizk6zdignv5jfkjgr38cnlxdupx9td29na82qergntown0jfx0hprobrwqhyh6q59wwspah7ztl+zyrns25j1v9at29quzkldw</latexit> <latexit sha1_base64="wsz5rmw8cwo8oy0jbztjxbo84hs=">aaacwxiclvfds8mwfe2rbnn+tffos3aopo1wbpvhmpxfxwnwddyy0vrui6zptvjxlp1jx0twrwhmh6buvngg5hdovcnnszhyprtjvfv2yupaovhal29sbm3vvhb37lwssqoetxgioyfrwjkatzpnoznkihhior0+xk/89jnixrjxp0cpbdezcnznlggj9sppl72cpyxxa/shdjjiw5hoyv7g2kdrovhrwu77/zzarn+nnncvunpqzhr4mbhzuknzthqvvz9kabad0jqtpbquk+ogj1izymfc9jmfkagpzabdqwwjqqx5njoxpjrkhpujnetopfv/duqkvmouh6bsjdhui95e/mvrzrp/hurmpjkgqwcx9toodyinoeoisacajwwhvdizk6zdignv5jfkjgr38cnlxdupx9td29na82qergntown0jfx0hprobrwqhyh6q59wwspah7ztl+zyrns25j1v9at29quzkldw</latexit> = A = Scalar TENSOR NETWORKS Notation and Visualization 2 3 a i = A i = = Vector A ij = = Matrix A ijk` = A ijk` = = Tensor 14

15 <latexit sha1_base64="iopidmcvwb55+b9azaw1bz0vyng=">aaacexicbvdlssnafj3uv42vqasxbgal4kokiqi7qhuxfywtncfmppn2zgqszizcdfkl/8ct/oarcesxupddnlrd2oqbc4dz7uxee8kmuals+8uolswula/uv8219y3nlwt7506mucdexsllrtdekjdkiauoyqsbcykskjfogf9vfuebceltfqtggfetnoa0ohgplqxwnpcgncsifrdlund72comlayzwa27ay8b/xjnshpginzgfxv9focj4qozjgxpstplf0goihkpts+xjem4rgps05sjhei/gd9qwkot9ggucl1cwbh6e6jaizsjjnsd1bly3qve/7xerqizv6a8yxxhelioyhlukazsgh0qcfzspancgupbir4igbdsmc1sutr+lktynpkq/hl3uhnedg5ogq3lat51sa8owbfwwclogwvqbi7aoatp4aw8gk/gm/fufexaa8z0zhfmwpj8arfdnbs=</latexit> <latexit sha1_base64="iopidmcvwb55+b9azaw1bz0vyng=">aaacexicbvdlssnafj3uv42vqasxbgal4kokiqi7qhuxfywtncfmppn2zgqszizcdfkl/8ct/oarcesxupddnlrd2oqbc4dz7uxee8kmuals+8uolswula/uv8219y3nlwt7506mucdexsllrtdekjdkiauoyqsbcykskjfogf9vfuebceltfqtggfetnoa0ohgplqxwnpcgncsifrdlund72comlayzwa27ay8b/xjnshpginzgfxv9focj4qozjgxpstplf0goihkpts+xjem4rgps05sjhei/gd9qwkot9ggucl1cwbh6e6jaizsjjnsd1bly3qve/7xerqizv6a8yxxhelioyhlukazsgh0qcfzspancgupbir4igbdsmc1sutr+lktynpkq/hl3uhnedg5ogq3lat51sa8owbfwwclogwvqbi7aoatp4aw8gk/gm/fufexaa8z0zhfmwpj8arfdnbs=</latexit> <latexit sha1_base64="iopidmcvwb55+b9azaw1bz0vyng=">aaacexicbvdlssnafj3uv42vqasxbgal4kokiqi7qhuxfywtncfmppn2zgqszizcdfkl/8ct/oarcesxupddnlrd2oqbc4dz7uxee8kmuals+8uolswula/uv8219y3nlwt7506mucdexsllrtdekjdkiauoyqsbcykskjfogf9vfuebceltfqtggfetnoa0ohgplqxwnpcgncsifrdlund72comlayzwa27ay8b/xjnshpginzgfxv9focj4qozjgxpstplf0goihkpts+xjem4rgps05sjhei/gd9qwkot9ggucl1cwbh6e6jaizsjjnsd1bly3qve/7xerqizv6a8yxxhelioyhlukazsgh0qcfzspancgupbir4igbdsmc1sutr+lktynpkq/hl3uhnedg5ogq3lat51sa8owbfwwclogwvqbi7aoatp4aw8gk/gm/fufexaa8z0zhfmwpj8arfdnbs=</latexit> <latexit sha1_base64="iopidmcvwb55+b9azaw1bz0vyng=">aaacexicbvdlssnafj3uv42vqasxbgal4kokiqi7qhuxfywtncfmppn2zgqszizcdfkl/8ct/oarcesxupddnlrd2oqbc4dz7uxee8kmuals+8uolswula/uv8219y3nlwt7506mucdexsllrtdekjdkiauoyqsbcykskjfogf9vfuebceltfqtggfetnoa0ohgplqxwnpcgncsifrdlund72comlayzwa27ay8b/xjnshpginzgfxv9focj4qozjgxpstplf0goihkpts+xjem4rgps05sjhei/gd9qwkot9ggucl1cwbh6e6jaizsjjnsd1bly3qve/7xerqizv6a8yxxhelioyhlukazsgh0qcfzspancgupbir4igbdsmc1sutr+lktynpkq/hl3uhnedg5ogq3lat51sa8owbfwwclogwvqbi7aoatp4aw8gk/gm/fufexaa8z0zhfmwpj8arfdnbs=</latexit> <latexit sha1_base64="mo2atz9sjjutm6gpmd564ganmba=">aaacahicbvbns8naej3urxq/qh69lbbbu0leug9vlx4rgftoq9lst+3azsbsboqacvifenu/4em8+k+8+0pctdny1gcdj/dmmjkxxjwp7tjfvmlpewv1rbxub2xube9udvfuvzriqj0s8ui2aqwoz4j6mmlow7gkoaw4bqaj69xvplkpwctu9dimfoghgvuzwdpirctuyh4y2+5wqk7nmqatercgvsjq6fz+or2ijcevmncsvnt1yu2nwgpgom3stqjojmkid2jbuifdqvx0cm+gjozsq/1imhiatds/eykolrqhgekmsr6qes8x//paie6f+yktcakpinnf/yqjhah8edrjkhlnx4zgipm5fzehlphoe9hmfs1gt1keizsfwilxtmoxnff2tfq/kvipwwecwjg4caz1uiegeecawwu8wpv1bl1bh9bntlvkftp7mapr6xda3jbc</latexit> <latexit sha1_base64="mo2atz9sjjutm6gpmd564ganmba=">aaacahicbvbns8naej3urxq/qh69lbbbu0leug9vlx4rgftoq9lst+3azsbsboqacvifenu/4em8+k+8+0pctdny1gcdj/dmmjkxxjwp7tjfvmlpewv1rbxub2xube9udvfuvzriqj0s8ui2aqwoz4j6mmlow7gkoaw4bqaj69xvplkpwctu9dimfoghgvuzwdpirctuyh4y2+5wqk7nmqatercgvsjq6fz+or2ijcevmncsvnt1yu2nwgpgom3stqjojmkid2jbuifdqvx0cm+gjozsq/1imhiatds/eykolrqhgekmsr6qes8x//paie6f+yktcakpinnf/yqjhah8edrjkhlnx4zgipm5fzehlphoe9hmfs1gt1keizsfwilxtmoxnff2tfq/kvipwwecwjg4caz1uiegeecawwu8wpv1bl1bh9bntlvkftp7mapr6xda3jbc</latexit> <latexit sha1_base64="mo2atz9sjjutm6gpmd564ganmba=">aaacahicbvbns8naej3urxq/qh69lbbbu0leug9vlx4rgftoq9lst+3azsbsboqacvifenu/4em8+k+8+0pctdny1gcdj/dmmjkxxjwp7tjfvmlpewv1rbxub2xube9udvfuvzriqj0s8ui2aqwoz4j6mmlow7gkoaw4bqaj69xvplkpwctu9dimfoghgvuzwdpirctuyh4y2+5wqk7nmqatercgvsjq6fz+or2ijcevmncsvnt1yu2nwgpgom3stqjojmkid2jbuifdqvx0cm+gjozsq/1imhiatds/eykolrqhgekmsr6qes8x//paie6f+yktcakpinnf/yqjhah8edrjkhlnx4zgipm5fzehlphoe9hmfs1gt1keizsfwilxtmoxnff2tfq/kvipwwecwjg4caz1uiegeecawwu8wpv1bl1bh9bntlvkftp7mapr6xda3jbc</latexit> <latexit sha1_base64="mo2atz9sjjutm6gpmd564ganmba=">aaacahicbvbns8naej3urxq/qh69lbbbu0leug9vlx4rgftoq9lst+3azsbsboqacvifenu/4em8+k+8+0pctdny1gcdj/dmmjkxxjwp7tjfvmlpewv1rbxub2xube9udvfuvzriqj0s8ui2aqwoz4j6mmlow7gkoaw4bqaj69xvplkpwctu9dimfoghgvuzwdpirctuyh4y2+5wqk7nmqatercgvsjq6fz+or2ijcevmncsvnt1yu2nwgpgom3stqjojmkid2jbuifdqvx0cm+gjozsq/1imhiatds/eykolrqhgekmsr6qes8x//paie6f+yktcakpinnf/yqjhah8edrjkhlnx4zgipm5fzehlphoe9hmfs1gt1keizsfwilxtmoxnff2tfq/kvipwwecwjg4caz1uiegeecawwu8wpv1bl1bh9bntlvkftp7mapr6xda3jbc</latexit> <latexit sha1_base64="rq+uxvchirixp4zke39hxf3+jdy=">aaaci3icbvdltsjafj3ic/gfunqzkzjahrtgrn2bblxiikjcstmdbmfk+mbmsojnv8g/8a/c6g+4mm5cupndbkelau9yk5nz7s2999gbz1lp+peww1ldw9/ibxa2tnd294r7b/fsdwwfjvw5l9o2kcczb03ffid2iic4noewpbxo/dyyhgs+d6cmaxrd0veywyhriwqvk6zl1iashtvjkyqzh0p5aalnlribmarzgnetkbjfhyjvlolvfqq8tiymlfcghlx8mxs+dv3wfoveyo6hb6obeaey5raxzfbcqoiq9kgtui+4ilvr9kuynyrkdzu+smptekr+nyiik+xetzpo9ag56kxif14nvm5fn2jeecrw6gyre3ksfjzmg3tmafv8khbcbutuxxrabkeqsxfui2ldxzinxvgmyzk0t6uxvep2rfs7yvljoyn0jmriqoeohm5qazurru/obb2in+1ze9c+tm9za07lzg7rhltvx8lhpfa=</latexit> <latexit sha1_base64="rq+uxvchirixp4zke39hxf3+jdy=">aaaci3icbvdltsjafj3ic/gfunqzkzjahrtgrn2bblxiikjcstmdbmfk+mbmsojnv8g/8a/c6g+4mm5cupndbkelau9yk5nz7s2999gbz1lp+peww1ldw9/ibxa2tnd294r7b/fsdwwfjvw5l9o2kcczb03ffid2iic4noewpbxo/dyyhgs+d6cmaxrd0veywyhriwqvk6zl1iashtvjkyqzh0p5aalnlribmarzgnetkbjfhyjvlolvfqq8tiymlfcghlx8mxs+dv3wfoveyo6hb6obeaey5raxzfbcqoiq9kgtui+4ilvr9kuynyrkdzu+smptekr+nyiik+xetzpo9ag56kxif14nvm5fn2jeecrw6gyre3ksfjzmg3tmafv8khbcbutuxxrabkeqsxfui2ldxzinxvgmyzk0t6uxvep2rfs7yvljoyn0jmriqoeohm5qazurru/obb2in+1ze9c+tm9za07lzg7rhltvx8lhpfa=</latexit> <latexit sha1_base64="rq+uxvchirixp4zke39hxf3+jdy=">aaaci3icbvdltsjafj3ic/gfunqzkzjahrtgrn2bblxiikjcstmdbmfk+mbmsojnv8g/8a/c6g+4mm5cupndbkelau9yk5nz7s2999gbz1lp+peww1ldw9/ibxa2tnd294r7b/fsdwwfjvw5l9o2kcczb03ffid2iic4noewpbxo/dyyhgs+d6cmaxrd0veywyhriwqvk6zl1iashtvjkyqzh0p5aalnlribmarzgnetkbjfhyjvlolvfqq8tiymlfcghlx8mxs+dv3wfoveyo6hb6obeaey5raxzfbcqoiq9kgtui+4ilvr9kuynyrkdzu+smptekr+nyiik+xetzpo9ag56kxif14nvm5fn2jeecrw6gyre3ksfjzmg3tmafv8khbcbutuxxrabkeqsxfui2ldxzinxvgmyzk0t6uxvep2rfs7yvljoyn0jmriqoeohm5qazurru/obb2in+1ze9c+tm9za07lzg7rhltvx8lhpfa=</latexit> <latexit sha1_base64="rq+uxvchirixp4zke39hxf3+jdy=">aaaci3icbvdltsjafj3ic/gfunqzkzjahrtgrn2bblxiikjcstmdbmfk+mbmsojnv8g/8a/c6g+4mm5cupndbkelau9yk5nz7s2999gbz1lp+peww1ldw9/ibxa2tnd294r7b/fsdwwfjvw5l9o2kcczb03ffid2iic4noewpbxo/dyyhgs+d6cmaxrd0veywyhriwqvk6zl1iashtvjkyqzh0p5aalnlribmarzgnetkbjfhyjvlolvfqq8tiymlfcghlx8mxs+dv3wfoveyo6hb6obeaey5raxzfbcqoiq9kgtui+4ilvr9kuynyrkdzu+smptekr+nyiik+xetzpo9ag56kxif14nvm5fn2jeecrw6gyre3ksfjzmg3tmafv8khbcbutuxxrabkeqsxfui2ldxzinxvgmyzk0t6uxvep2rfs7yvljoyn0jmriqoeohm5qazurru/obb2in+1ze9c+tm9za07lzg7rhltvx8lhpfa=</latexit> <latexit sha1_base64="6/q+x0nnfxufcxtl441chdi5hv0=">aaacihicbvdltsjafj36rhyhlt1mjbryknayqdvqjutmregoaabdfezmpmvmsojn/8c/8a/c6g+4mi517yfyqhccnuqmj+fcm3vvcqngltbnl2npewv1bt23kd/c2t7zlezt3ys/ljg0sm982xkriowk0tbum9ikjehczatpdq5tvzkiulff3olxqdoc9qt1key6kzzcsc2j6emmbtim6qjahok+riyqxu5ucoblehfloj93ckwzyk4af4mvksliuhckp3bxxyenqmoglgpbzqa7ezkaykbivb0qeia8qd3stqhankhonpknhsej0owel5msgk7uvxmr4kqnuzt0pveqes8v//paofyuohevqaijwnnfxsig9meaduxssbbm44qglglyk8r9jbhwsyqzwzqdpmzplnz8ciukcvq5rfi3z8xqvzzpdhyci1acfjghvxad6qabmhgcl+avvbnpxrvxyxxow5emboyazmd4/gva/qmj</latexit> <latexit sha1_base64="6/q+x0nnfxufcxtl441chdi5hv0=">aaacihicbvdltsjafj36rhyhlt1mjbryknayqdvqjutmregoaabdfezmpmvmsojn/8c/8a/c6g+4mi517yfyqhccnuqmj+fcm3vvcqngltbnl2npewv1bt23kd/c2t7zlezt3ys/ljg0sm982xkriowk0tbum9ikjehczatpdq5tvzkiulff3olxqdoc9qt1key6kzzcsc2j6emmbtim6qjahok+riyqxu5ucoblehfloj93ckwzyk4af4mvksliuhckp3bxxyenqmoglgpbzqa7ezkaykbivb0qeia8qd3stqhankhonpknhsej0owel5msgk7uvxmr4kqnuzt0pveqes8v//paofyuohevqaijwnnfxsig9meaduxssbbm44qglglyk8r9jbhwsyqzwzqdpmzplnz8ciukcvq5rfi3z8xqvzzpdhyci1acfjghvxad6qabmhgcl+avvbnpxrvxyxxow5emboyazmd4/gva/qmj</latexit> <latexit sha1_base64="6/q+x0nnfxufcxtl441chdi5hv0=">aaacihicbvdltsjafj36rhyhlt1mjbryknayqdvqjutmregoaabdfezmpmvmsojn/8c/8a/c6g+4mi517yfyqhccnuqmj+fcm3vvcqngltbnl2npewv1bt23kd/c2t7zlezt3ys/ljg0sm982xkriowk0tbum9ikjehczatpdq5tvzkiulff3olxqdoc9qt1key6kzzcsc2j6emmbtim6qjahok+riyqxu5ucoblehfloj93ckwzyk4af4mvksliuhckp3bxxyenqmoglgpbzqa7ezkaykbivb0qeia8qd3stqhankhonpknhsej0owel5msgk7uvxmr4kqnuzt0pveqes8v//paofyuohevqaijwnnfxsig9meaduxssbbm44qglglyk8r9jbhwsyqzwzqdpmzplnz8ciukcvq5rfi3z8xqvzzpdhyci1acfjghvxad6qabmhgcl+avvbnpxrvxyxxow5emboyazmd4/gva/qmj</latexit> <latexit sha1_base64="6/q+x0nnfxufcxtl441chdi5hv0=">aaacihicbvdltsjafj36rhyhlt1mjbryknayqdvqjutmregoaabdfezmpmvmsojn/8c/8a/c6g+4mi517yfyqhccnuqmj+fcm3vvcqngltbnl2npewv1bt23kd/c2t7zlezt3ys/ljg0sm982xkriowk0tbum9ikjehczatpdq5tvzkiulff3olxqdoc9qt1key6kzzcsc2j6emmbtim6qjahok+riyqxu5ucoblehfloj93ckwzyk4af4mvksliuhckp3bxxyenqmoglgpbzqa7ezkaykbivb0qeia8qd3stqhankhonpknhsej0owel5msgk7uvxmr4kqnuzt0pveqes8v//paofyuohevqaijwnnfxsig9meaduxssbbm44qglglyk8r9jbhwsyqzwzqdpmzplnz8ciukcvq5rfi3z8xqvzzpdhyci1acfjghvxad6qabmhgcl+avvbnpxrvxyxxow5emboyazmd4/gva/qmj</latexit> <latexit sha1_base64="o2iei5+35efdcth4sxy3e82gmvi=">aaacfhicbvdlssnafj34rpevdaebwslutuleuhetblxwmlbqhjcz3rzjjw9njouaav6ff+bwf8cvuhxv3g8xabuwrqcuhm65l3vv8slopdlnb21hcwl5zbwwpq9vbg5tgzu7dzkmbqwbhjwudy9i4cwawzhforejil7hoe71r3k/pgahwrjcqmeejk+6aeswslqmucz+1u1k7l7fas6pu9ych5gncnx1dd01imbzhahpe2tcimicmmv8tnohjx0ifoveyqzlrspjifcmckj1viwhirrputdmaeb8ke4y+ihfr5nsxp1qzbuopfl/titel3loe1mnt1rpznq5+j/xjfxn3elyemukajpe1ik5vihoa8ftjoaqpswioyjlt2lai4jqlcu2tuwx/moax2lnhjbp7jpyrdm6os1wlif5fnabokqlzkezvehxqizsrnetekgv6e171t61d+1z3lqgtwb20bs0r19+1z3p</latexit> <latexit sha1_base64="o2iei5+35efdcth4sxy3e82gmvi=">aaacfhicbvdlssnafj34rpevdaebwslutuleuhetblxwmlbqhjcz3rzjjw9njouaav6ff+bwf8cvuhxv3g8xabuwrqcuhm65l3vv8slopdlnb21hcwl5zbwwpq9vbg5tgzu7dzkmbqwbhjwudy9i4cwawzhforejil7hoe71r3k/pgahwrjcqmeejk+6aeswslqmucz+1u1k7l7fas6pu9ych5gncnx1dd01imbzhahpe2tcimicmmv8tnohjx0ifoveyqzlrspjifcmckj1viwhirrputdmaeb8ke4y+ihfr5nsxp1qzbuopfl/titel3loe1mnt1rpznq5+j/xjfxn3elyemukajpe1ik5vihoa8ftjoaqpswioyjlt2lai4jqlcu2tuwx/moax2lnhjbp7jpyrdm6os1wlif5fnabokqlzkezvehxqizsrnetekgv6e171t61d+1z3lqgtwb20bs0r19+1z3p</latexit> <latexit sha1_base64="o2iei5+35efdcth4sxy3e82gmvi=">aaacfhicbvdlssnafj34rpevdaebwslutuleuhetblxwmlbqhjcz3rzjjw9njouaav6ff+bwf8cvuhxv3g8xabuwrqcuhm65l3vv8slopdlnb21hcwl5zbwwpq9vbg5tgzu7dzkmbqwbhjwudy9i4cwawzhforejil7hoe71r3k/pgahwrjcqmeejk+6aeswslqmucz+1u1k7l7fas6pu9ych5gncnx1dd01imbzhahpe2tcimicmmv8tnohjx0ifoveyqzlrspjifcmckj1viwhirrputdmaeb8ke4y+ihfr5nsxp1qzbuopfl/titel3loe1mnt1rpznq5+j/xjfxn3elyemukajpe1ik5vihoa8ftjoaqpswioyjlt2lai4jqlcu2tuwx/moax2lnhjbp7jpyrdm6os1wlif5fnabokqlzkezvehxqizsrnetekgv6e171t61d+1z3lqgtwb20bs0r19+1z3p</latexit> <latexit sha1_base64="o2iei5+35efdcth4sxy3e82gmvi=">aaacfhicbvdlssnafj34rpevdaebwslutuleuhetblxwmlbqhjcz3rzjjw9njouaav6ff+bwf8cvuhxv3g8xabuwrqcuhm65l3vv8slopdlnb21hcwl5zbwwpq9vbg5tgzu7dzkmbqwbhjwudy9i4cwawzhforejil7hoe71r3k/pgahwrjcqmeejk+6aeswslqmucz+1u1k7l7fas6pu9ych5gncnx1dd01imbzhahpe2tcimicmmv8tnohjx0ifoveyqzlrspjifcmckj1viwhirrputdmaeb8ke4y+ihfr5nsxp1qzbuopfl/titel3loe1mnt1rpznq5+j/xjfxn3elyemukajpe1ik5vihoa8ftjoaqpswioyjlt2lai4jqlcu2tuwx/moax2lnhjbp7jpyrdm6os1wlif5fnabokqlzkezvehxqizsrnetekgv6e171t61d+1z3lqgtwb20bs0r19+1z3p</latexit> TENSOR NETWORKS Notation and Visualization Matricize Vectorize A ijk` A (ij)(k`) A pq A (ijk`) A m Tensorize A ij A ij A (pq)(mn) 15

16 TENSOR NETWORKS Notation and Visualization inner product outer product SVD 16

17 <latexit sha1_base64="6tmhaxhdcavfrhbjw01k2z9ny0e=">aaacsxicbzdns8mwgmbtzy85v6oevqshmefgowt1iex38tjbuce6s5pmw1ialiqvrunf58wtn/8ilx5upjmtfxtzhcdd73nfn8njryxkzvkvrqg4tlyywlorr29sbm2bo7t3mowfjm0cslb0psqjo5y0fvwmdcnbuoax0vhgzanfesbc0pdfqkle+geacjqggcmnxbndugl1berwoyp9uelixz7cc+gwvcrhrodomwzej1x7kj21ivsh1dnsz0g+nns887hixbni1axzwuvh56ic8mq55rpjhzgocfeyisl7thwpfokeopirtozekkqij9gq9ltkkccyn8yisoghjj4cheifrucm/p5iucdljpb0z4dusm57u/if14vv4kyfub7finccxtsigvqhnoykfsoivmyibckc6rdcpeicyaxtl+sq7pkvl4p2vxzes29oko2rpi0s2achoapscaoa4bq0qbtg8ahewtv4mj6mn+pt+mpac0y+swf+vkh4dcpsr3s=</latexit> <latexit sha1_base64="6tmhaxhdcavfrhbjw01k2z9ny0e=">aaacsxicbzdns8mwgmbtzy85v6oevqshmefgowt1iex38tjbuce6s5pmw1ialiqvrunf58wtn/8ilx5upjmtfxtzhcdd73nfn8njryxkzvkvrqg4tlyywlorr29sbm2bo7t3mowfjm0cslb0psqjo5y0fvwmdcnbuoax0vhgzanfesbc0pdfqkle+geacjqggcmnxbndugl1berwoyp9uelixz7cc+gwvcrhrodomwzej1x7kj21ivsh1dnsz0g+nns887hixbni1axzwuvh56ic8mq55rpjhzgocfeyisl7thwpfokeopirtozekkqij9gq9ltkkccyn8yisoghjj4cheifrucm/p5iucdljpb0z4dusm57u/if14vv4kyfub7finccxtsigvqhnoykfsoivmyibckc6rdcpeicyaxtl+sq7pkvl4p2vxzes29oko2rpi0s2achoapscaoa4bq0qbtg8ahewtv4mj6mn+pt+mpac0y+swf+vkh4dcpsr3s=</latexit> <latexit sha1_base64="6tmhaxhdcavfrhbjw01k2z9ny0e=">aaacsxicbzdns8mwgmbtzy85v6oevqshmefgowt1iex38tjbuce6s5pmw1ialiqvrunf58wtn/8ilx5upjmtfxtzhcdd73nfn8njryxkzvkvrqg4tlyywlorr29sbm2bo7t3mowfjm0cslb0psqjo5y0fvwmdcnbuoax0vhgzanfesbc0pdfqkle+geacjqggcmnxbndugl1berwoyp9uelixz7cc+gwvcrhrodomwzej1x7kj21ivsh1dnsz0g+nns887hixbni1axzwuvh56ic8mq55rpjhzgocfeyisl7thwpfokeopirtozekkqij9gq9ltkkccyn8yisoghjj4cheifrucm/p5iucdljpb0z4dusm57u/if14vv4kyfub7finccxtsigvqhnoykfsoivmyibckc6rdcpeicyaxtl+sq7pkvl4p2vxzes29oko2rpi0s2achoapscaoa4bq0qbtg8ahewtv4mj6mn+pt+mpac0y+swf+vkh4dcpsr3s=</latexit> <latexit sha1_base64="6tmhaxhdcavfrhbjw01k2z9ny0e=">aaacsxicbzdns8mwgmbtzy85v6oevqshmefgowt1iex38tjbuce6s5pmw1ialiqvrunf58wtn/8ilx5upjmtfxtzhcdd73nfn8njryxkzvkvrqg4tlyywlorr29sbm2bo7t3mowfjm0cslb0psqjo5y0fvwmdcnbuoax0vhgzanfesbc0pdfqkle+geacjqggcmnxbndugl1berwoyp9uelixz7cc+gwvcrhrodomwzej1x7kj21ivsh1dnsz0g+nns887hixbni1axzwuvh56ic8mq55rpjhzgocfeyisl7thwpfokeopirtozekkqij9gq9ltkkccyn8yisoghjj4cheifrucm/p5iucdljpb0z4dusm57u/if14vv4kyfub7finccxtsigvqhnoykfsoivmyibckc6rdcpeicyaxtl+sq7pkvl4p2vxzes29oko2rpi0s2achoapscaoa4bq0qbtg8ahewtv4mj6mn+pt+mpac0y+swf+vkh4dcpsr3s=</latexit> CP DECOMPOSITION Canonical Polyadic Decomposition A i1 i 2 i n = r C (1) i 1 r C(2) i 2 r C(n) i n r 17

18 <latexit sha1_base64="6tmhaxhdcavfrhbjw01k2z9ny0e=">aaacsxicbzdns8mwgmbtzy85v6oevqshmefgowt1iex38tjbuce6s5pmw1ialiqvrunf58wtn/8ilx5upjmtfxtzhcdd73nfn8njryxkzvkvrqg4tlyywlorr29sbm2bo7t3mowfjm0cslb0psqjo5y0fvwmdcnbuoax0vhgzanfesbc0pdfqkle+geacjqggcmnxbndugl1berwoyp9uelixz7cc+gwvcrhrodomwzej1x7kj21ivsh1dnsz0g+nns887hixbni1axzwuvh56ic8mq55rpjhzgocfeyisl7thwpfokeopirtozekkqij9gq9ltkkccyn8yisoghjj4cheifrucm/p5iucdljpb0z4dusm57u/if14vv4kyfub7finccxtsigvqhnoykfsoivmyibckc6rdcpeicyaxtl+sq7pkvl4p2vxzes29oko2rpi0s2achoapscaoa4bq0qbtg8ahewtv4mj6mn+pt+mpac0y+swf+vkh4dcpsr3s=</latexit> <latexit sha1_base64="6tmhaxhdcavfrhbjw01k2z9ny0e=">aaacsxicbzdns8mwgmbtzy85v6oevqshmefgowt1iex38tjbuce6s5pmw1ialiqvrunf58wtn/8ilx5upjmtfxtzhcdd73nfn8njryxkzvkvrqg4tlyywlorr29sbm2bo7t3mowfjm0cslb0psqjo5y0fvwmdcnbuoax0vhgzanfesbc0pdfqkle+geacjqggcmnxbndugl1berwoyp9uelixz7cc+gwvcrhrodomwzej1x7kj21ivsh1dnsz0g+nns887hixbni1axzwuvh56ic8mq55rpjhzgocfeyisl7thwpfokeopirtozekkqij9gq9ltkkccyn8yisoghjj4cheifrucm/p5iucdljpb0z4dusm57u/if14vv4kyfub7finccxtsigvqhnoykfsoivmyibckc6rdcpeicyaxtl+sq7pkvl4p2vxzes29oko2rpi0s2achoapscaoa4bq0qbtg8ahewtv4mj6mn+pt+mpac0y+swf+vkh4dcpsr3s=</latexit> <latexit sha1_base64="6tmhaxhdcavfrhbjw01k2z9ny0e=">aaacsxicbzdns8mwgmbtzy85v6oevqshmefgowt1iex38tjbuce6s5pmw1ialiqvrunf58wtn/8ilx5upjmtfxtzhcdd73nfn8njryxkzvkvrqg4tlyywlorr29sbm2bo7t3mowfjm0cslb0psqjo5y0fvwmdcnbuoax0vhgzanfesbc0pdfqkle+geacjqggcmnxbndugl1berwoyp9uelixz7cc+gwvcrhrodomwzej1x7kj21ivsh1dnsz0g+nns887hixbni1axzwuvh56ic8mq55rpjhzgocfeyisl7thwpfokeopirtozekkqij9gq9ltkkccyn8yisoghjj4cheifrucm/p5iucdljpb0z4dusm57u/if14vv4kyfub7finccxtsigvqhnoykfsoivmyibckc6rdcpeicyaxtl+sq7pkvl4p2vxzes29oko2rpi0s2achoapscaoa4bq0qbtg8ahewtv4mj6mn+pt+mpac0y+swf+vkh4dcpsr3s=</latexit> <latexit sha1_base64="6tmhaxhdcavfrhbjw01k2z9ny0e=">aaacsxicbzdns8mwgmbtzy85v6oevqshmefgowt1iex38tjbuce6s5pmw1ialiqvrunf58wtn/8ilx5upjmtfxtzhcdd73nfn8njryxkzvkvrqg4tlyywlorr29sbm2bo7t3mowfjm0cslb0psqjo5y0fvwmdcnbuoax0vhgzanfesbc0pdfqkle+geacjqggcmnxbndugl1berwoyp9uelixz7cc+gwvcrhrodomwzej1x7kj21ivsh1dnsz0g+nns887hixbni1axzwuvh56ic8mq55rpjhzgocfeyisl7thwpfokeopirtozekkqij9gq9ltkkccyn8yisoghjj4cheifrucm/p5iucdljpb0z4dusm57u/if14vv4kyfub7finccxtsigvqhnoykfsoivmyibckc6rdcpeicyaxtl+sq7pkvl4p2vxzes29oko2rpi0s2achoapscaoa4bq0qbtg8ahewtv4mj6mn+pt+mpac0y+swf+vkh4dcpsr3s=</latexit> CP DECOMPOSITION A i1 i 2 i n = r C (1) i 1 r C(2) i 2 r C(n) i n r 18

19 CP DECOMPOSITION Properties Analogous to SVD for Tensors Rank is the size of the diagonal core tensor in it s CP Finding the minimal rank is NP-hard Truncated SVD is the best rank-k approximation is NOT true. Uniqueness of the factors Matrix decompositions are not 19

20 <latexit sha1_base64="l9ynuhfbihs23kj/yv0czku0uz4=">aaab53icbvbns8naej3ur1q/qh69lbbbu0leua9c0yvhfowttkfsttn27wytdjdccf0fxjyoepuvefpfug1z0nyha4/3zpizfyaca+o6305hzxvtfao4wdra3tndk+8fpog4vqx9fotytukquxcjvufgyctrsknqydmc3u795hmqzwn5b8yjbhedsn7njborna675ypbdwcgy8tlsqvy1lvlr04vzmme0jbbtw57bmkcjcrdmcbjqznqtcgb0qg2lzu0qh1ks0mn5mqqpdkpls1pyez9pzhrsotxfnroijqhxvsm4n9eozx9yydjmkknsjzf1e8fmtgzfk16xcezymwjzyrbwwkbukwzsdmubaje4svlxd+rxlw9xnmldponuyqjoizt8oacanahdfcbacizvmkb8+i8oo/ox7y14oqzh/ahzucp+1+mlq==</latexit> <latexit sha1_base64="l9ynuhfbihs23kj/yv0czku0uz4=">aaab53icbvbns8naej3ur1q/qh69lbbbu0leua9c0yvhfowttkfsttn27wytdjdccf0fxjyoepuvefpfug1z0nyha4/3zpizfyaca+o6305hzxvtfao4wdra3tndk+8fpog4vqx9fotytukquxcjvufgyctrsknqydmc3u795hmqzwn5b8yjbhedsn7njborna675ypbdwcgy8tlsqvy1lvlr04vzmme0jbbtw57bmkcjcrdmcbjqznqtcgb0qg2lzu0qh1ks0mn5mqqpdkpls1pyez9pzhrsotxfnroijqhxvsm4n9eozx9yydjmkknsjzf1e8fmtgzfk16xcezymwjzyrbwwkbukwzsdmubaje4svlxd+rxlw9xnmldponuyqjoizt8oacanahdfcbacizvmkb8+i8oo/ox7y14oqzh/ahzucp+1+mlq==</latexit> <latexit sha1_base64="l9ynuhfbihs23kj/yv0czku0uz4=">aaab53icbvbns8naej3ur1q/qh69lbbbu0leua9c0yvhfowttkfsttn27wytdjdccf0fxjyoepuvefpfug1z0nyha4/3zpizfyaca+o6305hzxvtfao4wdra3tndk+8fpog4vqx9fotytukquxcjvufgyctrsknqydmc3u795hmqzwn5b8yjbhedsn7njborna675ypbdwcgy8tlsqvy1lvlr04vzmme0jbbtw57bmkcjcrdmcbjqznqtcgb0qg2lzu0qh1ks0mn5mqqpdkpls1pyez9pzhrsotxfnroijqhxvsm4n9eozx9yydjmkknsjzf1e8fmtgzfk16xcezymwjzyrbwwkbukwzsdmubaje4svlxd+rxlw9xnmldponuyqjoizt8oacanahdfcbacizvmkb8+i8oo/ox7y14oqzh/ahzucp+1+mlq==</latexit> CP DECOMPOSITION Properties A vector s CP Decomposition is itself. =<latexit sha1_base64="l9ynuhfbihs23kj/yv0czku0uz4=">aaab53icbvbns8naej3ur1q/qh69lbbbu0leua9c0yvhfowttkfsttn27wytdjdccf0fxjyoepuvefpfug1z0nyha4/3zpizfyaca+o6305hzxvtfao4wdra3tndk+8fpog4vqx9fotytukquxcjvufgyctrsknqydmc3u795hmqzwn5b8yjbhedsn7njborna675ypbdwcgy8tlsqvy1lvlr04vzmme0jbbtw57bmkcjcrdmcbjqznqtcgb0qg2lzu0qh1ks0mn5mqqpdkpls1pyez9pzhrsotxfnroijqhxvsm4n9eozx9yydjmkknsjzf1e8fmtgzfk16xcezymwjzyrbwwkbukwzsdmubaje4svlxd+rxlw9xnmldponuyqjoizt8oacanahdfcbacizvmkb8+i8oo/ox7y14oqzh/ahzucp+1+mlq==</latexit> A matrix s CP Decomposition is the SVD. CP-ALS to compute a CP Decomposition from a 3D+ Tensor. Choose rank. Fix all but one core tensor and solve linear least squares to fit. Continue. 20

21 <latexit sha1_base64="3gau8xc0s+bn+bf9fg9nendpaao=">aaacwnicbzfbs8mwgibtetjcpmzdntfbisjiaiugxggelvrswbnbwkoazs4stuuscqp0t3qlf/4vme2kqpodwmvzfm8ox8kem6ud592yfxaxlmv1luzzdw19o7w59ajivblajtgpzt/einimafczzwk/krrhiae9chjd+l0xkhwlxyoejjsi8lngi0awngi15cxkghiz8qbphrfwkcgrw3n4gzkjxim8cssc+0fw+ik7ca/zmmuavpk3y55jlkykva6yualybbxatscpc84ltxjtunudar36w5ikerwackzuwhushwryaky4zrt+qmicyqq/04grakdubvk5mxzugzkeo1iajtqs6c9ehiolplfooiosx+qvv8d/vegqr6dbxkssairi7kbryqgoytfoogsses2nrmaimbkrjgmsmdhmoxpmco7fj8+lrtc567j3x+2lq2oadbal9sabcmejuac34a50aqfv4noqwxxrw16wv+zmrnw2qsw2+fx2zhfexref</latexit> <latexit sha1_base64="3gau8xc0s+bn+bf9fg9nendpaao=">aaacwnicbzfbs8mwgibtetjcpmzdntfbisjiaiugxggelvrswbnbwkoazs4stuuscqp0t3qlf/4vme2kqpodwmvzfm8ox8kem6ud592yfxaxlmv1luzzdw19o7w59ajivblajtgpzt/einimafczzwk/krrhiae9chjd+l0xkhwlxyoejjsi8lngi0awngi15cxkghiz8qbphrfwkcgrw3n4gzkjxim8cssc+0fw+ik7ca/zmmuavpk3y55jlkykva6yualybbxatscpc84ltxjtunudar36w5ikerwackzuwhushwryaky4zrt+qmicyqq/04grakdubvk5mxzugzkeo1iajtqs6c9ehiolplfooiosx+qvv8d/vegqr6dbxkssairi7kbryqgoytfoogsses2nrmaimbkrjgmsmdhmoxpmco7fj8+lrtc567j3x+2lq2oadbal9sabcmejuac34a50aqfv4noqwxxrw16wv+zmrnw2qsw2+fx2zhfexref</latexit> <latexit sha1_base64="3gau8xc0s+bn+bf9fg9nendpaao=">aaacwnicbzfbs8mwgibtetjcpmzdntfbisjiaiugxggelvrswbnbwkoazs4stuuscqp0t3qlf/4vme2kqpodwmvzfm8ox8kem6ud592yfxaxlmv1luzzdw19o7w59ajivblajtgpzt/einimafczzwk/krrhiae9chjd+l0xkhwlxyoejjsi8lngi0awngi15cxkghiz8qbphrfwkcgrw3n4gzkjxim8cssc+0fw+ik7ca/zmmuavpk3y55jlkykva6yualybbxatscpc84ltxjtunudar36w5ikerwackzuwhushwryaky4zrt+qmicyqq/04grakdubvk5mxzugzkeo1iajtqs6c9ehiolplfooiosx+qvv8d/vegqr6dbxkssairi7kbryqgoytfoogsses2nrmaimbkrjgmsmdhmoxpmco7fj8+lrtc567j3x+2lq2oadbal9sabcmejuac34a50aqfv4noqwxxrw16wv+zmrnw2qsw2+fx2zhfexref</latexit> <latexit sha1_base64="3gau8xc0s+bn+bf9fg9nendpaao=">aaacwnicbzfbs8mwgibtetjcpmzdntfbisjiaiugxggelvrswbnbwkoazs4stuuscqp0t3qlf/4vme2kqpodwmvzfm8ox8kem6ud592yfxaxlmv1luzzdw19o7w59ajivblajtgpzt/einimafczzwk/krrhiae9chjd+l0xkhwlxyoejjsi8lngi0awngi15cxkghiz8qbphrfwkcgrw3n4gzkjxim8cssc+0fw+ik7ca/zmmuavpk3y55jlkykva6yualybbxatscpc84ltxjtunudar36w5ikerwackzuwhushwryaky4zrt+qmicyqq/04grakdubvk5mxzugzkeo1iajtqs6c9ehiolplfooiosx+qvv8d/vegqr6dbxkssairi7kbryqgoytfoogsses2nrmaimbkrjgmsmdhmoxpmco7fj8+lrtc567j3x+2lq2oadbal9sabcmejuac34a50aqfv4noqwxxrw16wv+zmrnw2qsw2+fx2zhfexref</latexit> TUCKER DECOMPOSITION A i1 i 2 i n = G r1 r 2 r n C (1) i 1 r 1 C (2) i 2 r 2 C (n) i n r n 21

22 <latexit sha1_base64="3gau8xc0s+bn+bf9fg9nendpaao=">aaacwnicbzfbs8mwgibtetjcpmzdntfbisjiaiugxggelvrswbnbwkoazs4stuuscqp0t3qlf/4vme2kqpodwmvzfm8ox8kem6ud592yfxaxlmv1luzzdw19o7w59ajivblajtgpzt/einimafczzwk/krrhiae9chjd+l0xkhwlxyoejjsi8lngi0awngi15cxkghiz8qbphrfwkcgrw3n4gzkjxim8cssc+0fw+ik7ca/zmmuavpk3y55jlkykva6yualybbxatscpc84ltxjtunudar36w5ikerwackzuwhushwryaky4zrt+qmicyqq/04grakdubvk5mxzugzkeo1iajtqs6c9ehiolplfooiosx+qvv8d/vegqr6dbxkssairi7kbryqgoytfoogsses2nrmaimbkrjgmsmdhmoxpmco7fj8+lrtc567j3x+2lq2oadbal9sabcmejuac34a50aqfv4noqwxxrw16wv+zmrnw2qsw2+fx2zhfexref</latexit> <latexit sha1_base64="3gau8xc0s+bn+bf9fg9nendpaao=">aaacwnicbzfbs8mwgibtetjcpmzdntfbisjiaiugxggelvrswbnbwkoazs4stuuscqp0t3qlf/4vme2kqpodwmvzfm8ox8kem6ud592yfxaxlmv1luzzdw19o7w59ajivblajtgpzt/einimafczzwk/krrhiae9chjd+l0xkhwlxyoejjsi8lngi0awngi15cxkghiz8qbphrfwkcgrw3n4gzkjxim8cssc+0fw+ik7ca/zmmuavpk3y55jlkykva6yualybbxatscpc84ltxjtunudar36w5ikerwackzuwhushwryaky4zrt+qmicyqq/04grakdubvk5mxzugzkeo1iajtqs6c9ehiolplfooiosx+qvv8d/vegqr6dbxkssairi7kbryqgoytfoogsses2nrmaimbkrjgmsmdhmoxpmco7fj8+lrtc567j3x+2lq2oadbal9sabcmejuac34a50aqfv4noqwxxrw16wv+zmrnw2qsw2+fx2zhfexref</latexit> <latexit sha1_base64="3gau8xc0s+bn+bf9fg9nendpaao=">aaacwnicbzfbs8mwgibtetjcpmzdntfbisjiaiugxggelvrswbnbwkoazs4stuuscqp0t3qlf/4vme2kqpodwmvzfm8ox8kem6ud592yfxaxlmv1luzzdw19o7w59ajivblajtgpzt/einimafczzwk/krrhiae9chjd+l0xkhwlxyoejjsi8lngi0awngi15cxkghiz8qbphrfwkcgrw3n4gzkjxim8cssc+0fw+ik7ca/zmmuavpk3y55jlkykva6yualybbxatscpc84ltxjtunudar36w5ikerwackzuwhushwryaky4zrt+qmicyqq/04grakdubvk5mxzugzkeo1iajtqs6c9ehiolplfooiosx+qvv8d/vegqr6dbxkssairi7kbryqgoytfoogsses2nrmaimbkrjgmsmdhmoxpmco7fj8+lrtc567j3x+2lq2oadbal9sabcmejuac34a50aqfv4noqwxxrw16wv+zmrnw2qsw2+fx2zhfexref</latexit> <latexit sha1_base64="3gau8xc0s+bn+bf9fg9nendpaao=">aaacwnicbzfbs8mwgibtetjcpmzdntfbisjiaiugxggelvrswbnbwkoazs4stuuscqp0t3qlf/4vme2kqpodwmvzfm8ox8kem6ud592yfxaxlmv1luzzdw19o7w59ajivblajtgpzt/einimafczzwk/krrhiae9chjd+l0xkhwlxyoejjsi8lngi0awngi15cxkghiz8qbphrfwkcgrw3n4gzkjxim8cssc+0fw+ik7ca/zmmuavpk3y55jlkykva6yualybbxatscpc84ltxjtunudar36w5ikerwackzuwhushwryaky4zrt+qmicyqq/04grakdubvk5mxzugzkeo1iajtqs6c9ehiolplfooiosx+qvv8d/vegqr6dbxkssairi7kbryqgoytfoogsses2nrmaimbkrjgmsmdhmoxpmco7fj8+lrtc567j3x+2lq2oadbal9sabcmejuac34a50aqfv4noqwxxrw16wv+zmrnw2qsw2+fx2zhfexref</latexit> TUCKER DECOMPOSITION A i1 i 2 i n = G r1 r 2 r n C (1) i 1 r 1 C (2) i 2 r 2 C (n) i n r n CP is a special case of Tucker when core is superdiagonal and R 1 = R 2 = = R n Called High-Order SVD (HOSVD/MLSVD) when core and factor tensors are orthogonal 22

23 TUCKER DECOMPOSITION Algorithms Computing a Tucker Decomposition from a tensor 23

24 <latexit sha1_base64="4n1myzincplwyxmzhkhqzmox/dg=">aaactxicbvhls8mwhe7ny3o+ph69bicwquzbbfugthfxomg5wtzdmmvbwjqwjbvg6v/orbz5z3jxocjmxq+6+ypax/fi44sxcqa0bb9auaxlldv8ya24vrg5tv3a2b1xqsqjbzkab7ltyuu5e7spmea0huqkfy/tljeut/xwi5wkbejot0la8/fqsaejwbsklegvihlyghjhl/qdrsbdiogxsp4qv5yjbmvyqjosuqnshqe0o6nde5lojkwzbgyqm0mserozyseovlardjpwetgzkinsgqj00u0hjpkp0irjptqohepejkvmhnok2i0udtez4yhtgciwt1uvtuti4kfh+naqsloehin7oxfjx6mj7xmnj/vizwtt8j+te+nbes9miow0fwr20cdiuadw2i3sm0mj5hmdmjhm3bwsezayapmdrvocm//krdb0qxdv5/a0xlvo2iiafxaaksabz6agbkadnaebt+anfibp69l6t76s75k1z2wzpfbncvkfpiyxpa==</latexit> <latexit sha1_base64="4n1myzincplwyxmzhkhqzmox/dg=">aaactxicbvhls8mwhe7ny3o+ph69bicwquzbbfugthfxomg5wtzdmmvbwjqwjbvg6v/orbz5z3jxocjmxq+6+ypax/fi44sxcqa0bb9auaxlldv8ya24vrg5tv3a2b1xqsqjbzkab7ltyuu5e7spmea0huqkfy/tljeut/xwi5wkbejot0la8/fqsaejwbsklegvihlyghjhl/qdrsbdiogxsp4qv5yjbmvyqjosuqnshqe0o6nde5lojkwzbgyqm0mserozyseovlardjpwetgzkinsgqj00u0hjpkp0irjptqohepejkvmhnok2i0udtez4yhtgciwt1uvtuti4kfh+naqsloehin7oxfjx6mj7xmnj/vizwtt8j+te+nbes9miow0fwr20cdiuadw2i3sm0mj5hmdmjhm3bwsezayapmdrvocm//krdb0qxdv5/a0xlvo2iiafxaaksabz6agbkadnaebt+anfibp69l6t76s75k1z2wzpfbncvkfpiyxpa==</latexit> <latexit sha1_base64="4n1myzincplwyxmzhkhqzmox/dg=">aaactxicbvhls8mwhe7ny3o+ph69bicwquzbbfugthfxomg5wtzdmmvbwjqwjbvg6v/orbz5z3jxocjmxq+6+ypax/fi44sxcqa0bb9auaxlldv8ya24vrg5tv3a2b1xqsqjbzkab7ltyuu5e7spmea0huqkfy/tljeut/xwi5wkbejot0la8/fqsaejwbsklegvihlyghjhl/qdrsbdiogxsp4qv5yjbmvyqjosuqnshqe0o6nde5lojkwzbgyqm0mserozyseovlardjpwetgzkinsgqj00u0hjpkp0irjptqohepejkvmhnok2i0udtez4yhtgciwt1uvtuti4kfh+naqsloehin7oxfjx6mj7xmnj/vizwtt8j+te+nbes9miow0fwr20cdiuadw2i3sm0mj5hmdmjhm3bwsezayapmdrvocm//krdb0qxdv5/a0xlvo2iiafxaaksabz6agbkadnaebt+anfibp69l6t76s75k1z2wzpfbncvkfpiyxpa==</latexit> <latexit sha1_base64="4n1myzincplwyxmzhkhqzmox/dg=">aaactxicbvhls8mwhe7ny3o+ph69bicwquzbbfugthfxomg5wtzdmmvbwjqwjbvg6v/orbz5z3jxocjmxq+6+ypax/fi44sxcqa0bb9auaxlldv8ya24vrg5tv3a2b1xqsqjbzkab7ltyuu5e7spmea0huqkfy/tljeut/xwi5wkbejot0la8/fqsaejwbsklegvihlyghjhl/qdrsbdiogxsp4qv5yjbmvyqjosuqnshqe0o6nde5lojkwzbgyqm0mserozyseovlardjpwetgzkinsgqj00u0hjpkp0irjptqohepejkvmhnok2i0udtez4yhtgciwt1uvtuti4kfh+naqsloehin7oxfjx6mj7xmnj/vizwtt8j+te+nbes9miow0fwr20cdiuadw2i3sm0mj5hmdmjhm3bwsezayapmdrvocm//krdb0qxdv5/a0xlvo2iiafxaaksabz6agbkadnaebt+anfibp69l6t76s75k1z2wzpfbncvkfpiyxpa==</latexit> TENSOR RING DECOMPOSITION A i1 i 2 i n = C (1) r 1 i 1 r 2 C (2) r 2 i 2 r 3 C (n) r n i n r 1 24

25 <latexit sha1_base64="996/we2ljjnp7a57s+vlgzoz7ek=">aaacr3icbzdns8mwgmbt+txn19sjl+aqjshoq6aehokuhic4n9hmslnshqvpsvjhlp55xjx682/w4khfo2nxg26+ehj4pc+bno8xcqa0bb9ahyxfpewv4mppbx1jc6u8vxongkgs2iibd2thw4pyjmhlm81pj5qu+x6nbw/csp32i5wkbejwt0la9/fiscejwbueyugsxqw5dlmwrwabvpahkcal2lipq85hkrksuqnshwxmtzkbzkoi4wzni3lataoxsnkbulli1+xs4lxwcleb+trr+au3cejku6ejx0p1htvu/rhlzqinsakxkrpimsyj2jvsyj+qfpwvkcadqwzwgehzhiyz/b0ry1+pie+zpi/1g5r1uvif14308kwfmxfgmgoy/daw4lahmg0vdpikrpojezhizv4vkgcsmdgm+5ipwzl98rxoubxzmnnzuqlf5w0uwr7yb1xggfnqb9egcvqagcfwbj7ap/vsvvtf1vc0wrdynv3wzwrwd4n6rvw=</latexit> <latexit sha1_base64="996/we2ljjnp7a57s+vlgzoz7ek=">aaacr3icbzdns8mwgmbt+txn19sjl+aqjshoq6aehokuhic4n9hmslnshqvpsvjhlp55xjx682/w4khfo2nxg26+ehj4pc+bno8xcqa0bb9ahyxfpewv4mppbx1jc6u8vxongkgs2iibd2thw4pyjmhlm81pj5qu+x6nbw/csp32i5wkbejwt0la9/fiscejwbueyugsxqw5dlmwrwabvpahkcal2lipq85hkrksuqnshwxmtzkbzkoi4wzni3lataoxsnkbulli1+xs4lxwcleb+trr+au3cejku6ejx0p1htvu/rhlzqinsakxkrpimsyj2jvsyj+qfpwvkcadqwzwgehzhiyz/b0ry1+pie+zpi/1g5r1uvif14308kwfmxfgmgoy/daw4lahmg0vdpikrpojezhizv4vkgcsmdgm+5ipwzl98rxoubxzmnnzuqlf5w0uwr7yb1xggfnqb9egcvqagcfwbj7ap/vsvvtf1vc0wrdynv3wzwrwd4n6rvw=</latexit> <latexit sha1_base64="996/we2ljjnp7a57s+vlgzoz7ek=">aaacr3icbzdns8mwgmbt+txn19sjl+aqjshoq6aehokuhic4n9hmslnshqvpsvjhlp55xjx682/w4khfo2nxg26+ehj4pc+bno8xcqa0bb9ahyxfpewv4mppbx1jc6u8vxongkgs2iibd2thw4pyjmhlm81pj5qu+x6nbw/csp32i5wkbejwt0la9/fiscejwbueyugsxqw5dlmwrwabvpahkcal2lipq85hkrksuqnshwxmtzkbzkoi4wzni3lataoxsnkbulli1+xs4lxwcleb+trr+au3cejku6ejx0p1htvu/rhlzqinsakxkrpimsyj2jvsyj+qfpwvkcadqwzwgehzhiyz/b0ry1+pie+zpi/1g5r1uvif14308kwfmxfgmgoy/daw4lahmg0vdpikrpojezhizv4vkgcsmdgm+5ipwzl98rxoubxzmnnzuqlf5w0uwr7yb1xggfnqb9egcvqagcfwbj7ap/vsvvtf1vc0wrdynv3wzwrwd4n6rvw=</latexit> <latexit sha1_base64="996/we2ljjnp7a57s+vlgzoz7ek=">aaacr3icbzdns8mwgmbt+txn19sjl+aqjshoq6aehokuhic4n9hmslnshqvpsvjhlp55xjx682/w4khfo2nxg26+ehj4pc+bno8xcqa0bb9ahyxfpewv4mppbx1jc6u8vxongkgs2iibd2thw4pyjmhlm81pj5qu+x6nbw/csp32i5wkbejwt0la9/fiscejwbueyugsxqw5dlmwrwabvpahkcal2lipq85hkrksuqnshwxmtzkbzkoi4wzni3lataoxsnkbulli1+xs4lxwcleb+trr+au3cejku6ejx0p1htvu/rhlzqinsakxkrpimsyj2jvsyj+qfpwvkcadqwzwgehzhiyz/b0ry1+pie+zpi/1g5r1uvif14308kwfmxfgmgoy/daw4lahmg0vdpikrpojezhizv4vkgcsmdgm+5ipwzl98rxoubxzmnnzuqlf5w0uwr7yb1xggfnqb9egcvqagcfwbj7ap/vsvvtf1vc0wrdynv3wzwrwd4n6rvw=</latexit> TENSOR RING DECOMPOSITION Often found as the Tensor Train A i1 i 2 i n = C (1) i 1 r 2 C (2) r 2 i 2 r 3 C (n) r n i n 25

26 TENSOR RING DECOMPOSITION Algorithms Similar algorithms for computation from a tensor: Direct HOSVD Iterative ALS Adaptive Rank ALS Block-wise adaptive rank ALS Q. Zhao, G Zhou, S. Xie, L Zhang, A Cichocki. Tensor Ring Decomposition. arxiv: [cs.na] Jun

27 TENSOR RING DECOMPOSITION Properties Interpretation as a hierarchical method: Relation to Kernel methods Hierarchical (H-matrix) decomposition Translational invariance E. Corona, A. Rahimian, D. Zorin. A Tensor Train Accelerated Solver For Integral Equation in Complex Geometries. Journal of Computational Physics, Volume 334,

28 <latexit sha1_base64="uztt9lax+prwgbgamohpfoxpync=">aaacb3icbvhlahsxfnvmmzz1+ndqtaftedufe2ngjpb2exdrtzcp1e3adsqdjzyiakrbuhmwg9f5v+76ed30c6qxp8fnekfwoa90dzqwwnlmkp9r/odhzqphu09ae0+fpx/r3n/53dvsctkrvlt3lokxwhk5qyvanhvoqp5qezpejwv99fo6r6z5hstczno4mgqhbgcgepvme68uz4opzykz6knizkwp6fi86rh+ilfagdykuisziyqxfm0z1paus/twkm5bgm47kdejjlomd//kwyjwdjczjouh9wfrqjc0c8lbp2azfwuudqon3k9zuuc8aodkallqzuovcxbxccgnaroic8yrdwer+j4wgv1yf45buma3exxk3i/zndhzwet/v6vj/2ntehcf5puyrynsim1fi1jttlrun2bksyf6gqaip8kuvfyca4hhj1qhbhb3yffbzdj4ogbfd7ujz00bu+q1eud6hjejmijfyamzeef+rz3otfq2+h2/ig9iurhguzppkh8m7v8bd5u6ya==</latexit> <latexit sha1_base64="uztt9lax+prwgbgamohpfoxpync=">aaacb3icbvhlahsxfnvmmzz1+ndqtaftedufe2ngjpb2exdrtzcp1e3adsqdjzyiakrbuhmwg9f5v+76ed30c6qxp8fnekfwoa90dzqwwnlmkp9r/odhzqphu09ae0+fpx/r3n/53dvsctkrvlt3lokxwhk5qyvanhvoqp5qezpejwv99fo6r6z5hstczno4mgqhbgcgepvme68uz4opzykz6knizkwp6fi86rh+ilfagdykuisziyqxfm0z1paus/twkm5bgm47kdejjlomd//kwyjwdjczjouh9wfrqjc0c8lbp2azfwuudqon3k9zuuc8aodkallqzuovcxbxccgnaroic8yrdwer+j4wgv1yf45buma3exxk3i/zndhzwet/v6vj/2ntehcf5puyrynsim1fi1jttlrun2bksyf6gqaip8kuvfyca4hhj1qhbhb3yffbzdj4ogbfd7ujz00bu+q1eud6hjejmijfyamzeef+rz3otfq2+h2/ig9iurhguzppkh8m7v8bd5u6ya==</latexit> <latexit sha1_base64="uztt9lax+prwgbgamohpfoxpync=">aaacb3icbvhlahsxfnvmmzz1+ndqtaftedufe2ngjpb2exdrtzcp1e3adsqdjzyiakrbuhmwg9f5v+76ed30c6qxp8fnekfwoa90dzqwwnlmkp9r/odhzqphu09ae0+fpx/r3n/53dvsctkrvlt3lokxwhk5qyvanhvoqp5qezpejwv99fo6r6z5hstczno4mgqhbgcgepvme68uz4opzykz6knizkwp6fi86rh+ilfagdykuisziyqxfm0z1paus/twkm5bgm47kdejjlomd//kwyjwdjczjouh9wfrqjc0c8lbp2azfwuudqon3k9zuuc8aodkallqzuovcxbxccgnaroic8yrdwer+j4wgv1yf45buma3exxk3i/zndhzwet/v6vj/2ntehcf5puyrynsim1fi1jttlrun2bksyf6gqaip8kuvfyca4hhj1qhbhb3yffbzdj4ogbfd7ujz00bu+q1eud6hjejmijfyamzeef+rz3otfq2+h2/ig9iurhguzppkh8m7v8bd5u6ya==</latexit> <latexit sha1_base64="uztt9lax+prwgbgamohpfoxpync=">aaacb3icbvhlahsxfnvmmzz1+ndqtaftedufe2ngjpb2exdrtzcp1e3adsqdjzyiakrbuhmwg9f5v+76ed30c6qxp8fnekfwoa90dzqwwnlmkp9r/odhzqphu09ae0+fpx/r3n/53dvsctkrvlt3lokxwhk5qyvanhvoqp5qezpejwv99fo6r6z5hstczno4mgqhbgcgepvme68uz4opzykz6knizkwp6fi86rh+ilfagdykuisziyqxfm0z1paus/twkm5bgm47kdejjlomd//kwyjwdjczjouh9wfrqjc0c8lbp2azfwuudqon3k9zuuc8aodkallqzuovcxbxccgnaroic8yrdwer+j4wgv1yf45buma3exxk3i/zndhzwet/v6vj/2ntehcf5puyrynsim1fi1jttlrun2bksyf6gqaip8kuvfyca4hhj1qhbhb3yffbzdj4ogbfd7ujz00bu+q1eud6hjejmijfyamzeef+rz3otfq2+h2/ig9iurhguzppkh8m7v8bd5u6ya==</latexit> KRONECKER DECOMPOSITION A i1 i 2 i n = C a (1) 1 a 2 a n C (2) b 1 b 2 b n C (m) d 1 d 2 d n 28

29 KRONECKER DECOMPOSITION Algorithms and Properties Similar algorithms for computation from a tensor: Direct SVD Iterative ALS (KPCA) Iterative Lanczos Relation to Perfect shuffles + Z-order curves Butterfly algorithms and factorizations 29

30 KRONECKER DECOMPOSITION Properties 30

31 EXOTIC DECOMPOSITIONS Hierarchical Tucker Decomposition Any tensor network without cycles where all nodes have degree 3 or less. 31

32 EXOTIC DECOMPOSITIONS Other Compositions Construct arbitrary rich structure that reflects any a-priori knowledge of the structure of inputs and outputs 32

33 TENSOR DECOMPOSITIONS IN DEEP LEARNING Compress and accelerate layers in Deep Learning 33

34 DEEP NETWORKS Fully connected layers take up a lot of space! TOTAL PARAMS FC PARAMS % FC LAYER AlexNet 61,100,840 58,631,144 96% VGG ,667, ,642,856 86% ResNet-50 25,557,032 2,049,000 8% ResNet ,549,160 2,049, % 34

35 A general tensor requires O(I N ) storage I is the maximum mode dimension N is the tensor order DEEP NETWORKS Compression Exponential savings Storage Compute 35

36 DEEP NETWORKS Observation: In CNNs (and other networks) Fully Connected Layers flatten data 36

37 CP DECOMPOSITIONS in machine learning Application in latent variable models Single topic models Gaussian mixture models (GMM) Latent Dirichlet allocation (LDA) Hidden Markov models (HMM) But in Deep Learning? E. Allman,C. Matias, J. Rhodes. Identifiability of parameters in latent structure models with many observed variables. Ann. Stat. 37 (2009) A. Anandkumar, R. Ge, D. Hsu, S. Kakade, M. Talgarsky. Tensor Decompositions for Learning Latent Variable Models. Journal of Machine Learning Research 15, Jan

38 CP DECOMPOSITIONS in deep learning Compact representation for matrices and tensors. Efficient application of linear algebra operations. Replace a fully connected layer with a CP decomposed layer Initialize from a trained network or randomly Fine-tune Match the modal structure of the input and output 38

39 TCL LAYER Special case of CP Layer with R = 1 and input/output mode fusion Fully Connected R = 1 Matricization TCL Application

40 TCL LAYER AlexNet, CIFAR-100 J. Kossaifi, A. Khanna, Z. Lipton, T. Furanello, A. Anandkumar. Tensor Contraction Layers for Parsimonious Deep Nets IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, 2017, pp

41 TCL LAYER VGG-19, CIFAR-100 J. Kossaifi, A. Khanna, Z. Lipton, T. Furanello, A. Anandkumar. Tensor Contraction Layers for Parsimonious Deep Nets IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, 2017, pp

42 CP LAYER Resnet-32, CIFAR-10 X. Cao, G. Rabusseau, J. Pineau. Tensor Regression Networks with various Low-Rank Tensor Approximations. arxiv: [cs.lg] Dec

43 CP LAYER Training ImageNet 43

44 CP LAYER Initialization from Pretrained ImageNet Demonstrates initialization from existing networks to gain a large head-start in training Fine-tuning still very important 44

45 <latexit sha1_base64="3gau8xc0s+bn+bf9fg9nendpaao=">aaacwnicbzfbs8mwgibtetjcpmzdntfbisjiaiugxggelvrswbnbwkoazs4stuuscqp0t3qlf/4vme2kqpodwmvzfm8ox8kem6ud592yfxaxlmv1luzzdw19o7w59ajivblajtgpzt/einimafczzwk/krrhiae9chjd+l0xkhwlxyoejjsi8lngi0awngi15cxkghiz8qbphrfwkcgrw3n4gzkjxim8cssc+0fw+ik7ca/zmmuavpk3y55jlkykva6yualybbxatscpc84ltxjtunudar36w5ikerwackzuwhushwryaky4zrt+qmicyqq/04grakdubvk5mxzugzkeo1iajtqs6c9ehiolplfooiosx+qvv8d/vegqr6dbxkssairi7kbryqgoytfoogsses2nrmaimbkrjgmsmdhmoxpmco7fj8+lrtc567j3x+2lq2oadbal9sabcmejuac34a50aqfv4noqwxxrw16wv+zmrnw2qsw2+fx2zhfexref</latexit> <latexit sha1_base64="3gau8xc0s+bn+bf9fg9nendpaao=">aaacwnicbzfbs8mwgibtetjcpmzdntfbisjiaiugxggelvrswbnbwkoazs4stuuscqp0t3qlf/4vme2kqpodwmvzfm8ox8kem6ud592yfxaxlmv1luzzdw19o7w59ajivblajtgpzt/einimafczzwk/krrhiae9chjd+l0xkhwlxyoejjsi8lngi0awngi15cxkghiz8qbphrfwkcgrw3n4gzkjxim8cssc+0fw+ik7ca/zmmuavpk3y55jlkykva6yualybbxatscpc84ltxjtunudar36w5ikerwackzuwhushwryaky4zrt+qmicyqq/04grakdubvk5mxzugzkeo1iajtqs6c9ehiolplfooiosx+qvv8d/vegqr6dbxkssairi7kbryqgoytfoogsses2nrmaimbkrjgmsmdhmoxpmco7fj8+lrtc567j3x+2lq2oadbal9sabcmejuac34a50aqfv4noqwxxrw16wv+zmrnw2qsw2+fx2zhfexref</latexit> <latexit sha1_base64="3gau8xc0s+bn+bf9fg9nendpaao=">aaacwnicbzfbs8mwgibtetjcpmzdntfbisjiaiugxggelvrswbnbwkoazs4stuuscqp0t3qlf/4vme2kqpodwmvzfm8ox8kem6ud592yfxaxlmv1luzzdw19o7w59ajivblajtgpzt/einimafczzwk/krrhiae9chjd+l0xkhwlxyoejjsi8lngi0awngi15cxkghiz8qbphrfwkcgrw3n4gzkjxim8cssc+0fw+ik7ca/zmmuavpk3y55jlkykva6yualybbxatscpc84ltxjtunudar36w5ikerwackzuwhushwryaky4zrt+qmicyqq/04grakdubvk5mxzugzkeo1iajtqs6c9ehiolplfooiosx+qvv8d/vegqr6dbxkssairi7kbryqgoytfoogsses2nrmaimbkrjgmsmdhmoxpmco7fj8+lrtc567j3x+2lq2oadbal9sabcmejuac34a50aqfv4noqwxxrw16wv+zmrnw2qsw2+fx2zhfexref</latexit> <latexit sha1_base64="3gau8xc0s+bn+bf9fg9nendpaao=">aaacwnicbzfbs8mwgibtetjcpmzdntfbisjiaiugxggelvrswbnbwkoazs4stuuscqp0t3qlf/4vme2kqpodwmvzfm8ox8kem6ud592yfxaxlmv1luzzdw19o7w59ajivblajtgpzt/einimafczzwk/krrhiae9chjd+l0xkhwlxyoejjsi8lngi0awngi15cxkghiz8qbphrfwkcgrw3n4gzkjxim8cssc+0fw+ik7ca/zmmuavpk3y55jlkykva6yualybbxatscpc84ltxjtunudar36w5ikerwackzuwhushwryaky4zrt+qmicyqq/04grakdubvk5mxzugzkeo1iajtqs6c9ehiolplfooiosx+qvv8d/vegqr6dbxkssairi7kbryqgoytfoogsses2nrmaimbkrjgmsmdhmoxpmco7fj8+lrtc567j3x+2lq2oadbal9sabcmejuac34a50aqfv4noqwxxrw16wv+zmrnw2qsw2+fx2zhfexref</latexit> TUCKER LAYERS A nice geometric interpretation: Reveal latent features in each mode. Core tensor, G, yields relatives importance of all combined features. A i1 i 2 i n = G r1 r 2 r n C (1) i 1 r 1 C (2) i 2 r 2 C (n) i n r n Straight forward compression and application Core tensor is of the same order. 45

46 TUCKER LAYERS Replace Fully Connected Resnet-32, CIFAR-10 X. Cao, G. Rabusseau, J. Pineau. Tensor Regression Networks with various Low-Rank Tensor Approximations. arxiv: [cs.lg] Dec

47 TUCKER LAYERS Replace Fully Connected TCL (CP) J. Kossaifi, Z. Lipton, A. Khanna, T. Furanello, A. Anandkumar. Tensor Contraction and Regression Networks.. arxiv: [cs.lg] Nov 2017 TRL (Tucker) 47

48 TUCKER LAYERS Performance and Compression Resnet-101 on ImageNet: Compression of the FullyConnected Layer J. Kossaifi, Z. Lipton, A. Khanna, T. Furanello, A. Anandkumar. Tensor Contraction and Regression Networks.. arxiv: [cs.lg] Nov

49 TENSOR RING LAYERS Compression for fully connected layers and convolutional layers 49

50 TENSOR TRAIN LAYERS Replace Fully Connected Resnet-32, CIFAR-10 X. Cao, G. Rabusseau, J. Pineau. Tensor Regression Networks with various Low-Rank Tensor Approximations. arxiv: [cs.lg] Dec

51 TENSOR TRAIN LAYERS Replace Fully Connected VGG-16/19, ImageNet A. Novikov, D. Podoprikhin, A. Osokin, D. Vetrov. Tensorizing Neural Networks. arxiv: [cs.lg] Dec

52 TENSOR TRAIN LAYERS Replace Fully Connected ImageNet 52

53 TENSOR TRAIN LAYERS Replace Convolutional Resnet-like, CIFAR-10 VGG-like, CIFAR-10 T. Garipov, D. Podoprikhin, A. Novikov, D. Vetrov. Ultimate Tensorization: Compressing Convolutional and FC Layers Alike. arxiv: [cs.lg] Nov

54 KRONECKER LAYERS Compact representation of advanced linear operators Preservation of properties of linear operators 54

55 KRONECKER LAYERS Fully Connected Layer S. Zhou, J. Wu. Compression of Fully-Connected Layer in Neural Network by Kronecker Product. Advanced Computational Intelligence (ICACI) IEEE

56 <latexit sha1_base64="mi6ifyuolwhrmlt0gdms5bp2qom=">aaacqnicbzdltgixfiy7xhfvqes3jcqeopizyqluigw0cyejcjeb0ul0okfzsdsxico8mxtfwj0v4mafgrculamxav5jkz/foac9/a2ausf1/uvbwfxaxllnraxxnza3tjm7u3fcdzkmnewznzcsjaijhqljkhlpbjwg12kkbvxlo3r9gxbbfa8qbwfpuajruydijbxqzo5nf8keriy6ixmmtn2zh0fqzoogg0hzgjrdpmnyonlcjni0hyd0mlvjepllrmnz2cm2i51mvi/oiec8msymcyaqddlppu3j0cwexawj0tt0qlyixcxfjmrpmxqkqlipuqspridcilprkkemdxwxoenzdtwje/p3ikkueapxup2jpcvsbqt/qzvd6zy3iuofosqehj/khaxkh44chtblbes2uazhttwuepcqr1iq2nmqbgp2y/omvixcfizb02zpcpjgcuyda5adbjgdjxafkqagmhger+adfghp2pv2qx2nwxe0ycwemjl2/qp5ak9f</latexit> <latexit sha1_base64="mi6ifyuolwhrmlt0gdms5bp2qom=">aaacqnicbzdltgixfiy7xhfvqes3jcqeopizyqluigw0cyejcjeb0ul0okfzsdsxico8mxtfwj0v4mafgrculamxav5jkz/foac9/a2ausf1/uvbwfxaxllnraxxnza3tjm7u3fcdzkmnewznzcsjaijhqljkhlpbjwg12kkbvxlo3r9gxbbfa8qbwfpuajruydijbxqzo5nf8keriy6ixmmtn2zh0fqzoogg0hzgjrdpmnyonlcjni0hyd0mlvjepllrmnz2cm2i51mvi/oiec8msymcyaqddlppu3j0cwexawj0tt0qlyixcxfjmrpmxqkqlipuqspridcilprkkemdxwxoenzdtwje/p3ikkueapxup2jpcvsbqt/qzvd6zy3iuofosqehj/khaxkh44chtblbes2uazhttwuepcqr1iq2nmqbgp2y/omvixcfizb02zpcpjgcuyda5adbjgdjxafkqagmhger+adfghp2pv2qx2nwxe0ycwemjl2/qp5ak9f</latexit> <latexit sha1_base64="mi6ifyuolwhrmlt0gdms5bp2qom=">aaacqnicbzdltgixfiy7xhfvqes3jcqeopizyqluigw0cyejcjeb0ul0okfzsdsxico8mxtfwj0v4mafgrculamxav5jkz/foac9/a2ausf1/uvbwfxaxllnraxxnza3tjm7u3fcdzkmnewznzcsjaijhqljkhlpbjwg12kkbvxlo3r9gxbbfa8qbwfpuajruydijbxqzo5nf8keriy6ixmmtn2zh0fqzoogg0hzgjrdpmnyonlcjni0hyd0mlvjepllrmnz2cm2i51mvi/oiec8msymcyaqddlppu3j0cwexawj0tt0qlyixcxfjmrpmxqkqlipuqspridcilprkkemdxwxoenzdtwje/p3ikkueapxup2jpcvsbqt/qzvd6zy3iuofosqehj/khaxkh44chtblbes2uazhttwuepcqr1iq2nmqbgp2y/omvixcfizb02zpcpjgcuyda5adbjgdjxafkqagmhger+adfghp2pv2qx2nwxe0ycwemjl2/qp5ak9f</latexit> <latexit sha1_base64="mi6ifyuolwhrmlt0gdms5bp2qom=">aaacqnicbzdltgixfiy7xhfvqes3jcqeopizyqluigw0cyejcjeb0ul0okfzsdsxico8mxtfwj0v4mafgrculamxav5jkz/foac9/a2ausf1/uvbwfxaxllnraxxnza3tjm7u3fcdzkmnewznzcsjaijhqljkhlpbjwg12kkbvxlo3r9gxbbfa8qbwfpuajruydijbxqzo5nf8keriy6ixmmtn2zh0fqzoogg0hzgjrdpmnyonlcjni0hyd0mlvjepllrmnz2cm2i51mvi/oiec8msymcyaqddlppu3j0cwexawj0tt0qlyixcxfjmrpmxqkqlipuqspridcilprkkemdxwxoenzdtwje/p3ikkueapxup2jpcvsbqt/qzvd6zy3iuofosqehj/khaxkh44chtblbes2uazhttwuepcqr1iq2nmqbgp2y/omvixcfizb02zpcpjgcuyda5adbjgdjxafkqagmhger+adfghp2pv2qx2nwxe0ycwemjl2/qp5ak9f</latexit> <latexit sha1_base64="ucnhaqn0jsyhhpzxkckz9fjnvww=">aaacrnicbvdlsgmxfl1t3/vvdekmwatdlekr1ixgy+nswarq1pdjpbrmtibkjlcg/p0bt+78btcuvnyaqv34oha4nhmu9+zemvyow/apqiynt0xotc9uz+fmfxzrs8vnzurwyjyw2tjlidupvspbqfdly8xknkraxks3r6v/csetuyy9w34muwm/tlvpcy5eyrwra1yorhvrdsgeoboqnujmgbwcuv5khymqkw5onesjyjt6bnrebnd9ycvllmy09jlwq4encajyl9arqcmij6z22imnybozotdcutynm+ww3kiswg6qndzjjitbfi3bnqbcb+0wwx4gzn0rmekz61+kzkh+nyh44lw/ixwy4xjjfnul+j/xzrg30y1umuuou/g1qjdrgoaupzjywslq9z3hwip/kxe33hkbvvqql4h+/vjf0mo2dhv0dku+fzhqyxpwyq02gmi27mmxnealbnzdm7zcw/aqvatvwcdxtbkmzlbgbyrwctbkr3g=</latexit> <latexit sha1_base64="ucnhaqn0jsyhhpzxkckz9fjnvww=">aaacrnicbvdlsgmxfl1t3/vvdekmwatdlekr1ixgy+nswarq1pdjpbrmtibkjlcg/p0bt+78btcuvnyaqv34oha4nhmu9+zemvyow/apqiynt0xotc9uz+fmfxzrs8vnzurwyjyw2tjlidupvspbqfdly8xknkraxks3r6v/csetuyy9w34muwm/tlvpcy5eyrwra1yorhvrdsgeoboqnujmgbwcuv5khymqkw5onesjyjt6bnrebnd9ycvllmy09jlwq4encajyl9arqcmij6z22imnybozotdcutynm+ww3kiswg6qndzjjitbfi3bnqbcb+0wwx4gzn0rmekz61+kzkh+nyh44lw/ixwy4xjjfnul+j/xzrg30y1umuuou/g1qjdrgoaupzjywslq9z3hwip/kxe33hkbvvqql4h+/vjf0mo2dhv0dku+fzhqyxpwyq02gmi27mmxnealbnzdm7zcw/aqvatvwcdxtbkmzlbgbyrwctbkr3g=</latexit> <latexit sha1_base64="ucnhaqn0jsyhhpzxkckz9fjnvww=">aaacrnicbvdlsgmxfl1t3/vvdekmwatdlekr1ixgy+nswarq1pdjpbrmtibkjlcg/p0bt+78btcuvnyaqv34oha4nhmu9+zemvyow/apqiynt0xotc9uz+fmfxzrs8vnzurwyjyw2tjlidupvspbqfdly8xknkraxks3r6v/csetuyy9w34muwm/tlvpcy5eyrwra1yorhvrdsgeoboqnujmgbwcuv5khymqkw5onesjyjt6bnrebnd9ycvllmy09jlwq4encajyl9arqcmij6z22imnybozotdcutynm+ww3kiswg6qndzjjitbfi3bnqbcb+0wwx4gzn0rmekz61+kzkh+nyh44lw/ixwy4xjjfnul+j/xzrg30y1umuuou/g1qjdrgoaupzjywslq9z3hwip/kxe33hkbvvqql4h+/vjf0mo2dhv0dku+fzhqyxpwyq02gmi27mmxnealbnzdm7zcw/aqvatvwcdxtbkmzlbgbyrwctbkr3g=</latexit> <latexit sha1_base64="ucnhaqn0jsyhhpzxkckz9fjnvww=">aaacrnicbvdlsgmxfl1t3/vvdekmwatdlekr1ixgy+nswarq1pdjpbrmtibkjlcg/p0bt+78btcuvnyaqv34oha4nhmu9+zemvyow/apqiynt0xotc9uz+fmfxzrs8vnzurwyjyw2tjlidupvspbqfdly8xknkraxks3r6v/csetuyy9w34muwm/tlvpcy5eyrwra1yorhvrdsgeoboqnujmgbwcuv5khymqkw5onesjyjt6bnrebnd9ycvllmy09jlwq4encajyl9arqcmij6z22imnybozotdcutynm+ww3kiswg6qndzjjitbfi3bnqbcb+0wwx4gzn0rmekz61+kzkh+nyh44lw/ixwy4xjjfnul+j/xzrg30y1umuuou/g1qjdrgoaupzjywslq9z3hwip/kxe33hkbvvqql4h+/vjf0mo2dhv0dku+fzhqyxpwyq02gmi27mmxnealbnzdm7zcw/aqvatvwcdxtbkmzlbgbyrwctbkr3g=</latexit> KRONECKER LAYERS RNN Layer Kronecker product preserves unitarity Control RNN vanishing/exploding gradient problem on small Kronecker factors A i1 i 2 = C a (1) 1 a 2 C (2) b 1 b 2 C (m) d 1 d 2 L( )+ kc (i) C (i)t Ik 2 2 Complex-valued factors for compact unitary set! 56

57 KRONECKER LAYERS RNN Layer C. Jose, M. Cisse, F. Fleuret. Kronecker Recurrent Units. arxiv: [cs.lg] Dec