Data Mining & Machine Learning

Size: px

Start display at page:

Download "Data Mining & Machine Learning"

Pauline Ward
5 years ago
Views:

1 Data Mining & Machine Learning CS57300 Purdue University April 10,

2 Predicting Sequences 2

3 But first, a detour to Noise Contrastive Estimation 3

} Machine learning methods are much better at classifying examples than generating new ones In classification tasks, we use the exact derivatives to find a solution that maximizes the likelihood In

4 } Machine learning methods are much better at classifying examples than generating new ones In classification tasks, we use the exact derivatives to find a solution that maximizes the likelihood In generative tasks, we can only compute an estimate of the derivative } Because we have better techniques to classify data than to generate data Can we make generative tasks look more like classification tasks? Bruno Ribeiro 4

Learn a Classifier to Distinguishing Noise from Data This is key idea behind noise contrastive estimation (NCE): make generative tasks look like

5 Learn a Classifier to Distinguishing Noise from Data This is key idea behind noise contrastive estimation (NCE): make generative tasks look like classification tasks Pioneered by Hastie, Tibshirani, Friedman in The Elements of Statistical Learning in 2008, Section Unsupervised Learning as Supervised Learning The idea is quite simple, consider the task of learning to distinguish noise from the data, i.e., search for a good classifier f(x ;W) Label y 1 =1 (= true data) Training Data Examples Classifier. x 1 = x 2 = Random Noise Examples Label y=0 (= false data) True example generator: : noise generator 5

6 Generation Task Using Classifier Now use the learned classifier (which distinguishes noise from the data) to generate new data but how? Naïve approach: Generate examples from the random noise whatever gets classified as real data will be our generated examples x 1 = Random Noise Examples x 2 = What is the problem with this naïve approach? In very high dimensions (e.g., images), true random noise will not generate any interesting examples x 1 x 2 x 3 6

7 Go to ipython notebook 7

8 Back to sequences 8

9 Sequences } In this lecture we will focus on word sequences (a.k.a. text) } The techniques we see are applicable to any type of sequence } A sequence is a succession of elements from a set (likely finite) } We will write a sequence of n elements as x 1,, x n } The temporal ordering is key to learning the sequence 9

10 2017 Bruno Ribeiro Word Sequences Rank-3 word embedding } Embed words w.r.t. their sentences Bring me a constant woman to her husband Forgetting, like a good man your late censure of his wife } Under the Markov assumption P[bring, me, a, constant, woman, to, her, husband] = P[bring]P[me bring] P[a me] P[constant a] P[husband her] Go to ipython notebook

11 2017 Bruno Ribeiro Word Sequences & Embeddings Rank-3 word embedding } Embed words w.r.t. their sentences Bring me a constant woman to her husband Forgetting, like a good man your late censure of his wife husband man wife woman } Initial idea by (Chen et al. 2012) was named Latent Markov Embedding, rediscovered by (Mikolov et al. 2013) named word2vec } Main difference: Application: (Mikolov et al. 2013) paid attention to the composition of latent vectors in sentences Otherwise, techniques equivalent Chen, S., Moore, J. L., Turnbull, D., & Joachims, T. (2012). Playlist prediction via metric embedding. In ACM SIGKDD. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In NIPS

12 2017 Bruno Ribeiro Word2vec Embeddings Rank-3 word embedding How the math works Bring me a constant woman to her husband Forgetting, like a good man your late censure of his wife husband man woman wife } Conditional bag-of-words assumption (SKIPGRAM): All words are independent given the target word P[bring, constant, husband woman] = P[bring woman] P[constant woman] P[husband woman] P[forgetting, like, good, late, censure, wife man] = P[forgetting man] P[like man] P[good man] P[late man] P[censure man] P[wife man]

+ Embedding of woman as conditional word 2 nd coordinate of conditional word σ 2 u 2

13 Word2vec Type Embeddings III 1 st coordinate of output words 2 nd coord. of output words exp X = 1 st coordinate of conditional word σ 1 u 1 woman v 1 husband + Embedding of woman as conditional word 2 nd coordinate of conditional word σ 2 u 2 woman v 2 husband The machine learning challenge is not summing over all words 2017 Bruno Ribeiro

14 <latexit sha1_base64="cipmw6+acjie9pp8tt1pqfh35g4=">aaahohic3vvnbxmxen0uskr4auhixw0vkrqabsiqschsjs6vemgrmryiu428xiexyq9xtrdtau3p4mfxozhyyfadkdrtbycbpdu+z8zz2g9gdhbzpo3rfqss3bu//kc68rd26pgtp89w154fa5koqrtecqloa6wpzxhtgmy4py0vxslg9cqyf8r8j+duasajizojqs/wmgidrrabu3/16ynycbwreymsly9nfq3tbvprfmi1hipko9t7ziycp9o4gcc0rcfp2rhqie8nom/hnvz6tocwuedpyf3xxjpk1pcvdw6ws6p+6qbbdpobfkgrbjtooq76a8vfvvcsrndiei617rxc2pgwk8mip2nnszsnmrnjie2f5yzwerzu+/yyz52iovhdnjakvsig3dplslhoprebrapsrvq6lzpe5oslzvdbtyyke0mjuiqajbwzibj6ojapsgyfambemdguiiosmdfqtbks2dpgsq7t2lwggcdussl8s3bekj93w8pgv6upg5wfcquj1smcu90utbnqjz8vp2amzdb/rb9z1wmos0fqrc+ifajh4zb19gbqgkgghqm1noadx0azk1rzsiq9ttcnhbfz8m2txq8jgg4pnczu+kjnrizyuap9xn2ftrdrw/0n2gvoa/gipasdc4jmmqpgesdrkzfzned+ysdmqz1ro9wodxbn4qhpinaafpnk9ujp+qvodooxu7qdubcv9g37smvpgpbqeae6/ikhp65bfb3jzp+ctjd1kwj717v0f9j0rllrytuudg4ht8zuysnvdpfemdpitdcjbemfr0rbw5d308b/nrayetu4k14gezhdwjj7hq1c4zloxx8ufsfxu9mcgh++3dzzlx+mfeels+e0njbz3tlxdp0dp+sq50dlvbjvev3dqo5w96uhrehspes8cozg9ctptzacjq==</latexit> <latexit sha1_base64="cipmw6+acjie9pp8tt1pqfh35g4=">aaahohic3vvnbxmxen0uskr4auhixw0vkrqabsiqschsjs6vemgrmryiu428xiexyq9xtrdtau3p4mfxozhyyfadkdrtbycbpdu+z8zz2g9gdhbzpo3rfqss3bu//kc68rd26pgtp89w154fa5koqrtecqloa6wpzxhtgmy4py0vxslg9cqyf8r8j+duasajizojqs/wmgidrrabu3/16ynycbwreymsly9nfq3tbvprfmi1hipko9t7ziycp9o4gcc0rcfp2rhqie8nom/hnvz6tocwuedpyf3xxjpk1pcvdw6ws6p+6qbbdpobfkgrbjtooq76a8vfvvcsrndiei617rxc2pgwk8mip2nnszsnmrnjie2f5yzwerzu+/yyz52iovhdnjakvsig3dplslhoprebrapsrvq6lzpe5oslzvdbtyyke0mjuiqajbwzibj6ojapsgyfambemdguiiosmdfqtbks2dpgsq7t2lwggcdussl8s3bekj93w8pgv6upg5wfcquj1smcu90utbnqjz8vp2amzdb/rb9z1wmos0fqrc+ifajh4zb19gbqgkgghqm1noadx0azk1rzsiq9ttcnhbfz8m2txq8jgg4pnczu+kjnrizyuap9xn2ftrdrw/0n2gvoa/gipasdc4jmmqpgesdrkzfzned+ysdmqz1ro9wodxbn4qhpinaafpnk9ujp+qvodooxu7qdubcv9g37smvpgpbqeae6/ikhp65bfb3jzp+ctjd1kwj717v0f9j0rllrytuudg4ht8zuysnvdpfemdpitdcjbemfr0rbw5d308b/nrayetu4k14gezhdwjj7hq1c4zloxx8ufsfxu9mcgh++3dzzlx+mfeels+e0njbz3tlxdp0dp+sq50dlvbjvev3dqo5w96uhrehspes8cozg9ctptzacjq==</latexit> <latexit sha1_base64="cipmw6+acjie9pp8tt1pqfh35g4=">aaahohic3vvnbxmxen0uskr4auhixw0vkrqabsiqschsjs6vemgrmryiu428xiexyq9xtrdtau3p4mfxozhyyfadkdrtbycbpdu+z8zz2g9gdhbzpo3rfqss3bu//kc68rd26pgtp89w154fa5koqrtecqloa6wpzxhtgmy4py0vxslg9cqyf8r8j+duasajizojqs/wmgidrrabu3/16ynycbwreymsly9nfq3tbvprfmi1hipko9t7ziycp9o4gcc0rcfp2rhqie8nom/hnvz6tocwuedpyf3xxjpk1pcvdw6ws6p+6qbbdpobfkgrbjtooq76a8vfvvcsrndiei617rxc2pgwk8mip2nnszsnmrnjie2f5yzwerzu+/yyz52iovhdnjakvsig3dplslhoprebrapsrvq6lzpe5oslzvdbtyyke0mjuiqajbwzibj6ojapsgyfambemdguiiosmdfqtbks2dpgsq7t2lwggcdussl8s3bekj93w8pgv6upg5wfcquj1smcu90utbnqjz8vp2amzdb/rb9z1wmos0fqrc+ifajh4zb19gbqgkgghqm1noadx0azk1rzsiq9ttcnhbfz8m2txq8jgg4pnczu+kjnrizyuap9xn2ftrdrw/0n2gvoa/gipasdc4jmmqpgesdrkzfzned+ysdmqz1ro9wodxbn4qhpinaafpnk9ujp+qvodooxu7qdubcv9g37smvpgpbqeae6/ikhp65bfb3jzp+ctjd1kwj717v0f9j0rllrytuudg4ht8zuysnvdpfemdpitdcjbemfr0rbw5d308b/nrayetu4k14gezhdwjj7hq1c4zloxx8ufsfxu9mcgh++3dzzlx+mfeels+e0njbz3tlxdp0dp+sq50dlvbjvev3dqo5w96uhrehspes8cozg9ctptzacjq==</latexit> <latexit sha1_base64="cipmw6+acjie9pp8tt1pqfh35g4=">aaahohic3vvnbxmxen0uskr4auhixw0vkrqabsiqschsjs6vemgrmryiu428xiexyq9xtrdtau3p4mfxozhyyfadkdrtbycbpdu+z8zz2g9gdhbzpo3rfqss3bu//kc68rd26pgtp89w154fa5koqrtecqloa6wpzxhtgmy4py0vxslg9cqyf8r8j+duasajizojqs/wmgidrrabu3/16ynycbwreymsly9nfq3tbvprfmi1hipko9t7ziycp9o4gcc0rcfp2rhqie8nom/hnvz6tocwuedpyf3xxjpk1pcvdw6ws6p+6qbbdpobfkgrbjtooq76a8vfvvcsrndiei617rxc2pgwk8mip2nnszsnmrnjie2f5yzwerzu+/yyz52iovhdnjakvsig3dplslhoprebrapsrvq6lzpe5oslzvdbtyyke0mjuiqajbwzibj6ojapsgyfambemdguiiosmdfqtbks2dpgsq7t2lwggcdussl8s3bekj93w8pgv6upg5wfcquj1smcu90utbnqjz8vp2amzdb/rb9z1wmos0fqrc+ifajh4zb19gbqgkgghqm1noadx0azk1rzsiq9ttcnhbfz8m2txq8jgg4pnczu+kjnrizyuap9xn2ftrdrw/0n2gvoa/gipasdc4jmmqpgesdrkzfzned+ysdmqz1ro9wodxbn4qhpinaafpnk9ujp+qvodooxu7qdubcv9g37smvpgpbqeae6/ikhp65bfb3jzp+ctjd1kwj717v0f9j0rllrytuudg4ht8zuysnvdpfemdpitdcjbemfr0rbw5d308b/nrayetu4k14gezhdwjj7hq1c4zloxx8ufsfxu9mcgh++3dzzlx+mfeels+e0njbz3tlxdp0dp+sq50dlvbjvev3dqo5w96uhrehspes8cozg9ctptzacjq==</latexit> <latexit sha1_base64="20r+kfvojzrfz60s1mdkdrhzzoq=">aaahf3ic3vxnbhmxehyldsx8tiujly1vpidaabobqekvknfb4kcrsfspu1re7ysxyq9xtrdtau17wbfehbviyphn4awyza5c07s9gqswvvt5zj7p+bvljllbjfx9hwul164v1w4s36zfun3n7srq2r09ozlnomuuupogogyet6brurvwkgqgmhkwh41efv79i9cgq+sdhacqsjpiej8zath03gwtjzygoo/mh6sbfsufdg8etcuwqaqxe7i29doifcskjjyjakyv7ac2dfrbzgtk9sazkfi2ogpoxuc8nqmvyej3mkmbew30x15fafws602sz0mosmpgmsjise3qnpcvxot8vcz2n4eoj2lmiwflon4mpku8qgyv5hqyfwmelgmo5xpssdvlfswayvksbzusjq/pzjb0bfopgtpgewzi1u0sh51wpgikhmmqx84maqqmjblhqm5kvwksi6kedtrakrvjb/2pvx4jxaqdejhmskqaxi9d8lqpny8ipcidcwxwnlrdt6eefel6hleb236rhbp6o9hwib4a7ibw9lhnhzwjvwzn03/kbw15tf836jsgg+af8jw8mcjilnnnqmxhiopnyjpbycvnxgofvwa7k1rknonhe1wgiwirjjdht/dfcqbtskzqcuzcxz0l6srr+zyq2hijov4piaeusxw9ys//nlyxnvlu9q+f0v9j0zosuodqger4owqqbu2t0svfesztuhdjrxgvf7ytjy2exe1d/3lazidgadg8sim4uicvceobhuiz0t7/kmydvu6rjt1++2rj5031ycytb+qhazi2euz2ycuys7qeeu0+ke/kc+1j7uvta+1bgbq4uhhuk5lr+/4lktzp9a==</latexit> <latexit sha1_base64="20r+kfvojzrfz60s1mdkdrhzzoq=">aaahf3ic3vxnbhmxehyldsx8tiujly1vpidaabobqekvknfb4kcrsfspu1re7ysxyq9xtrdtau17wbfehbviyphn4awyza5c07s9gqswvvt5zj7p+bvljllbjfx9hwul164v1w4s36zfun3n7srq2r09ozlnomuuupogogyet6brurvwkgqgmhkwh41efv79i9cgq+sdhacqsjpiej8zath03gwtjzygoo/mh6sbfsufdg8etcuwqaqxe7i29doifcskjjyjakyv7ac2dfrbzgtk9sazkfi2ogpoxuc8nqmvyej3mkmbew30x15fafws602sz0mosmpgmsjise3qnpcvxot8vcz2n4eoj2lmiwflon4mpku8qgyv5hqyfwmelgmo5xpssdvlfswayvksbzusjq/pzjb0bfopgtpgewzi1u0sh51wpgikhmmqx84maqqmjblhqm5kvwksi6kedtrakrvjb/2pvx4jxaqdejhmskqaxi9d8lqpny8ipcidcwxwnlrdt6eefel6hleb236rhbp6o9hwib4a7ibw9lhnhzwjvwzn03/kbw15tf836jsgg+af8jw8mcjilnnnqmxhiopnyjpbycvnxgofvwa7k1rknonhe1wgiwirjjdht/dfcqbtskzqcuzcxz0l6srr+zyq2hijov4piaeusxw9ys//nlyxnvlu9q+f0v9j0zosuodqger4owqqbu2t0svfesztuhdjrxgvf7ytjy2exe1d/3lazidgadg8sim4uicvceobhuiz0t7/kmydvu6rjt1++2rj5031ycytb+qhazi2euz2ycuys7qeeu0+ke/kc+1j7uvta+1bgbq4uhhuk5lr+/4lktzp9a==</latexit> <latexit sha1_base64="20r+kfvojzrfz60s1mdkdrhzzoq=">aaahf3ic3vxnbhmxehyldsx8tiujly1vpidaabobqekvknfb4kcrsfspu1re7ysxyq9xtrdtau17wbfehbviyphn4awyza5c07s9gqswvvt5zj7p+bvljllbjfx9hwul164v1w4s36zfun3n7srq2r09ozlnomuuupogogyet6brurvwkgqgmhkwh41efv79i9cgq+sdhacqsjpiej8zath03gwtjzygoo/mh6sbfsufdg8etcuwqaqxe7i29doifcskjjyjakyv7ac2dfrbzgtk9sazkfi2ogpoxuc8nqmvyej3mkmbew30x15fafws602sz0mosmpgmsjise3qnpcvxot8vcz2n4eoj2lmiwflon4mpku8qgyv5hqyfwmelgmo5xpssdvlfswayvksbzusjq/pzjb0bfopgtpgewzi1u0sh51wpgikhmmqx84maqqmjblhqm5kvwksi6kedtrakrvjb/2pvx4jxaqdejhmskqaxi9d8lqpny8ipcidcwxwnlrdt6eefel6hleb236rhbp6o9hwib4a7ibw9lhnhzwjvwzn03/kbw15tf836jsgg+af8jw8mcjilnnnqmxhiopnyjpbycvnxgofvwa7k1rknonhe1wgiwirjjdht/dfcqbtskzqcuzcxz0l6srr+zyq2hijov4piaeusxw9ys//nlyxnvlu9q+f0v9j0zosuodqger4owqqbu2t0svfesztuhdjrxgvf7ytjy2exe1d/3lazidgadg8sim4uicvceobhuiz0t7/kmydvu6rjt1++2rj5031ycytb+qhazi2euz2ycuys7qeeu0+ke/kc+1j7uvta+1bgbq4uhhuk5lr+/4lktzp9a==</latexit> <latexit sha1_base64="sm8sxrapgkh1g+uhjbid190wsxc=">aaahjhic3vxdahqxfe7vrnx9a/vkvektc6u0y8yikeih4i3ghrxtd+wmjzm5oxs2mqxjpu02dd6nxuqlecdeeonl+ajmdoz1t9v2tkedw3w553w5j98jszrxpo3nfv+4dpnkyupq0rxm9rs3b91exrmzq2wukoxqyaxaj4ggzllymcxw2m8uebfx2iugl0r/3ieozwt6zowycavjutznlbhnoli+z4pxijymapgoglqi3rjekojgec3reoob54ffgzvuj+2dlcwfqsxplia1lbote76xmdaszrjludsdxeng6jak0ispwaztikch9nhcqyfbzh/jvltusw0ew6djlgitryjykykygt7tk41n+xq56t8lluuz3ebkq0t9ngmjcakjjpkcavjiauivc+vioickuooum8lsrm2k5lpozmqqzahkshfaslikfnztdrue1kyschypokzwd0gguioypk6d8x45mklw2pfv9ym3gng3latk4yhkiugap7lbq747blgkkslaqlvkm6pzctsdmklpu8jh0+v4ow22wi0mcqjun7eirz0zygmi29117yhe2mbt7zfovqdrwhphw+70oeemzd7jfdpioayx0zekaz5lyvuzt/1uzhx3kl5bxiu7enzjzo+e7l/itkzhvdufzlm6umfuvtslosfd44xt8u8joxgdq+gfe/7ntd3rldpt//op/z80nzlubkqlgrnbizazkgkkki/si2ygnalmtk39s52oaot4yprzn6zfzgcglm2lji/lc1jyolk5updm+kcfhxmw2+34rsdvhq9tva4fjcv0hz1abesjp2glvutbaadr9b59qj/q58bhxpfg18a3kvtsqs25i2zg48cv7ehvaw==</latexit> <latexit sha1_base64="sm8sxrapgkh1g+uhjbid190wsxc=">aaahjhic3vxdahqxfe7vrnx9a/vkvektc6u0y8yikeih4i3ghrxtd+wmjzm5oxs2mqxjpu02dd6nxuqlecdeeonl+ajmdoz1t9v2tkedw3w553w5j98jszrxpo3nfv+4dpnkyupq0rxm9rs3b91exrmzq2wukoxqyaxaj4ggzllymcxw2m8uebfx2iugl0r/3ieozwt6zowycavjutznlbhnoli+z4pxijymapgoglqi3rjekojgec3reoob54ffgzvuj+2dlcwfqsxplia1lbote76xmdaszrjludsdxeng6jak0ispwaztikch9nhcqyfbzh/jvltusw0ew6djlgitryjykykygt7tk41n+xq56t8lluuz3ebkq0t9ngmjcakjjpkcavjiauivc+vioickuooum8lsrm2k5lpozmqqzahkshfaslikfnztdrue1kyschypokzwd0gguioypk6d8x45mklw2pfv9ym3gng3latk4yhkiugap7lbq747blgkkslaqlvkm6pzctsdmklpu8jh0+v4ow22wi0mcqjun7eirz0zygmi29117yhe2mbt7zfovqdrwhphw+70oeemzd7jfdpioayx0zekaz5lyvuzt/1uzhx3kl5bxiu7enzjzo+e7l/itkzhvdufzlm6umfuvtslosfd44xt8u8joxgdq+gfe/7ntd3rldpt//op/z80nzlubkqlgrnbizazkgkkki/si2ygnalmtk39s52oaot4yprzn6zfzgcglm2lji/lc1jyolk5updm+kcfhxmw2+34rsdvhq9tva4fjcv0hz1abesjp2glvutbaadr9b59qj/q58bhxpfg18a3kvtsqs25i2zg48cv7ehvaw==</latexit> <latexit sha1_base64="sm8sxrapgkh1g+uhjbid190wsxc=">aaahjhic3vxdahqxfe7vrnx9a/vkvektc6u0y8yikeih4i3ghrxtd+wmjzm5oxs2mqxjpu02dd6nxuqlecdeeonl+ajmdoz1t9v2tkedw3w553w5j98jszrxpo3nfv+4dpnkyupq0rxm9rs3b91exrmzq2wukoxqyaxaj4ggzllymcxw2m8uebfx2iugl0r/3ieozwt6zowycavjutznlbhnoli+z4pxijymapgoglqi3rjekojgec3reoob54ffgzvuj+2dlcwfqsxplia1lbote76xmdaszrjludsdxeng6jak0ispwaztikch9nhcqyfbzh/jvltusw0ew6djlgitryjykykygt7tk41n+xq56t8lluuz3ebkq0t9ngmjcakjjpkcavjiauivc+vioickuooum8lsrm2k5lpozmqqzahkshfaslikfnztdrue1kyschypokzwd0gguioypk6d8x45mklw2pfv9ym3gng3latk4yhkiugap7lbq747blgkkslaqlvkm6pzctsdmklpu8jh0+v4ow22wi0mcqjun7eirz0zygmi29117yhe2mbt7zfovqdrwhphw+70oeemzd7jfdpioayx0zekaz5lyvuzt/1uzhx3kl5bxiu7enzjzo+e7l/itkzhvdufzlm6umfuvtslosfd44xt8u8joxgdq+gfe/7ntd3rldpt//op/z80nzlubkqlgrnbizazkgkkki/si2ygnalmtk39s52oaot4yprzn6zfzgcglm2lji/lc1jyolk5updm+kcfhxmw2+34rsdvhq9tva4fjcv0hz1abesjp2glvutbaadr9b59qj/q58bhxpfg18a3kvtsqs25i2zg48cv7ehvaw==</latexit> <latexit sha1_base64="sm8sxrapgkh1g+uhjbid190wsxc=">aaahjhic3vxdahqxfe7vrnx9a/vkvektc6u0y8yikeih4i3ghrxtd+wmjzm5oxs2mqxjpu02dd6nxuqlecdeeonl+ajmdoz1t9v2tkedw3w553w5j98jszrxpo3nfv+4dpnkyupq0rxm9rs3b91exrmzq2wukoxqyaxaj4ggzllymcxw2m8uebfx2iugl0r/3ieozwt6zowycavjutznlbhnoli+z4pxijymapgoglqi3rjekojgec3reoob54ffgzvuj+2dlcwfqsxplia1lbote76xmdaszrjludsdxeng6jak0ispwaztikch9nhcqyfbzh/jvltusw0ew6djlgitryjykykygt7tk41n+xq56t8lluuz3ebkq0t9ngmjcakjjpkcavjiauivc+vioickuooum8lsrm2k5lpozmqqzahkshfaslikfnztdrue1kyschypokzwd0gguioypk6d8x45mklw2pfv9ym3gng3latk4yhkiugap7lbq747blgkkslaqlvkm6pzctsdmklpu8jh0+v4ow22wi0mcqjun7eirz0zygmi29117yhe2mbt7zfovqdrwhphw+70oeemzd7jfdpioayx0zekaz5lyvuzt/1uzhx3kl5bxiu7enzjzo+e7l/itkzhvdufzlm6umfuvtslosfd44xt8u8joxgdq+gfe/7ntd3rldpt//op/z80nzlubkqlgrnbizazkgkkki/si2ygnalmtk39s52oaot4yprzn6zfzgcglm2lji/lc1jyolk5updm+kcfhxmw2+34rsdvhq9tva4fjcv0hz1abesjp2glvutbaadr9b59qj/q58bhxpfg18a3kvtsqs25i2zg48cv7ehvaw==</latexit> <latexit sha1_base64="6jplcbwszurx8pqdw+oaf+hhnae=">aaahcxic3vvlbxmxehylbup4txbcxnxwkqk00w5uvcquqrixja4uit6k7bj5vu5irb27sr1tu2v/ix+av0gpcgd2ozd0dqmjlewznw8+z/ibkr2kgmvjon8wfu/cxvpurnxr3n/w8nhj1bunhzrjfguhnbgjog6izolh7mbwi9hxqhirgwbhqfsuwi9omni8it+bscp8suyxh3jkdlggq0ppcbeye5wprlu1m+me9nqmbzbqufmxhwy9mocniypgire4b5ypjmkh0rrulmyzcomonzodkbh8bly74b2sbjodp1z4quhwxiaq1/5gbenccxoasrybkojwfddjjw+jmpwklje9tlou0iimwd884amoiwtat2dl9hy3aa/xmfhwiw0uvbmks6twexlapcrmrc9jhfm6rj+z4rvf8jjnditplwiycwwsxaimq64ynwicbqgkq7myjoki1eab5riue5sketpvzmwqjgiqsarvkykpe/owntw6r12fkxigijpypsyp0x3jnqv1yvmkzmyaz8u/9dfoeibgkwik2slnpcrx+nj6h4ywu0ggrrm1nobku6p5owt6rzk+pkswntnxfdtstvqyhsmgpwkvoe2bmy993o5uos/w9jzuo7+nbmv0wxglpayjcijmmumurk8ylk3is7t8mk+5gj7jtttndxbn4mhaqnaw/drjzdht81ec6adfvxul80pd3wvqypv5fsgh8rj0/fnctqebfb3lzp+cttdnkwj716f0f9j0rllretwshg4hl0mziizrxuv6ys1ycmmntju+34mkrs5upwf+mraymtokaf6qilc4gbphvt4ihwfcvfwoxduoux0xevxpz3pvy/1grkdnaao1kyt20r56j/braalok/qofqcfyxenzw3c2khcfxdqzlm0txqvfgg5j3d0</latexit> <latexit sha1_base64="6jplcbwszurx8pqdw+oaf+hhnae=">aaahcxic3vvlbxmxehylbup4txbcxnxwkqk00w5uvcquqrixja4uit6k7bj5vu5irb27sr1tu2v/ix+av0gpcgd2ozd0dqmjlewznw8+z/ibkr2kgmvjon8wfu/cxvpurnxr3n/w8nhj1bunhzrjfguhnbgjog6izolh7mbwi9hxqhirgwbhqfsuwi9omni8it+bscp8suyxh3jkdlggq0ppcbeye5wprlu1m+me9nqmbzbqufmxhwy9mocniypgire4b5ypjmkh0rrulmyzcomonzodkbh8bly74b2sbjodp1z4quhwxiaq1/5gbenccxoasrybkojwfddjjw+jmpwklje9tlou0iimwd884amoiwtat2dl9hy3aa/xmfhwiw0uvbmks6twexlapcrmrc9jhfm6rj+z4rvf8jjnditplwiycwwsxaimq64ynwicbqgkq7myjoki1eab5riue5sketpvzmwqjgiqsarvkykpe/owntw6r12fkxigijpypsyp0x3jnqv1yvmkzmyaz8u/9dfoeibgkwik2slnpcrx+nj6h4ywu0ggrrm1nobku6p5owt6rzk+pkswntnxfdtstvqyhsmgpwkvoe2bmy993o5uos/w9jzuo7+nbmv0wxglpayjcijmmumurk8ylk3is7t8mk+5gj7jtttndxbn4mhaqnaw/drjzdht81ec6adfvxul80pd3wvqypv5fsgh8rj0/fnctqebfb3lzp+cttdnkwj716f0f9j0rllretwshg4hl0mziizrxuv6ys1ycmmntju+34mkrs5upwf+mraymtokaf6qilc4gbphvt4ihwfcvfwoxduoux0xevxpz3pvy/1grkdnaao1kyt20r56j/braalok/qofqcfyxenzw3c2khcfxdqzlm0txqvfgg5j3d0</latexit> <latexit sha1_base64="6jplcbwszurx8pqdw+oaf+hhnae=">aaahcxic3vvlbxmxehylbup4txbcxnxwkqk00w5uvcquqrixja4uit6k7bj5vu5irb27sr1tu2v/ix+av0gpcgd2ozd0dqmjlewznw8+z/ibkr2kgmvjon8wfu/cxvpurnxr3n/w8nhj1bunhzrjfguhnbgjog6izolh7mbwi9hxqhirgwbhqfsuwi9omni8it+bscp8suyxh3jkdlggq0ppcbeye5wprlu1m+me9nqmbzbqufmxhwy9mocniypgire4b5ypjmkh0rrulmyzcomonzodkbh8bly74b2sbjodp1z4quhwxiaq1/5gbenccxoasrybkojwfddjjw+jmpwklje9tlou0iimwd884amoiwtat2dl9hy3aa/xmfhwiw0uvbmks6twexlapcrmrc9jhfm6rj+z4rvf8jjnditplwiycwwsxaimq64ynwicbqgkq7myjoki1eab5riue5sketpvzmwqjgiqsarvkykpe/owntw6r12fkxigijpypsyp0x3jnqv1yvmkzmyaz8u/9dfoeibgkwik2slnpcrx+nj6h4ywu0ggrrm1nobku6p5owt6rzk+pkswntnxfdtstvqyhsmgpwkvoe2bmy993o5uos/w9jzuo7+nbmv0wxglpayjcijmmumurk8ylk3is7t8mk+5gj7jtttndxbn4mhaqnaw/drjzdht81ec6adfvxul80pd3wvqypv5fsgh8rj0/fnctqebfb3lzp+cttdnkwj716f0f9j0rllretwshg4hl0mziizrxuv6ys1ycmmntju+34mkrs5upwf+mraymtokaf6qilc4gbphvt4ihwfcvfwoxduoux0xevxpz3pvy/1grkdnaao1kyt20r56j/braalok/qofqcfyxenzw3c2khcfxdqzlm0txqvfgg5j3d0</latexit> <latexit sha1_base64="6jplcbwszurx8pqdw+oaf+hhnae=">aaahcxic3vvlbxmxehylbup4txbcxnxwkqk00w5uvcquqrixja4uit6k7bj5vu5irb27sr1tu2v/ix+av0gpcgd2ozd0dqmjlewznw8+z/ibkr2kgmvjon8wfu/cxvpurnxr3n/w8nhj1bunhzrjfguhnbgjog6izolh7mbwi9hxqhirgwbhqfsuwi9omni8it+bscp8suyxh3jkdlggq0ppcbeye5wprlu1m+me9nqmbzbqufmxhwy9mocniypgire4b5ypjmkh0rrulmyzcomonzodkbh8bly74b2sbjodp1z4quhwxiaq1/5gbenccxoasrybkojwfddjjw+jmpwklje9tlou0iimwd884amoiwtat2dl9hy3aa/xmfhwiw0uvbmks6twexlapcrmrc9jhfm6rj+z4rvf8jjnditplwiycwwsxaimq64ynwicbqgkq7myjoki1eab5riue5sketpvzmwqjgiqsarvkykpe/owntw6r12fkxigijpypsyp0x3jnqv1yvmkzmyaz8u/9dfoeibgkwik2slnpcrx+nj6h4ywu0ggrrm1nobku6p5owt6rzk+pkswntnxfdtstvqyhsmgpwkvoe2bmy993o5uos/w9jzuo7+nbmv0wxglpayjcijmmumurk8ylk3is7t8mk+5gj7jtttndxbn4mhaqnaw/drjzdht81ec6adfvxul80pd3wvqypv5fsgh8rj0/fnctqebfb3lzp+cttdnkwj716f0f9j0rllretwshg4hl0mziizrxuv6ys1ycmmntju+34mkrs5upwf+mraymtokaf6qilc4gbphvt4ihwfcvfwoxduoux0xevxpz3pvy/1grkdnaao1kyt20r56j/braalok/qofqcfyxenzw3c2khcfxdqzlm0txqvfgg5j3d0</latexit> <latexit sha1_base64="rjx2viy0zv0wbbosgmdnfnknehm=">aaahg3ic3vxnbhmxehyldsx8pxdk4ljfcqindimqskhsjs5ihcgs/zgyq8rrnsrw7pxk9qznrx0tomklcencofacvadeh4wmaxsdcsyt9vpmfj7xn5ydpzxp43k/lpavxv9p3fi92bx1+87de621+/tazorchpvcqsoiaoasgt3ddifdvaereyedapyq8b9mqgkmk/dmmkioydbha0ajcaajvssg5si24hnked8/am14xa8cebh4ndha9dg9wlv5gcsszgisqznruu97qqktuyzrdnkzydskhi7jeprxhku6iqj0ae/kxdluo3+mb1k5lzg4tj4lwsk0norirqpirvq8rzbe5otnzvaitcxjmwmjrrinmo6nxiuuogykqoftbwhvzjwl6ygoqo0tbc5lsbarkuu8ozdbkdeokuvokuko8hm3nwx8wpskyfmkijpapsip6k5gmjp1yv1ymdymajxuaeluy9/6m2894m4pf5tamzvckcr+yom3a9f9kfld3npau8pto9kpnimisv9twmhb6/qhbbbbbqzxenxuykwo+2bekhb3epvey7y1htveb9crqm+bl46h3afxgsyya8b5dnfgnotpickat5lyfcydv5caxz0trycuwh0i6ysxr8/2x3fm07cq6grmql29c+rkm/mckk7xwun4p4scus5v9io9/3paxnrknbz//zt+t5qekdqgra0fc7ddifnqxehvxkthziw4e8xow/vno1hrymnvnoc/t4vmcawpmhdjhhcxsorbzxoh7pnwzz8ki2c/1/vdj9893dh5wz8yq+gheoq6yefp0q56jxbrhqjogj6gt+hz42pjs+nr41svurxucx6gudh4/gtsildh</latexit> <latexit sha1_base64="rjx2viy0zv0wbbosgmdnfnknehm=">aaahg3ic3vxnbhmxehyldsx8pxdk4ljfcqindimqskhsjs5ihcgs/zgyq8rrnsrw7pxk9qznrx0tomklcencofacvadeh4wmaxsdcsyt9vpmfj7xn5ydpzxp43k/lpavxv9p3fi92bx1+87de621+/tazorchpvcqsoiaoasgt3ddifdvaereyedapyq8b9mqgkmk/dmmkioydbha0ajcaajvssg5si24hnked8/am14xa8cebh4ndha9dg9wlv5gcsszgisqznruu97qqktuyzrdnkzydskhi7jeprxhku6iqj0ae/kxdluo3+mb1k5lzg4tj4lwsk0norirqpirvq8rzbe5otnzvaitcxjmwmjrrinmo6nxiuuogykqoftbwhvzjwl6ygoqo0tbc5lsbarkuu8ozdbkdeokuvokuko8hm3nwx8wpskyfmkijpapsip6k5gmjp1yv1ymdymajxuaeluy9/6m2894m4pf5tamzvckcr+yom3a9f9kfld3npau8pto9kpnimisv9twmhb6/qhbbbbbqzxenxuykwo+2bekhb3epvey7y1htveb9crqm+bl46h3afxgsyya8b5dnfgnotpickat5lyfcydv5caxz0trycuwh0i6ysxr8/2x3fm07cq6grmql29c+rkm/mckk7xwun4p4scus5v9io9/3paxnrknbz//zt+t5qekdqgra0fc7ddifnqxehvxkthziw4e8xow/vno1hrymnvnoc/t4vmcawpmhdjhhcxsorbzxoh7pnwzz8ki2c/1/vdj9893dh5wz8yq+gheoq6yefp0q56jxbrhqjogj6gt+hz42pjs+nr41svurxucx6gudh4/gtsildh</latexit> <latexit sha1_base64="rjx2viy0zv0wbbosgmdnfnknehm=">aaahg3ic3vxnbhmxehyldsx8pxdk4ljfcqindimqskhsjs5ihcgs/zgyq8rrnsrw7pxk9qznrx0tomklcencofacvadeh4wmaxsdcsyt9vpmfj7xn5ydpzxp43k/lpavxv9p3fi92bx1+87de621+/tazorchpvcqsoiaoasgt3ddifdvaereyedapyq8b9mqgkmk/dmmkioydbha0ajcaajvssg5si24hnked8/am14xa8cebh4ndha9dg9wlv5gcsszgisqznruu97qqktuyzrdnkzydskhi7jeprxhku6iqj0ae/kxdluo3+mb1k5lzg4tj4lwsk0norirqpirvq8rzbe5otnzvaitcxjmwmjrrinmo6nxiuuogykqoftbwhvzjwl6ygoqo0tbc5lsbarkuu8ozdbkdeokuvokuko8hm3nwx8wpskyfmkijpapsip6k5gmjp1yv1ymdymajxuaeluy9/6m2894m4pf5tamzvckcr+yom3a9f9kfld3npau8pto9kpnimisv9twmhb6/qhbbbbbqzxenxuykwo+2bekhb3epvey7y1htveb9crqm+bl46h3afxgsyya8b5dnfgnotpickat5lyfcydv5caxz0trycuwh0i6ysxr8/2x3fm07cq6grmql29c+rkm/mckk7xwun4p4scus5v9io9/3paxnrknbz//zt+t5qekdqgra0fc7ddifnqxehvxkthziw4e8xow/vno1hrymnvnoc/t4vmcawpmhdjhhcxsorbzxoh7pnwzz8ki2c/1/vdj9893dh5wz8yq+gheoq6yefp0q56jxbrhqjogj6gt+hz42pjs+nr41svurxucx6gudh4/gtsildh</latexit> Matrix Factorization as a Neural Network (v2) But neural networks can have isomorphic network representations KX Matrix factorization: X U V T = Neural network representation: k=1 ku k v T k 4X X 1,7 = ku 1,k v 7,k k=1 Element-wise product Linear activation Linear activation u 1 v 1 U<latexit sha1_base64="20r+kfvojzrfz60s1mdkdrhzzoq=">aaahf3ic3vxnbhmxehyldsx8tiujly1vpidaabobqekvknfb4kcrsfspu1re7ysxyq9xtrdtau17wbfehbviyphn4awyza5c07s9gqswvvt5zj7p+bvljllbjfx9hwul164v1w4s36zfun3n7srq2r09ozlnomuuupogogyet6brurvwkgqgmhkwh41efv79i9cgq+sdhacqsjpiej8zath03gwtjzygoo/mh6sbfsufdg8etcuwqaqxe7i29doifcskjjyjakyv7ac2dfrbzgtk9sazkfi2ogpoxuc8nqmvyej3mkmbew30x15fafws602sz0mosmpgmsjise3qnpcvxot8vcz2n4eoj2lmiwflon4mpku8qgyv5hqyfwmelgmo5xpssdvlfswayvksbzusjq/pzjb0bfopgtpgewzi1u0sh51wpgikhmmqx84maqqmjblhqm5kvwksi6kedtrakrvjb/2pvx4jxaqdejhmskqaxi9d8lqpny8ipcidcwxwnlrdt6eefel6hleb236rhbp6o9hwib4a7ibw9lhnhzwjvwzn03/kbw15tf836jsgg+af8jw8mcjilnnnqmxhiopnyjpbycvnxgofvwa7k1rknonhe1wgiwirjjdht/dfcqbtskzqcuzcxz0l6srr+zyq2hijov4piaeusxw9ys//nlyxnvlu9q+f0v9j0zosuodqger4owqqbu2t0svfesztuhdjrxgvf7ytjy2exe1d/3lazidgadg8sim4uicvceobhuiz0t7/kmydvu6rjt1++2rj5031ycytb+qhazi2euz2ycuys7qeeu0+ke/kc+1j7uvta+1bgbq4uhhuk5lr+/4lktzp9a==</latexit> V<latexit sha1_base64="rjx2viy0zv0wbbosgmdnfnknehm=">aaahg3ic3vxnbhmxehyldsx8pxdk4ljfcqindimqskhsjs5ihcgs/zgyq8rrnsrw7pxk9qznrx0tomklcencofacvadeh4wmaxsdcsyt9vpmfj7xn5ydpzxp43k/lpavxv9p3fi92bx1+87de621+/tazorchpvcqsoiaoasgt3ddifdvaereyedapyq8b9mqgkmk/dmmkioydbha0ajcaajvssg5si24hnked8/am14xa8cebh4ndha9dg9wlv5gcsszgisqznruu97qqktuyzrdnkzydskhi7jeprxhku6iqj0ae/kxdluo3+mb1k5lzg4tj4lwsk0norirqpirvq8rzbe5otnzvaitcxjmwmjrrinmo6nxiuuogykqoftbwhvzjwl6ygoqo0tbc5lsbarkuu8ozdbkdeokuvokuko8hm3nwx8wpskyfmkijpapsip6k5gmjp1yv1ymdymajxuaeluy9/6m2894m4pf5tamzvckcr+yom3a9f9kfld3npau8pto9kpnimisv9twmhb6/qhbbbbbqzxenxuykwo+2bekhb3epvey7y1htveb9crqm+bl46h3afxgsyya8b5dnfgnotpickat5lyfcydv5caxz0trycuwh0i6ysxr8/2x3fm07cq6grmql29c+rkm/mckk7xwun4p4scus5v9io9/3paxnrknbz//zt+t5qekdqgra0fc7ddifnqxehvxkthziw4e8xow/vno1hrymnvnoc/t4vmcawpmhdjhhcxsorbzxoh7pnwzz8ki2c/1/vdj9893dh5wz8yq+gheoq6yefp0q56jxbrhqjogj6gt+hz42pjs+nr41svurxucx6gudh4/gtsildh</latexit> One-hot encoding of user One-hot encoding of product

15 <latexit sha1_base64="20r+kfvojzrfz60s1mdkdrhzzoq=">aaahf3ic3vxnbhmxehyldsx8tiujly1vpidaabobqekvknfb4kcrsfspu1re7ysxyq9xtrdtau17wbfehbviyphn4awyza5c07s9gqswvvt5zj7p+bvljllbjfx9hwul164v1w4s36zfun3n7srq2r09ozlnomuuupogogyet6brurvwkgqgmhkwh41efv79i9cgq+sdhacqsjpiej8zath03gwtjzygoo/mh6sbfsufdg8etcuwqaqxe7i29doifcskjjyjakyv7ac2dfrbzgtk9sazkfi2ogpoxuc8nqmvyej3mkmbew30x15fafws602sz0mosmpgmsjise3qnpcvxot8vcz2n4eoj2lmiwflon4mpku8qgyv5hqyfwmelgmo5xpssdvlfswayvksbzusjq/pzjb0bfopgtpgewzi1u0sh51wpgikhmmqx84maqqmjblhqm5kvwksi6kedtrakrvjb/2pvx4jxaqdejhmskqaxi9d8lqpny8ipcidcwxwnlrdt6eefel6hleb236rhbp6o9hwib4a7ibw9lhnhzwjvwzn03/kbw15tf836jsgg+af8jw8mcjilnnnqmxhiopnyjpbycvnxgofvwa7k1rknonhe1wgiwirjjdht/dfcqbtskzqcuzcxz0l6srr+zyq2hijov4piaeusxw9ys//nlyxnvlu9q+f0v9j0zosuodqger4owqqbu2t0svfesztuhdjrxgvf7ytjy2exe1d/3lazidgadg8sim4uicvceobhuiz0t7/kmydvu6rjt1++2rj5031ycytb+qhazi2euz2ycuys7qeeu0+ke/kc+1j7uvta+1bgbq4uhhuk5lr+/4lktzp9a==</latexit> <latexit sha1_base64="20r+kfvojzrfz60s1mdkdrhzzoq=">aaahf3ic3vxnbhmxehyldsx8tiujly1vpidaabobqekvknfb4kcrsfspu1re7ysxyq9xtrdtau17wbfehbviyphn4awyza5c07s9gqswvvt5zj7p+bvljllbjfx9hwul164v1w4s36zfun3n7srq2r09ozlnomuuupogogyet6brurvwkgqgmhkwh41efv79i9cgq+sdhacqsjpiej8zath03gwtjzygoo/mh6sbfsufdg8etcuwqaqxe7i29doifcskjjyjakyv7ac2dfrbzgtk9sazkfi2ogpoxuc8nqmvyej3mkmbew30x15fafws602sz0mosmpgmsjise3qnpcvxot8vcz2n4eoj2lmiwflon4mpku8qgyv5hqyfwmelgmo5xpssdvlfswayvksbzusjq/pzjb0bfopgtpgewzi1u0sh51wpgikhmmqx84maqqmjblhqm5kvwksi6kedtrakrvjb/2pvx4jxaqdejhmskqaxi9d8lqpny8ipcidcwxwnlrdt6eefel6hleb236rhbp6o9hwib4a7ibw9lhnhzwjvwzn03/kbw15tf836jsgg+af8jw8mcjilnnnqmxhiopnyjpbycvnxgofvwa7k1rknonhe1wgiwirjjdht/dfcqbtskzqcuzcxz0l6srr+zyq2hijov4piaeusxw9ys//nlyxnvlu9q+f0v9j0zosuodqger4owqqbu2t0svfesztuhdjrxgvf7ytjy2exe1d/3lazidgadg8sim4uicvceobhuiz0t7/kmydvu6rjt1++2rj5031ycytb+qhazi2euz2ycuys7qeeu0+ke/kc+1j7uvta+1bgbq4uhhuk5lr+/4lktzp9a==</latexit> <latexit sha1_base64="20r+kfvojzrfz60s1mdkdrhzzoq=">aaahf3ic3vxnbhmxehyldsx8tiujly1vpidaabobqekvknfb4kcrsfspu1re7ysxyq9xtrdtau17wbfehbviyphn4awyza5c07s9gqswvvt5zj7p+bvljllbjfx9hwul164v1w4s36zfun3n7srq2r09ozlnomuuupogogyet6brurvwkgqgmhkwh41efv79i9cgq+sdhacqsjpiej8zath03gwtjzygoo/mh6sbfsufdg8etcuwqaqxe7i29doifcskjjyjakyv7ac2dfrbzgtk9sazkfi2ogpoxuc8nqmvyej3mkmbew30x15fafws602sz0mosmpgmsjise3qnpcvxot8vcz2n4eoj2lmiwflon4mpku8qgyv5hqyfwmelgmo5xpssdvlfswayvksbzusjq/pzjb0bfopgtpgewzi1u0sh51wpgikhmmqx84maqqmjblhqm5kvwksi6kedtrakrvjb/2pvx4jxaqdejhmskqaxi9d8lqpny8ipcidcwxwnlrdt6eefel6hleb236rhbp6o9hwib4a7ibw9lhnhzwjvwzn03/kbw15tf836jsgg+af8jw8mcjilnnnqmxhiopnyjpbycvnxgofvwa7k1rknonhe1wgiwirjjdht/dfcqbtskzqcuzcxz0l6srr+zyq2hijov4piaeusxw9ys//nlyxnvlu9q+f0v9j0zosuodqger4owqqbu2t0svfesztuhdjrxgvf7ytjy2exe1d/3lazidgadg8sim4uicvceobhuiz0t7/kmydvu6rjt1++2rj5031ycytb+qhazi2euz2ycuys7qeeu0+ke/kc+1j7uvta+1bgbq4uhhuk5lr+/4lktzp9a==</latexit> <latexit sha1_base64="rjx2viy0zv0wbbosgmdnfnknehm=">aaahg3ic3vxnbhmxehyldsx8pxdk4ljfcqindimqskhsjs5ihcgs/zgyq8rrnsrw7pxk9qznrx0tomklcencofacvadeh4wmaxsdcsyt9vpmfj7xn5ydpzxp43k/lpavxv9p3fi92bx1+87de621+/tazorchpvcqsoiaoasgt3ddifdvaereyedapyq8b9mqgkmk/dmmkioydbha0ajcaajvssg5si24hnked8/am14xa8cebh4ndha9dg9wlv5gcsszgisqznruu97qqktuyzrdnkzydskhi7jeprxhku6iqj0ae/kxdluo3+mb1k5lzg4tj4lwsk0norirqpirvq8rzbe5otnzvaitcxjmwmjrrinmo6nxiuuogykqoftbwhvzjwl6ygoqo0tbc5lsbarkuu8ozdbkdeokuvokuko8hm3nwx8wpskyfmkijpapsip6k5gmjp1yv1ymdymajxuaeluy9/6m2894m4pf5tamzvckcr+yom3a9f9kfld3npau8pto9kpnimisv9twmhb6/qhbbbbbqzxenxuykwo+2bekhb3epvey7y1htveb9crqm+bl46h3afxgsyya8b5dnfgnotpickat5lyfcydv5caxz0trycuwh0i6ysxr8/2x3fm07cq6grmql29c+rkm/mckk7xwun4p4scus5v9io9/3paxnrknbz//zt+t5qekdqgra0fc7ddifnqxehvxkthziw4e8xow/vno1hrymnvnoc/t4vmcawpmhdjhhcxsorbzxoh7pnwzz8ki2c/1/vdj9893dh5wz8yq+gheoq6yefp0q56jxbrhqjogj6gt+hz42pjs+nr41svurxucx6gudh4/gtsildh</latexit> <latexit sha1_base64="rjx2viy0zv0wbbosgmdnfnknehm=">aaahg3ic3vxnbhmxehyldsx8pxdk4ljfcqindimqskhsjs5ihcgs/zgyq8rrnsrw7pxk9qznrx0tomklcencofacvadeh4wmaxsdcsyt9vpmfj7xn5ydpzxp43k/lpavxv9p3fi92bx1+87de621+/tazorchpvcqsoiaoasgt3ddifdvaereyedapyq8b9mqgkmk/dmmkioydbha0ajcaajvssg5si24hnked8/am14xa8cebh4ndha9dg9wlv5gcsszgisqznruu97qqktuyzrdnkzydskhi7jeprxhku6iqj0ae/kxdluo3+mb1k5lzg4tj4lwsk0norirqpirvq8rzbe5otnzvaitcxjmwmjrrinmo6nxiuuogykqoftbwhvzjwl6ygoqo0tbc5lsbarkuu8ozdbkdeokuvokuko8hm3nwx8wpskyfmkijpapsip6k5gmjp1yv1ymdymajxuaeluy9/6m2894m4pf5tamzvckcr+yom3a9f9kfld3npau8pto9kpnimisv9twmhb6/qhbbbbbqzxenxuykwo+2bekhb3epvey7y1htveb9crqm+bl46h3afxgsyya8b5dnfgnotpickat5lyfcydv5caxz0trycuwh0i6ysxr8/2x3fm07cq6grmql29c+rkm/mckk7xwun4p4scus5v9io9/3paxnrknbz//zt+t5qekdqgra0fc7ddifnqxehvxkthziw4e8xow/vno1hrymnvnoc/t4vmcawpmhdjhhcxsorbzxoh7pnwzz8ki2c/1/vdj9893dh5wz8yq+gheoq6yefp0q56jxbrhqjogj6gt+hz42pjs+nr41svurxucx6gudh4/gtsildh</latexit> <latexit sha1_base64="rjx2viy0zv0wbbosgmdnfnknehm=">aaahg3ic3vxnbhmxehyldsx8pxdk4ljfcqindimqskhsjs5ihcgs/zgyq8rrnsrw7pxk9qznrx0tomklcencofacvadeh4wmaxsdcsyt9vpmfj7xn5ydpzxp43k/lpavxv9p3fi92bx1+87de621+/tazorchpvcqsoiaoasgt3ddifdvaereyedapyq8b9mqgkmk/dmmkioydbha0ajcaajvssg5si24hnked8/am14xa8cebh4ndha9dg9wlv5gcsszgisqznruu97qqktuyzrdnkzydskhi7jeprxhku6iqj0ae/kxdluo3+mb1k5lzg4tj4lwsk0norirqpirvq8rzbe5otnzvaitcxjmwmjrrinmo6nxiuuogykqoftbwhvzjwl6ygoqo0tbc5lsbarkuu8ozdbkdeokuvokuko8hm3nwx8wpskyfmkijpapsip6k5gmjp1yv1ymdymajxuaeluy9/6m2894m4pf5tamzvckcr+yom3a9f9kfld3npau8pto9kpnimisv9twmhb6/qhbbbbbqzxenxuykwo+2bekhb3epvey7y1htveb9crqm+bl46h3afxgsyya8b5dnfgnotpickat5lyfcydv5caxz0trycuwh0i6ysxr8/2x3fm07cq6grmql29c+rkm/mckk7xwun4p4scus5v9io9/3paxnrknbz//zt+t5qekdqgra0fc7ddifnqxehvxkthziw4e8xow/vno1hrymnvnoc/t4vmcawpmhdjhhcxsorbzxoh7pnwzz8ki2c/1/vdj9893dh5wz8yq+gheoq6yefp0q56jxbrhqjogj6gt+hz42pjs+nr41svurxucx6gudh4/gtsildh</latexit> Word2vec Embedding as Neural Network Neural network representation of word2vec: Linear activation Linear activation u 1 v 1 U<latexit sha1_base64="20r+kfvojzrfz60s1mdkdrhzzoq=">aaahf3ic3vxnbhmxehyldsx8tiujly1vpidaabobqekvknfb4kcrsfspu1re7ysxyq9xtrdtau17wbfehbviyphn4awyza5c07s9gqswvvt5zj7p+bvljllbjfx9hwul164v1w4s36zfun3n7srq2r09ozlnomuuupogogyet6brurvwkgqgmhkwh41efv79i9cgq+sdhacqsjpiej8zath03gwtjzygoo/mh6sbfsufdg8etcuwqaqxe7i29doifcskjjyjakyv7ac2dfrbzgtk9sazkfi2ogpoxuc8nqmvyej3mkmbew30x15fafws602sz0mosmpgmsjise3qnpcvxot8vcz2n4eoj2lmiwflon4mpku8qgyv5hqyfwmelgmo5xpssdvlfswayvksbzusjq/pzjb0bfopgtpgewzi1u0sh51wpgikhmmqx84maqqmjblhqm5kvwksi6kedtrakrvjb/2pvx4jxaqdejhmskqaxi9d8lqpny8ipcidcwxwnlrdt6eefel6hleb236rhbp6o9hwib4a7ibw9lhnhzwjvwzn03/kbw15tf836jsgg+af8jw8mcjilnnnqmxhiopnyjpbycvnxgofvwa7k1rknonhe1wgiwirjjdht/dfcqbtskzqcuzcxz0l6srr+zyq2hijov4piaeusxw9ys//nlyxnvlu9q+f0v9j0zosuodqger4owqqbu2t0svfesztuhdjrxgvf7ytjy2exe1d/3lazidgadg8sim4uicvceobhuiz0t7/kmydvu6rjt1++2rj5031ycytb+qhazi2euz2ycuys7qeeu0+ke/kc+1j7uvta+1bgbq4uhhuk5lr+/4lktzp9a==</latexit> V<latexit sha1_base64="rjx2viy0zv0wbbosgmdnfnknehm=">aaahg3ic3vxnbhmxehyldsx8pxdk4ljfcqindimqskhsjs5ihcgs/zgyq8rrnsrw7pxk9qznrx0tomklcencofacvadeh4wmaxsdcsyt9vpmfj7xn5ydpzxp43k/lpavxv9p3fi92bx1+87de621+/tazorchpvcqsoiaoasgt3ddifdvaereyedapyq8b9mqgkmk/dmmkioydbha0ajcaajvssg5si24hnked8/am14xa8cebh4ndha9dg9wlv5gcsszgisqznruu97qqktuyzrdnkzydskhi7jeprxhku6iqj0ae/kxdluo3+mb1k5lzg4tj4lwsk0norirqpirvq8rzbe5otnzvaitcxjmwmjrrinmo6nxiuuogykqoftbwhvzjwl6ygoqo0tbc5lsbarkuu8ozdbkdeokuvokuko8hm3nwx8wpskyfmkijpapsip6k5gmjp1yv1ymdymajxuaeluy9/6m2894m4pf5tamzvckcr+yom3a9f9kfld3npau8pto9kpnimisv9twmhb6/qhbbbbbqzxenxuykwo+2bekhb3epvey7y1htveb9crqm+bl46h3afxgsyya8b5dnfgnotpickat5lyfcydv5caxz0trycuwh0i6ysxr8/2x3fm07cq6grmql29c+rkm/mckk7xwun4p4scus5v9io9/3paxnrknbz//zt+t5qekdqgra0fc7ddifnqxehvxkthziw4e8xow/vno1hrymnvnoc/t4vmcawpmhdjhhcxsorbzxoh7pnwzz8ki2c/1/vdj9893dh5wz8yq+gheoq6yefp0q56jxbrhqjogj6gt+hz42pjs+nr41svurxucx6gudh4/gtsildh</latexit> One-hot encoding of output word One-hot encoding of input word

Deep Learning Basics Lecture 10: Neural Language Models. Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 10: Neural Language Models Princeton University COS 495 Instructor: Yingyu Liang Natural language Processing (NLP) The processing of the human languages by computers One of