Learning to Process Natural Language in Big Data Environment
|
|
|
- Christine Edwards
- 5 years ago
- Views:
Transcription
1 CCF ADL 2015 Nanchang Oc 11, 2015 Learning o Process Naural Language in Big Daa Environmen Hang Li Noah s Ark Lab Huawei Technologies
2 Par 2: Useful Deep Learning Tools
3 Powerful Deep Learning Tools (Unsupervised Neural Word Embedding Recurren Neural Neworks Convoluional Neural Neworks Recursive Neural Neworks
4 Neural Word Embedding
5 Neural Word Embedding Moivaion Represening words wih lower-dimensional (100~ real-valued vecors Unsupervised seing As inpu o neural neworks Tool: Word2Vec Mehod: SGNS (Skip-Gram wih Negaive Sampling
6 Skip-Gram wih Negaive Sampling (Mikolov e al., 2013 Inpu: occurrences beween words and conexs M w 1 c1 c2 c3 c4 c w 2 w Probabiliy model: 1 P( D 1 w, c ( wc 1 e 1 P( D 0 w, c ( wc 1 e wc wc
7 Skip-Gram wih Negaive Sampling Negaive sampling: randomly sample unobserved pair E c w,c N [log ( w c N~ P N ] Objecive in learning L w c #( w, clog ( wc k EC ~ P log ( wcn N Algorihm: sochasic gradien decen
8 Inerpreaion as Marix Facorizaion (Levy & Goldberg 2014 Poinwise Muual Informaion Marix M c1 c2 c3 c4 c5 w 1 w 2 w log P( w, c P( w P( c
9 Inerpreaion as Marix Facorizaion w 1 w 2 w 3 M c1 c2 c3 c4 c w 1 W M WC T marix facorizaion, equivalen o SGNS w 2 w Word Embedding
10 Word Represenaion: Neural Word Embedding w 1 M (Mikolov e al., 2013 c1 c2 c3 c4 c w 2 w log P( w, c P( w P( c W T M WC marix facorizaion w w 2 w word embedding or word2vec
11 Recurren Neural Nework
12 Recurren Neural Nework (RNN (Mikolov e al h 1 h he ca sa on he ma x h 1 h f ( h 1, x he ca sa. ma x
13 Simple Recurren Neural Nework h f ( h, x ( Whh 1 1 Wx x b h h1 x +1
14 Long Term Shor Memory (LSTM ( h 1, x (Hochreier & Schmidhuber, 1997 g h, x, x ( 1 i inpu gae forge gae m f ( h 1, x c oupu gae ( h 1 o oupu gae h Have a memory (vecor o memorize previous values Use inpu gae, oupu gae, forge gae Gae: elemen-wise produc wih vecor wih values in [0,1] i f o c g h ( W ( W ( W 1 anh( W i o ih fh oh g h h h 1 1 gh W h f 1 anh( c ix W W x fx ox c x x 1 b W gx i b b x f o b g
15 Gaed Recurren Uni (GRU h 1 ( x, h 1 r rese gae (Cho e al., 2014 Have a memory (vecor o memorize previous values Use rese gae, updae gae x h 1 g updae gae z h r z h g ( W z rh ( W zh h h 1 h anh( W 1 1 gh W ( r rx W zx x 1 (1 z g x h b r b z W gx x b g ( x, h 1
16 Recurren Neural Nework Language Model Model h p anh( W h P( x h x 1 1 x W x 1 x b hx sof max( Wh b Objecive of Learning 1 T log ˆ T 1 p x 1 x h 1 h x 1 x
17 Recurren Neural Nework (RNN (Mikolov e al On sequence of words Variable lengh Long dependency: LSTM or GRU he ca sa on he ma h 1 h f ( h 1, x he ca sa. ma x
18 Convoluional Neural Nework
19 Convoluional Neural Nework (CNN (Hu e al., 2014 Concaenaion he ca sa on he ma he ca sa on sa on he ma he ca sa on he ma max pooling he ca ca sa ca sa sa on sa on on he on he he ma he ca sa ca sa on sa on he on he ma convoluion he ca sa on he ma
20 Example: Image Convoluion Filer Dark pixel value = 1, ligh pixel value = Filer Leow Wee Kheng
21 Example: Image Convoluion Feaure Map
22 Convoluion z z z z z ( l, f i ( l, f i w ( l, f ( l1 i (0 i (0 i ( w, b [ x ( l, f is sigmoid T i, x ( l, f is oupu of T i1 z, x ( l1 i ype f funcion T ih1 b ] T ( l, f are parameers of is inpu from cancaenaed f 1,2,, F for locaion i in layer l neuron of is inpu for locaion i from layer l 1 l ype f in layer l word vecors for locaion i w ( l, f z i ( l, f ( l1 z i b ( l, f Filer feaure map neuron +1 convoluion
23 Max Pooling z z z ( l, f i ( l, f i is ( l1, f 2i1 max( z, z ( l1, f 2i ( l1, f 2i1 oupu of, z ( l1, f 2i pooling of are inpu of ype f pooling of for locaion i in layer l ype f for locaion i in layer l max pooling
24 Senence Classificaion Using Convoluional Neural Nework y f ( x sof max( Wz z CNN( x b (L z y concaenaion max pooling convoluion x
25 Convoluional Neural Nework (CNN (Hu e al. 2014, Blunsom e al Concaenaion he ca sa on he ma he ca sa on sa on he ma Robus parsing Shared parameer on same level Fixed lengh, zero padding he ca sa on he ma max pooling he ca ca sa sa on on he ca sa sa on on he he ma he ca sa ca sa on sa on he on he ma convoluion he ca sa on he ma
26 Recursive Neural Nework
27 Recursive Neural Nework (Socher e al., 2013 he ca sa on he ma he ca sa on he ma
28 Recursive Neural Nework p score U p c1 p anhw c2 b c1 c2
29 Learning of Recursive Neural Nework The score of a ree is he sum of he scores of is nodes. Max margin parsing (, ( ( n s y x s y nodes n rees : greedily searched ( on incorrec ree :penaly, (, (, ( max, ( ( x Z z y z y z x s y x s L x Z z
30 Recursive Neural Nework (RNN (Socher e al On parse ree of senence Learning is based on max margin parsing he ca sa on he ma he ca sa on he ma
31 Learning of Senence Represenaion
32 Represenaion of Word Meaning dog ca puppy kien Using high-dimensional real-valued vecors o represen he meaning of words
33 Represenaion of Senence Meaning New finding: This is possible Mary is loved by John Mary loves John John loves Mary Using high-dimensional real-valued vecors o represen he meaning of senences
34 Recen Breakhrough in Disribuional Linguisics From words o senences Composiional Represening synax, semanics, even pragmaics
35 How Is Learning of Senence Meaning Possible? Deep neural neworks (complicaed non-linear models Big Daa Task-oriened Error-driven and gradien-based
36 Naural Language Tasks Classificaion: assigning a label o a sring s c Generaion: creaing a sring Maching: maching wo srings s, s R Translaion: ransforming one sring o anoher s Srucured predicion: mapping sring o srucure s s'
37 Naural Language Applicaions Can Be Formalized as Tasks Classificaion Senimen analysis Generaion Language modeling Maching Search Quesion answering Translaion Machine ranslaion Naural language dialogue (single urn Tex summarizaion Paraphrasing Srucured Predicion Informaion Exracion Parsing
38 Learning of Represenaions Classificaion in Tasks s Generaion Maching s, Translaion s r r s(r r c Srucured Predicion s s' r R
39 Our Observaion Unsupervised word-embedding (e.g., Word2Vec is needed, only when here is no enough daa for supervised word-embedding Convoluional Neural Nework is suiable for maching asks Recurren Neural Nework is suiable for generaion asks No observed so far ha Recursive Neural Nework works beer han he oher wo models
40 References Tomas Mikolov, Marin Karafiá, Lukas Burge, Jan Cernocký, and Sanjeev Khudanpur. Recurren Neural Nework based Language Model. InerSpeech Omer Levy, Yoav Goldberg, and Ido Dagan. Improving Disribuional Similariy wih Lessons Learned from Word Embeddings. TACL 2015 pp Tomas Mikolov, Ilya Suskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Disribued Represenaions of Words and Phrases and Their Composiionaliy. NIPS 2013, pp Hochreier, S., & Schmidhuber, J. Long Shor-Term Memory. Neural Compuaion, 9(8, , Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014. Learning Phrase Represenaions using RNN Encoder- Decoder for Saisical Machine Translaion. arxiv: Hu, B., Lu, Z., Li, H., & Chen, Q. Convoluional Neural Nework Archiecures for Maching Naural Language Senences. NIPS 2014 (pp Blunsom, P., Grefensee, E., & Kalchbrenner, N. (2014. A Convoluional neural nework for modeling senences. ACL Socher, Richard, John Bauer, Chrisopher D. Manning, and Andrew Y. Ng. "Parsing wih composiional vecor grammars." ACL 2013.
41 Thank you!
Deep Learning: Theory, Techniques & Applications - Recurrent Neural Networks -
Deep Learning: Theory, Techniques & Applicaions - Recurren Neural Neworks - Prof. Maeo Maeucci maeo.maeucci@polimi.i Deparmen of Elecronics, Informaion and Bioengineering Arificial Inelligence and Roboics
Natural Language Processing and Deep Learning
Lecure Series: Dialogue beween Brain Science and Arificial Inelligence Tsinghua Universi Nov. 30, 206 Naural Language Processing and Deep Learning Hang Li Huawei Noah s Ark Lab This Lecure Overview of
Deep Multi-Task Learning with Shared Memory
Deep Muli-Task Learning wih Shared Memory Pengfei Liu Xipeng Qiu Xuanjing Huang Shanghai Key Laboraory of Inelligen Informaion Processing, Fudan Universiy School of Compuer Science, Fudan Universiy 825
A Generalized Recurrent Neural Architecture for Text Classification with Multi-Task Learning
Proceedings of he Tweny-Sixh Inernaional Join Conference on Arificial Inelligence (IJCAI-17) A Generalized Recurren Neural Archiecure for Tex Classificaion wih Muli-Task Learning Honglun Zhang 1, Liqiang
CS224n: Natural Language Processing with Deep Learning 1 Lecture Notes: Part V Language Models, RNN, GRU and LSTM 2 Winter 2019
CS224n: Naural Language Processing wih Deep Learning 1 Lecure Noes: Par V Language Models, RNN, GRU and LSTM 2 Winer 2019 1 Course Insrucors: Chrisopher Manning, Richard Socher 2 Auhors: Milad Mohammadi,
Shortcut Sequence Tagging
Shorcu Sequence Tagging Huijia Wu 1,3, Jiajun Zhang 1,2, and Chengqing Zong 1,2,3 1 Naional Laboraory of Paern Recogniion, Insiue of Auomaion, CAS 2 CAS Cener for Excellence in Brain Science and Inelligence
a) b) (Natural Language Processing; NLP) (Deep Learning) Bag of words White House RGB [1] IBM
![a) b) (Natural Language Processing; NLP) (Deep Learning) Bag of words White House RGB [1] IBM](/thumbs/92/107948882.jpg "a) b) (Natural Language Processing; NLP) (Deep Learning) Bag of words White House RGB [1] IBM")
c 1. (Natural Language Processing; NLP) (Deep Learning) RGB IBM 135 8511 5 6 52 yutat@jp.ibm.com a) b) 2. 1 0 2 1 Bag of words White House 2 [1] 2015 4 Copyright c by ORSJ. Unauthorized reproduction of
Simplified Gating in Long Short-term Memory (LSTM) Recurrent Neural Networks
Simplified Gaing in Long Shor-erm Memory (LSTM) Recurren Neural Neworks Yuzhen Lu and Fahi M. Salem Circuis, Sysems, and Neural Neworks (CSANN) Lab Deparmen of Biosysems and Agriculural Engineering Deparmen
CHAPTER 10 VALIDATION OF TEST WITH ARTIFICAL NEURAL NETWORK
175 CHAPTER 10 VALIDATION OF TEST WITH ARTIFICAL NEURAL NETWORK 10.1 INTRODUCTION Amongs he research work performed, he bes resuls of experimenal work are validaed wih Arificial Neural Nework. From he
Tensorial Recurrent Neural Networks for Longitudinal Data Analysis
1 Tensorial Recurren Neural Neworks for Longiudinal Daa Analysis Mingyuan Bai Boyan Zhang and Junbin Gao arxiv:1708.00185v1 [cs.lg] 1 Aug 2017 Absrac Tradiional Recurren Neural Neworks assume vecorized
Zürich. ETH Master Course: L Autonomous Mobile Robots Localization II
Roland Siegwar Margaria Chli Paul Furgale Marco Huer Marin Rufli Davide Scaramuzza ETH Maser Course: 151-0854-00L Auonomous Mobile Robos Localizaion II ACT and SEE For all do, (predicion updae / ACT),
Ensamble methods: Boosting
Lecure 21 Ensamble mehods: Boosing Milos Hauskrech milos@cs.pi.edu 5329 Senno Square Schedule Final exam: April 18: 1:00-2:15pm, in-class Term projecs April 23 & April 25: a 1:00-2:30pm in CS seminar room
Ensamble methods: Bagging and Boosting
Lecure 21 Ensamble mehods: Bagging and Boosing Milos Hauskrech milos@cs.pi.edu 5329 Senno Square Ensemble mehods Mixure of expers Muliple base models (classifiers, regressors), each covers a differen par
Deep learning for Natural Language Processing and Machine Translation
Deep learning for Natural Language Processing and Machine Translation 2015.10.16 Seung-Hoon Na Contents Introduction: Neural network, deep learning Deep learning for Natural language processing Neural
Neural Networks for NLP. COMP-599 Nov 30, 2016
Neural Networks for NLP COMP-599 Nov 30, 2016 Outline Neural networks and deep learning: introduction Feedforward neural networks word2vec Complex neural network architectures Convolutional neural networks
Collaborative Recurrent Autoencoder: Recommend while Learning to Fill in the Blanks
Collaboraive Recurren Auoencoder: Recommend while Learning o Fill in he Blanks Hao Wang, Xingjian Shi, Di-Yan Yeung Hong Kong Universiy of Science and Technology {hwangaz,xshiab,dyyeung}@cse.us.hk Absrac
An Empirical Study on Energy Disaggregation via Deep Learning
Advances in Inelligen Sysems Research, volume 133 2nd Inernaional Conference on Arificial Inelligence and Indusrial Engineering (AIIE2016) An Empirical Sudy on Energy Disaggregaion via Deep Learning Wan
Speaker Adaptation Techniques For Continuous Speech Using Medium and Small Adaptation Data Sets. Constantinos Boulis
Speaker Adapaion Techniques For Coninuous Speech Using Medium and Small Adapaion Daa Ses Consaninos Boulis Ouline of he Presenaion Inroducion o he speaker adapaion problem Maximum Likelihood Sochasic Transformaions
Dimitri Solomatine. D.P. Solomatine. Data-driven modelling (part 2). 2
Daa-driven modelling. Par. Daa-driven Arificial di Neural modelling. Newors Par Dimiri Solomaine Arificial neural newors D.P. Solomaine. Daa-driven modelling par. 1 Arificial neural newors ANN: main pes
Réseaux de neurones récurrents Handwriting Recognition with Long Short-Term Memory Networks
Réseaux de neurones récurrens Handwriing Recogniion wih Long Shor-Term Memory Neworks Dr. Marcus Eichenberger-Liwicki DFKI, Germany Marcus.Liwicki@dfki.de Handwriing Recogniion (Sae of he Ar) Transform
Hidden Markov Models
Hidden Markov Models Probabilisic reasoning over ime So far, we ve mosly deal wih episodic environmens Excepions: games wih muliple moves, planning In paricular, he Bayesian neworks we ve seen so far describe
Application of a Stochastic-Fuzzy Approach to Modeling Optimal Discrete Time Dynamical Systems by Using Large Scale Data Processing
Applicaion of a Sochasic-Fuzzy Approach o Modeling Opimal Discree Time Dynamical Sysems by Using Large Scale Daa Processing AA WALASZE-BABISZEWSA Deparmen of Compuer Engineering Opole Universiy of Technology
An recursive analytical technique to estimate time dependent physical parameters in the presence of noise processes
WHAT IS A KALMAN FILTER An recursive analyical echnique o esimae ime dependen physical parameers in he presence of noise processes Example of a ime and frequency applicaion: Offse beween wo clocks PREDICTORS,
Pattern Classification (VI) 杜俊
Paern lassificaion VI 杜俊 jundu@usc.edu.cn Ouline Bayesian Decision Theory How o make he oimal decision? Maximum a oserior MAP decision rule Generaive Models Join disribuion of observaion and label sequences
arxiv: v1 [cs.cl] 21 Nov 2017
![arxiv: v1 [cs.cl] 21 Nov 2017](/thumbs/76/73575719.jpg "arxiv: v1 [cs.cl] 21 Nov 2017")
Cross Temporal Recurren Neworks for Ranking Quesion Answer Pairs Yi Tay 1, Luu Anh Tuan 2 and Siu Cheung Hui 3 1, 3 Nanyang Technological Universiy School of Compuer Science and Engineering, Singapore
Improved Learning through Augmenting the Loss
Improved Learning through Augmenting the Loss Hakan Inan inanh@stanford.edu Khashayar Khosravi khosravi@stanford.edu Abstract We present two improvements to the well-known Recurrent Neural Network Language
Latent Spaces and Matrix Factorization
Compuaional Linguisics Laen Spaces and Marix Facorizaion Dierich Klakow FR 4.7 Allgemeine Linguisik (Compuerlinguisik) Universiä des Saarlandes Summer 0 Goal Goal: rea documen clusering and word clusering
Fine-grained Opinion Mining with Recurrent Neural Networks and Word Embeddings
Fine-grained Opinion Mining wih Recurren Neural Neworks and Word Embeddings Pengfei Liu 1, Shafiq Joy 2 and Helen Meng 1 1 Deparmen of Sysems Engineering and Engineering Managemen, The Chinese Universiy
CS230: Lecture 8 Word2Vec applications + Recurrent Neural Networks with Attention
CS23: Lecture 8 Word2Vec applications + Recurrent Neural Networks with Attention Today s outline We will learn how to: I. Word Vector Representation i. Training - Generalize results with word vectors -
Power Market Price Forecasting via Deep Learning
Power Marke Price Forecasing via Deep Learning Yongli Zhu, Renchang Dai, Guangyi Liu, Zhiwei Wang GEIRI Norh America Graph Compuing and Grid Modernizaion Deparmen San Jose, California yzhu16@vols.uk.edu
Introduction to RNNs!
Introduction to RNNs Arun Mallya Best viewed with Computer Modern fonts installed Outline Why Recurrent Neural Networks (RNNs)? The Vanilla RNN unit The RNN forward pass Backpropagation refresher The RNN
Latent Spaces and Matrix Factorization
Compuaional Linguisics Laen Spaces and Marix Facorizaion Sefan Thaer & Dierich Klakow FR 4.7 Allgemeine Linguisik (Compuerlinguisik) Universiä des Saarlandes Summer 2013 Goal Goal: rea documen clusering
Orientation. Connections between network coding and stochastic network theory. Outline. Bruce Hajek. Multicast with lost packets
Connecions beween nework coding and sochasic nework heory Bruce Hajek Orienaion On Thursday, Ralf Koeer discussed nework coding: coding wihin he nework Absrac: Randomly generaed coded informaion blocks
ST2352. Stochastic Processes constructed via Conditional Simulation. 09/02/2014 ST2352 Week 4 1
ST35 Sochasic Processes consruced via Condiional Simulaion 09/0/014 ST35 Week 4 1 Sochasic Processes consruced via Condiional Simulaion Markov Processes Simulaing Random Tex Google Sugges n grams Random
Kriging Models Predicting Atrazine Concentrations in Surface Water Draining Agricultural Watersheds
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Kriging Models Predicing Arazine Concenraions in Surface Waer Draining Agriculural Waersheds Paul L. Mosquin, Jeremy Aldworh, Wenlin Chen Supplemenal Maerial Number
An EM based training algorithm for recurrent neural networks
An EM based raining algorihm for recurren neural neworks Jan Unkelbach, Sun Yi, and Jürgen Schmidhuber IDSIA,Galleria 2, 6928 Manno, Swizerland {jan.unkelbach,yi,juergen}@idsia.ch hp://www.idsia.ch Absrac.
Lecture 10 - Model Identification
Lecure - odel Idenificaion Wha is ssem idenificaion? Direc impulse response idenificaion Linear regression Regularizaion Parameric model ID nonlinear LS Conrol Engineering - Wha is Ssem Idenificaion? Experimen
High Order LSTM/GRU. Wenjie Luo. January 19, 2016
High Order LSTM/GRU Wenjie Luo January 19, 2016 1 Introduction RNN is a powerful model for sequence data but suffers from gradient vanishing and explosion, thus difficult to be trained to capture long
Retrieval Models. Boolean and Vector Space Retrieval Models. Common Preprocessing Steps. Boolean Model. Boolean Retrieval Model
1 Boolean and Vecor Space Rerieval Models Many slides in his secion are adaped from Prof. Joydeep Ghosh (UT ECE) who in urn adaped hem from Prof. Dik Lee (Univ. of Science and Tech, Hong Kong) Rerieval
Y. Xiang, Learning Bayesian Networks 1
Learning Bayesian Neworks Objecives Acquisiion of BNs Technical conex of BN learning Crierion of sound srucure learning BN srucure learning in 2 seps BN CPT esimaion Reference R.E. Neapolian: Learning
Natural Language Processing (CSEP 517): Machine Translation (Continued), Summarization, & Finale
Natural Language Processing (CSEP 517): Machine Translation (Continued), Summarization, & Finale Noah Smith c 2017 University of Washington nasmith@cs.washington.edu May 22, 2017 1 / 30 To-Do List Online
Solutions to the Exam Digital Communications I given on the 11th of June = 111 and g 2. c 2
Soluions o he Exam Digial Communicaions I given on he 11h of June 2007 Quesion 1 (14p) a) (2p) If X and Y are independen Gaussian variables, hen E [ XY ]=0 always. (Answer wih RUE or FALSE) ANSWER: False.
Non-parametric techniques. Instance Based Learning. NN Decision Boundaries. Nearest Neighbor Algorithm. Distance metric important
on-parameric echniques Insance Based Learning AKA: neares neighbor mehods, non-parameric, lazy, memorybased, or case-based learning Copyrigh 2005 by David Helmbold 1 Do no fi a model (as do LTU, decision
Non-parametric techniques. Instance Based Learning. NN Decision Boundaries. Nearest Neighbor Algorithm. Distance metric important
on-parameric echniques Insance Based Learning AKA: neares neighbor mehods, non-parameric, lazy, memorybased, or case-based learning Copyrigh 2005 by David Helmbold 1 Do no fi a model (as do LDA, logisic
ELE 538B: Large-Scale Optimization for Data Science. Introduction. Yuxin Chen Princeton University, Spring 2018
ELE 538B: Large-Scale Opimizaion for Daa Science Inroducion Yuxin Chen Princeon Universiy, Spring 2018 Surge of daa-inensive applicaions Widespread applicaions in large-scale daa science and learning 2.5
arxiv: v1 [stat.ml] 24 May 2016 Abstract
![arxiv: v1 [stat.ml] 24 May 2016 Abstract](/thumbs/72/67260081.jpg "arxiv: v1 [stat.ml] 24 May 2016 Abstract")
Sequenial Neural Models wih Sochasic Layers Marco Fraccaro Søren Kaae Sønderby Ulrich Paque Ole Winher Technical Universiy of Denmark Universiy of Copenhagen, Denmark arxiv:1605.07571v1 [sa.ml] 24 May
Article from. Predictive Analytics and Futurism. July 2016 Issue 13
Aricle from Predicive Analyics and Fuurism July 6 Issue An Inroducion o Incremenal Learning By Qiang Wu and Dave Snell Machine learning provides useful ools for predicive analyics The ypical machine learning
Probabilistic Robotics SLAM
Probabilisic Roboics SLAM The SLAM Problem SLAM is he process by which a robo builds a map of he environmen and, a he same ime, uses his map o compue is locaion Localizaion: inferring locaion given a map
Object tracking: Using HMMs to estimate the geographical location of fish
Objec racking: Using HMMs o esimae he geographical locaion of fish 02433 - Hidden Markov Models Marin Wæver Pedersen, Henrik Madsen Course week 13 MWP, compiled June 8, 2011 Objecive: Locae fish from agging
Combining Static and Dynamic Information for Clinical Event Prediction
Combining Static and Dynamic Information for Clinical Event Prediction Cristóbal Esteban 1, Antonio Artés 2, Yinchong Yang 1, Oliver Staeck 3, Enrique Baca-García 4 and Volker Tresp 1 1 Siemens AG and
The Rosenblatt s LMS algorithm for Perceptron (1958) is built around a linear neuron (a neuron with a linear
In The name of God Lecure4: Percepron and AALIE r. Majid MjidGhoshunih Inroducion The Rosenbla s LMS algorihm for Percepron 958 is buil around a linear neuron a neuron ih a linear acivaion funcion. Hoever,
Linear Gaussian State Space Models
Linear Gaussian Sae Space Models Srucural Time Series Models Level and Trend Models Basic Srucural Model (BSM Dynamic Linear Models Sae Space Model Represenaion Level, Trend, and Seasonal Models Time Varying
EE-559 Deep learning LSTM and GRU
EE-559 Deep learning 11.2. LSTM and GRU François Fleuret https://fleuret.org/ee559/ Mon Feb 18 13:33:24 UTC 2019 ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE The Long-Short Term Memory unit (LSTM) by Hochreiter
arxiv: v1 [eess.as] 6 Nov 2018
![arxiv: v1 [eess.as] 6 Nov 2018](/thumbs/89/101189899.jpg "arxiv: v1 [eess.as] 6 Nov 2018")
LANGUAGE MODEL INTEGRATION BASED ON MEMORY CONTROL FOR SEQUENCE TO SEQUENCE SPEECH RECOGNITION Jaejin Cho 1, Shinji Waanabe 1, Takaaki Hori 2, Murali Karhick Baskar 3, Hirofumi Inaguma 4, Jesus Villalba
Probabilistic Robotics The Sparse Extended Information Filter
Probabilisic Roboics The Sparse Exended Informaion Filer MSc course Arificial Inelligence 2018 hps://saff.fnwi.uva.nl/a.visser/educaion/probabilisicroboics/ Arnoud Visser Inelligen Roboics Lab Informaics
Robot Motion Model EKF based Localization EKF SLAM Graph SLAM
Robo Moion Model EKF based Localizaion EKF SLAM Graph SLAM General Robo Moion Model Robo sae v r Conrol a ime Sae updae model Noise model of robo conrol Noise model of conrol Robo moion model
Deep Convolutional Recurrent Network for Segmentation-free Offline Handwritten Japanese Text Recognition
2017 14h IAPR Inernaional Conference on Documen Analysis and Recogniion Deep Convoluional Recurren Nework for Segmenaion-free Offline Handwrien Japanese Tex Recogniion Nam-Tuan Ly Dep. of Compuer Science
References are appeared in the last slide. Last update: (1393/08/19)
SYSEM IDEIFICAIO Ali Karimpour Associae Professor Ferdowsi Universi of Mashhad References are appeared in he las slide. Las updae: 0..204 393/08/9 Lecure 5 lecure 5 Parameer Esimaion Mehods opics o be
Deep Learning and Lexical, Syntactic and Semantic Analysis. Wanxiang Che and Yue Zhang
Deep Learning and Lexical, Syntactic and Semantic Analysis Wanxiang Che and Yue Zhang 2016-10 Part 2: Introduction to Deep Learning Part 2.1: Deep Learning Background What is Machine Learning? From Data
INTRODUCTION TO MACHINE LEARNING 3RD EDITION
ETHEM ALPAYDIN The MIT Press, 2014 Lecure Slides for INTRODUCTION TO MACHINE LEARNING 3RD EDITION alpaydin@boun.edu.r hp://www.cmpe.boun.edu.r/~ehem/i2ml3e CHAPTER 2: SUPERVISED LEARNING Learning a Class
Institute for Mathematical Methods in Economics. University of Technology Vienna. Singapore, May Manfred Deistler
MULTIVARIATE TIME SERIES ANALYSIS AND FORECASTING Manfred Deisler E O S Economerics and Sysems Theory Insiue for Mahemaical Mehods in Economics Universiy of Technology Vienna Singapore, May 2004 Inroducion
Hidden Markov Models. Adapted from. Dr Catherine Sweeney-Reed s slides
Hidden Markov Models Adaped from Dr Caherine Sweeney-Reed s slides Summary Inroducion Descripion Cenral in HMM modelling Exensions Demonsraion Specificaion of an HMM Descripion N - number of saes Q = {q
CSC321 Lecture 10 Training RNNs
CSC321 Lecture 10 Training RNNs Roger Grosse and Nitish Srivastava February 23, 2015 Roger Grosse and Nitish Srivastava CSC321 Lecture 10 Training RNNs February 23, 2015 1 / 18 Overview Last time, we saw
Stability and Bifurcation in a Neural Network Model with Two Delays
Inernaional Mahemaical Forum, Vol. 6, 11, no. 35, 175-1731 Sabiliy and Bifurcaion in a Neural Nework Model wih Two Delays GuangPing Hu and XiaoLing Li School of Mahemaics and Physics, Nanjing Universiy
Learning a Class from Examples. Training set X. Class C 1. Class C of a family car. Output: Input representation: x 1 : price, x 2 : engine power
Alpaydin Chaper, Michell Chaper 7 Alpaydin slides are in urquoise. Ehem Alpaydin, copyrigh: The MIT Press, 010. alpaydin@boun.edu.r hp://www.cmpe.boun.edu.r/ ehem/imle All oher slides are based on Michell.
arxiv: v1 [cs.cv] 6 Dec 2017
![arxiv: v1 [cs.cv] 6 Dec 2017](/thumbs/78/78192488.jpg "arxiv: v1 [cs.cv] 6 Dec 2017")
Show-and-Fool: Crafing Adversarial Examples for Neural Image Capioning Hongge Chen 1, Huan Zhang 2, Pin-Yu Chen 3, Jinfeng Yi 4, Cho-Jui Hsieh 2 1 Massachuses Insiue of Technology, Cambridge, MA 02139
Hardware-Software Co-design of Slimmed Optical Neural Networks
Hardware-Sofware Co-design of Slimmed Opical Neural Neworks Zheng Zhao 1, Derong Liu 1, Meng Li 1, Zhoufeng Ying 1, Lu Zhang 2, Biying Xu 1, Bei Yu 2, Ray Chen 1, David Pan 1 The Universiy of Texas a Ausin
word2vec Parameter Learning Explained
word2vec Parameter Learning Explained Xin Rong ronxin@umich.edu Abstract The word2vec model and application by Mikolov et al. have attracted a great amount of attention in recent two years. The vector
Content-Based Shape Retrieval Using Different Shape Descriptors: A Comparative Study Dengsheng Zhang and Guojun Lu
Conen-Based Shape Rerieval Using Differen Shape Descripors: A Comparaive Sudy Dengsheng Zhang and Guojun Lu Gippsland School of Compuing and Informaion Technology Monash Universiy Churchill, Vicoria 3842
Image Processing, Image Recognition, Computer Vision, Image Understanding. Sound Processing, Speech Recognition, Auditory Scene Understanding
Image Processing, Image Recogniion, Compuer Vision, Image Undersanding Sound Processing, Speech Recogniion, Audiory Scene Undersanding Mahemaical Theory of Paern Recogniion Takashi Masuyama m@i.kyoo-u.ac.jp
State-Space Models. Initialization, Estimation and Smoothing of the Kalman Filter
Sae-Space Models Iniializaion, Esimaion and Smoohing of he Kalman Filer Iniializaion of he Kalman Filer The Kalman filer shows how o updae pas predicors and he corresponding predicion error variances when
Technical Report Doc ID: TR March-2013 (Last revision: 23-February-2016) On formulating quadratic functions in optimization models.
Technical Repor Doc ID: TR--203 06-March-203 (Las revision: 23-Februar-206) On formulaing quadraic funcions in opimizaion models. Auhor: Erling D. Andersen Convex quadraic consrains quie frequenl appear
Learning a Class from Examples. Training set X. Class C 1. Class C of a family car. Output: Input representation: x 1 : price, x 2 : engine power
Alpaydin Chaper, Michell Chaper 7 Alpaydin slides are in urquoise. Ehem Alpaydin, copyrigh: The MIT Press, 010. alpaydin@boun.edu.r hp://www.cmpe.boun.edu.r/ ehem/imle All oher slides are based on Michell.
arxiv: v1 [cs.sd] 21 Sep 2017
![arxiv: v1 [cs.sd] 21 Sep 2017](/thumbs/81/83627266.jpg "arxiv: v1 [cs.sd] 21 Sep 2017")
DEEP RECURRENT NMF FOR SPEECH SEPARATION BY UNFOLDING ITERATIVE THRESHOLDING Sco Wisdom, 1 Thomas Powers, 1 James Pion, 1,2 Les Alas, 1 1 Deparmen of Elecrical Engineering, Universiy of Washingon, Seale,
1 Review of Zero-Sum Games
COS 5: heoreical Machine Learning Lecurer: Rob Schapire Lecure #23 Scribe: Eugene Brevdo April 30, 2008 Review of Zero-Sum Games Las ime we inroduced a mahemaical model for wo player zero-sum games. Any
Elements of Computer Graphics
CS580: Compuer Graphics Min H. Kim KAIST School of Compuing Elemens of Compuer Graphics Geomery Maerial model Ligh Rendering Virual phoography 2 Foundaions of Compuer Graphics A PINHOLE CAMERA IN 3D 3
Probabilistic Robotics SLAM
Probabilisic Roboics SLAM The SLAM Problem SLAM is he process by which a robo builds a map of he environmen and, a he same ime, uses his map o compue is locaion Localizaion: inferring locaion given a map
Layer Trajectory LSTM
Layer Trajecory LSTM Jinyu Li, Changliang Liu, Yifan Gong Microsof AI and Research {jinyli, chanliu, ygong}@microsof.com Absrac I is popular o sack LSTM layers o ge beer modeling power, especially when
Collaborative Recurrent Neural Networks for Dynamic Recommender Systems
JMLR: Workshop and Conference Proceedings 63:366 381, 2016 ACML 2016 Collaboraive Recurren Neural Neworks for Dynamic Recommender Sysems Young-Jun Ko Lucas Maysre Mahias Grossglauser École Polyechnique
Removing Useless Productions of a Context Free Grammar through Petri Net
Journal of Compuer Science 3 (7): 494-498, 2007 ISSN 1549-3636 2007 Science Publicaions Removing Useless Producions of a Conex Free Grammar hrough Peri Ne Mansoor Al-A'ali and Ali A Khan Deparmen of Compuer
Sequential Neural Models with Stochastic Layers
Sequenial Neural Models wih Sochasic Layers Marco Fraccaro Søren Kaae Sønderby Ulrich Paque * Ole Winher Technical Universiy of Denmark Universiy of Copenhagen * Google DeepMind Absrac How can we efficienly
Globally Coherent Text Generation with Neural Checklist Models
Globally Coheren Te Generaion wih Neural Checklis Models Chloé Kiddon Luke Zelemoyer Yejin Choi Compuer Science & Engineering Universiy of Washingon {chloe, lsz, yejin}@cs.washingon.edu Absrac Recurren
GMM - Generalized Method of Moments
GMM - Generalized Mehod of Momens Conens GMM esimaion, shor inroducion 2 GMM inuiion: Maching momens 2 3 General overview of GMM esimaion. 3 3. Weighing marix...........................................
Learning to Jointly Translate and Predict Dropped Pronouns with a Shared Reconstruction Mechanism
Learning o Joinly Translae and Predic Dropped Pronouns wih a Shared Reconsrucion Mechanism Longyue Wang Tencen AI Lab vinnylywang@encen.com Andy Way Dublin Ciy Universiy andy.way@adapcenre.ie Absrac Pronouns
Implementation and Optimization of Differentiable Neural Computers
Implemenaion and Opimizaion of Differeniable Neural Compuers Carol Hsin Graduae Suden in Compuaional & Mahemaical Engineering Sanford Universiy cshsin[a]sanford.edu Absrac We implemened and opimized Differeniable
text classification 3: neural networks
text classification 3: neural networks CS 585, Fall 2018 Introduction to Natural Language Processing http://people.cs.umass.edu/~miyyer/cs585/ Mohit Iyyer College of Information and Computer Sciences University
Natural Language Processing (CSEP 517): Language Models, Continued
Natural Language Processing (CSEP 517): Language Models, Continued Noah Smith c 2017 University of Washington nasmith@cs.washington.edu April 3, 2017 1 / 102 To-Do List Online quiz: due Sunday Print, sign,
Bayesian Paragraph Vectors
Bayesian Paragraph Vectors Geng Ji 1, Robert Bamler 2, Erik B. Sudderth 1, and Stephan Mandt 2 1 Department of Computer Science, UC Irvine, {gji1, sudderth}@uci.edu 2 Disney Research, firstname.lastname@disneyresearch.com
arxiv: v1 [cs.cl] 22 Jun 2015
![arxiv: v1 [cs.cl] 22 Jun 2015](/thumbs/84/90648195.jpg "arxiv: v1 [cs.cl] 22 Jun 2015")
Neural Transformation Machine: A New Architecture for Sequence-to-Sequence Learning arxiv:1506.06442v1 [cs.cl] 22 Jun 2015 Fandong Meng 1 Zhengdong Lu 2 Zhaopeng Tu 2 Hang Li 2 and Qun Liu 1 1 Institute
Tasty Coffee example
Lecure Slides for (Binary) Classificaion: Learning a Class from labeled Examples ITRODUCTIO TO Machine Learning ETHEM ALPAYDI The MIT Press, 00 (modified by dph, 0000) CHAPTER : Supervised Learning Things
Augmented Reality II - Kalman Filters - Gudrun Klinker May 25, 2004
Augmened Realiy II Kalman Filers Gudrun Klinker May 25, 2004 Ouline Moivaion Discree Kalman Filer Modeled Process Compuing Model Parameers Algorihm Exended Kalman Filer Kalman Filer for Sensor Fusion Lieraure
CSE/NB 528 Lecture 14: From Supervised to Reinforcement Learning (Chapter 9) R. Rao, 528: Lecture 14
CSE/NB 58 Lecure 14: From Supervised o Reinforcemen Learning Chaper 9 1 Recall from las ime: Sigmoid Neworks Oupu v T g w u g wiui w Inpu nodes u = u 1 u u 3 T i Sigmoid oupu funcion: 1 g a 1 a e 1 ga
Notes on online convex optimization
Noes on online convex opimizaion Karl Sraos Online convex opimizaion (OCO) is a principled framework for online learning: OnlineConvexOpimizaion Inpu: convex se S, number of seps T For =, 2,..., T : Selec
Probabilistic Robotics
Probabilisic Roboics Bayes Filer Implemenaions Gaussian filers Bayes Filer Reminder Predicion bel p u bel d Correcion bel η p z bel Gaussians : ~ π e p N p - Univariae / / : ~ μ μ μ e p Ν p d π Mulivariae
Using the Kalman filter Extended Kalman filter
Using he Kalman filer Eended Kalman filer Doz. G. Bleser Prof. Sricker Compuer Vision: Objec and People Tracking SA- Ouline Recap: Kalman filer algorihm Using Kalman filers Eended Kalman filer algorihm
Georey E. Hinton. University oftoronto. Technical Report CRG-TR February 22, Abstract
Parameer Esimaion for Linear Dynamical Sysems Zoubin Ghahramani Georey E. Hinon Deparmen of Compuer Science Universiy oftorono 6 King's College Road Torono, Canada M5S A4 Email: zoubin@cs.orono.edu Technical
GeoMAN: Multi-level Attention Networks for Geo-sensory Time Series Prediction
GeoMAN: Muli-level Aenion Neworks for Geo-sensory Time Series Predicion Yuxuan Liang 1,2, Songyu Ke 3,2, Junbo Zhang 2,4, Xiuwen Yi 4,2, Yu Zheng 2,1,3,4 1 School of Compuer Science and Technology, Xidian
YNU-HPCC at SemEval-2018 Task 2: Multi-ensemble Bi-GRU Model with Attention Mechanism for Multilingual Emoji Prediction
YNU-HPCC a SemEval-2018 Task 2: Muli-ensemble Bi- Model wih Aenion Mechanism for Mulilingual Emoji Predicion Nan Wang, Jin Wang and Xuejie Zhang School of Informaion Science and Engineering Yunnan Universiy
LEARNING HARD ALIGNMENTS WITH VARIATIONAL INFERENCE. Dieterich Lawson, Chung-Cheng Chiu, George Tucker, Colin Raffel, Kevin Swersky, Navdeep Jaitly
LEARNING HARD ALIGNMENTS WITH VARIATIONAL INFERENCE Dieerich Lawson, Chung-Cheng Chiu, George Tucker, Colin Raffel, Kevin Swersky, Navdeep Jaily Google Brain {dieerichl,chungchengc,gj,craffel,kswersky,ndjaily}@google.com
WATER LEVEL TRACKING WITH CONDENSATION ALGORITHM
WATER LEVEL TRACKING WITH CONDENSATION ALGORITHM Shinsuke KOBAYASHI, Shogo MURAMATSU, Hisakazu KIKUCHI, Masahiro IWAHASHI Dep. of Elecrical and Elecronic Eng., Niigaa Universiy, 8050 2-no-cho Igarashi,