Neural Networks. Adapted from slides by Tim Finin and Marie desjardins. Some material adapted from lecture notes by Lise Getoor and Ron Parr
|
|
- Emil Hutchinson
- 5 years ago
- Views:
Transcription
1 Neural Networks Adapted from sldes by Tm Fnn and Mare desjardns. Some materal adapted from lecture notes by Lse Getoor and Ron Parr
2 Neural func1on Bran func1on (thought) occurs as the result of the frng of neurons Neurons connect to each other through synapses, whch propagate ac+on poten+al (electrcal mpulses) by releasng neurotransm1ers Synapses can be exctatory (poten1al- ncreasng) or nhbtory (poten1al- decreasng), and have varyng ac+va+on thresholds Learnng occurs as a result of the synapses plas+ccty: They exhbt long- term changes n connec1on strength There are about neurons and about synapses n the human bran! 2
3 Bology of a neuron 3
4 Bran structure Dfferent areas of the bran have dfferent func1ons Some areas seem to have the same func1on n all humans (e.g., Broca s regon for motor speech); the overall layout s generally consstent Some areas are more plas1c, and vary n ther func1on; also, the lower- level structure and func1on vary greatly We don t know how dfferent func1ons are assgned or acqured Partly the result of the physcal layout / connec1on to nputs (sensors) and outputs (effectors) Partly the result of experence (learnng) We really don t understand how ths neural structure leads to what we perceve as conscousness or thought Ar1fcal neural networks are not nearly as complex or ntrcate as the actual bran structure 4
5 Comparson of compu1ng power INFORMATION CIRCA 2012 Computer Human Bran Computaton Unts 10-core Xeon: 10 9 Gates Neurons Storage Unts 10 9 bts RAM, bts dsk neurons, synapses Cycle tme 10-9 sec 10-3 sec Bandwdth 10 9 bts/sec bts/sec Computers are way faster than neurons But there are a lot more neurons than we can reasonably model n modern dgtal computers, and they all fre n parallel Neural networks are desgned to be massvely parallel The bran s effec1vely a bllon 1mes faster 5
6 Neural networks Output unts Hdden unts Layered feed-forward network Neural networks are made up of nodes or unts, connected by lnks Each lnk has an assocated weght and ac+va+on level Each node has an nput func+on (typcally summng over weghted nputs), an ac+va+on func+on, and an output 6 Input unts
7 Model of a neuron Neuron modeled as a unt weghts on nput unt to, w net nput to unt s: n = " Ac1va1on func1on g() determnes the neuron s output g() s typcally a sgmod output s ether 0 or 1 (no par1al ac1va1on) 7 w! o
8 Execu1ng neural networks Input unts are set by some exteror func1on (thnk of these as sensors), whch causes ther output lnks to be ac+vated at the specfed level Workng forward through the network, the nput func+on of each unt s appled to compute the nput value Usually ths s ust the weghted sum of the ac1va1on on the lnks feedng nto ths node The ac+va+on func+on transforms ths nput func1on nto a fnal value Typcally ths s a nonlnear func1on, o_en a sgmod func1on correspondng to the threshold of that node 8
9 Learnng rules Rosenblab (1959) suggested that f a target output value s provded for a sngle neuron wth fxed nputs, can ncrementally change weghts to learn to produce these outputs usng the perceptron learnng rule assumes bnary valued nput/outputs assumes a sngle lnear threshold unt 9
10 Perceptron learnng rule If the target output for unt s t w = w + η( t o ) o Equvalent to the ntu1ve rules: If output s correct, don t change the weghts If output s low (o =0, t =1), ncrement weghts for all the nputs whch are 1 If output s hgh (o =1, t =0), decrement weghts for all nputs whch are 1 Must also adust threshold. Or equvalently assume there s a weght w 0 for an extra nput unt that has an output of 1. 10
11 Perceptron learnng algorthm Repeatedly terate through examples adus1ng weghts accordng to the perceptron learnng rule un1l all outputs are correct In1alze the weghts to all zero (or random) Un1l outputs for all tranng examples are correct for each tranng example e do compute the current output o compare t to the target t and update weghts Each execu1on of outer loop s called an epoch For mul1ple category problems, learn a separate perceptron for each category and assgn to the class whose perceptron most exceeds ts threshold 11
12 Representa1on lmta1ons of a perceptron Perceptrons can only represent lnear threshold func1ons and can therefore only learn func1ons whch lnearly separate the data..e., the pos1ve and nega1ve examples are separable by a hyperplane n n- dmensonal space <W,X> - θ = 0 > 0 on ths sde < 0 on ths sde 12
13 Perceptron learnablty Perceptron Convergence Theorem: If there s a set of weghts that s consstent wth the tranng data (.e., the data s lnearly separable), the perceptron learnng algorthm wll converge (Mncksy & Papert, 1969) Unfortunately, many func1ons (lke party) cannot be represented by LTU 13
14 Learnng: Backpropaga1on Smlar to perceptron learnng algorthm, we cycle through our examples f the output of the network s correct, no changes are made f there s an error, the weghts are adusted to reduce the error The trck s to assess the blame for the error and dvde t among the contrbu1ng weghts 14
15 Output layer As n perceptron learnng algorthm, we want to mnmze dfference between target output and the output actually computed W = W + α a Err g ʹ (n ) Δ W = = Err actvaton of hdden unt (T O ) dervatve of actvaton functon W g ʹ (n ) + α a 15 Δ
16 Hdden layers Need to defne error; we do error backpropaga1on. Intu1on: Each hdden node s responsble for some frac1on of the error Δ I n each of the output nodes to whch t connects. Δ I dvded accordng to the strength of the connec1on between hdden node and the output node and propagated back to provde the Δ values for the hdden layer: update rule: Δ W k = = gʹ (n W k 16 ) + α W I k Δ Δ
17 Backproga1on algorthm Compute the Δ values for the output unts usng the observed error Star1ng wth output layer, repeat the followng for each layer n the network, un1l earlest hdden layer s reached: propagate the Δ values back to the prevous layer update the weghts between the two layers 17
18 Backprop ssues Backprop s the cockroach of machne learnng. It s ugly, and annoyng, but you ust can t get rd of t. - Geoff Hnton Problems: black box local mnma 18
19 Restrcted Boltzmann Machnes and Deep Belef Networks Sldes from: Geoffrey Hnton, Sue Becker, Yann Le Cun, Yoshua Bengo, Frank Wood
20 Mo1va1ons Supervsed tranng of deep models (e.g. many- layered NNets) s dffcult (op1mza1on problem) Shallow models (SVMs, one- hdden- layer NNets, boos1ng, etc ) are unlkely canddates for learnng hgh- level abstrac1ons needed for AI Unsupervsed learnng could do local- learnng (each module tres ts best to model what t sees) Inference (+ learnng) s ntractable n drected graphcal models wth many hdden varables Current unsupervsed learnng methods don t easly extend to learn mul1ple levels of representa1on
21 Belef Nets A belef net s a drected acyclc graph composed of stochas1c varables. Can observe some of the varables and we would lke to solve two problems: The nference problem: Infer the states of the unobserved varables. The learnng problem: Adust the nterac1ons between varables to make the network more lkely to generate the observed data. stochas1c hdden cause vsble effect Use nets composed of layers of stochas1c bnary varables wth weghted connec1ons. Later, we wll generalze to other types of varable.
22 Stochas1c bnary neurons These have a state of 1 or 0 whch s a stochas1c func1on of the neuron s bas, b, and the nput t receves from other neurons. p( s = 1 1 ) = 1+ exp( b 1 s w ) p( = 1) s b + s 0 w
23 Stochas1c unts Replace the bnary threshold unts by bnary stochas1c unts that make based random decsons. The temperature controls the amount of nose. Decreasng all the energy gaps between confgura1ons s equvalent to rasng the nose level. ) ( ) ( ) ( = = Δ = = Δ = + = + = T E T w s s E s E E gap Energy e e s p temperature
24 The Energy of a ont confgura1on bnary state of unt n ont confgura1on v, h E( v, h) = vh s unts b < s vh s vh w Energy wth confgura1on v on the vsble unts and h on the hdden unts bas of unt weght between unts and ndexes every non- den1cal par of and once
25 Weghts à Energes à Probabl1es Each possble ont confgura1on of the vsble and hdden unts has an energy The energy s determned by the weghts and bases (as n a Hopfeld net). The energy of a ont confgura1on of the vsble and hdden unts determnes ts probablty: p( v, h) E( v, h) e The probablty of a confgura1on over the vsble unts s found by summng the probabl1es of all the ont confgura1ons that contan t.
26 Restrcted Boltzmann Machnes Restrct the connec1vty to make learnng easer. Only one layer of hdden unts. Deal wth more layers later No connec1ons between hdden unts. In an RBM, the hdden unts are cond1onally ndependent gven the vsble states. So can quckly get an unbased sample from the posteror dstrbu1on when gven a data- vector. Ths s a bg advantage over drected belef nets hdden vsble
27 Restrcted Boltzmann Machnes In an RBM, the hdden unts are cond1onally ndependent gven the vsble states. It only takes one step to reach thermal equlbrum when the vsble unts are clamped. Can quckly get the exact value of : < s s > v hdden vsble
28 A pcture of the Boltzmann machne learnng algorthm for an RBM 0 < s s > 1 < s s > < s s > a fantasy t = 0 t = 1 t = 2 t = nfnty Start wth a tranng vector on the vsble unts. Then alternate between upda1ng all the hdden unts n parallel and upda1ng all the vsble unts n parallel. Δw = ε ( < s s > 0 < ss > )
29 Contras1ve dvergence learnng: A quck way to learn an RBM 0 < s s > 1 < s s > t = 0 t = 1 data reconstruc1on Start wth a tranng vector on the vsble unts. Update all the hdden unts n parallel Update the all the vsble unts n parallel to get a reconstruc1on. Update the hdden unts agan. Δw = ε ( < s s > 0 < s s > 1 ) Ths s not followng the gradent of the log lkelhood. But t works well. When we consder nfnte drected nets t wll be easy to see why t works.
30 How to learn a set of features that are good for reconstruc1ng mages of the dgt 2 50 bnary feature neurons Increment weghts between an ac1ve pxel and an ac1ve feature 50 bnary feature neurons Decrement weghts between an ac1ve pxel and an ac1ve feature 16 x 16 pxel mage data (realty) 16 x 16 pxel mage reconstruc1on (beber than realty)
31 Usng an RBM to learn a model of a dgt class Reconstruc1ons by model traned on 2 s Data Reconstruc1ons by model traned on 3 s 0 < s s > 1 < s s > 100 hdden unts (features) data reconstruc1on 256 vsble unts (pxels)
32 The fnal 50 x 256 weghts Each neuron grabs a dfferent feature.
33 How well can we reconstruct the dgt mages from the bnary feature ac1va1ons? Data Reconstruc1on from ac1vated bnary features Data Reconstruc1on from ac1vated bnary features New test mages from the dgt class that the model was traned on Images from an unfamlar dgt class (the network tres to see every mage as a 2)
34 Deep Belef Networks
35 Dvde and conquer mul1layer learnng Re- represen1ng the data: Each 1me the base learner s called, t passes a transformed verson of the data to the next learner. Can we learn a deep, dense DAG one layer at a 1me, star1ng at the bobom, and s1ll guarantee that learnng each layer mproves the overall model of the tranng data? Ths seems very unlkely. Surely we need to know the weghts n hgher layers to learn lower layers?
36 Mul1layer contras1ve dvergence Start by learnng one hdden layer. Then re- present the data as the ac1v1es of the hdden unts. The same learnng algorthm can now be appled to the re- presented data. Can we prove that each step of ths greedy learnng mproves the log probablty of the data under the overall model? What s the overall model?
37 Learnng a deep drected network etc. W T Frst learn wth all the weghts 1ed Ths s exactly equvalent to learnng an RBM Contras1ve dvergence learnng s equvalent to gnorng the small derva1ves contrbuted by the 1ed weghts between deeper layers. h0 v0 W h2 v2 h1 v1 h0 v0 W W W W W T T
38 Then freeze the frst layer of weghts n both drec1ons and learn the remanng weghts (s1ll 1ed together). Ths s equvalent to learnng another RBM, usng the aggregated posteror dstrbu1on of h0 as the data. v1 h0 W T W frozen etc. h2 v2 h1 v1 h0 v0 W W W W W T T T W frozen
39 A smplfed verson wth all hdden layers the same sze The RBM at the top can be vewed as shorthand for an nfnte drected net. When learnng W1 we can vew the model n two qute dfferent ways: The model s an RBM composed of the data layer and h1. The model s an nfnte DAG wth 1ed weghts. A_er learnng W1 we un1e t from the other weght matrces. We then learn W2 whch s s1ll 1ed to all the matrces above t. T W 2 T W 1 h3 h2 h1 data W 3 W 2 W 1
40 A neural network model of dgt recogn1on The top two layers form a restrcted Boltzmann machne whose free energy landscape models the low dmensonal manfolds of the dgts. The valleys have names: 10 label unts 2000 top- level unts 500 unts The model learns a ont densty for labels and mages. To perform recogn1on we can start wth a neutral state of the label unts and do one or two tera1ons of the top- level RBM. Or we can ust compute the free energy of the RBM wth each of the 10 labels 500 unts 28 x 28 pxel mage
41 Show the move of the network genera1ng dgts (avalable at
Hopfield networks and Boltzmann machines. Geoffrey Hinton et al. Presented by Tambet Matiisen
Hopfeld networks and Boltzmann machnes Geoffrey Hnton et al. Presented by Tambet Matsen 18.11.2014 Hopfeld network Bnary unts Symmetrcal connectons http://www.nnwj.de/hopfeld-net.html Energy functon The
More informationFor now, let us focus on a specific model of neurons. These are simplified from reality but can achieve remarkable results.
Neural Networks : Dervaton compled by Alvn Wan from Professor Jtendra Malk s lecture Ths type of computaton s called deep learnng and s the most popular method for many problems, such as computer vson
More informationCSC321 Tutorial 9: Review of Boltzmann machines and simulated annealing
CSC321 Tutoral 9: Revew of Boltzmann machnes and smulated annealng (Sldes based on Lecture 16-18 and selected readngs) Yue L Emal: yuel@cs.toronto.edu Wed 11-12 March 19 Fr 10-11 March 21 Outlne Boltzmann
More informationWeek 5: Neural Networks
Week 5: Neural Networks Instructor: Sergey Levne Neural Networks Summary In the prevous lecture, we saw how we can construct neural networks by extendng logstc regresson. Neural networks consst of multple
More informationMultilayer Perceptrons and Backpropagation. Perceptrons. Recap: Perceptrons. Informatics 1 CG: Lecture 6. Mirella Lapata
Multlayer Perceptrons and Informatcs CG: Lecture 6 Mrella Lapata School of Informatcs Unversty of Ednburgh mlap@nf.ed.ac.uk Readng: Kevn Gurney s Introducton to Neural Networks, Chapters 5 6.5 January,
More informationMultilayer Perceptron (MLP)
Multlayer Perceptron (MLP) Seungjn Cho Department of Computer Scence and Engneerng Pohang Unversty of Scence and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjn@postech.ac.kr 1 / 20 Outlne
More informationNeural Networks. Perceptrons and Backpropagation. Silke Bussen-Heyen. 5th of Novemeber Universität Bremen Fachbereich 3. Neural Networks 1 / 17
Neural Networks Perceptrons and Backpropagaton Slke Bussen-Heyen Unverstät Bremen Fachberech 3 5th of Novemeber 2012 Neural Networks 1 / 17 Contents 1 Introducton 2 Unts 3 Network structure 4 Snglelayer
More informationAdmin NEURAL NETWORKS. Perceptron learning algorithm. Our Nervous System 10/25/16. Assignment 7. Class 11/22. Schedule for the rest of the semester
0/25/6 Admn Assgnment 7 Class /22 Schedule for the rest of the semester NEURAL NETWORKS Davd Kauchak CS58 Fall 206 Perceptron learnng algorthm Our Nervous System repeat untl convergence (or for some #
More informationModel of Neurons. CS 416 Artificial Intelligence. Early History of Neural Nets. Cybernetics. McCulloch-Pitts Neurons. Hebbian Modification.
Page 1 Model of Neurons CS 416 Artfcal Intellgence Lecture 18 Neural Nets Chapter 20 Multple nputs/dendrtes (~10,000!!!) Cell body/soma performs computaton Sngle output/axon Computaton s typcally modeled
More informationEEE 241: Linear Systems
EEE : Lnear Systems Summary #: Backpropagaton BACKPROPAGATION The perceptron rule as well as the Wdrow Hoff learnng were desgned to tran sngle layer networks. They suffer from the same dsadvantage: they
More informationUnsupervised Learning
Unsupervsed Learnng Kevn Swngler What s Unsupervsed Learnng? Most smply, t can be thought of as learnng to recognse and recall thngs Recognton I ve seen that before Recall I ve seen that before and I can
More informationUsing deep belief network modelling to characterize differences in brain morphometry in schizophrenia
Usng deep belef network modellng to characterze dfferences n bran morphometry n schzophrena Walter H. L. Pnaya * a ; Ary Gadelha b ; Orla M. Doyle c ; Crstano Noto b ; André Zugman d ; Qurno Cordero b,
More informationKernel Methods and SVMs Extension
Kernel Methods and SVMs Extenson The purpose of ths document s to revew materal covered n Machne Learnng 1 Supervsed Learnng regardng support vector machnes (SVMs). Ths document also provdes a general
More informationEnsemble Methods: Boosting
Ensemble Methods: Boostng Ncholas Ruozz Unversty of Texas at Dallas Based on the sldes of Vbhav Gogate and Rob Schapre Last Tme Varance reducton va baggng Generate new tranng data sets by samplng wth replacement
More information10-701/ Machine Learning, Fall 2005 Homework 3
10-701/15-781 Machne Learnng, Fall 2005 Homework 3 Out: 10/20/05 Due: begnnng of the class 11/01/05 Instructons Contact questons-10701@autonlaborg for queston Problem 1 Regresson and Cross-valdaton [40
More informationOnline Classification: Perceptron and Winnow
E0 370 Statstcal Learnng Theory Lecture 18 Nov 8, 011 Onlne Classfcaton: Perceptron and Wnnow Lecturer: Shvan Agarwal Scrbe: Shvan Agarwal 1 Introducton In ths lecture we wll start to study the onlne learnng
More informationInternet Engineering. Jacek Mazurkiewicz, PhD Softcomputing. Part 3: Recurrent Artificial Neural Networks Self-Organising Artificial Neural Networks
Internet Engneerng Jacek Mazurkewcz, PhD Softcomputng Part 3: Recurrent Artfcal Neural Networks Self-Organsng Artfcal Neural Networks Recurrent Artfcal Neural Networks Feedback sgnals between neurons Dynamc
More informationLecture 23: Artificial neural networks
Lecture 23: Artfcal neural networks Broad feld that has developed over the past 20 to 30 years Confluence of statstcal mechancs, appled math, bology and computers Orgnal motvaton: mathematcal modelng of
More informationNonlinear Classifiers II
Nonlnear Classfers II Nonlnear Classfers: Introducton Classfers Supervsed Classfers Lnear Classfers Perceptron Least Squares Methods Lnear Support Vector Machne Nonlnear Classfers Part I: Mult Layer Neural
More informationOther NN Models. Reinforcement learning (RL) Probabilistic neural networks
Other NN Models Renforcement learnng (RL) Probablstc neural networks Support vector machne (SVM) Renforcement learnng g( (RL) Basc deas: Supervsed dlearnng: (delta rule, BP) Samples (x, f(x)) to learn
More informationKernels in Support Vector Machines. Based on lectures of Martin Law, University of Michigan
Kernels n Support Vector Machnes Based on lectures of Martn Law, Unversty of Mchgan Non Lnear separable problems AND OR NOT() The XOR problem cannot be solved wth a perceptron. XOR Per Lug Martell - Systems
More informationINF 5860 Machine learning for image classification. Lecture 3 : Image classification and regression part II Anne Solberg January 31, 2018
INF 5860 Machne learnng for mage classfcaton Lecture 3 : Image classfcaton and regresson part II Anne Solberg January 3, 08 Today s topcs Multclass logstc regresson and softma Regularzaton Image classfcaton
More informationLecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
More informationProblem Set 9 Solutions
Desgn and Analyss of Algorthms May 4, 2015 Massachusetts Insttute of Technology 6.046J/18.410J Profs. Erk Demane, Srn Devadas, and Nancy Lynch Problem Set 9 Solutons Problem Set 9 Solutons Ths problem
More information1 Convex Optimization
Convex Optmzaton We wll consder convex optmzaton problems. Namely, mnmzaton problems where the objectve s convex (we assume no constrants for now). Such problems often arse n machne learnng. For example,
More informationGeneralized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
More informationCHALMERS, GÖTEBORGS UNIVERSITET. SOLUTIONS to RE-EXAM for ARTIFICIAL NEURAL NETWORKS. COURSE CODES: FFR 135, FIM 720 GU, PhD
CHALMERS, GÖTEBORGS UNIVERSITET SOLUTIONS to RE-EXAM for ARTIFICIAL NEURAL NETWORKS COURSE CODES: FFR 35, FIM 72 GU, PhD Tme: Place: Teachers: Allowed materal: Not allowed: January 2, 28, at 8 3 2 3 SB
More informationSupporting Information
Supportng Informaton The neural network f n Eq. 1 s gven by: f x l = ReLU W atom x l + b atom, 2 where ReLU s the element-wse rectfed lnear unt, 21.e., ReLUx = max0, x, W atom R d d s the weght matrx to
More informationMultilayer neural networks
Lecture Multlayer neural networks Mlos Hauskrecht mlos@cs.ptt.edu 5329 Sennott Square Mdterm exam Mdterm Monday, March 2, 205 In-class (75 mnutes) closed book materal covered by February 25, 205 Multlayer
More informationIntroduction to the Introduction to Artificial Neural Network
Introducton to the Introducton to Artfcal Neural Netork Vuong Le th Hao Tang s sldes Part of the content of the sldes are from the Internet (possbly th modfcatons). The lecturer does not clam any onershp
More informationModule 3 LOSSY IMAGE COMPRESSION SYSTEMS. Version 2 ECE IIT, Kharagpur
Module 3 LOSSY IMAGE COMPRESSION SYSTEMS Verson ECE IIT, Kharagpur Lesson 6 Theory of Quantzaton Verson ECE IIT, Kharagpur Instructonal Objectves At the end of ths lesson, the students should be able to:
More informationLecture 10 Support Vector Machines II
Lecture 10 Support Vector Machnes II 22 February 2016 Taylor B. Arnold Yale Statstcs STAT 365/665 1/28 Notes: Problem 3 s posted and due ths upcomng Frday There was an early bug n the fake-test data; fxed
More informationMultigradient for Neural Networks for Equalizers 1
Multgradent for Neural Netorks for Equalzers 1 Chulhee ee, Jnook Go and Heeyoung Km Department of Electrcal and Electronc Engneerng Yonse Unversty 134 Shnchon-Dong, Seodaemun-Ku, Seoul 1-749, Korea ABSTRACT
More informationLimited Dependent Variables
Lmted Dependent Varables. What f the left-hand sde varable s not a contnuous thng spread from mnus nfnty to plus nfnty? That s, gven a model = f (, β, ε, where a. s bounded below at zero, such as wages
More informationMATH 567: Mathematical Techniques in Data Science Lab 8
1/14 MATH 567: Mathematcal Technques n Data Scence Lab 8 Domnque Gullot Departments of Mathematcal Scences Unversty of Delaware Aprl 11, 2017 Recall We have: a (2) 1 = f(w (1) 11 x 1 + W (1) 12 x 2 + W
More informationBoostrapaggregating (Bagging)
Boostrapaggregatng (Baggng) An ensemble meta-algorthm desgned to mprove the stablty and accuracy of machne learnng algorthms Can be used n both regresson and classfcaton Reduces varance and helps to avod
More informationArtificial Intelligence Bayesian Networks
Artfcal Intellgence Bayesan Networks Adapted from sldes by Tm Fnn and Mare desjardns. Some materal borrowed from Lse Getoor. 1 Outlne Bayesan networks Network structure Condtonal probablty tables Condtonal
More informationPattern Classification
Pattern Classfcaton All materals n these sldes ere taken from Pattern Classfcaton (nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wley & Sons, 000 th the permsson of the authors and the publsher
More informationMulti-layer neural networks
Lecture 0 Mult-layer neural networks Mlos Hauskrecht mlos@cs.ptt.edu 5329 Sennott Square Lnear regresson w Lnear unts f () Logstc regresson T T = w = p( y =, w) = g( w ) w z f () = p ( y = ) w d w d Gradent
More informationSDMML HT MSc Problem Sheet 4
SDMML HT 06 - MSc Problem Sheet 4. The recever operatng characterstc ROC curve plots the senstvty aganst the specfcty of a bnary classfer as the threshold for dscrmnaton s vared. Let the data space be
More informationFeature Selection: Part 1
CSE 546: Machne Learnng Lecture 5 Feature Selecton: Part 1 Instructor: Sham Kakade 1 Regresson n the hgh dmensonal settng How do we learn when the number of features d s greater than the sample sze n?
More informationMLE and Bayesian Estimation. Jie Tang Department of Computer Science & Technology Tsinghua University 2012
MLE and Bayesan Estmaton Je Tang Department of Computer Scence & Technology Tsnghua Unversty 01 1 Lnear Regresson? As the frst step, we need to decde how we re gong to represent the functon f. One example:
More information1 Input-Output Mappings. 2 Hebbian Failure. 3 Delta Rule Success.
Task Learnng 1 / 27 1 Input-Output Mappngs. 2 Hebban Falure. 3 Delta Rule Success. Input-Output Mappngs 2 / 27 0 1 2 3 4 5 6 7 8 9 Output 3 8 2 7 Input 5 6 0 9 1 4 Make approprate: Response gven stmulus.
More informationWhich Separator? Spring 1
Whch Separator? 6.034 - Sprng 1 Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng 3 Margn of a pont " # y (w $ + b) proportonal
More informationCSE 546 Midterm Exam, Fall 2014(with Solution)
CSE 546 Mdterm Exam, Fall 014(wth Soluton) 1. Personal nfo: Name: UW NetID: Student ID:. There should be 14 numbered pages n ths exam (ncludng ths cover sheet). 3. You can use any materal you brought:
More informationCS 3710: Visual Recognition Classification and Detection. Adriana Kovashka Department of Computer Science January 13, 2015
CS 3710: Vsual Recognton Classfcaton and Detecton Adrana Kovashka Department of Computer Scence January 13, 2015 Plan for Today Vsual recognton bascs part 2: Classfcaton and detecton Adrana s research
More informationP R. Lecture 4. Theory and Applications of Pattern Recognition. Dept. of Electrical and Computer Engineering /
Theory and Applcatons of Pattern Recognton 003, Rob Polkar, Rowan Unversty, Glassboro, NJ Lecture 4 Bayes Classfcaton Rule Dept. of Electrcal and Computer Engneerng 0909.40.0 / 0909.504.04 Theory & Applcatons
More informationNatural Images, Gaussian Mixtures and Dead Leaves Supplementary Material
Natural Images, Gaussan Mxtures and Dead Leaves Supplementary Materal Danel Zoran Interdscplnary Center for Neural Computaton Hebrew Unversty of Jerusalem Israel http://www.cs.huj.ac.l/ danez Yar Wess
More informationThe Gaussian classifier. Nuno Vasconcelos ECE Department, UCSD
he Gaussan classfer Nuno Vasconcelos ECE Department, UCSD Bayesan decson theory recall that we have state of the world X observatons g decson functon L[g,y] loss of predctng y wth g Bayes decson rule s
More informationSupport Vector Machines. Vibhav Gogate The University of Texas at dallas
Support Vector Machnes Vbhav Gogate he Unversty of exas at dallas What We have Learned So Far? 1. Decson rees. Naïve Bayes 3. Lnear Regresson 4. Logstc Regresson 5. Perceptron 6. Neural networks 7. K-Nearest
More informationVideo Data Analysis. Video Data Analysis, B-IT
Lecture Vdeo Data Analyss Deformable Snakes Segmentaton Neural networks Lecture plan:. Segmentaton by morphologcal watershed. Deformable snakes 3. Segmentaton va classfcaton of patterns 4. Concept of a
More information1 Derivation of Rate Equations from Single-Cell Conductance (Hodgkin-Huxley-like) Equations
Physcs 171/271 -Davd Klenfeld - Fall 2005 (revsed Wnter 2011) 1 Dervaton of Rate Equatons from Sngle-Cell Conductance (Hodgkn-Huxley-lke) Equatons We consder a network of many neurons, each of whch obeys
More informationCIS526: Machine Learning Lecture 3 (Sept 16, 2003) Linear Regression. Preparation help: Xiaoying Huang. x 1 θ 1 output... θ M x M
CIS56: achne Learnng Lecture 3 (Sept 6, 003) Preparaton help: Xaoyng Huang Lnear Regresson Lnear regresson can be represented by a functonal form: f(; θ) = θ 0 0 +θ + + θ = θ = 0 ote: 0 s a dummy attrbute
More informationNatural Language Processing and Information Retrieval
Natural Language Processng and Informaton Retreval Support Vector Machnes Alessandro Moschtt Department of nformaton and communcaton technology Unversty of Trento Emal: moschtt@ds.untn.t Summary Support
More informationSupport Vector Machines
/14/018 Separatng boundary, defned by w Support Vector Machnes CISC 5800 Professor Danel Leeds Separatng hyperplane splts class 0 and class 1 Plane s defned by lne w perpendcular to plan Is data pont x
More informationCSE 252C: Computer Vision III
CSE 252C: Computer Vson III Lecturer: Serge Belonge Scrbe: Catherne Wah LECTURE 15 Kernel Machnes 15.1. Kernels We wll study two methods based on a specal knd of functon k(x, y) called a kernel: Kernel
More informationHomework Assignment 3 Due in class, Thursday October 15
Homework Assgnment 3 Due n class, Thursday October 15 SDS 383C Statstcal Modelng I 1 Rdge regresson and Lasso 1. Get the Prostrate cancer data from http://statweb.stanford.edu/~tbs/elemstatlearn/ datasets/prostate.data.
More information), it produces a response (output function g (x)
Lnear Systems Revew Notes adapted from notes by Mchael Braun Typcally n electrcal engneerng, one s concerned wth functons of tme, such as a voltage waveform System descrpton s therefore defned n the domans
More informationLogistic Classifier CISC 5800 Professor Daniel Leeds
lon 9/7/8 Logstc Classfer CISC 58 Professor Danel Leeds Classfcaton strategy: generatve vs. dscrmnatve Generatve, e.g., Bayes/Naïve Bayes: 5 5 Identfy probablty dstrbuton for each class Determne class
More informationDeep Learning. Boyang Albert Li, Jie Jay Tan
Deep Learnng Boyang Albert L, Je Jay Tan An Unrelated Vdeo A bcycle controller learned usng NEAT (Stanley) What do you mean, deep? Shallow Hdden Markov models ANNs wth one hdden layer Manually selected
More informationPolynomial Regression Models
LINEAR REGRESSION ANALYSIS MODULE XII Lecture - 6 Polynomal Regresson Models Dr. Shalabh Department of Mathematcs and Statstcs Indan Insttute of Technology Kanpur Test of sgnfcance To test the sgnfcance
More information8 Derivation of Network Rate Equations from Single- Cell Conductance Equations
Physcs 178/278 - Davd Klenfeld - Wnter 2015 8 Dervaton of Network Rate Equatons from Sngle- Cell Conductance Equatons We consder a network of many neurons, each of whch obeys a set of conductancebased,
More informationVQ widely used in coding speech, image, and video
at Scalar quantzers are specal cases of vector quantzers (VQ): they are constraned to look at one sample at a tme (memoryless) VQ does not have such constrant better RD perfomance expected Source codng
More informationMaximal Margin Classifier
CS81B/Stat41B: Advanced Topcs n Learnng & Decson Makng Mamal Margn Classfer Lecturer: Mchael Jordan Scrbes: Jana van Greunen Corrected verson - /1/004 1 References/Recommended Readng 1.1 Webstes www.kernel-machnes.org
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Luca Trevisan September 5, 2017
U.C. Berkeley CS94: Beyond Worst-Case Analyss Handout 4s Luca Trevsan September 5, 07 Summary of Lecture 4 In whch we ntroduce semdefnte programmng and apply t to Max Cut. Semdefnte Programmng Recall that
More informationNeural networks. Nuno Vasconcelos ECE Department, UCSD
Neural networs Nuno Vasconcelos ECE Department, UCSD Classfcaton a classfcaton problem has two types of varables e.g. X - vector of observatons (features) n the world Y - state (class) of the world x X
More information8 Derivation of Network Rate Equations from Single- Cell Conductance Equations
Physcs 178/278 - Davd Klenfeld - Wnter 2019 8 Dervaton of Network Rate Equatons from Sngle- Cell Conductance Equatons Our goal to derve the form of the abstract quanttes n rate equatons, such as synaptc
More informationErrors for Linear Systems
Errors for Lnear Systems When we solve a lnear system Ax b we often do not know A and b exactly, but have only approxmatons  and ˆb avalable. Then the best thng we can do s to solve ˆx ˆb exactly whch
More informationLinear Classification, SVMs and Nearest Neighbors
1 CSE 473 Lecture 25 (Chapter 18) Lnear Classfcaton, SVMs and Nearest Neghbors CSE AI faculty + Chrs Bshop, Dan Klen, Stuart Russell, Andrew Moore Motvaton: Face Detecton How do we buld a classfer to dstngush
More informationEM and Structure Learning
EM and Structure Learnng Le Song Machne Learnng II: Advanced Topcs CSE 8803ML, Sprng 2012 Partally observed graphcal models Mxture Models N(μ 1, Σ 1 ) Z X N N(μ 2, Σ 2 ) 2 Gaussan mxture model Consder
More information9 Derivation of Rate Equations from Single-Cell Conductance (Hodgkin-Huxley-like) Equations
Physcs 171/271 - Chapter 9R -Davd Klenfeld - Fall 2005 9 Dervaton of Rate Equatons from Sngle-Cell Conductance (Hodgkn-Huxley-lke) Equatons We consder a network of many neurons, each of whch obeys a set
More informationSupport Vector Machines
Separatng boundary, defned by w Support Vector Machnes CISC 5800 Professor Danel Leeds Separatng hyperplane splts class 0 and class 1 Plane s defned by lne w perpendcular to plan Is data pont x n class
More informationRadial-Basis Function Networks
Radal-Bass uncton Networs v.0 March 00 Mchel Verleysen Radal-Bass uncton Networs - Radal-Bass uncton Networs p Orgn: Cover s theorem p Interpolaton problem p Regularzaton theory p Generalzed RBN p Unversal
More informationSolving Nonlinear Differential Equations by a Neural Network Method
Solvng Nonlnear Dfferental Equatons by a Neural Network Method Luce P. Aarts and Peter Van der Veer Delft Unversty of Technology, Faculty of Cvlengneerng and Geoscences, Secton of Cvlengneerng Informatcs,
More informationarxiv: v3 [cs.ne] 18 Jan 2019
Boltzmann machnes for tme-seres Takayuk Osogam IBM Research - Tokyo arxv:1708.06004v3 [cs.ne] 18 Jan 2019 osogam@p.bm.com verson 1.0.1 Abstract We revew Boltzmann machnes extended for tme-seres. These
More informationLossy Compression. Compromise accuracy of reconstruction for increased compression.
Lossy Compresson Compromse accuracy of reconstructon for ncreased compresson. The reconstructon s usually vsbly ndstngushable from the orgnal mage. Typcally, one can get up to 0:1 compresson wth almost
More informationNeural Networks. Neural Network Motivation. Why Neural Networks? Comments on Blue Gene. More Comments on Blue Gene
Motvaton for non-lnear Classfers Neural Networs CPS 27 Ron Parr Lnear methods are wea Mae strong assumptons Can only express relatvely smple functons of nputs Comng up wth good features can be hard Why
More informationLogistic Regression. CAP 5610: Machine Learning Instructor: Guo-Jun QI
Logstc Regresson CAP 561: achne Learnng Instructor: Guo-Jun QI Bayes Classfer: A Generatve model odel the posteror dstrbuton P(Y X) Estmate class-condtonal dstrbuton P(X Y) for each Y Estmate pror dstrbuton
More informationx = , so that calculated
Stat 4, secton Sngle Factor ANOVA notes by Tm Plachowsk n chapter 8 we conducted hypothess tests n whch we compared a sngle sample s mean or proporton to some hypotheszed value Chapter 9 expanded ths to
More informationMarkov Chain Monte Carlo Lecture 6
where (x 1,..., x N ) X N, N s called the populaton sze, f(x) f (x) for at least one {1, 2,..., N}, and those dfferent from f(x) are called the tral dstrbutons n terms of mportance samplng. Dfferent ways
More informationDr. Shalabh Department of Mathematics and Statistics Indian Institute of Technology Kanpur
Analyss of Varance and Desgn of Experment-I MODULE VII LECTURE - 3 ANALYSIS OF COVARIANCE Dr Shalabh Department of Mathematcs and Statstcs Indan Insttute of Technology Kanpur Any scentfc experment s performed
More informationHidden Markov Models
CM229S: Machne Learnng for Bonformatcs Lecture 12-05/05/2016 Hdden Markov Models Lecturer: Srram Sankararaman Scrbe: Akshay Dattatray Shnde Edted by: TBD 1 Introducton For a drected graph G we can wrte
More informationChapter 8 SCALAR QUANTIZATION
Outlne Chapter 8 SCALAR QUANTIZATION Yeuan-Kuen Lee [ CU, CSIE ] 8.1 Overvew 8. Introducton 8.4 Unform Quantzer 8.5 Adaptve Quantzaton 8.6 Nonunform Quantzaton 8.7 Entropy-Coded Quantzaton Ch 8 Scalar
More informationLecture 4. Instructor: Haipeng Luo
Lecture 4 Instructor: Hapeng Luo In the followng lectures, we focus on the expert problem and study more adaptve algorthms. Although Hedge s proven to be worst-case optmal, one may wonder how well t would
More informationNUMERICAL DIFFERENTIATION
NUMERICAL DIFFERENTIATION 1 Introducton Dfferentaton s a method to compute the rate at whch a dependent output y changes wth respect to the change n the ndependent nput x. Ths rate of change s called the
More informationImage classification. Given the bag-of-features representations of images from different classes, how do we learn a model for distinguishing i them?
Image classfcaton Gven te bag-of-features representatons of mages from dfferent classes ow do we learn a model for dstngusng tem? Classfers Learn a decson rule assgnng bag-offeatures representatons of
More informationGaussian Conditional Random Field Network for Semantic Segmentation - Supplementary Material
Gaussan Condtonal Random Feld Networ for Semantc Segmentaton - Supplementary Materal Ravtea Vemulapall, Oncel Tuzel *, Mng-Yu Lu *, and Rama Chellappa Center for Automaton Research, UMIACS, Unversty of
More informationStructure and Drive Paul A. Jensen Copyright July 20, 2003
Structure and Drve Paul A. Jensen Copyrght July 20, 2003 A system s made up of several operatons wth flow passng between them. The structure of the system descrbes the flow paths from nputs to outputs.
More informationLecture 12: Classification
Lecture : Classfcaton g Dscrmnant functons g The optmal Bayes classfer g Quadratc classfers g Eucldean and Mahalanobs metrcs g K Nearest Neghbor Classfers Intellgent Sensor Systems Rcardo Guterrez-Osuna
More informationLearning Theory: Lecture Notes
Learnng Theory: Lecture Notes Lecturer: Kamalka Chaudhur Scrbe: Qush Wang October 27, 2012 1 The Agnostc PAC Model Recall that one of the constrants of the PAC model s that the data dstrbuton has to be
More informationTuring Machines (intro)
CHAPTER 3 The Church-Turng Thess Contents Turng Machnes defntons, examples, Turng-recognzable and Turng-decdable languages Varants of Turng Machne Multtape Turng machnes, non-determnstc Turng Machnes,
More informationarxiv: v1 [cs.lg] 17 Jan 2019
LECTURE NOTES arxv:90.05639v [cs.lg] 7 Jan 209 Artfcal Neural Networks B. MEHLIG Department of Physcs Unversty of Gothenburg Göteborg, Sweden 209 PREFACE These are lecture notes for my course on Artfcal
More informationThe Geometry of Logit and Probit
The Geometry of Logt and Probt Ths short note s meant as a supplement to Chapters and 3 of Spatal Models of Parlamentary Votng and the notaton and reference to fgures n the text below s to those two chapters.
More informationNegative Binomial Regression
STATGRAPHICS Rev. 9/16/2013 Negatve Bnomal Regresson Summary... 1 Data Input... 3 Statstcal Model... 3 Analyss Summary... 4 Analyss Optons... 7 Plot of Ftted Model... 8 Observed Versus Predcted... 10 Predctons...
More informationGaussian Mixture Models
Lab Gaussan Mxture Models Lab Objectve: Understand the formulaton of Gaussan Mxture Models (GMMs) and how to estmate GMM parameters. You ve already seen GMMs as the observaton dstrbuton n certan contnuous
More informationLECTURE NOTES. Artifical Neural Networks. B. MEHLIG (course home page)
LECTURE NOTES Artfcal Neural Networks B. MEHLIG (course home page) Department of Physcs Unversty of Gothenburg Göteborg, Sweden 208 PREFACE These are lecture notes for my course on Artfcal Neural Networks
More informationp 1 c 2 + p 2 c 2 + p 3 c p m c 2
Where to put a faclty? Gven locatons p 1,..., p m n R n of m houses, want to choose a locaton c n R n for the fre staton. Want c to be as close as possble to all the house. We know how to measure dstance
More informationCourse 395: Machine Learning - Lectures
Course 395: Machne Learnng - Lectures Lecture 1-2: Concept Learnng (M. Pantc Lecture 3-4: Decson Trees & CC Intro (M. Pantc Lecture 5-6: Artfcal Neural Networks (S.Zaferou Lecture 7-8: Instance ased Learnng
More informationA how to guide to second quantization method.
Phys. 67 (Graduate Quantum Mechancs Sprng 2009 Prof. Pu K. Lam. Verson 3 (4/3/2009 A how to gude to second quantzaton method. -> Second quantzaton s a mathematcal notaton desgned to handle dentcal partcle
More informationMachine Learning & Data Mining CS/CNS/EE 155. Lecture 4: Regulariza5on, Sparsity & Lasso
Machne Learnng Data Mnng CS/CS/EE 155 Lecture 4: Regularza5on, Sparsty Lasso 1 Recap: Complete Ppelne S = {(x, y )} Tranng Data f (x, b) = T x b Model Class(es) L(a, b) = (a b) 2 Loss Func5on,b L( y, f
More information