Manning & Schuetze, FSNLP (c)1999, 2001
|
|
- Brendan Holmes
- 5 years ago
- Views:
Transcription
1 page Maxmum Entropy Modelng 589 Mannng & Schuetze, FSNLP (c)1999, 2001 a decson tree that detects spam. Fndng the rght features s paramount for ths task, so desgn your feature set carefully. Exercse 16.4 [ ] Another mportant applcaton of text categorzaton s the detecton of adult content, that s, content that s not approprate for chldren because t s sexually explct. Collect tranng and test sets of adult and non-adult materal from the World Wde Web and buld a decson tree that can block access to adult materal. Exercse 16.5 [ ] Collect a reasonable amount of text wrtten by yourself and by a frend. You may want to break up ndvdual texts (e.g., term papers) nto smaller peces to get a large enough set. Buld a decson tree that automatcally determnes whether you are the author of a pece of text. Note that t s often the lttle words that gve an author away (for example, the relatve frequences of words lke because or though). Exercse 16.6 [ ] Download a set of Englsh and non-englsh texts from the World Wde Web or use some other multlngual source. Buld a decson tree that can dstngush between Englsh and non-englsh texts. (See also exercse 6.10.) 16.2 Maxmum Entropy Modelng Maxmum entropy modelng s a framework for ntegratng nformaton from many heterogeneous nformaton sources for classfcaton. The data for a classfcaton problem s descrbed as a (potentally large) number of features. These features can be qute complex and allow the expermenter to make use of pror knowledge about what types of nformaton are expected to be mportant for classfcaton. Each feature corresponds to a constrant on the model. We then compute the maxmum entropy model, the model wth maxmum entropy of all the models that satsfy the constrants. Ths term may ntally seem perverse, snce we have spent most of the book tryng to mnmze the (cross) entropy of data accordng to models, but the dea s that we do not want to go beyond the data. If we chose a model wth less entropy, we would add nformaton to the model that s not justfed by the emprcal evdence avalable to us. Choosng the maxmum entropy model s motvated by the desre to preserve as much uncertanty as possble. We have smplfed matters n ths chapter by neglectng the problem of feature selecton (we use the same 20 features throughout). In maxmum entropy modelng, feature selecton and tranng are usually ntegrated.
2 page Text Categorzaton Mannng & Schuetze, FSNLP (c)1999, 2001 (16.3) Recall that s j s the term weght for word n Reuters artcle j. Note that the use of bnary features s dfferent from the rest of ths chapter: The other classfers use the magntude of the weght, not just the presence or absence of a word. 5 For a gven set of features, we frst compute the expectaton of each feature based on the tranng set. Each feature then defnes the constrant that the expectaton of the feature n our fnal maxmum entropy model must be the same as ths emprcal expectaton. Of all probablty ds- trbutons that obey these constrants, we attempt to fnd the maxmum entropy dstrbuton, the one wth the hghest entropy. One can show that there exsts a unque such maxmum entropy dstrbuton and there exsts an algorthm, generalzed teratve scalng, whch s guaranteed to converge to t. The model class for the partcular varety of maxmum entropy modelng that we ntroduce here s loglnear models of the followng form: emprcal expectaton maxmum entropy dstrbuton loglnear models (16.4) Ideally, ths enables us to specfy all potentally relevant nformaton at the begnnng, and then to let the tranng procedure worry about how to come up wth the best model for classfcaton. We wll only ntroduce the basc method here and refer the reader to the Further Readng for feature selecton. The features f are bnary functons that can be used to characterze any property of a par ( x, c), where xs a vector representng an nput element (n our case the 20-dmensonal vector of word weghts representng an artcle as n table 16.3), and c s the class label (1 f the artcle s n the earnngs category, 0 otherwse). For text categorzaton, we defne features as follows: { 1 f sj > 0andc=1 f ( x j,c)= 0 otherwse p( x, c) = 1 Z K =1 α f ( x,c) where K s the number of features, α s the weght for feature f and Z s a normalzng constant (commonly called the partton functon ), used 5. The maxmum entropy approach s not n prncple lmted to bnary features. Generalzed teratve scalng, whch we ntroduce below, requres merely that weghts are non-negatve and that ther sum s bounded. However, bnary features have generally been employed because they mprove the effcency of the computatonally ntensve reestmaton procedure.
3 page Maxmum Entropy Modelng 591 Mannng & Schuetze, FSNLP (c)1999, 2001 features (16.5) Generalzed teratve scalng s a procedure for fndng the maxmum en- tropy dstrbuton p of form (16.4) that obeys the followng set of constrants: generalzed teratve scalng to ensure that a probablty dstrbuton results. To use the model for text categorzaton, we compute p( x, 0) and p( x, 1) and, n the smplest case, choose the class label wth the greater probablty. Note that, n ths secton, features contan nformaton about the class of the object n addton to the measurements of the object we want to classfy. Here, we are followng most publcatons on maxmum entropy modelng n defnng feature n ths sense. The more common use of the term feature (whch we have adopted for the rest of the book) s that t only refers to some characterstc of the object, ndependent of the class the object s a member of. Equaton (16.4) defnes a loglnear model because, f we take logs on both sdes, then log p s a lnear combnaton of the logs of the weghts: log p( x, c) = log Z + K f ( x, c) log α =1 Loglnear models are a general and very mportant class of models for classfcaton wth categorcal varables. Other examples of the class are logstc regresson (McCullagh and Nelder 1989), decomposable models (Bruce and Webe 1999), and the HMMs andpcfgs of earler chapters. We ntroduce the maxmum entropy modelng approach here because maxmum entropy models have recently been wdely used n Statstcal NLP and because t s an applcaton of the mportant maxmum entropy prncple Generalzed teratve scalng (16.6) (16.7) E p f = E p f In other words, the expected value of f for p s the same as the expected value for the emprcal dstrbuton p (n other words, for the tranng set). The algorthm requres that the sum of the features for each possble ( x, c) be equal to a constant C: 6 x, c f ( x, c) = C 6. See Berger et al. (1996) for Improved Iteratve Scalng, a varant of generalzed teratve scalng that does not mpose ths constrant.
4 page Text Categorzaton Mannng & Schuetze, FSNLP (c)1999, 2001 (16.8) (16.9) (16.10) In order to fulfll ths requrement, we defne C as the greatest possble feature sum (over all possble data, not just observed data): K C max f ( x, c) x,c =1 and add a feature f K+1 that s defned as follows: K f K+1 ( x, c) = C f ( x, c) =1 Note that ths feature s not bnary, n contrast to the others. E p f s defned as (secton 2.1.5): E p f = x,c p( x, c)f ( x, c) where the sum s over the event space, that s, all possble vectors x and class labels c. The emprcal expectaton s easy to compute: E p f = p( x, c)f ( x, c) = 1 N f ( x j,c) N x,c j=1 where N s the number of elements n the tranng set and we use the fact that the emprcal probablty for a par that doesn t occur n the tranng set s 0. In general, the maxmum entropy dstrbuton E p f cannot be computed effcently snce t would nvolve summng over all possble combnatons of x and c, a huge or nfnte set. Instead, people use the followng approxmaton where only emprcally observed x are consdered (Lau 1994: 25): E p f p( x) p(c x)f ( x, c) = 1 N p(c x j )f ( x j,c) N x,c j=1 c where c stll ranges over all possble classes, n our case c {0,1}. Now we have all the peces to state the generalzed teratve scalng algorthm: 1. Intalze {α (1) }. Any ntalzaton wll do, but usually we choose α (1) = 1, 1 j K + 1. Compute E p f as shown above. Set n = Compute p (n) ( x, c) for the dstrbuton p (n) gven by the {α (n) } for each element ( x, c) n the tranng set: (16.11) p (n) ( x, c) = 1 K+1 ( (n) α Z =1 ) f ( x,c) where Z = K+1 ( (n) α x,c =1 ) f ( x,c)
5 page Maxmum Entropy Modelng 593 Mannng & Schuetze, FSNLP (c)1999, 2001 (16.12) x c proft earnngs f 1 f 2 β = f 1 log α 1 + f 2 log α 2 2 β (0) (0) (1) (1) Table 16.6 An example of a maxmum entropy dstrbuton n the form of equaton (16.4). The vector x conssts of a sngle element, ndcatng the presence orabsenceofthewordproft n the artcle. There are two classes (member of earnngs or not). Feature f 1 s 1 f and only f the artcle s n earnngs and proft occurs. f 2 s the fller feature f K+1. For one partcular choce of the parameters, namely log α 1 = 2.0 and log α 2 = 1.0, we get after normalzaton (Z = = 10) the followng maxmum entropy dstrbuton: p(0, 0) = p(0, 1) = p(1, 0) = 2/Z = 0.2 andp(1,1)=4/z = 0.4. An example of a data set wth the same emprcal dstrbuton s ((0, 0), (0, 1), (1, 0), (1, 1), (1, 1)). 3. Compute E p (n) f for all 1 K + 1 accordng to equaton (16.10). 4. Update the parameters α : α (n+1) ( ) 1 = α (n) C E p f E p (n) f 5. If the parameters of the procedure have converged, stop, otherwse ncrement n and go to 2. We present the algorthm n ths form for readablty. In an actual mplementaton, t s more convenent to do the computatons usng logarthms. One can show that ths procedure converges to a dstrbuton p that obeys the constrants (16.6), and that of all such dstrbutons t s the one that maxmzes the entropy H(p) and the lkelhood of the data. Darroch and Ratclff (1972) show that ths dstrbuton always exsts and s unque. A toy example of a maxmum entropy dstrbuton that generalzed teratve scalng wll converge to s shown n table Exercse 16.7 [ ] What are the classfcaton decsons for the dstrbuton n table 16.6? Compute P( earnngs proft) and P( earnngs proft).
6 page Text Categorzaton Mannng & Schuetze, FSNLP (c)1999, 2001 Does proft Is topc earnngs? occur? YES NO YES 20 9 NO 8 13 Table 16.7 An emprcal dstrbuton whose correspondng maxmum entropy dstrbuton s the one n table Exercse 16.8 [ ] Show that the dstrbuton n table 16.6 s a fxed pont for generalzed teratve scalng. That s, computng one teraton should leave the dstrbuton unchanged. Exercse 16.9 [ ] Consder the dstrbuton n table Show that for the features defned n table 16.6, ths dstrbuton has the same feature expectatons E p as the one n table Exercse [ ] Compute a number of teratons of generalzed teratve scalng for the data n table 16.7 (usng the features defned n table 16.6). The procedure should converge towards the dstrbuton n table Exercse [ ] Select one of exercses 16.3 through 16.6 and buld a maxmum entropy model for the correspondng text categorzaton task Applcaton to text categorzaton We have already suggested how to defne approprate features for text categorzaton n equaton (16.3). For the task of dentfyng Reuters earnngs artcles we end up wth 20 features, each correspondng to one of the selected words, and the f K+1 feature ntroduced at the start of the last subsecton, defned so that the features added to C = 20. We traned on the 9603 tranng set artcles. Table 16.8 shows the weghts found by generalzed teratve scalng after convergence (500 teratons). The features wth the hghest weghts are cts, proft, net and losst. IfweuseP( earnngs x) > P( earnngs x) as our decson rule, we get the classfcaton results n table Classfcaton accuracy s 96.2%. An mportant queston n an mplementaton s when to stop the teraton. One way to test for convergence s to compute the log dfference
7 page Maxmum Entropy Modelng 595 Mannng & Schuetze, FSNLP (c)1999, 2001 Word Feature weght w α log e α vs mln cts ; & loss " proft dlrs pct s s that net lt at f K Table 16.8 Feature weghts n maxmum entropy modelng for the category earnngs n Reuters. earnngs earnngs correct? assgned? YES NO YES NO Table 16.9 Classfcaton results for the dstrbuton correspondng to table 16.8 on the test set. Classfcaton accuracy s 96.2%.
8 page Text Categorzaton Mannng & Schuetze, FSNLP (c)1999, 2001 Nave Bayes lnear regresson between emprcal and estmated feature expectatons (log E p log E p (n)), whch should approach zero. Rstad (1996) recommends to also look at the largest α when dong teratve scalng. If the largest weght becomes too large, then ths ndcates a problem wth ether the data representaton or the mplementaton. When s the maxmum entropy framework presented n ths secton approprate as a classfcaton method? A characterstc of the maxmum entropy systems that are currently practcal for large data sets s the restrcton to bnary features. Ths s a shortcomng n some stuatons. In text categorzaton, we often need a noton of strength of evdence whch goes beyond smply recordng presence or absence of evdence. But ths does not appear to have hurt us much here. (Ths perhaps partly reflects how easy ths classfcaton problem s: Smply classfyng based on whether the cts feature s non-zero yelds an accuracy of 91.2%.) Generalzed teratve scalng can also be computatonally expensve due to slow convergence (but see (Lau 1994) for suggestons for speedng up convergence). For bnary classfcaton, the loglnear model defnes a lnear separator that s n prncple no more powerful than Nave Bayes or lnear regresson, classfers that can be traned more effcently. However, t s mportant to stress that, apart from the theoretcal power of a classfcaton method, the tranng procedure s crucal. Unlke Nave Bayes, generalzed teratve scalng takes dependence between features nto account: f one duplcated a feature, then the weght of each nstance of t would be halved. If feature dependence s not expected to a be a problem, then Nave Bayes s a better choce than maxmum entropy modelng. Fnally, the lack of smoothng can also cause problems. For example, f we have a feature that always predcts a certan class, then ths feature may get an excessvely hgh weght. One way to deal wth ths s to smooth the emprcal data by addng events that dd not occur. In practce, features that occur less than fve tmes are usually elmnated. One of the strengths of maxmum entropy modelng s that t offers a framework for specfyng all possbly relevant nformaton. The attracton of the method les n the fact that arbtrarly complex features can be defned f the expermenter beleves that these features may contrbute useful nformaton for the classfcaton decson. For example, Berger et al. (1996: 57) defne a feature for the translaton of the preposton n from Englsh to French that s 1 f and only f n s translated as pendant and n s followed by the word weeks wthn three words. There s also no need to worry about heterogenety of features or weghtng fea-
Generalized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
More informationFor now, let us focus on a specific model of neurons. These are simplified from reality but can achieve remarkable results.
Neural Networks : Dervaton compled by Alvn Wan from Professor Jtendra Malk s lecture Ths type of computaton s called deep learnng and s the most popular method for many problems, such as computer vson
More informationEnsemble Methods: Boosting
Ensemble Methods: Boostng Ncholas Ruozz Unversty of Texas at Dallas Based on the sldes of Vbhav Gogate and Rob Schapre Last Tme Varance reducton va baggng Generate new tranng data sets by samplng wth replacement
More informationEvaluation for sets of classes
Evaluaton for Tet Categorzaton Classfcaton accuracy: usual n ML, the proporton of correct decsons, Not approprate f the populaton rate of the class s low Precson, Recall and F 1 Better measures 21 Evaluaton
More informationKernel Methods and SVMs Extension
Kernel Methods and SVMs Extenson The purpose of ths document s to revew materal covered n Machne Learnng 1 Supervsed Learnng regardng support vector machnes (SVMs). Ths document also provdes a general
More informationLearning from Data 1 Naive Bayes
Learnng from Data 1 Nave Bayes Davd Barber dbarber@anc.ed.ac.uk course page : http://anc.ed.ac.uk/ dbarber/lfd1/lfd1.html c Davd Barber 2001, 2002 1 Learnng from Data 1 : c Davd Barber 2001,2002 2 1 Why
More informationLecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
More informationBoostrapaggregating (Bagging)
Boostrapaggregatng (Baggng) An ensemble meta-algorthm desgned to mprove the stablty and accuracy of machne learnng algorthms Can be used n both regresson and classfcaton Reduces varance and helps to avod
More informationHomework Assignment 3 Due in class, Thursday October 15
Homework Assgnment 3 Due n class, Thursday October 15 SDS 383C Statstcal Modelng I 1 Rdge regresson and Lasso 1. Get the Prostrate cancer data from http://statweb.stanford.edu/~tbs/elemstatlearn/ datasets/prostate.data.
More informationLearning Theory: Lecture Notes
Learnng Theory: Lecture Notes Lecturer: Kamalka Chaudhur Scrbe: Qush Wang October 27, 2012 1 The Agnostc PAC Model Recall that one of the constrants of the PAC model s that the data dstrbuton has to be
More informationFeature Selection: Part 1
CSE 546: Machne Learnng Lecture 5 Feature Selecton: Part 1 Instructor: Sham Kakade 1 Regresson n the hgh dmensonal settng How do we learn when the number of features d s greater than the sample sze n?
More informationANSWERS. Problem 1. and the moment generating function (mgf) by. defined for any real t. Use this to show that E( U) var( U)
Econ 413 Exam 13 H ANSWERS Settet er nndelt 9 deloppgaver, A,B,C, som alle anbefales å telle lkt for å gøre det ltt lettere å stå. Svar er gtt . Unfortunately, there s a prntng error n the hnt of
More information4DVAR, according to the name, is a four-dimensional variational method.
4D-Varatonal Data Assmlaton (4D-Var) 4DVAR, accordng to the name, s a four-dmensonal varatonal method. 4D-Var s actually a drect generalzaton of 3D-Var to handle observatons that are dstrbuted n tme. The
More informationLimited Dependent Variables
Lmted Dependent Varables. What f the left-hand sde varable s not a contnuous thng spread from mnus nfnty to plus nfnty? That s, gven a model = f (, β, ε, where a. s bounded below at zero, such as wages
More informationSupport Vector Machines. Vibhav Gogate The University of Texas at dallas
Support Vector Machnes Vbhav Gogate he Unversty of exas at dallas What We have Learned So Far? 1. Decson rees. Naïve Bayes 3. Lnear Regresson 4. Logstc Regresson 5. Perceptron 6. Neural networks 7. K-Nearest
More information2E Pattern Recognition Solutions to Introduction to Pattern Recognition, Chapter 2: Bayesian pattern classification
E395 - Pattern Recognton Solutons to Introducton to Pattern Recognton, Chapter : Bayesan pattern classfcaton Preface Ths document s a soluton manual for selected exercses from Introducton to Pattern Recognton
More information10-701/ Machine Learning, Fall 2005 Homework 3
10-701/15-781 Machne Learnng, Fall 2005 Homework 3 Out: 10/20/05 Due: begnnng of the class 11/01/05 Instructons Contact questons-10701@autonlaborg for queston Problem 1 Regresson and Cross-valdaton [40
More informationLINEAR REGRESSION ANALYSIS. MODULE IX Lecture Multicollinearity
LINEAR REGRESSION ANALYSIS MODULE IX Lecture - 30 Multcollnearty Dr. Shalabh Department of Mathematcs and Statstcs Indan Insttute of Technology Kanpur 2 Remedes for multcollnearty Varous technques have
More informationModule 3 LOSSY IMAGE COMPRESSION SYSTEMS. Version 2 ECE IIT, Kharagpur
Module 3 LOSSY IMAGE COMPRESSION SYSTEMS Verson ECE IIT, Kharagpur Lesson 6 Theory of Quantzaton Verson ECE IIT, Kharagpur Instructonal Objectves At the end of ths lesson, the students should be able to:
More information3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X
Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number
More informationLecture 12: Discrete Laplacian
Lecture 12: Dscrete Laplacan Scrbe: Tanye Lu Our goal s to come up wth a dscrete verson of Laplacan operator for trangulated surfaces, so that we can use t n practce to solve related problems We are mostly
More informationCSC 411 / CSC D11 / CSC C11
18 Boostng s a general strategy for learnng classfers by combnng smpler ones. The dea of boostng s to take a weak classfer that s, any classfer that wll do at least slghtly better than chance and use t
More informationDifference Equations
Dfference Equatons c Jan Vrbk 1 Bascs Suppose a sequence of numbers, say a 0,a 1,a,a 3,... s defned by a certan general relatonshp between, say, three consecutve values of the sequence, e.g. a + +3a +1
More informationLecture 4: Universal Hash Functions/Streaming Cont d
CSE 5: Desgn and Analyss of Algorthms I Sprng 06 Lecture 4: Unversal Hash Functons/Streamng Cont d Lecturer: Shayan Oves Gharan Aprl 6th Scrbe: Jacob Schreber Dsclamer: These notes have not been subjected
More informationErrors for Linear Systems
Errors for Lnear Systems When we solve a lnear system Ax b we often do not know A and b exactly, but have only approxmatons  and ˆb avalable. Then the best thng we can do s to solve ˆx ˆb exactly whch
More informationChapter Newton s Method
Chapter 9. Newton s Method After readng ths chapter, you should be able to:. Understand how Newton s method s dfferent from the Golden Secton Search method. Understand how Newton s method works 3. Solve
More informationCSci 6974 and ECSE 6966 Math. Tech. for Vision, Graphics and Robotics Lecture 21, April 17, 2006 Estimating A Plane Homography
CSc 6974 and ECSE 6966 Math. Tech. for Vson, Graphcs and Robotcs Lecture 21, Aprl 17, 2006 Estmatng A Plane Homography Overvew We contnue wth a dscusson of the major ssues, usng estmaton of plane projectve
More informationOnline Classification: Perceptron and Winnow
E0 370 Statstcal Learnng Theory Lecture 18 Nov 8, 011 Onlne Classfcaton: Perceptron and Wnnow Lecturer: Shvan Agarwal Scrbe: Shvan Agarwal 1 Introducton In ths lecture we wll start to study the onlne learnng
More informationSingular Value Decomposition: Theory and Applications
Sngular Value Decomposton: Theory and Applcatons Danel Khashab Sprng 2015 Last Update: March 2, 2015 1 Introducton A = UDV where columns of U and V are orthonormal and matrx D s dagonal wth postve real
More informationCS 3710: Visual Recognition Classification and Detection. Adriana Kovashka Department of Computer Science January 13, 2015
CS 3710: Vsual Recognton Classfcaton and Detecton Adrana Kovashka Department of Computer Scence January 13, 2015 Plan for Today Vsual recognton bascs part 2: Classfcaton and detecton Adrana s research
More informationMMA and GCMMA two methods for nonlinear optimization
MMA and GCMMA two methods for nonlnear optmzaton Krster Svanberg Optmzaton and Systems Theory, KTH, Stockholm, Sweden. krlle@math.kth.se Ths note descrbes the algorthms used n the author s 2007 mplementatons
More informationLinear Approximation with Regularization and Moving Least Squares
Lnear Approxmaton wth Regularzaton and Movng Least Squares Igor Grešovn May 007 Revson 4.6 (Revson : March 004). 5 4 3 0.5 3 3.5 4 Contents: Lnear Fttng...4. Weghted Least Squares n Functon Approxmaton...
More informationSupporting Information
Supportng Informaton The neural network f n Eq. 1 s gven by: f x l = ReLU W atom x l + b atom, 2 where ReLU s the element-wse rectfed lnear unt, 21.e., ReLUx = max0, x, W atom R d d s the weght matrx to
More informationLogistic Regression. CAP 5610: Machine Learning Instructor: Guo-Jun QI
Logstc Regresson CAP 561: achne Learnng Instructor: Guo-Jun QI Bayes Classfer: A Generatve model odel the posteror dstrbuton P(Y X) Estmate class-condtonal dstrbuton P(X Y) for each Y Estmate pror dstrbuton
More informationMultilayer Perceptron (MLP)
Multlayer Perceptron (MLP) Seungjn Cho Department of Computer Scence and Engneerng Pohang Unversty of Scence and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjn@postech.ac.kr 1 / 20 Outlne
More informationLecture 12: Classification
Lecture : Classfcaton g Dscrmnant functons g The optmal Bayes classfer g Quadratc classfers g Eucldean and Mahalanobs metrcs g K Nearest Neghbor Classfers Intellgent Sensor Systems Rcardo Guterrez-Osuna
More informationThe Expectation-Maximization Algorithm
The Expectaton-Maxmaton Algorthm Charles Elan elan@cs.ucsd.edu November 16, 2007 Ths chapter explans the EM algorthm at multple levels of generalty. Secton 1 gves the standard hgh-level verson of the algorthm.
More informationProblem Set 9 Solutions
Desgn and Analyss of Algorthms May 4, 2015 Massachusetts Insttute of Technology 6.046J/18.410J Profs. Erk Demane, Srn Devadas, and Nancy Lynch Problem Set 9 Solutons Problem Set 9 Solutons Ths problem
More informationWeek 5: Neural Networks
Week 5: Neural Networks Instructor: Sergey Levne Neural Networks Summary In the prevous lecture, we saw how we can construct neural networks by extendng logstc regresson. Neural networks consst of multple
More informationPsychology 282 Lecture #24 Outline Regression Diagnostics: Outliers
Psychology 282 Lecture #24 Outlne Regresson Dagnostcs: Outlers In an earler lecture we studed the statstcal assumptons underlyng the regresson model, ncludng the followng ponts: Formal statement of assumptons.
More informationPredictive Analytics : QM901.1x Prof U Dinesh Kumar, IIMB. All Rights Reserved, Indian Institute of Management Bangalore
Sesson Outlne Introducton to classfcaton problems and dscrete choce models. Introducton to Logstcs Regresson. Logstc functon and Logt functon. Maxmum Lkelhood Estmator (MLE) for estmaton of LR parameters.
More informationLecture 10 Support Vector Machines II
Lecture 10 Support Vector Machnes II 22 February 2016 Taylor B. Arnold Yale Statstcs STAT 365/665 1/28 Notes: Problem 3 s posted and due ths upcomng Frday There was an early bug n the fake-test data; fxed
More informationADVANCED MACHINE LEARNING ADVANCED MACHINE LEARNING
1 ADVANCED ACHINE LEARNING ADVANCED ACHINE LEARNING Non-lnear regresson technques 2 ADVANCED ACHINE LEARNING Regresson: Prncple N ap N-dm. nput x to a contnuous output y. Learn a functon of the type: N
More informationNatural Language Processing and Information Retrieval
Natural Language Processng and Informaton Retreval Support Vector Machnes Alessandro Moschtt Department of nformaton and communcaton technology Unversty of Trento Emal: moschtt@ds.untn.t Summary Support
More informationModule 9. Lecture 6. Duality in Assignment Problems
Module 9 1 Lecture 6 Dualty n Assgnment Problems In ths lecture we attempt to answer few other mportant questons posed n earler lecture for (AP) and see how some of them can be explaned through the concept
More informationNegative Binomial Regression
STATGRAPHICS Rev. 9/16/2013 Negatve Bnomal Regresson Summary... 1 Data Input... 3 Statstcal Model... 3 Analyss Summary... 4 Analyss Optons... 7 Plot of Ftted Model... 8 Observed Versus Predcted... 10 Predctons...
More informationIntroduction to Vapor/Liquid Equilibrium, part 2. Raoult s Law:
CE304, Sprng 2004 Lecture 4 Introducton to Vapor/Lqud Equlbrum, part 2 Raoult s Law: The smplest model that allows us do VLE calculatons s obtaned when we assume that the vapor phase s an deal gas, and
More informationINF 5860 Machine learning for image classification. Lecture 3 : Image classification and regression part II Anne Solberg January 31, 2018
INF 5860 Machne learnng for mage classfcaton Lecture 3 : Image classfcaton and regresson part II Anne Solberg January 3, 08 Today s topcs Multclass logstc regresson and softma Regularzaton Image classfcaton
More information3.1 ML and Empirical Distribution
67577 Intro. to Machne Learnng Fall semester, 2008/9 Lecture 3: Maxmum Lkelhood/ Maxmum Entropy Dualty Lecturer: Amnon Shashua Scrbe: Amnon Shashua 1 In the prevous lecture we defned the prncple of Maxmum
More informationFREQUENCY DISTRIBUTIONS Page 1 of The idea of a frequency distribution for sets of observations will be introduced,
FREQUENCY DISTRIBUTIONS Page 1 of 6 I. Introducton 1. The dea of a frequency dstrbuton for sets of observatons wll be ntroduced, together wth some of the mechancs for constructng dstrbutons of data. Then
More informationxp(x µ) = 0 p(x = 0 µ) + 1 p(x = 1 µ) = µ
CSE 455/555 Sprng 2013 Homework 7: Parametrc Technques Jason J. Corso Computer Scence and Engneerng SUY at Buffalo jcorso@buffalo.edu Solutons by Yngbo Zhou Ths assgnment does not need to be submtted and
More informationVQ widely used in coding speech, image, and video
at Scalar quantzers are specal cases of vector quantzers (VQ): they are constraned to look at one sample at a tme (memoryless) VQ does not have such constrant better RD perfomance expected Source codng
More informationCalculation of time complexity (3%)
Problem 1. (30%) Calculaton of tme complexty (3%) Gven n ctes, usng exhaust search to see every result takes O(n!). Calculaton of tme needed to solve the problem (2%) 40 ctes:40! dfferent tours 40 add
More informationSociété de Calcul Mathématique SA
Socété de Calcul Mathématque SA Outls d'ade à la décson Tools for decson help Probablstc Studes: Normalzng the Hstograms Bernard Beauzamy December, 202 I. General constructon of the hstogram Any probablstc
More informationCSE 546 Midterm Exam, Fall 2014(with Solution)
CSE 546 Mdterm Exam, Fall 014(wth Soluton) 1. Personal nfo: Name: UW NetID: Student ID:. There should be 14 numbered pages n ths exam (ncludng ths cover sheet). 3. You can use any materal you brought:
More information8/25/17. Data Modeling. Data Modeling. Data Modeling. Patrice Koehl Department of Biological Sciences National University of Singapore
8/5/17 Data Modelng Patrce Koehl Department of Bologcal Scences atonal Unversty of Sngapore http://www.cs.ucdavs.edu/~koehl/teachng/bl59 koehl@cs.ucdavs.edu Data Modelng Ø Data Modelng: least squares Ø
More informationNotes on Frequency Estimation in Data Streams
Notes on Frequency Estmaton n Data Streams In (one of) the data streamng model(s), the data s a sequence of arrvals a 1, a 2,..., a m of the form a j = (, v) where s the dentty of the tem and belongs to
More informationLinear Regression Analysis: Terminology and Notation
ECON 35* -- Secton : Basc Concepts of Regresson Analyss (Page ) Lnear Regresson Analyss: Termnology and Notaton Consder the generc verson of the smple (two-varable) lnear regresson model. It s represented
More informationLectures - Week 4 Matrix norms, Conditioning, Vector Spaces, Linear Independence, Spanning sets and Basis, Null space and Range of a Matrix
Lectures - Week 4 Matrx norms, Condtonng, Vector Spaces, Lnear Independence, Spannng sets and Bass, Null space and Range of a Matrx Matrx Norms Now we turn to assocatng a number to each matrx. We could
More informationVapnik-Chervonenkis theory
Vapnk-Chervonenks theory Rs Kondor June 13, 2008 For the purposes of ths lecture, we restrct ourselves to the bnary supervsed batch learnng settng. We assume that we have an nput space X, and an unknown
More informationMarkov Chain Monte Carlo (MCMC), Gibbs Sampling, Metropolis Algorithms, and Simulated Annealing Bioinformatics Course Supplement
Markov Chan Monte Carlo MCMC, Gbbs Samplng, Metropols Algorthms, and Smulated Annealng 2001 Bonformatcs Course Supplement SNU Bontellgence Lab http://bsnuackr/ Outlne! Markov Chan Monte Carlo MCMC! Metropols-Hastngs
More information8 : Learning in Fully Observed Markov Networks. 1 Why We Need to Learn Undirected Graphical Models. 2 Structural Learning for Completely Observed MRF
10-708: Probablstc Graphcal Models 10-708, Sprng 2014 8 : Learnng n Fully Observed Markov Networks Lecturer: Erc P. Xng Scrbes: Meng Song, L Zhou 1 Why We Need to Learn Undrected Graphcal Models In the
More informationSupport Vector Machines CS434
Support Vector Machnes CS434 Lnear Separators Many lnear separators exst that perfectly classfy all tranng examples Whch of the lnear separators s the best? Intuton of Margn Consder ponts A, B, and C We
More informationExercises. 18 Algorithms
18 Algorthms Exercses 0.1. In each of the followng stuatons, ndcate whether f = O(g), or f = Ω(g), or both (n whch case f = Θ(g)). f(n) g(n) (a) n 100 n 200 (b) n 1/2 n 2/3 (c) 100n + log n n + (log n)
More information8.4 COMPLEX VECTOR SPACES AND INNER PRODUCTS
SECTION 8.4 COMPLEX VECTOR SPACES AND INNER PRODUCTS 493 8.4 COMPLEX VECTOR SPACES AND INNER PRODUCTS All the vector spaces you have studed thus far n the text are real vector spaces because the scalars
More informationEcon107 Applied Econometrics Topic 3: Classical Model (Studenmund, Chapter 4)
I. Classcal Assumptons Econ7 Appled Econometrcs Topc 3: Classcal Model (Studenmund, Chapter 4) We have defned OLS and studed some algebrac propertes of OLS. In ths topc we wll study statstcal propertes
More informationLinear Classification, SVMs and Nearest Neighbors
1 CSE 473 Lecture 25 (Chapter 18) Lnear Classfcaton, SVMs and Nearest Neghbors CSE AI faculty + Chrs Bshop, Dan Klen, Stuart Russell, Andrew Moore Motvaton: Face Detecton How do we buld a classfer to dstngush
More informationCollege of Computer & Information Science Fall 2009 Northeastern University 20 October 2009
College of Computer & Informaton Scence Fall 2009 Northeastern Unversty 20 October 2009 CS7880: Algorthmc Power Tools Scrbe: Jan Wen and Laura Poplawsk Lecture Outlne: Prmal-dual schema Network Desgn:
More informationLecture 4. Instructor: Haipeng Luo
Lecture 4 Instructor: Hapeng Luo In the followng lectures, we focus on the expert problem and study more adaptve algorthms. Although Hedge s proven to be worst-case optmal, one may wonder how well t would
More informationx = , so that calculated
Stat 4, secton Sngle Factor ANOVA notes by Tm Plachowsk n chapter 8 we conducted hypothess tests n whch we compared a sngle sample s mean or proporton to some hypotheszed value Chapter 9 expanded ths to
More informationTHE SUMMATION NOTATION Ʃ
Sngle Subscrpt otaton THE SUMMATIO OTATIO Ʃ Most of the calculatons we perform n statstcs are repettve operatons on lsts of numbers. For example, we compute the sum of a set of numbers, or the sum of the
More informationChapter 5. Solution of System of Linear Equations. Module No. 6. Solution of Inconsistent and Ill Conditioned Systems
Numercal Analyss by Dr. Anta Pal Assstant Professor Department of Mathematcs Natonal Insttute of Technology Durgapur Durgapur-713209 emal: anta.bue@gmal.com 1 . Chapter 5 Soluton of System of Lnear Equatons
More informationLecture 7: Boltzmann distribution & Thermodynamics of mixing
Prof. Tbbtt Lecture 7 etworks & Gels Lecture 7: Boltzmann dstrbuton & Thermodynamcs of mxng 1 Suggested readng Prof. Mark W. Tbbtt ETH Zürch 13 März 018 Molecular Drvng Forces Dll and Bromberg: Chapters
More informationQuestion Classification Using Language Modeling
Queston Classfcaton Usng Language Modelng We L Center for Intellgent Informaton Retreval Department of Computer Scence Unversty of Massachusetts, Amherst, MA 01003 ABSTRACT Queston classfcaton assgns a
More informationCOS 521: Advanced Algorithms Game Theory and Linear Programming
COS 521: Advanced Algorthms Game Theory and Lnear Programmng Moses Charkar February 27, 2013 In these notes, we ntroduce some basc concepts n game theory and lnear programmng (LP). We show a connecton
More informationClustering & Unsupervised Learning
Clusterng & Unsupervsed Learnng Ken Kreutz-Delgado (Nuno Vasconcelos) ECE 175A Wnter 2012 UCSD Statstcal Learnng Goal: Gven a relatonshp between a feature vector x and a vector y, and d data samples (x,y
More informationOutline. Bayesian Networks: Maximum Likelihood Estimation and Tree Structure Learning. Our Model and Data. Outline
Outlne Bayesan Networks: Maxmum Lkelhood Estmaton and Tree Structure Learnng Huzhen Yu janey.yu@cs.helsnk.f Dept. Computer Scence, Unv. of Helsnk Probablstc Models, Sprng, 200 Notces: I corrected a number
More informationSimultaneous Optimization of Berth Allocation, Quay Crane Assignment and Quay Crane Scheduling Problems in Container Terminals
Smultaneous Optmzaton of Berth Allocaton, Quay Crane Assgnment and Quay Crane Schedulng Problems n Contaner Termnals Necat Aras, Yavuz Türkoğulları, Z. Caner Taşkın, Kuban Altınel Abstract In ths work,
More informationOpen Systems: Chemical Potential and Partial Molar Quantities Chemical Potential
Open Systems: Chemcal Potental and Partal Molar Quanttes Chemcal Potental For closed systems, we have derved the followng relatonshps: du = TdS pdv dh = TdS + Vdp da = SdT pdv dg = VdP SdT For open systems,
More informationCredit Card Pricing and Impact of Adverse Selection
Credt Card Prcng and Impact of Adverse Selecton Bo Huang and Lyn C. Thomas Unversty of Southampton Contents Background Aucton model of credt card solctaton - Errors n probablty of beng Good - Errors n
More informationFoundations of Arithmetic
Foundatons of Arthmetc Notaton We shall denote the sum and product of numbers n the usual notaton as a 2 + a 2 + a 3 + + a = a, a 1 a 2 a 3 a = a The notaton a b means a dvdes b,.e. ac = b where c s an
More information= z 20 z n. (k 20) + 4 z k = 4
Problem Set #7 solutons 7.2.. (a Fnd the coeffcent of z k n (z + z 5 + z 6 + z 7 + 5, k 20. We use the known seres expanson ( n+l ( z l l z n below: (z + z 5 + z 6 + z 7 + 5 (z 5 ( + z + z 2 + z + 5 5
More informationEEE 241: Linear Systems
EEE : Lnear Systems Summary #: Backpropagaton BACKPROPAGATION The perceptron rule as well as the Wdrow Hoff learnng were desgned to tran sngle layer networks. They suffer from the same dsadvantage: they
More informationHopfield Training Rules 1 N
Hopfeld Tranng Rules To memorse a sngle pattern Suppose e set the eghts thus - = p p here, s the eght beteen nodes & s the number of nodes n the netor p s the value requred for the -th node What ll the
More informationMDL-Based Unsupervised Attribute Ranking
MDL-Based Unsupervsed Attrbute Rankng Zdravko Markov Computer Scence Department Central Connectcut State Unversty New Brtan, CT 06050, USA http://www.cs.ccsu.edu/~markov/ markovz@ccsu.edu MDL-Based Unsupervsed
More informationThe Geometry of Logit and Probit
The Geometry of Logt and Probt Ths short note s meant as a supplement to Chapters and 3 of Spatal Models of Parlamentary Votng and the notaton and reference to fgures n the text below s to those two chapters.
More informationCanonical transformations
Canoncal transformatons November 23, 2014 Recall that we have defned a symplectc transformaton to be any lnear transformaton M A B leavng the symplectc form nvarant, Ω AB M A CM B DΩ CD Coordnate transformatons,
More informationBayesian predictive Configural Frequency Analysis
Psychologcal Test and Assessment Modelng, Volume 54, 2012 (3), 285-292 Bayesan predctve Confgural Frequency Analyss Eduardo Gutérrez-Peña 1 Abstract Confgural Frequency Analyss s a method for cell-wse
More informationSome modelling aspects for the Matlab implementation of MMA
Some modellng aspects for the Matlab mplementaton of MMA Krster Svanberg krlle@math.kth.se Optmzaton and Systems Theory Department of Mathematcs KTH, SE 10044 Stockholm September 2004 1. Consdered optmzaton
More informationp 1 c 2 + p 2 c 2 + p 3 c p m c 2
Where to put a faclty? Gven locatons p 1,..., p m n R n of m houses, want to choose a locaton c n R n for the fre staton. Want c to be as close as possble to all the house. We know how to measure dstance
More informationWhich Separator? Spring 1
Whch Separator? 6.034 - Sprng 1 Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng 3 Margn of a pont " # y (w $ + b) proportonal
More information1 The Mistake Bound Model
5-850: Advanced Algorthms CMU, Sprng 07 Lecture #: Onlne Learnng and Multplcatve Weghts February 7, 07 Lecturer: Anupam Gupta Scrbe: Bryan Lee,Albert Gu, Eugene Cho he Mstake Bound Model Suppose there
More informationStructure and Drive Paul A. Jensen Copyright July 20, 2003
Structure and Drve Paul A. Jensen Copyrght July 20, 2003 A system s made up of several operatons wth flow passng between them. The structure of the system descrbes the flow paths from nputs to outputs.
More informationCommon loop optimizations. Example to improve locality. Why Dependence Analysis. Data Dependence in Loops. Goal is to find best schedule:
15-745 Lecture 6 Data Dependence n Loops Copyrght Seth Goldsten, 2008 Based on sldes from Allen&Kennedy Lecture 6 15-745 2005-8 1 Common loop optmzatons Hostng of loop-nvarant computatons pre-compute before
More informationFinding Dense Subgraphs in G(n, 1/2)
Fndng Dense Subgraphs n Gn, 1/ Atsh Das Sarma 1, Amt Deshpande, and Rav Kannan 1 Georga Insttute of Technology,atsh@cc.gatech.edu Mcrosoft Research-Bangalore,amtdesh,annan@mcrosoft.com Abstract. Fndng
More informationMAXIMUM A POSTERIORI TRANSDUCTION
MAXIMUM A POSTERIORI TRANSDUCTION LI-WEI WANG, JU-FU FENG School of Mathematcal Scences, Peng Unversty, Bejng, 0087, Chna Center for Informaton Scences, Peng Unversty, Bejng, 0087, Chna E-MIAL: {wanglw,
More informationClassification as a Regression Problem
Target varable y C C, C,, ; Classfcaton as a Regresson Problem { }, 3 L C K To treat classfcaton as a regresson problem we should transform the target y nto numercal values; The choce of numercal class
More informationComposite Hypotheses testing
Composte ypotheses testng In many hypothess testng problems there are many possble dstrbutons that can occur under each of the hypotheses. The output of the source s a set of parameters (ponts n a parameter
More informationWe present the algorithm first, then derive it later. Assume access to a dataset {(x i, y i )} n i=1, where x i R d and y i { 1, 1}.
CS 189 Introducton to Machne Learnng Sprng 2018 Note 26 1 Boostng We have seen that n the case of random forests, combnng many mperfect models can produce a snglodel that works very well. Ths s the dea
More informationSIO 224. m(r) =(ρ(r),k s (r),µ(r))
SIO 224 1. A bref look at resoluton analyss Here s some background for the Masters and Gubbns resoluton paper. Global Earth models are usually found teratvely by assumng a startng model and fndng small
More information