LECTURE NOTE #12 PROF. ALAN YUILLE
|
|
- June Miles
- 6 years ago
- Views:
Transcription
1 LECTURE NOTE #12 PROF. ALAN YUILLE 1. Clustering, K-mens, nd EM Tsk: set of unlbeled dt D = {x 1,..., x n } Decompose into clsses w 1,..., w M where M is unknown. Lern clss models p(x w)) Discovery of new clsses (concepts) This is form of unsupervised lerning. How do people lern grmmr? Is it innte? Coded in DNA? Or is it lern in n unsupervised mnner? Prcticl Motivtions for unsupervised lerning - lbeling lrge dtsets is hrd, costly, nd time consuming. Also tht the dt is generted by complex model which involves hidden vribles. Why is this clled unsupervised? It mens tht there re some hidden/ltent/missing vribles tht re not specified by the trining dt. I.e., the trining dt is {x 1,..., x n } nd we ssume tht it is generted by model P (x w)p (w), where the w re unknown nd cn be clled ltent/hidden/missing (Sttisticins cll them missing dt, Neurl Network reserchers cll them hidden units, Mchine Lerning reserchers cll them ltent vribles ). By contrst, in previous lectures, we ssumed model p(x) which ws either n exponentil distribution or non-prmetric distribution. We cn think of this s lerning model P (x, w) nd we cn ssume tht this model is of exponentil form P (x, w) = 1 Z[λ] exp{λ φ(x, w)}, i.e. n exponentil distribution 1
2 2 PROF. ALAN YUILLE defined over the observer units x nd the unobserved units w (ltent/hidden/missing). This model cn cpture extremely complex models stochstic grmmrs on nturl lnguge, hierrchicl models of objects (e.g. my tlk in the BCE symposium), probbilistic models of resoning. But these models re hrder to del with thn models without hidden vribles. The models require: (i) lerning the structure how the hidden nd observed units interct (see lter lectures) which is clled structure induction nd which is very difficult, (ii) lerning the prmeters λ of the model if the structure is known this is non-convex optimiztion problem nd so it will hve multiple minim. We will describe the bsic lgorithm for doing this the EM lgorithm lter in this lecture. (Note: there is lso mchine lerning lgorithm clled ltent SVM which is like n pproximtion to EM). Note, there re lso non-prmetric methods for lerning P (x, w). Note tht there is lso semi-supervised lerning (e.g, Prof. Jin s tlk in the BCE Symposium). This mens tht some informtion is supplied for some of the hidden sttes i.e. you hve some dt {x 1,..., x n } which is unsupervised, nd some dt which is supervised {(x n+1, h n+1 ),...(x n+m, h n+m )}. 2. K-mens lgorithm Bsic Assumption the dt D is clustered round (unknown) men vlues m 1,..., m k, see figure (1). This is n ssumption bout the structure of the model. Figure 1. The k-mens lgorithm ssumes tht the dt cn be clustered round (unknown) men vlues m 1,..., m k. Equivlently, tht the dt is generted by mixture of Gussins with fixed covrince nd with these mens.
3 LECTURE NOTE #12 3 Seek to ssocite the dt to these mens, nd to simultneously estimte the mens. For now, we ssume tht the number of mens is known i.e. k is fixed. Ide: decompose D = k =1 D, where D is dt ssocited with men m Men of Fit F ({D }) = k =1 x D (x m ) 2 Wnt to select {m }, nd ssignment x D to minimize the fit F ( ). 3. Assignment Vrible: V i = 1, if dt x i is ssigned to m V i = 0, otherwise Constrint k =1 V i = 1, i, (i.e. ech dtpoint x i is ssigned to only one clss m ). Note: don t need ssignment vribles yet. - so tht ll dt- Deterministic K-mens: 1. Initilize prtition {D 0 : = 1 to k} of the dt (E.G. rndomly choose points x nd put them into set, D1 0, D0 2,..., D0 k points re in exctly one set) 2. Compute the men of ech cluster D, m = 1 w x D x 3. For i=1 to N, compute d (x i ) = x i m 2 Assign x i to cluster D s.t. = rg min{d (x i ),..., d k (x i )} 4. Repet steps 2 & 3 until convergence.
4 4 PROF. ALAN YUILLE 4. Comment: The k-men lgorithm is not gurnteed to converge to the best fit of F (D) It will lmost lwys converge to fixed point. Typiclly, you cn improve k-mens by giving it different initil conditions. Then select the solution which hs best fit. Evlute solution with different vlues of k. If the number of mens (clusters) is too lrge, then you will usully find some of the men will be close together, see figure (2). i.e do postprocessing to eliminte mens which re too close to ech other. Figure 2. There re three clusters nd four mens. mens will usully be ssigned to one cluster. In prctice, two
5 LECTURE NOTE # A softer version of k-mens the Expecttion-Mximiztion (EM) lgorithm. Assign dtpoint x i to ech cluster with probbility (P 1,..., P k ) 1. Initilize prtition. E.G. rndomly chose k points s centres m 1, m 2,..., m k 2. For j=1 to N, Compute distnces d (x j ) = x j m 2 Compute the probbility tht x j belongs to ω P (x j ) = 1 e 1 2σ 2 (x j m ) 2 (2πσ 2 )d/2 3. Compute the men nd vrince for ech cluster: m = 1 D x D xp (x) σ 2 = 1 D x D (x m ) 2 P (x) Repet steps 2 & 3 until convergence. 6. Deeper Understnding Modeling dt with hidden vribles P (x, h, θ)
6 6 PROF. ALAN YUILLE * x : observed dt * h : hidden vrible *θ : model prmeter Exmple: dt is generted by mixture of Gussin distribution with unknown mens nd covrince. The vrible h indictes which Gussin generted the dt. P (x h =, θ ) = 1 e 1 2 (x µ )Σ 1 (x µ (2π) Σ 1/2 ) Assume P (h = θ) = eφ c eφc And prior probbility P (θ) on {φ, θ } 7. Mixture of Gussin exmple P (x, h, θ) = p(x h, θ)p(h θ)p(θ), see figure (3). Now perform MAP estimtion of θ from dt exmples D = {x 1,.., x n } p(d h, θ) = p({x i,..., x n } h, θ) = N i=1 p(x i h, θ) i.i.d. ssumption.
7 LECTURE NOTE #12 7 Figure 3. The dt x is generted by hidden vrible h by probbility model with unknown prmeters θ. p(θ D) = h p(θ, h D) How to estimte ˆθ from D? Answer (most populr), the Expecttion Mximiztion (EM) lgorithm. Comment: stndrd k-mens lgorithm cn be thought of s limiting cse of EM for mixture of Gussins - where the covrince is fixed to be the identity mtrix. 8. The EM Algorithm p(θ D) = h p(θ, h D) Define new distribution q(h) Minimize F (θ, q) = log p(θ D) + q(h) h q(h) log p(h θ,d)
8 8 PROF. ALAN YUILLE * Kullbck-Leibler divergence: h q(h) log q(h) p(h θ,d) 0 Note, the minimum occurs t ˆθ = rg min θ { log p(θ D)} = rg mx θ p(θ w) nd t ˆq(h) = p(h ˆθ, D) (Becuse the Kullbck-Liebler divergence ttins its minimum t ˆq(h) = p(h ˆθ, w)). 9. We cn re-express the Free Energy p(h θ,d) F (θ, q) = log p(θ D) + q(h) h q(h) log = h q(h) log p(θ D) + h q(h) log q(h) h q(h) log p(h θ, D) = h q(h) log{p(θ D)p(h θ, D)} + h q(h) log q(h) F (θ, q) = h q(h) log p(h, θ D) + h q(h) log q(h) EM minimize F (θ, q) w.r.t. θ&q lterntively. This is clled coordinte descent, see figure (4). Here is n intuition for this lgorithm. You live in city with hills, like Seoul, nd the streets re rrnged in horizontl grid of blocks. You strt on hill (t high ltitude) nd wnt to decrese your ltitude. You cn either wlk in the North-South direction or in the Est-West direction. You look in ll directions nd must choose to wlk North, South, Est, or West. You see tht North nd Est re uphill, so you do not choose them. You see tht you cn wlk South nd decrese your ltitude by only 10 meters before the street strts going uphill. You look West nd see tht you cn decrese your ltitude by 100 meters by wlking in tht direction (until tht street lso strts going uphill). So you i.e. coordinte descent chooses to wlk West nd stop when the street strts going uphill (i.e. you hve lost 100 meters). The you stop, look gin in the directions North, South, Est, West nd wlk in the direction to mximize your decrese in ltitude. The you repet until you re t plce where ll directions re uphill. Note: coordinte descent will be importnt when the dimension of the spce is very big. In two-dimensions, like the city exmple, the number of choices is smll so if you hve moved Est-West lst time then you hve to move North-South the next time. In high-dimensions, there re n enormous number of choices. So the lgorithm will often choose only to move in smll number of possible directions, nd the other directions will be irrelevnt. This will be importnt for AdBoost lerning.
9 LECTURE NOTE #12 9 Note: you my not be ble to clculte how much you will decrese ltitude by wlking in one direction. This will depend on the specific problem. Figure 4. The coordinte descent lgorithm. You strt t θ t, q t. You fix q t nd decrese F (θ, q) by chnging θ until you rech θ t+1, where θ F (θ, qt ) = 0. Then fix θ t+1 nd decrese F (θ, q) by chnging q until you rech q t+1, where q F (θt+1, q) = 0. Step 1: Fix q t, set θ t+1 = rg min θ { h q(h) log p(h, θ D)} Step 2: Fix θ t, set q t+1 (h) = p(h θ t, D) Iterte steps 1 & 2 until convergence. Note: this lgorithm is gurnteed to converge to locl minimum of F (θ, q), but there is no gurntee tht it will converge to the globl minimum. You cn use multiple strting points nd pick the solution which hs lowest vlue of F (., ) or you cn use extr knowledge bout the problem to hve good strting point, or strting points. Or you cn use stochstic lgorithm. 10. Exponentil EM Exmple & Decision Trees EM tkes simple form for exponentil distributions. P (x, h λ) = eλφ(x,h) z[λ]
10 10 PROF. ALAN YUILLE Generl exponentil form, where the h re hidden vribles / ltent vribles. P (x λ) = Σ h P (x, h λ) Two steps of EM Step 1: q t+1 (h) = P (h x, λ t ) = P (x,h λt ) Σ xp ( x,h λ t ) Step 2: λ t+1 = ARG λ MIN h qt+1 (h){log P (x, h λ)} λ t+1 = ARG x MIN { h qt+1 φ(x, h)λ + log z[λ]} Differentiting gives : x,h P (x, h λt+1 )φ(x, h) = h qt+1 (h)φ(x, h) 11. Mixture of Gussin Exmple 1 P (y i V i = 1, θ) = e 1 2 (y i µ )T Σ 1 (y (2π) d/2 Σ 1/2 i µ ) dtpoint y i is generted by th model. We cn write : P (y i {V i }, θ) = 1 (2π) d/2 e V i{ 1 2 (y i µ )T Σ 1 (y i µ )+ 1 2 log 1/2 } By i.i.d ssumption, P ({y i } {V i }, θ) = i P (y i {V i }, θ) = 1 (2π) nd/2 e i V i{ 1 2 (y i µ )T Σ 1 (y i µ )+ 1 2 log 1/2 }
11 LECTURE NOTE #12 11 Let the prior probbility on the {V i } be uniform unknown P ({V i }) = constnt. Then P ({y i }, {V i } θ) = e Z i V i{ 1 2 (y i µ )T Σ 1 (y i µ )+ 1 2 log 1/2 } *Z : Normliztion constnt. 12. To pply EM to this problem, E-step: Q(θ θ (t) ) = v {log P (y, V θ)}p (V θt ) M-step: Determine θ (t+1) by mximizing Q(θ θ (t) ) P ({V i } θ) = i P i(v i θ). T Σ 1 (y i µ ) 1 2 log } b e V ib { 1 2 (y µ ) T Σ 1 (y µ ) 1 i b b i b 2 log b } P i (V i θ) = e V i { 1 2 (y i µ ) Note : P i(v i = 1 θ) = 1. Also P i (V i θ) is biggest for the model µ, Σ the dt point. closest to Let V i V i P i (V i θ) = q i E-step: Q(θ θ (t) ) = q i { 1 2 (y i µ ) T Σ 1 (y i µ ) log }
12 12 PROF. ALAN YUILLE 13. M-step: minimize w.r.t. µ & gives µ = i 1 q ( i i q iy i ) weighted sum i q i(y i µ )(y i µ ) T = 1 i q i Hence, for this cse (mixture of Gussins) we cn perform the E nd M steps nlyticlly. Extend the probbility model, Let the number of clsses be rndom vrible k p(k) = exp λk Z[λ] p(y v, µ,, k)p(k)
Lecture 14. Clustering, K-means, and EM
Lecture 14. Clustering, K-means, and EM Prof. Alan Yuille Spring 2014 Outline 1. Clustering 2. K-means 3. EM 1 Clustering Task: Given a set of unlabeled data D = {x 1,..., x n }, we do the following: 1.
More informationContinuous Random Variables
STAT/MATH 395 A - PROBABILITY II UW Winter Qurter 217 Néhémy Lim Continuous Rndom Vribles Nottion. The indictor function of set S is rel-vlued function defined by : { 1 if x S 1 S (x) if x S Suppose tht
More informationChapter 5 : Continuous Random Variables
STAT/MATH 395 A - PROBABILITY II UW Winter Qurter 216 Néhémy Lim Chpter 5 : Continuous Rndom Vribles Nottions. N {, 1, 2,...}, set of nturl numbers (i.e. ll nonnegtive integers); N {1, 2,...}, set of ll
More informationSolution for Assignment 1 : Intro to Probability and Statistics, PAC learning
Solution for Assignment 1 : Intro to Probbility nd Sttistics, PAC lerning 10-701/15-781: Mchine Lerning (Fll 004) Due: Sept. 30th 004, Thursdy, Strt of clss Question 1. Bsic Probbility ( 18 pts) 1.1 (
More informationBayesian Networks: Approximate Inference
pproches to inference yesin Networks: pproximte Inference xct inference Vrillimintion Join tree lgorithm pproximte inference Simplify the structure of the network to mkxct inferencfficient (vritionl methods,
More information19 Optimal behavior: Game theory
Intro. to Artificil Intelligence: Dle Schuurmns, Relu Ptrscu 1 19 Optiml behvior: Gme theory Adversril stte dynmics hve to ccount for worst cse Compute policy π : S A tht mximizes minimum rewrd Let S (,
More informationMath Lecture 23
Mth 8 - Lecture 3 Dyln Zwick Fll 3 In our lst lecture we delt with solutions to the system: x = Ax where A is n n n mtrix with n distinct eigenvlues. As promised, tody we will del with the question of
More informationCS 188 Introduction to Artificial Intelligence Fall 2018 Note 7
CS 188 Introduction to Artificil Intelligence Fll 2018 Note 7 These lecture notes re hevily bsed on notes originlly written by Nikhil Shrm. Decision Networks In the third note, we lerned bout gme trees
More information1 Probability Density Functions
Lis Yn CS 9 Continuous Distributions Lecture Notes #9 July 6, 28 Bsed on chpter by Chris Piech So fr, ll rndom vribles we hve seen hve been discrete. In ll the cses we hve seen in CS 9, this ment tht our
More information1 Online Learning and Regret Minimization
2.997 Decision-Mking in Lrge-Scle Systems My 10 MIT, Spring 2004 Hndout #29 Lecture Note 24 1 Online Lerning nd Regret Minimiztion In this lecture, we consider the problem of sequentil decision mking in
More informationLecture 21: Order statistics
Lecture : Order sttistics Suppose we hve N mesurements of sclr, x i =, N Tke ll mesurements nd sort them into scending order x x x 3 x N Define the mesured running integrl S N (x) = 0 for x < x = i/n for
More informationProperties of Integrals, Indefinite Integrals. Goals: Definition of the Definite Integral Integral Calculations using Antiderivatives
Block #6: Properties of Integrls, Indefinite Integrls Gols: Definition of the Definite Integrl Integrl Clcultions using Antiderivtives Properties of Integrls The Indefinite Integrl 1 Riemnn Sums - 1 Riemnn
More informationNon-Linear & Logistic Regression
Non-Liner & Logistic Regression If the sttistics re boring, then you've got the wrong numbers. Edwrd R. Tufte (Sttistics Professor, Yle University) Regression Anlyses When do we use these? PART 1: find
More informationReinforcement Learning
Reinforcement Lerning Tom Mitchell, Mchine Lerning, chpter 13 Outline Introduction Comprison with inductive lerning Mrkov Decision Processes: the model Optiml policy: The tsk Q Lerning: Q function Algorithm
More informationReinforcement learning II
CS 1675 Introduction to Mchine Lerning Lecture 26 Reinforcement lerning II Milos Huskrecht milos@cs.pitt.edu 5329 Sennott Squre Reinforcement lerning Bsics: Input x Lerner Output Reinforcement r Critic
More informationMath 426: Probability Final Exam Practice
Mth 46: Probbility Finl Exm Prctice. Computtionl problems 4. Let T k (n) denote the number of prtitions of the set {,..., n} into k nonempty subsets, where k n. Argue tht T k (n) kt k (n ) + T k (n ) by
More informationAdministrivia CSE 190: Reinforcement Learning: An Introduction
Administrivi CSE 190: Reinforcement Lerning: An Introduction Any emil sent to me bout the course should hve CSE 190 in the subject line! Chpter 4: Dynmic Progrmming Acknowledgment: A good number of these
More informationMIXED MODELS (Sections ) I) In the unrestricted model, interactions are treated as in the random effects model:
1 2 MIXED MODELS (Sections 17.7 17.8) Exmple: Suppose tht in the fiber breking strength exmple, the four mchines used were the only ones of interest, but the interest ws over wide rnge of opertors, nd
More informationCS667 Lecture 6: Monte Carlo Integration 02/10/05
CS667 Lecture 6: Monte Crlo Integrtion 02/10/05 Venkt Krishnrj Lecturer: Steve Mrschner 1 Ide The min ide of Monte Crlo Integrtion is tht we cn estimte the vlue of n integrl by looking t lrge number of
More informationThe Regulated and Riemann Integrals
Chpter 1 The Regulted nd Riemnn Integrls 1.1 Introduction We will consider severl different pproches to defining the definite integrl f(x) dx of function f(x). These definitions will ll ssign the sme vlue
More informationRecitation 3: More Applications of the Derivative
Mth 1c TA: Pdric Brtlett Recittion 3: More Applictions of the Derivtive Week 3 Cltech 2012 1 Rndom Question Question 1 A grph consists of the following: A set V of vertices. A set E of edges where ech
More informationBellman Optimality Equation for V*
Bellmn Optimlity Eqution for V* The vlue of stte under n optiml policy must equl the expected return for the best ction from tht stte: V (s) mx Q (s,) A(s) mx A(s) mx A(s) Er t 1 V (s t 1 ) s t s, t s
More information3.4 Numerical integration
3.4. Numericl integrtion 63 3.4 Numericl integrtion In mny economic pplictions it is necessry to compute the definite integrl of relvlued function f with respect to "weight" function w over n intervl [,
More informationMonte Carlo method in solving numerical integration and differential equation
Monte Crlo method in solving numericl integrtion nd differentil eqution Ye Jin Chemistry Deprtment Duke University yj66@duke.edu Abstrct: Monte Crlo method is commonly used in rel physics problem. The
More informationLecture 3 Gaussian Probability Distribution
Introduction Lecture 3 Gussin Probbility Distribution Gussin probbility distribution is perhps the most used distribution in ll of science. lso clled bell shped curve or norml distribution Unlike the binomil
More informationReview of Calculus, cont d
Jim Lmbers MAT 460 Fll Semester 2009-10 Lecture 3 Notes These notes correspond to Section 1.1 in the text. Review of Clculus, cont d Riemnn Sums nd the Definite Integrl There re mny cses in which some
More informationOverview of Calculus I
Overview of Clculus I Prof. Jim Swift Northern Arizon University There re three key concepts in clculus: The limit, the derivtive, nd the integrl. You need to understnd the definitions of these three things,
More information2D1431 Machine Learning Lab 3: Reinforcement Learning
2D1431 Mchine Lerning Lb 3: Reinforcement Lerning Frnk Hoffmnn modified by Örjn Ekeberg December 7, 2004 1 Introduction In this lb you will lern bout dynmic progrmming nd reinforcement lerning. It is ssumed
More informationOrthogonal Polynomials
Mth 4401 Gussin Qudrture Pge 1 Orthogonl Polynomils Orthogonl polynomils rise from series solutions to differentil equtions, lthough they cn be rrived t in vriety of different mnners. Orthogonl polynomils
More informationNUMERICAL INTEGRATION. The inverse process to differentiation in calculus is integration. Mathematically, integration is represented by.
NUMERICAL INTEGRATION 1 Introduction The inverse process to differentition in clculus is integrtion. Mthemticlly, integrtion is represented by f(x) dx which stnds for the integrl of the function f(x) with
More informationThe First Fundamental Theorem of Calculus. If f(x) is continuous on [a, b] and F (x) is any antiderivative. f(x) dx = F (b) F (a).
The Fundmentl Theorems of Clculus Mth 4, Section 0, Spring 009 We now know enough bout definite integrls to give precise formultions of the Fundmentl Theorems of Clculus. We will lso look t some bsic emples
More informationDiscrete Mathematics and Probability Theory Spring 2013 Anant Sahai Lecture 17
EECS 70 Discrete Mthemtics nd Proility Theory Spring 2013 Annt Shi Lecture 17 I.I.D. Rndom Vriles Estimting the is of coin Question: We wnt to estimte the proportion p of Democrts in the US popultion,
More informationBases for Vector Spaces
Bses for Vector Spces 2-26-25 A set is independent if, roughly speking, there is no redundncy in the set: You cn t uild ny vector in the set s liner comintion of the others A set spns if you cn uild everything
More informationLecture 20: Numerical Integration III
cs4: introduction to numericl nlysis /8/0 Lecture 0: Numericl Integrtion III Instructor: Professor Amos Ron Scribes: Mrk Cowlishw, Yunpeng Li, Nthnel Fillmore For the lst few lectures we hve discussed
More informationTaylor Polynomial Inequalities
Tylor Polynomil Inequlities Ben Glin September 17, 24 Abstrct There re instnces where we my wish to pproximte the vlue of complicted function round given point by constructing simpler function such s polynomil
More informationIntegral equations, eigenvalue, function interpolation
Integrl equtions, eigenvlue, function interpoltion Mrcin Chrząszcz mchrzsz@cernch Monte Crlo methods, 26 My, 2016 1 / Mrcin Chrząszcz (Universität Zürich) Integrl equtions, eigenvlue, function interpoltion
More informationData Assimilation. Alan O Neill Data Assimilation Research Centre University of Reading
Dt Assimiltion Aln O Neill Dt Assimiltion Reserch Centre University of Reding Contents Motivtion Univrite sclr dt ssimiltion Multivrite vector dt ssimiltion Optiml Interpoltion BLUE 3d-Vritionl Method
More informationCS 188: Artificial Intelligence Spring 2007
CS 188: Artificil Intelligence Spring 2007 Lecture 3: Queue-Bsed Serch 1/23/2007 Srini Nrynn UC Berkeley Mny slides over the course dpted from Dn Klein, Sturt Russell or Andrew Moore Announcements Assignment
More informationMulti-Armed Bandits: Non-adaptive and Adaptive Sampling
CSE 547/Stt 548: Mchine Lerning for Big Dt Lecture Multi-Armed Bndits: Non-dptive nd Adptive Smpling Instructor: Shm Kkde 1 The (stochstic) multi-rmed bndit problem The bsic prdigm is s follows: K Independent
More informationNormal Distribution. Lecture 6: More Binomial Distribution. Properties of the Unit Normal Distribution. Unit Normal Distribution
Norml Distribution Lecture 6: More Binomil Distribution If X is rndom vrible with norml distribution with men µ nd vrince σ 2, X N (µ, σ 2, then P(X = x = f (x = 1 e 1 (x µ 2 2 σ 2 σ Sttistics 104 Colin
More information8 Laplace s Method and Local Limit Theorems
8 Lplce s Method nd Locl Limit Theorems 8. Fourier Anlysis in Higher DImensions Most of the theorems of Fourier nlysis tht we hve proved hve nturl generliztions to higher dimensions, nd these cn be proved
More informationChapters 4 & 5 Integrals & Applications
Contents Chpters 4 & 5 Integrls & Applictions Motivtion to Chpters 4 & 5 2 Chpter 4 3 Ares nd Distnces 3. VIDEO - Ares Under Functions............................................ 3.2 VIDEO - Applictions
More informationIntroduction to Group Theory
Introduction to Group Theory Let G be n rbitrry set of elements, typiclly denoted s, b, c,, tht is, let G = {, b, c, }. A binry opertion in G is rule tht ssocites with ech ordered pir (,b) of elements
More informationAdvanced Calculus: MATH 410 Notes on Integrals and Integrability Professor David Levermore 17 October 2004
Advnced Clculus: MATH 410 Notes on Integrls nd Integrbility Professor Dvid Levermore 17 October 2004 1. Definite Integrls In this section we revisit the definite integrl tht you were introduced to when
More informationp-adic Egyptian Fractions
p-adic Egyptin Frctions Contents 1 Introduction 1 2 Trditionl Egyptin Frctions nd Greedy Algorithm 2 3 Set-up 3 4 p-greedy Algorithm 5 5 p-egyptin Trditionl 10 6 Conclusion 1 Introduction An Egyptin frction
More informationTHE EXISTENCE-UNIQUENESS THEOREM FOR FIRST-ORDER DIFFERENTIAL EQUATIONS.
THE EXISTENCE-UNIQUENESS THEOREM FOR FIRST-ORDER DIFFERENTIAL EQUATIONS RADON ROSBOROUGH https://intuitiveexplntionscom/picrd-lindelof-theorem/ This document is proof of the existence-uniqueness theorem
More information1.4 Nonregular Languages
74 1.4 Nonregulr Lnguges The number of forml lnguges over ny lphbet (= decision/recognition problems) is uncountble On the other hnd, the number of regulr expressions (= strings) is countble Hence, ll
More informationLecture 1. Functional series. Pointwise and uniform convergence.
1 Introduction. Lecture 1. Functionl series. Pointwise nd uniform convergence. In this course we study mongst other things Fourier series. The Fourier series for periodic function f(x) with period 2π is
More information38.2. The Uniform Distribution. Introduction. Prerequisites. Learning Outcomes
The Uniform Distribution 8. Introduction This Section introduces the simplest type of continuous probbility distribution which fetures continuous rndom vrible X with probbility density function f(x) which
More information4 7x =250; 5 3x =500; Read section 3.3, 3.4 Announcements: Bell Ringer: Use your calculator to solve
Dte: 3/14/13 Objective: SWBAT pply properties of exponentil functions nd will pply properties of rithms. Bell Ringer: Use your clcultor to solve 4 7x =250; 5 3x =500; HW Requests: Properties of Log Equtions
More information12 TRANSFORMING BIVARIATE DENSITY FUNCTIONS
1 TRANSFORMING BIVARIATE DENSITY FUNCTIONS Hving seen how to trnsform the probbility density functions ssocited with single rndom vrible, the next logicl step is to see how to trnsform bivrite probbility
More informationODE: Existence and Uniqueness of a Solution
Mth 22 Fll 213 Jerry Kzdn ODE: Existence nd Uniqueness of Solution The Fundmentl Theorem of Clculus tells us how to solve the ordinry differentil eqution (ODE) du = f(t) dt with initil condition u() =
More informationRiemann Sums and Riemann Integrals
Riemnn Sums nd Riemnn Integrls Jmes K. Peterson Deprtment of Biologicl Sciences nd Deprtment of Mthemticl Sciences Clemson University August 26, 2013 Outline 1 Riemnn Sums 2 Riemnn Integrls 3 Properties
More informationReview of Gaussian Quadrature method
Review of Gussin Qudrture method Nsser M. Asi Spring 006 compiled on Sundy Decemer 1, 017 t 09:1 PM 1 The prolem To find numericl vlue for the integrl of rel vlued function of rel vrile over specific rnge
More information1 The Lagrange interpolation formula
Notes on Qudrture 1 The Lgrnge interpoltion formul We briefly recll the Lgrnge interpoltion formul. The strting point is collection of N + 1 rel points (x 0, y 0 ), (x 1, y 1 ),..., (x N, y N ), with x
More informationDiscrete Mathematics and Probability Theory Summer 2014 James Cook Note 17
CS 70 Discrete Mthemtics nd Proility Theory Summer 2014 Jmes Cook Note 17 I.I.D. Rndom Vriles Estimting the is of coin Question: We wnt to estimte the proportion p of Democrts in the US popultion, y tking
More informationFinite Automata Theory and Formal Languages TMV027/DIT321 LP4 2018
Finite Automt Theory nd Forml Lnguges TMV027/DIT321 LP4 2018 Lecture 10 An Bove April 23rd 2018 Recp: Regulr Lnguges We cn convert between FA nd RE; Hence both FA nd RE ccept/generte regulr lnguges; More
More informationRiemann Sums and Riemann Integrals
Riemnn Sums nd Riemnn Integrls Jmes K. Peterson Deprtment of Biologicl Sciences nd Deprtment of Mthemticl Sciences Clemson University August 26, 203 Outline Riemnn Sums Riemnn Integrls Properties Abstrct
More informationReview of basic calculus
Review of bsic clculus This brief review reclls some of the most importnt concepts, definitions, nd theorems from bsic clculus. It is not intended to tech bsic clculus from scrtch. If ny of the items below
More informationTutorial 4. b a. h(f) = a b a ln 1. b a dx = ln(b a) nats = log(b a) bits. = ln λ + 1 nats. = log e λ bits. = ln 1 2 ln λ + 1. nats. = ln 2e. bits.
Tutoril 4 Exercises on Differentil Entropy. Evlute the differentil entropy h(x) f ln f for the following: () The uniform distribution, f(x) b. (b) The exponentil density, f(x) λe λx, x 0. (c) The Lplce
More informationRiemann Integrals and the Fundamental Theorem of Calculus
Riemnn Integrls nd the Fundmentl Theorem of Clculus Jmes K. Peterson Deprtment of Biologicl Sciences nd Deprtment of Mthemticl Sciences Clemson University September 16, 2013 Outline Grphing Riemnn Sums
More informationMath 135, Spring 2012: HW 7
Mth 3, Spring : HW 7 Problem (p. 34 #). SOLUTION. Let N the number of risins per cookie. If N is Poisson rndom vrible with prmeter λ, then nd for this to be t lest.99, we need P (N ) P (N ) ep( λ) λ ln(.)
More informationSection 14.3 Arc Length and Curvature
Section 4.3 Arc Length nd Curvture Clculus on Curves in Spce In this section, we ly the foundtions for describing the movement of n object in spce.. Vector Function Bsics In Clc, formul for rc length in
More informationMath 8 Winter 2015 Applications of Integration
Mth 8 Winter 205 Applictions of Integrtion Here re few importnt pplictions of integrtion. The pplictions you my see on n exm in this course include only the Net Chnge Theorem (which is relly just the Fundmentl
More informationBest Approximation. Chapter The General Case
Chpter 4 Best Approximtion 4.1 The Generl Cse In the previous chpter, we hve seen how n interpolting polynomil cn be used s n pproximtion to given function. We now wnt to find the best pproximtion to given
More informationConvert the NFA into DFA
Convert the NF into F For ech NF we cn find F ccepting the sme lnguge. The numer of sttes of the F could e exponentil in the numer of sttes of the NF, ut in prctice this worst cse occurs rrely. lgorithm:
More informationRiemann is the Mann! (But Lebesgue may besgue to differ.)
Riemnn is the Mnn! (But Lebesgue my besgue to differ.) Leo Livshits My 2, 2008 1 For finite intervls in R We hve seen in clss tht every continuous function f : [, b] R hs the property tht for every ɛ >
More informationReasoning with Bayesian Networks
Complexity of Probbilistic Inference Compiling Byesin Networks Resoning with Byesin Networks Lecture 5: Complexity of Probbilistic Inference, Compiling Byesin Networks Jinbo Hung NICTA nd ANU Jinbo Hung
More informationPredict Global Earth Temperature using Linier Regression
Predict Globl Erth Temperture using Linier Regression Edwin Swndi Sijbt (23516012) Progrm Studi Mgister Informtik Sekolh Teknik Elektro dn Informtik ITB Jl. Gnesh 10 Bndung 40132, Indonesi 23516012@std.stei.itb.c.id
More informationFinite Automata. Informatics 2A: Lecture 3. John Longley. 22 September School of Informatics University of Edinburgh
Lnguges nd Automt Finite Automt Informtics 2A: Lecture 3 John Longley School of Informtics University of Edinburgh jrl@inf.ed.c.uk 22 September 2017 1 / 30 Lnguges nd Automt 1 Lnguges nd Automt Wht is
More informationHandout: Natural deduction for first order logic
MATH 457 Introduction to Mthemticl Logic Spring 2016 Dr Json Rute Hndout: Nturl deduction for first order logic We will extend our nturl deduction rules for sententil logic to first order logic These notes
More informationQuadratic Forms. Quadratic Forms
Qudrtic Forms Recll the Simon & Blume excerpt from n erlier lecture which sid tht the min tsk of clculus is to pproximte nonliner functions with liner functions. It s ctully more ccurte to sy tht we pproximte
More informationSUMMER KNOWHOW STUDY AND LEARNING CENTRE
SUMMER KNOWHOW STUDY AND LEARNING CENTRE Indices & Logrithms 2 Contents Indices.2 Frctionl Indices.4 Logrithms 6 Exponentil equtions. Simplifying Surds 13 Opertions on Surds..16 Scientific Nottion..18
More informationRead section 3.3, 3.4 Announcements:
Dte: 3/1/13 Objective: SWBAT pply properties of exponentil functions nd will pply properties of rithms. Bell Ringer: 1. f x = 3x 6, find the inverse, f 1 x., Using your grphing clcultor, Grph 1. f x,f
More informationParse trees, ambiguity, and Chomsky normal form
Prse trees, miguity, nd Chomsky norml form In this lecture we will discuss few importnt notions connected with contextfree grmmrs, including prse trees, miguity, nd specil form for context-free grmmrs
More informationCMPSCI 250: Introduction to Computation. Lecture #31: What DFA s Can and Can t Do David Mix Barrington 9 April 2014
CMPSCI 250: Introduction to Computtion Lecture #31: Wht DFA s Cn nd Cn t Do Dvid Mix Brrington 9 April 2014 Wht DFA s Cn nd Cn t Do Deterministic Finite Automt Forml Definition of DFA s Exmples of DFA
More informationMORE FUNCTION GRAPHING; OPTIMIZATION. (Last edited October 28, 2013 at 11:09pm.)
MORE FUNCTION GRAPHING; OPTIMIZATION FRI, OCT 25, 203 (Lst edited October 28, 203 t :09pm.) Exercise. Let n be n rbitrry positive integer. Give n exmple of function with exctly n verticl symptotes. Give
More informationEnergy Bands Energy Bands and Band Gap. Phys463.nb Phenomenon
Phys463.nb 49 7 Energy Bnds Ref: textbook, Chpter 7 Q: Why re there insultors nd conductors? Q: Wht will hppen when n electron moves in crystl? In the previous chpter, we discussed free electron gses,
More informationWe partition C into n small arcs by forming a partition of [a, b] by picking s i as follows: a = s 0 < s 1 < < s n = b.
Mth 255 - Vector lculus II Notes 4.2 Pth nd Line Integrls We begin with discussion of pth integrls (the book clls them sclr line integrls). We will do this for function of two vribles, but these ides cn
More informationInfinite Geometric Series
Infinite Geometric Series Finite Geometric Series ( finite SUM) Let 0 < r < 1, nd let n be positive integer. Consider the finite sum It turns out there is simple lgebric expression tht is equivlent to
More informationGoals: Determine how to calculate the area described by a function. Define the definite integral. Explore the relationship between the definite
Unit #8 : The Integrl Gols: Determine how to clculte the re described by function. Define the definite integrl. Eplore the reltionship between the definite integrl nd re. Eplore wys to estimte the definite
More informationJim Lambers MAT 169 Fall Semester Lecture 4 Notes
Jim Lmbers MAT 169 Fll Semester 2009-10 Lecture 4 Notes These notes correspond to Section 8.2 in the text. Series Wht is Series? An infinte series, usully referred to simply s series, is n sum of ll of
More informationThings to Memorize: A Partial List. January 27, 2017
Things to Memorize: A Prtil List Jnury 27, 2017 Chpter 2 Vectors - Bsic Fcts A vector hs mgnitude (lso clled size/length/norm) nd direction. It does not hve fixed position, so the sme vector cn e moved
More informationMath 1B, lecture 4: Error bounds for numerical methods
Mth B, lecture 4: Error bounds for numericl methods Nthn Pflueger 4 September 0 Introduction The five numericl methods descried in the previous lecture ll operte by the sme principle: they pproximte the
More informationMath 360: A primitive integral and elementary functions
Mth 360: A primitive integrl nd elementry functions D. DeTurck University of Pennsylvni October 16, 2017 D. DeTurck Mth 360 001 2017C: Integrl/functions 1 / 32 Setup for the integrl prtitions Definition:
More information4.4 Areas, Integrals and Antiderivatives
. res, integrls nd ntiderivtives 333. Ares, Integrls nd Antiderivtives This section explores properties of functions defined s res nd exmines some connections mong res, integrls nd ntiderivtives. In order
More informationNumerical integration
2 Numericl integrtion This is pge i Printer: Opque this 2. Introduction Numericl integrtion is problem tht is prt of mny problems in the economics nd econometrics literture. The orgniztion of this chpter
More informationMath 520 Final Exam Topic Outline Sections 1 3 (Xiao/Dumas/Liaw) Spring 2008
Mth 520 Finl Exm Topic Outline Sections 1 3 (Xio/Dums/Liw) Spring 2008 The finl exm will be held on Tuesdy, My 13, 2-5pm in 117 McMilln Wht will be covered The finl exm will cover the mteril from ll of
More informationTheoretical foundations of Gaussian quadrature
Theoreticl foundtions of Gussin qudrture 1 Inner product vector spce Definition 1. A vector spce (or liner spce) is set V = {u, v, w,...} in which the following two opertions re defined: (A) Addition of
More information1B40 Practical Skills
B40 Prcticl Skills Comining uncertinties from severl quntities error propgtion We usully encounter situtions where the result of n experiment is given in terms of two (or more) quntities. We then need
More informationPhysics 201 Lab 3: Measurement of Earth s local gravitational field I Data Acquisition and Preliminary Analysis Dr. Timothy C. Black Summer I, 2018
Physics 201 Lb 3: Mesurement of Erth s locl grvittionl field I Dt Acquisition nd Preliminry Anlysis Dr. Timothy C. Blck Summer I, 2018 Theoreticl Discussion Grvity is one of the four known fundmentl forces.
More information{ } = E! & $ " k r t +k +1
Chpter 4: Dynmic Progrmming Objectives of this chpter: Overview of collection of clssicl solution methods for MDPs known s dynmic progrmming (DP) Show how DP cn be used to compute vlue functions, nd hence,
More information20 MATHEMATICS POLYNOMIALS
0 MATHEMATICS POLYNOMIALS.1 Introduction In Clss IX, you hve studied polynomils in one vrible nd their degrees. Recll tht if p(x) is polynomil in x, the highest power of x in p(x) is clled the degree of
More informationCS 188: Artificial Intelligence Fall Announcements
CS 188: Artificil Intelligence Fll 2009 Lecture 20: Prticle Filtering 11/5/2009 Dn Klein UC Berkeley Announcements Written 3 out: due 10/12 Project 4 out: due 10/19 Written 4 proly xed, Project 5 moving
More informationSpectral Regularization for Max-Margin Sequence Tagging
Spectrl Regulriztion for Mx-Mrgin Sequence Tgging Aridn Quttoni Borj Blle Xvier Crrers Amir Globerson pq Universitt Politècnic de Ctluny now t p q McGill University p q The Hebrew University of Jeruslem
More informationChapter 4: Dynamic Programming
Chpter 4: Dynmic Progrmming Objectives of this chpter: Overview of collection of clssicl solution methods for MDPs known s dynmic progrmming (DP) Show how DP cn be used to compute vlue functions, nd hence,
More informationp(t) dt + i 1 re it ireit dt =
Note: This mteril is contined in Kreyszig, Chpter 13. Complex integrtion We will define integrls of complex functions long curves in C. (This is bit similr to [relvlued] line integrls P dx + Q dy in R2.)
More informationdt. However, we might also be curious about dy
Section 0. The Clculus of Prmetric Curves Even though curve defined prmetricly my not be function, we cn still consider concepts such s rtes of chnge. However, the concepts will need specil tretment. For
More informationTopic 1 Notes Jeremy Orloff
Topic 1 Notes Jerem Orloff 1 Introduction to differentil equtions 1.1 Gols 1. Know the definition of differentil eqution. 2. Know our first nd second most importnt equtions nd their solutions. 3. Be ble
More informationThe Wave Equation I. MA 436 Kurt Bryan
1 Introduction The Wve Eqution I MA 436 Kurt Bryn Consider string stretching long the x xis, of indeterminte (or even infinite!) length. We wnt to derive n eqution which models the motion of the string
More information