Support Vector Machines. Jie Tang Knowledge Engineering Group Department of Computer Science and Technology Tsinghua University 2012
|
|
- Dora Eaton
- 5 years ago
- Views:
Transcription
1 Support Vector Machnes Je Tang Knowledge Engneerng Group Department of Computer Scence and Technology Tsnghua Unversty
2 Outlne What s a Support Vector Machne? Solvng SVMs Kernel Trcks 2
3 What s a Support Vector Machne SVM s related to statstcal learnng theory [3] SVM was frst ntroduced n 1992 [1] SVM becomes popular because of ts success n handwrtten dgt recognton 1.1% test error rate for SVM. Ths s the same as the error rates of a carefully constructed neural network, Leet 4. See Secton 5.11 n [2] or the dscusson n [3] for detals SVM s now regarded as an mportant example of kernel methods, one of the key area n machne learnng [1] B.E. Boser et al. A Tranng Algorthm for Optmal Margn Classfers. Proceedngs of the Ffth Annual Workshop on Computatonal Learnng Theory , Pttsburgh, [2] L. Bottou et al. Comparson of classfer methods: a case study n handwrtten dgt recognton. Proceedngs of the 12th IAPR Internatonal Conference on Pattern Recognton, vol. 2, pp , [3] V. Vapnk. The ature of Statstcal Learnng Theory. 2 nd edton, Sprnger,
4 Classfcaton Problem Gven a tranng set S={(x 1, y 1 ),(x 2, y 2 ),,(x, y )}, and x X=R m, =1,2,, To learn a functon g(x), and make the decson functon f(x)=sgn(g(x)) can classfy new nput x So ths s a supervsed batch learnng method Lnear classfer g(x) = (w T x + b) # sgn(g(x)) = 1,g(x) 0 & $ ' % 1,g(x) < 0( f (x) = sgn(g(x)) 4
5 What s a good Decson Boundary? Consder a two-class, lnearly separable classfcaton problem Many decson boundares! The Perceptron algorthm can be used to fnd such a boundary Dfferent algorthms have been proposed Are all decson boundares equally good? Class +1 Class -1 5
6 6 Geometrc Interpretaton
7 Affne Set Lne through x 1 and x 2 : all ponts x =θx 1 + (1 θ)x 2 θ R Affne set: contans the lne through any two dstnct ponts n the set Affne functon: f: R n ->R m s affne f f(x)=ax+b wth A number of the followng pages are from Boyd s sldes A R m n,b R m 7
8 Convex Set Lne segment between x 1 and x 2 : all ponts x =θx 1 + (1 θ)x 2 0 θ 1 Convex set: contans lne segment between any two ponts n the set x 1, x 2 C, 0 θ 1 =>θx 1 + (1 θ)x 2 C Examples (one convex, two nonconvex sets) 8
9 Hyperplanes and Halfspaces Hyperplane: set of the form {x a T x = b}(a 0) Halfspace: set of the form {x a T x b}(a 0) a s the normal vector Hyperplanes are affne and convex; halfspaces are convex. 9
10 Bsector based Decson Boundary k k conv( S) = x = λjx j λj = 1, λj 0, j = 1, L, k j= 1 j= 1 d Class -1 Class +1 c m 10
11 Formalzaton 1 1 mn β c d mnβ β x β x 2 2 st.. β = 1, β = 1 y = 1 y = 1 0 β 1, = [1, m] j j j j y = 1 y = 1 j The objectve s to solve all the β,. Then we can obtan the two ponts havng the closest dstance by c= β x d = β x j j y = 1 y = 1 ext we compute the hyperplane w T x + b = 0 by m 1 w= c d = βyx b= (( c d ) g ( c+ d )) 2 Fnally, we make predcton by f( x) = sgn(( w T x) + b) j 11
12 12 Maxmal Margn
13 Large-margn Decson Boundary The decson boundary should be as far away from the data of both classes as possble We should maxmze the margn, m Dstance between the orgn and the lne w T x=-b s -b/ w Class -1 m = 2γ w Class +1 m T wx+ b= γ 13 T wx+ b= γ
14 Formalzaton maxγ, wb, 2γ w equal to mn γ,w,b w 2γ ote: we have constrants s.t. w T x () + b γ,1 k, y =1 w T x ( j) + b < γ,k < j, y j = 1 equal to y () (w T x () + b) γ,1 Snce we can arbtrarly scale w and b wthout changng anythng, We ntroduce the scalng constrant γ=1 mn w,b w 2 s.t. Change to 2-norm space w 2 y () (w T x () + b) 1,1 2 loss functon Ths s a constraned optmzaton problem. 14
15 Loss Functon Then we result n the Lasso loss functon mn w,b 1 2 w 2 s.t. y () (b + w T x () ) 1 Another popular loss func: Hnge loss + penalty mn w,b [1 y () (b + w T x)] + + λ 2 =1 w 2 15
16 16 Loss Functon (cont.) Emprcal loss functon Structural loss functon mn w,b [1 y () (b + wx () )] + =1 mn w,b [1 y () (b + wx () )] + + λ 2 w 2 =1 where w 2 ndcates the complexty of the model, t s also named as penalty There are many knds of formulaton of the loss functon
17 Optmal Margn Classfers For the problem: w 2 mn w,b 2 s.t. y () (w T x () + b) 1,1 We can wrte lagrangan form: L(w,b,α) = w 2 2 α [ y() (w T x () + b) 1] s.t. α 0,1 WHY? Let us revew generalzed Lagrangan =0 17
18 Revew Convex Optmzaton and Lagrange Dualty 18
19 Convex Functon f: R n ->R s convex f dom(f) s a convex set and f (θx + (1 θ)y) θ f (x) + (1 θ) f ( y) for all x, y dom( f ),0 θ 1 f s concave f f s convex f s strctly convex f dom(f) s convex and f (θx + (1 θ)y) <θ f (x) + (1 θ) f ( y) for all x, y dom( f ), x y,0 θ 1 19
20 Frst-order Condton f s dfferentable f dom(f) s open and the gradent exsts at each # f (x) = % $ f (x) x 1 x dom( f ), f (x) x 2 1 st -order condton: dfferentable f wth convex doman s convex ff f ( y) f (x) + f (x) T *( y x),..., f (x) x n & ( ' for all x, y dom( f ) 20
21 Second-order Condton f s twce dfferentable f dom(f) s open and the Hessan exsts at each 2 nd -order condton: for twce dfferentable f wth convex doman f s convex ff for all 2 f (x) j = 2 f (x) x x j,, j =1,...,, x dom( f ) x dom( f ) 2 f (x) 0 x dom( f ) If 2 f (x) > 0 for all, then f s strctly convex. 21
22 Convex Optmzaton Problem 22 Standard form convex optmzaton problem f 0,f 1,,f k are convex; Equalty constrants are affne Important property: feasble set of a convex optmzaton problem s convex Example mn f ( x) 0 st.. f ( x) 0, = 1,..., k T a x b = 0, = 1,..., l mn f ( x) = x + x st f x ( ) = 2 0 (1 + xx ) x h x x x 2 1( ) = ( 1 + 2) = 0 f 0 s convex; feasble set {(x 1, x 2 ) x 1 = x 2 0} s convex ot a convex problem snce f 1 s not convex, h 1 s not affne
23 Lagrange Dualty When solvng optmzaton problems wth constrants, lagrange dualty s always used to obtan the soluton of the prmal problem through solvng the dual problem. Prmal optmzaton problem If f(x), c (x), h j (x) are contnuously dfferentable functons defned n R n, then the followng optmzaton problem s called prmal optmzaton problem mn f( x) x R n st.. g ( x) 0, = 1,..., k h ( x) = 0, j = 1,..., l j 23
24 24 Prmal Optmzaton Problem To solve the prmal optmzaton problem, we defne the generalzed Lagrangan: Lx (, αβ, ) = f() x αg() x βh() x st.. α 0 where α and β j are Lagrange multplers. Consder the functon: Assume some x volates any of the prmal constrants (.e., f ether g (x)<0 or h j (x) 0 for some ), then we can verfy that Snce f g (x)<0 for some, we can set α as + ; f h j (x) 0 for some, we can set β j h j (x) as +, and set other α and β j as 0. In contrast, f the constrants are ndeed satsfed for a partcular value of x, then Therefore: θ P (x) = max L(x,α,β) α,β:α 0 k j j = 1 j= 1 Here, f we consder the mnmzaton problem l mn f( x) n x R θ ( x) = max [ f( x) α g ( x) β h ( x)] = P j j αβα, : 0 = 1 j= 1 k " $ θ P (x) = # f (x) %$ mn x θ P (x) = mn x l If x satsfes prmal constrants otherwse max α,β:α 0 Here P stands for prmal. L(x,α,β) st.. g ( x) 0, = 1,..., k h ( x) = 0, j = 1,..., l j Prmal problem We see that the prmal problem s represented by the mn max problem of the generalzed Lagrangan,.e., p* = mnθ x P (x)
25 Dual optmzaton problem: Dual Optmzaton Problem max θ α,β:α 0 D (α,β) = max mn L(x,α,β) α,β:α 0 x Ths s exactly the same as the prmal problem, except that the order of the max and the mn are now exchanged. We also defne the optmal value of the dual problem s objectve to be How are the prmal and the dual problems related? It can easly shown that: Proof: d* = max θ α,β:α 0 D (x) d* = max mn α,β:α 0 x L(x,α,β) mn x max α,β:α 0 L(x,α,β) = p * θ D (α,β) = mn L(x,α,β) L(x,α,β) max L(x,α,β) =θ x α,β:α 0 P (x) So θ D (α,β) θ P (x) Because the prmal and dual problem both have the optmal value, thus max θ α,β:α 0 D (α,β) mnθ x P (x) mn f( x) n x R st.. g ( x) 0, = 1,..., k h ( x) = 0, j = 1,..., l j Prmal problem.e., d* = max mn α,β:α 0 x L(x,α,β) mn x max α,β:α 0 L(x,α,β) = p * 25
26 KKT Condtons 26 Under certan condtons, we wll have d*=p* So that we can solve the dual problem n leu of the prmal problem. Then what s the condtons? Suppose (1) f and g are convex, and h (x) s affne. (2) the constrants g are (strctly) feasble; ths means that there exsts some x so that g (x) >0 for all. Under the above assumptons, there must exst x*, α* and β* so that x* s the soluton to the prmal problem, α*, β* are the soluton to the dual problem and p*=d*=l(x*, α*, β* ). The necessary and suffcent condtons are KKT (Karush-Kuhn-Tucher) condtons: L(x *,α *,β * ) = 0, [1, ] x L(x *,α *,β * ) α L(x *,α *,β * ) β = 0, [1,k] = 0, [1,l] α * g (x * ) = 0, [1,k] g (x * ) 0, [1,k] α * 0, [1,k] KKT dual complementarty condton. If α *>0, then g (x)=0
27 Back to Our Optmal Margn Classfers 27
28 Optmal Margn Classfers For the problem: w 2 mn w,b 2 s.t. y () (w T x () + b) 1,1 We can wrte lagrangan form: L(w,b,α) = w 2 2 α [ y() (w T x () + b) 1] s.t. α 0,1 Then our problem becomes: wb, =0 mn max Lwb (,, α) α If certan constrants are satsfed, then we have max mn Lwb α wb, (,, α) 28
29 Solve the Dual Problem max α mn w,b L(w,b,α) = w 2 s.t. α 0,1 2 α [ y() (w T x () + b) 1] Let us frst solve the nsde mnmal problem by settng the gradent of L(w, b, a) w.r.t. w and b to zero, we have =1 Lwb (,, α) w () () = w α y x = = 1 w= α y x = 1 () () 0 Lwb (,, α) b () = α y = = 1 0 Then let us substtute the two equatons nto L(w, b, a) to solve the maxmal problem 29
30 Solve the Dual Problem ( ) w = α y x ow we have: ( ) =1 and α y ( ) =0 =1 w = α y ( ) x ( ) back to L(w, b, a) Substtute =1 1 ( ) ( ) ( j) ( j) ( ) L(b,α ) = α y x α j y x α [ y ( α j y ( j ) x ( j ) x ( ) + b) 1] 2 =1 j=1 =1 j=1 1 ( ) ( j ) ( ) ( j ) ( ) ( j ) ( j ) ( ) ( ) = α α j y y x x α y α j y x x b α y + α 2 =1 j=1 =1 j=1 =1 =0 1 ( ) ( j ) ( ) ( j ) ( ) = α α j y y x x b α y + α 2 =1 j=1 =1 =1 because α y ( ) = 0, we obtan =1 1 ( ) ( j ) ( ) ( j ) L(α ) = α α j y y x x + α 2 =1 j=1 =1 The new objectve functon s a functon of a only It s known as the dual problem: f we know w, we know all a; vce versa 30
31 The Dual Problem (cont.) The orgnal problem, also known as prmal problem w 2 mn w,b 2 s.t. y () (w T x () + b) 1,1 mn w,b max α L(w,b,α) = w 2 s.t. α 0,1 2 α [ y() (w T x () + b) 1] =1 The dual problem max mn (,, α) α Lwb wb, max α 1 α 2 α j y () y ( j) x () x ( j) + α =1 j=1 s.t. α > 0,1 ; α y = 0 =1 =1 Propertes of a when we ntroduce the Lagrange multplers The result when we dfferentate the orgnal Lagrangan w.r.t. b 31
32 Relatonshp between Prmal and Dual Problems d = max mn L( x, αβ, ) mn max L( x, αβ, ) = p * * αβ, x x αβ, ote: f under some condtons, d*=p* We can solve the dual problem n leu of the prmal problem What s the condtons? The famous KKT condtons (Karush-Kuhn-Tucker condtons) Lw b w * * * (,, α ) Lw b b * * * (,, α ) * * α ( ) ( ) = 1 * ( ) = α y = 0 (2) In our case = 1 * ( ) * ( ) * () * () * y w x + b * = w y x = 0, [1, ] (1) ( ) 1 0, [1, ] 0, [1, ] α ( y ( w x + b ) 1) = 0, [1, ] (3) α What s KKT In Lagrangan formula Lx (, αβ, ) = f() x αg() x βh() x st.. α 0 k KKT condtons are j j = 0 j= 0 l L(x *,α *,β * ) x = 0, [1, ] L(x *,α *,β * ) α L(x *,α *,β * ) β = 0, [1,k] = 0, [1,l] α * g (x * ) = 0, [1,k] g (x * ) 0, [1,k] α * 0, [1,k] 32
33 ow We Have Then, what we have the maxmum optmum problem wth respect to α: 33 Ths s a quadratc programmng (QP) problem, A global maxmum of a can always be found Then solve w by Fnally solve b: Snce there s at least one α j* >0 (f all α j* =0, from equaton (1) we know that w*=0, however w*=0 s not the optmal soluton). Then from equaton (3), we know that Because y (j) y (j) =1, then max α 1 α 2 α j y () y ( j) x () x ( j) + α w = j=1 =1 α y () x () y ( j) (w * x ( j) + b * ) 1= 0 ( j) * ( j) ( j) ( ) ( ) ( j) α = 1 b= y w x = y y x x =1 s.t. α > 0,1 ; α y = 0 =1 =1 Characterstcs of the Soluton Many of the a are zero - w s a lnear combnaton of a small number of data ponts - Ths sparse representaton can be vewed as data compresson as n the constructon of knn classfer x wth non-zero a are called support vectors (SV) - The decson boundary s determned only by the SV
34 A Geometrcal Interpretaton Class -1 α 8 =0.6 α 10 =0 α 5 =0 α 7 =0 α 2 =0 α 4 =0 α 9 =0 Class +1 α 3 =0 α 6 =1.4 α 1 =0.8 34
35 How to Predct For a new sample x, we can predct t by: T ( α () () T ) = 1 wx+ b= y x x+ b = α + = 1 () () y x, x b classfy x as class +1 f the sum s postve, and class -1 otherwse ote: w need not be formed explctly 35
36 on-separable What s non-separable case? Class -1 We allow error ξ n classfcaton; t s based on the output of the dscrmnant functon w T x+b ξ approxmates the number of msclassfed samples Class +1 36
37 on-lnear Cases What s non-lnear case? 37
38 on-separable case The formalzaton of the optmal problem becomes : mn w,b,ξ s.t. w 2 +C ξ =1 y(w T x () + b) 1 ξ,1 ξ 0,1 Thus, examples are now permtted to have margn less than 1, and f an example has functonal margn 1-ξ (wth ξ>0), we would pay a cost of the objectve functon by ncreased by Cξ. The parameter C controls the relatve weghtng between the twn goals of makng the w 2 small and of ensurng that most examples have functonal margn at least 1. 38
39 Lagrangan Soluton Agan, we have the lagrangan form : L(w,b,ξ,α,σ ) = w 2 +C ξ α [ y(w T x () + b) 1+ξ ] σ ξ s.t. σ 0;α 0 max α L(α) = α 1 2 =1, j=1 s.t. C α 0, [1, ] α y () = 0 =1 α = + α () T () 0 y ( w x b) 1 () T () = C y w x + b ( ) 1 () T () 0 < α < C y ( w x + b) = 1 =1 =1 y () y ( j) α α j x (), x ( j) What s the dfference from the separable form??!! + =1 KKT condtons: L(w,b,α) w = 0, [1, ] L(w,b,α) ξ = 0, [1, ] L(w,b,α) = 0 b α ( y () (w T x () + b) 1 ξ ) = 0, [1, ] y () (w T x () + b) 1 ξ 0, [1, ] α 0,σ 0, [1, ] 39
40 How to tran SVM Solvng the quadratc programmng optmzaton problem drectly to tran the SVM s very slow when the tranng data grows large. Sequental mnmal optmzaton (SMO) algorthm, due to John Platt. Frst, let us ntroduce coordnate ascent algorthm: Loop untl convergence: { For =1,, m { a :=argmax a L(a 1,, a -1, a, a +1,, a m ) } } 40
41 Coordnate ascent s ok? Is t suffcent? α 1 y (1) = α 1 = y (1) =2 =2 α y () α y () 41
42 SMO Change the algorthm by: ths s just SMO Repeat untl convergence { 1. select some par a and a j to update next. (usng a heurstc that tres to pck the two that wll allow us to make the bggest progress towards the global maxmum). 2. reoptmze L(a) wth respect to a and a j, whle holdng all the other a. } Many approaches have been proposed -e.g., Loqo, cplex, etc. (see α 1 y (1) +α 2 y (2) = =3 α y () = ς α 1 = (ς α 2 y (2) )y (1) L(a) = L((ς α 2 y (2) )y (1),α 2,...,α ) 42
43 SMO(2) La ( ) = L(( ς α y ) y, α,..., α ) (2) (1) 2 2 m Ths s a quadratc functon n a 2. I.e. t can be wrtten as: aα + bα + c
44 Solvng a 2 aα + bα + c For the quadratc functon, we can smply solve t by settng ts dervatve to zero. Let us use a 2 new, unclpped as the resultng value. H f > H α α α new, unclpped L f( α2 < L) new, unclpped ( α2 ) new new new, unclpped 2 = 2 f ( L 2 H) Havng fnd a 2, we can go back to fnd the optmal a 1. Please read Platt s paper f you want to read more detals 44
45 Thanks! HP: Emal: 45
Lecture 10 Support Vector Machines. Oct
Lecture 10 Support Vector Machnes Oct - 20-2008 Lnear Separators Whch of the lnear separators s optmal? Concept of Margn Recall that n Perceptron, we learned that the convergence rate of the Perceptron
More informationNatural Language Processing and Information Retrieval
Natural Language Processng and Informaton Retreval Support Vector Machnes Alessandro Moschtt Department of nformaton and communcaton technology Unversty of Trento Emal: moschtt@ds.untn.t Summary Support
More informationLecture 10 Support Vector Machines II
Lecture 10 Support Vector Machnes II 22 February 2016 Taylor B. Arnold Yale Statstcs STAT 365/665 1/28 Notes: Problem 3 s posted and due ths upcomng Frday There was an early bug n the fake-test data; fxed
More informationSupport Vector Machines CS434
Support Vector Machnes CS434 Lnear Separators Many lnear separators exst that perfectly classfy all tranng examples Whch of the lnear separators s the best? + + + + + + + + + Intuton of Margn Consder ponts
More informationSupport Vector Machines
Support Vector Machnes Konstantn Tretyakov (kt@ut.ee) MTAT.03.227 Machne Learnng So far So far Supervsed machne learnng Lnear models Non-lnear models Unsupervsed machne learnng Generc scaffoldng So far
More informationMaximal Margin Classifier
CS81B/Stat41B: Advanced Topcs n Learnng & Decson Makng Mamal Margn Classfer Lecturer: Mchael Jordan Scrbes: Jana van Greunen Corrected verson - /1/004 1 References/Recommended Readng 1.1 Webstes www.kernel-machnes.org
More informationSupport Vector Machines. Vibhav Gogate The University of Texas at dallas
Support Vector Machnes Vbhav Gogate he Unversty of exas at dallas What We have Learned So Far? 1. Decson rees. Naïve Bayes 3. Lnear Regresson 4. Logstc Regresson 5. Perceptron 6. Neural networks 7. K-Nearest
More informationWhich Separator? Spring 1
Whch Separator? 6.034 - Sprng 1 Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng 3 Margn of a pont " # y (w $ + b) proportonal
More informationSupport Vector Machines
Support Vector Machnes Konstantn Tretyakov (kt@ut.ee) MTAT.03.227 Machne Learnng So far Supervsed machne learnng Lnear models Least squares regresson Fsher s dscrmnant, Perceptron, Logstc model Non-lnear
More information18-660: Numerical Methods for Engineering Design and Optimization
8-66: Numercal Methods for Engneerng Desgn and Optmzaton n L Department of EE arnege Mellon Unversty Pttsburgh, PA 53 Slde Overve lassfcaton Support vector machne Regularzaton Slde lassfcaton Predct categorcal
More informationUVA CS / Introduc8on to Machine Learning and Data Mining. Lecture 10: Classifica8on with Support Vector Machine (cont.
UVA CS 4501-001 / 6501 007 Introduc8on to Machne Learnng and Data Mnng Lecture 10: Classfca8on wth Support Vector Machne (cont. ) Yanjun Q / Jane Unversty of Vrgna Department of Computer Scence 9/6/14
More informationKristin P. Bennett. Rensselaer Polytechnic Institute
Support Vector Machnes and Other Kernel Methods Krstn P. Bennett Mathematcal Scences Department Rensselaer Polytechnc Insttute Support Vector Machnes (SVM) A methodology for nference based on Statstcal
More informationMultilayer Perceptron (MLP)
Multlayer Perceptron (MLP) Seungjn Cho Department of Computer Scence and Engneerng Pohang Unversty of Scence and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjn@postech.ac.kr 1 / 20 Outlne
More informationKernel Methods and SVMs Extension
Kernel Methods and SVMs Extenson The purpose of ths document s to revew materal covered n Machne Learnng 1 Supervsed Learnng regardng support vector machnes (SVMs). Ths document also provdes a general
More informationLagrange Multipliers Kernel Trick
Lagrange Multplers Kernel Trck Ncholas Ruozz Unversty of Texas at Dallas Based roughly on the sldes of Davd Sontag General Optmzaton A mathematcal detour, we ll come back to SVMs soon! subject to: f x
More informationLecture 3: Dual problems and Kernels
Lecture 3: Dual problems and Kernels C4B Machne Learnng Hlary 211 A. Zsserman Prmal and dual forms Lnear separablty revsted Feature mappng Kernels for SVMs Kernel trck requrements radal bass functons SVM
More informationSupport Vector Machines
CS 2750: Machne Learnng Support Vector Machnes Prof. Adrana Kovashka Unversty of Pttsburgh February 17, 2016 Announcement Homework 2 deadlne s now 2/29 We ll have covered everythng you need today or at
More informationprinceton univ. F 17 cos 521: Advanced Algorithm Design Lecture 7: LP Duality Lecturer: Matt Weinberg
prnceton unv. F 17 cos 521: Advanced Algorthm Desgn Lecture 7: LP Dualty Lecturer: Matt Wenberg Scrbe: LP Dualty s an extremely useful tool for analyzng structural propertes of lnear programs. Whle there
More informationSolutions HW #2. minimize. Ax = b. Give the dual problem, and make the implicit equality constraints explicit. Solution.
Solutons HW #2 Dual of general LP. Fnd the dual functon of the LP mnmze subject to c T x Gx h Ax = b. Gve the dual problem, and make the mplct equalty constrants explct. Soluton. 1. The Lagrangan s L(x,
More informationLinear Classification, SVMs and Nearest Neighbors
1 CSE 473 Lecture 25 (Chapter 18) Lnear Classfcaton, SVMs and Nearest Neghbors CSE AI faculty + Chrs Bshop, Dan Klen, Stuart Russell, Andrew Moore Motvaton: Face Detecton How do we buld a classfer to dstngush
More informationSupport Vector Machines CS434
Support Vector Machnes CS434 Lnear Separators Many lnear separators exst that perfectly classfy all tranng examples Whch of the lnear separators s the best? Intuton of Margn Consder ponts A, B, and C We
More informationLecture 6: Support Vector Machines
Lecture 6: Support Vector Machnes Marna Melă mmp@stat.washngton.edu Department of Statstcs Unversty of Washngton November, 2018 Lnear SVM s The margn and the expected classfcaton error Maxmum Margn Lnear
More informationConvex Optimization. Optimality conditions. (EE227BT: UC Berkeley) Lecture 9 (Optimality; Conic duality) 9/25/14. Laurent El Ghaoui.
Convex Optmzaton (EE227BT: UC Berkeley) Lecture 9 (Optmalty; Conc dualty) 9/25/14 Laurent El Ghaou Organsatonal Mdterm: 10/7/14 (1.5 hours, n class, double-sded cheat sheet allowed) Project: Intal proposal
More informationCS 3710: Visual Recognition Classification and Detection. Adriana Kovashka Department of Computer Science January 13, 2015
CS 3710: Vsual Recognton Classfcaton and Detecton Adrana Kovashka Department of Computer Scence January 13, 2015 Plan for Today Vsual recognton bascs part 2: Classfcaton and detecton Adrana s research
More informationLecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
More informationRecap: the SVM problem
Machne Learnng 0-70/5-78 78 Fall 0 Advanced topcs n Ma-Margn Margn Learnng Erc Xng Lecture 0 Noveber 0 Erc Xng @ CMU 006-00 Recap: the SVM proble We solve the follong constraned opt proble: a s.t. J 0
More informationSolutions to exam in SF1811 Optimization, Jan 14, 2015
Solutons to exam n SF8 Optmzaton, Jan 4, 25 3 3 O------O -4 \ / \ / The network: \/ where all lnks go from left to rght. /\ / \ / \ 6 O------O -5 2 4.(a) Let x = ( x 3, x 4, x 23, x 24 ) T, where the varable
More informationFeature Selection: Part 1
CSE 546: Machne Learnng Lecture 5 Feature Selecton: Part 1 Instructor: Sham Kakade 1 Regresson n the hgh dmensonal settng How do we learn when the number of features d s greater than the sample sze n?
More informationPattern Classification
Pattern Classfcaton All materals n these sldes ere taken from Pattern Classfcaton (nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wley & Sons, 000 th the permsson of the authors and the publsher
More informationChapter 6 Support vector machine. Séparateurs à vaste marge
Chapter 6 Support vector machne Séparateurs à vaste marge Méthode de classfcaton bnare par apprentssage Introdute par Vladmr Vapnk en 1995 Repose sur l exstence d un classfcateur lnéare Apprentssage supervsé
More informationLogistic Regression. CAP 5610: Machine Learning Instructor: Guo-Jun QI
Logstc Regresson CAP 561: achne Learnng Instructor: Guo-Jun QI Bayes Classfer: A Generatve model odel the posteror dstrbuton P(Y X) Estmate class-condtonal dstrbuton P(X Y) for each Y Estmate pror dstrbuton
More information1 Convex Optimization
Convex Optmzaton We wll consder convex optmzaton problems. Namely, mnmzaton problems where the objectve s convex (we assume no constrants for now). Such problems often arse n machne learnng. For example,
More informationMachine Learning. What is a good Decision Boundary? Support Vector Machines
Machne Learnng 0-70/5 70/5-78 78 Sprng 200 Support Vector Machnes Erc Xng Lecture 7 March 5 200 Readng: Chap. 6&7 C.B book and lsted papers Erc Xng @ CMU 2006-200 What s a good Decson Boundar? Consder
More informationLecture 20: November 7
0-725/36-725: Convex Optmzaton Fall 205 Lecturer: Ryan Tbshran Lecture 20: November 7 Scrbes: Varsha Chnnaobreddy, Joon Sk Km, Lngyao Zhang Note: LaTeX template courtesy of UC Berkeley EECS dept. Dsclamer:
More informationGeneralized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
More informationChapter Newton s Method
Chapter 9. Newton s Method After readng ths chapter, you should be able to:. Understand how Newton s method s dfferent from the Golden Secton Search method. Understand how Newton s method works 3. Solve
More informationMMA and GCMMA two methods for nonlinear optimization
MMA and GCMMA two methods for nonlnear optmzaton Krster Svanberg Optmzaton and Systems Theory, KTH, Stockholm, Sweden. krlle@math.kth.se Ths note descrbes the algorthms used n the author s 2007 mplementatons
More informationCollege of Computer & Information Science Fall 2009 Northeastern University 20 October 2009
College of Computer & Informaton Scence Fall 2009 Northeastern Unversty 20 October 2009 CS7880: Algorthmc Power Tools Scrbe: Jan Wen and Laura Poplawsk Lecture Outlne: Prmal-dual schema Network Desgn:
More information17 Support Vector Machines
17 We now dscuss an nfluental and effectve classfcaton algorthm called (SVMs). In addton to ther successes n many classfcaton problems, SVMs are responsble for ntroducng and/or popularzng several mportant
More informationAssortment Optimization under MNL
Assortment Optmzaton under MNL Haotan Song Aprl 30, 2017 1 Introducton The assortment optmzaton problem ams to fnd the revenue-maxmzng assortment of products to offer when the prces of products are fxed.
More information10-701/ Machine Learning, Fall 2005 Homework 3
10-701/15-781 Machne Learnng, Fall 2005 Homework 3 Out: 10/20/05 Due: begnnng of the class 11/01/05 Instructons Contact questons-10701@autonlaborg for queston Problem 1 Regresson and Cross-valdaton [40
More informationAPPENDIX A Some Linear Algebra
APPENDIX A Some Lnear Algebra The collecton of m, n matrces A.1 Matrces a 1,1,..., a 1,n A = a m,1,..., a m,n wth real elements a,j s denoted by R m,n. If n = 1 then A s called a column vector. Smlarly,
More informationSome modelling aspects for the Matlab implementation of MMA
Some modellng aspects for the Matlab mplementaton of MMA Krster Svanberg krlle@math.kth.se Optmzaton and Systems Theory Department of Mathematcs KTH, SE 10044 Stockholm September 2004 1. Consdered optmzaton
More information6.854J / J Advanced Algorithms Fall 2008
MIT OpenCourseWare http://ocw.mt.edu 6.854J / 18.415J Advanced Algorthms Fall 2008 For nformaton about ctng these materals or our Terms of Use, vst: http://ocw.mt.edu/terms. 18.415/6.854 Advanced Algorthms
More informationADVANCED MACHINE LEARNING ADVANCED MACHINE LEARNING
1 ADVANCED ACHINE LEARNING ADVANCED ACHINE LEARNING Non-lnear regresson technques 2 ADVANCED ACHINE LEARNING Regresson: Prncple N ap N-dm. nput x to a contnuous output y. Learn a functon of the type: N
More informationCSci 6974 and ECSE 6966 Math. Tech. for Vision, Graphics and Robotics Lecture 21, April 17, 2006 Estimating A Plane Homography
CSc 6974 and ECSE 6966 Math. Tech. for Vson, Graphcs and Robotcs Lecture 21, Aprl 17, 2006 Estmatng A Plane Homography Overvew We contnue wth a dscusson of the major ssues, usng estmaton of plane projectve
More informationEEE 241: Linear Systems
EEE : Lnear Systems Summary #: Backpropagaton BACKPROPAGATION The perceptron rule as well as the Wdrow Hoff learnng were desgned to tran sngle layer networks. They suffer from the same dsadvantage: they
More informationCSE 252C: Computer Vision III
CSE 252C: Computer Vson III Lecturer: Serge Belonge Scrbe: Catherne Wah LECTURE 15 Kernel Machnes 15.1. Kernels We wll study two methods based on a specal knd of functon k(x, y) called a kernel: Kernel
More informationSupport Vector Machines
Separatng boundary, defned by w Support Vector Machnes CISC 5800 Professor Danel Leeds Separatng hyperplane splts class 0 and class 1 Plane s defned by lne w perpendcular to plan Is data pont x n class
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Luca Trevisan September 5, 2017
U.C. Berkeley CS94: Beyond Worst-Case Analyss Handout 4s Luca Trevsan September 5, 07 Summary of Lecture 4 In whch we ntroduce semdefnte programmng and apply t to Max Cut. Semdefnte Programmng Recall that
More informationCHAPTER 7 CONSTRAINED OPTIMIZATION 2: SQP AND GRG
Chapter 7: Constraned Optmzaton CHAPER 7 CONSRAINED OPIMIZAION : SQP AND GRG Introducton In the prevous chapter we eamned the necessary and suffcent condtons for a constraned optmum. We dd not, however,
More informationA Local Variational Problem of Second Order for a Class of Optimal Control Problems with Nonsmooth Objective Function
A Local Varatonal Problem of Second Order for a Class of Optmal Control Problems wth Nonsmooth Objectve Functon Alexander P. Afanasev Insttute for Informaton Transmsson Problems, Russan Academy of Scences,
More information14 Lagrange Multipliers
Lagrange Multplers 14 Lagrange Multplers The Method of Lagrange Multplers s a powerful technque for constraned optmzaton. Whle t has applcatons far beyond machne learnng t was orgnally developed to solve
More informationYong Joon Ryang. 1. Introduction Consider the multicommodity transportation problem with convex quadratic cost function. 1 2 (x x0 ) T Q(x x 0 )
Kangweon-Kyungk Math. Jour. 4 1996), No. 1, pp. 7 16 AN ITERATIVE ROW-ACTION METHOD FOR MULTICOMMODITY TRANSPORTATION PROBLEMS Yong Joon Ryang Abstract. The optmzaton problems wth quadratc constrants often
More informationEnsemble Methods: Boosting
Ensemble Methods: Boostng Ncholas Ruozz Unversty of Texas at Dallas Based on the sldes of Vbhav Gogate and Rob Schapre Last Tme Varance reducton va baggng Generate new tranng data sets by samplng wth replacement
More informationCS246: Mining Massive Datasets Jure Leskovec, Stanford University
CS246: Mnng Massve Datasets Jure Leskovec, Stanford Unversty http://cs246.stanford.edu 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, http://cs246.stanford.edu 2 Hgh dm. data Graph data Infnte
More informationImage classification. Given the bag-of-features representations of images from different classes, how do we learn a model for distinguishing i them?
Image classfcaton Gven te bag-of-features representatons of mages from dfferent classes ow do we learn a model for dstngusng tem? Classfers Learn a decson rule assgnng bag-offeatures representatons of
More informationMachine Learning. Support Vector Machines. Eric Xing , Fall Lecture 9, October 6, 2015
Machne Learnng 0-70 Fall 205 Support Vector Machnes Erc Xng Lecture 9 Octoer 6 205 Readng: Chap. 6&7 C.B ook and lsted papers Erc Xng @ CMU 2006-205 What s a good Decson Boundar? Consder a nar classfcaton
More informationON REGULARISATION PARAMETER TRANSFORMATION OF SUPPORT VECTOR MACHINES. Hong-Gunn Chew Cheng-Chew Lim. (Communicated by the associate editor name)
Manuscrpt submtted to AIMS Journals Volume X, Number 0X, XX 200X Webste: http://aimscences.org pp. X XX ON REGULARISATION PARAMETER TRANSFORMATION OF SUPPORT VECTOR MACHINES Hong-Gunn Chew Cheng-Chew Lm
More informationCSCI B609: Foundations of Data Science
CSCI B609: Foundatons of Data Scence Lecture 13/14: Gradent Descent, Boostng and Learnng from Experts Sldes at http://grgory.us/data-scence-class.html Grgory Yaroslavtsev http://grgory.us Constraned Convex
More informationA NEW ALGORITHM FOR FINDING THE MINIMUM DISTANCE BETWEEN TWO CONVEX HULLS. Dougsoo Kaown, B.Sc., M.Sc. Dissertation Prepared for the Degree of
A NEW ALGORITHM FOR FINDING THE MINIMUM DISTANCE BETWEEN TWO CONVEX HULLS Dougsoo Kaown, B.Sc., M.Sc. Dssertaton Prepared for the Degree of DOCTOR OF PHILOSOPHY UNIVERSITY OF NORTH TEXAS May 2009 APPROVED:
More informationThe Minimum Universal Cost Flow in an Infeasible Flow Network
Journal of Scences, Islamc Republc of Iran 17(2): 175-180 (2006) Unversty of Tehran, ISSN 1016-1104 http://jscencesutacr The Mnmum Unversal Cost Flow n an Infeasble Flow Network H Saleh Fathabad * M Bagheran
More informationCSC 411 / CSC D11 / CSC C11
18 Boostng s a general strategy for learnng classfers by combnng smpler ones. The dea of boostng s to take a weak classfer that s, any classfer that wll do at least slghtly better than chance and use t
More informationCOS 521: Advanced Algorithms Game Theory and Linear Programming
COS 521: Advanced Algorthms Game Theory and Lnear Programmng Moses Charkar February 27, 2013 In these notes, we ntroduce some basc concepts n game theory and lnear programmng (LP). We show a connecton
More informationUsing T.O.M to Estimate Parameter of distributions that have not Single Exponential Family
IOSR Journal of Mathematcs IOSR-JM) ISSN: 2278-5728. Volume 3, Issue 3 Sep-Oct. 202), PP 44-48 www.osrjournals.org Usng T.O.M to Estmate Parameter of dstrbutons that have not Sngle Exponental Famly Jubran
More informationC4B Machine Learning Answers II. = σ(z) (1 σ(z)) 1 1 e z. e z = σ(1 σ) (1 + e z )
C4B Machne Learnng Answers II.(a) Show that for the logstc sgmod functon dσ(z) dz = σ(z) ( σ(z)) A. Zsserman, Hlary Term 20 Start from the defnton of σ(z) Note that Then σ(z) = σ = dσ(z) dz = + e z e z
More informationMLE and Bayesian Estimation. Jie Tang Department of Computer Science & Technology Tsinghua University 2012
MLE and Bayesan Estmaton Je Tang Department of Computer Scence & Technology Tsnghua Unversty 01 1 Lnear Regresson? As the frst step, we need to decde how we re gong to represent the functon f. One example:
More informationKernels in Support Vector Machines. Based on lectures of Martin Law, University of Michigan
Kernels n Support Vector Machnes Based on lectures of Martn Law, Unversty of Mchgan Non Lnear separable problems AND OR NOT() The XOR problem cannot be solved wth a perceptron. XOR Per Lug Martell - Systems
More informationSupporting Information
Supportng Informaton The neural network f n Eq. 1 s gven by: f x l = ReLU W atom x l + b atom, 2 where ReLU s the element-wse rectfed lnear unt, 21.e., ReLUx = max0, x, W atom R d d s the weght matrx to
More informationSupport Vector Machines
/14/018 Separatng boundary, defned by w Support Vector Machnes CISC 5800 Professor Danel Leeds Separatng hyperplane splts class 0 and class 1 Plane s defned by lne w perpendcular to plan Is data pont x
More informationModule 9. Lecture 6. Duality in Assignment Problems
Module 9 1 Lecture 6 Dualty n Assgnment Problems In ths lecture we attempt to answer few other mportant questons posed n earler lecture for (AP) and see how some of them can be explaned through the concept
More informationModule 3 LOSSY IMAGE COMPRESSION SYSTEMS. Version 2 ECE IIT, Kharagpur
Module 3 LOSSY IMAGE COMPRESSION SYSTEMS Verson ECE IIT, Kharagpur Lesson 6 Theory of Quantzaton Verson ECE IIT, Kharagpur Instructonal Objectves At the end of ths lesson, the students should be able to:
More informationTraining Support Vector Machines with Particle Swarms
Tranng Support Vector Machnes wth Partcle Swarms U Paquet Department of Computer Scence Unversty of Pretora South Afrca Emal: upaquet@cs.up.ac.za AP Engelbrecht Department of Computer Scence Unversty of
More informationDepartment of Computer Science Artificial Intelligence Research Laboratory. Iowa State University MACHINE LEARNING
Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory MACHINE LEARNING Vasant Honavar Artfcal Intellgence Research Laboratory Department of Computer Scence Bonformatcs
More informationP R. Lecture 4. Theory and Applications of Pattern Recognition. Dept. of Electrical and Computer Engineering /
Theory and Applcatons of Pattern Recognton 003, Rob Polkar, Rowan Unversty, Glassboro, NJ Lecture 4 Bayes Classfcaton Rule Dept. of Electrcal and Computer Engneerng 0909.40.0 / 0909.504.04 Theory & Applcatons
More informationPHYS 705: Classical Mechanics. Calculus of Variations II
1 PHYS 705: Classcal Mechancs Calculus of Varatons II 2 Calculus of Varatons: Generalzaton (no constrant yet) Suppose now that F depends on several dependent varables : We need to fnd such that has a statonary
More informationLinear, affine, and convex sets and hulls In the sequel, unless otherwise specified, X will denote a real vector space.
Lnear, affne, and convex sets and hulls In the sequel, unless otherwse specfed, X wll denote a real vector space. Lnes and segments. Gven two ponts x, y X, we defne xy = {x + t(y x) : t R} = {(1 t)x +
More informationAdvanced Introduction to Machine Learning
Advanced Introducton to Machne Learnng 10715, Fall 2014 The Kernel Trck, Reproducng Kernel Hlbert Space, and the Representer Theorem Erc Xng Lecture 6, September 24, 2014 Readng: Erc Xng @ CMU, 2014 1
More informationCOMPLEX NUMBERS AND QUADRATIC EQUATIONS
COMPLEX NUMBERS AND QUADRATIC EQUATIONS INTRODUCTION We know that x 0 for all x R e the square of a real number (whether postve, negatve or ero) s non-negatve Hence the equatons x, x, x + 7 0 etc are not
More informationMACHINE APPLIED MACHINE LEARNING LEARNING. Gaussian Mixture Regression
11 MACHINE APPLIED MACHINE LEARNING LEARNING MACHINE LEARNING Gaussan Mture Regresson 22 MACHINE APPLIED MACHINE LEARNING LEARNING Bref summary of last week s lecture 33 MACHINE APPLIED MACHINE LEARNING
More information10) Activity analysis
3C3 Mathematcal Methods for Economsts (6 cr) 1) Actvty analyss Abolfazl Keshvar Ph.D. Aalto Unversty School of Busness Sldes orgnally by: Tmo Kuosmanen Updated by: Abolfazl Keshvar 1 Outlne Hstorcal development
More informationFMA901F: Machine Learning Lecture 5: Support Vector Machines. Cristian Sminchisescu
FMA901F: Machne Learnng Lecture 5: Support Vector Machnes Crstan Smnchsescu Back to Bnary Classfcaton Setup We are gven a fnte, possbly nosy, set of tranng data:,, 1,..,. Each nput s pared wth a bnary
More information3.1 ML and Empirical Distribution
67577 Intro. to Machne Learnng Fall semester, 2008/9 Lecture 3: Maxmum Lkelhood/ Maxmum Entropy Dualty Lecturer: Amnon Shashua Scrbe: Amnon Shashua 1 In the prevous lecture we defned the prncple of Maxmum
More information2E Pattern Recognition Solutions to Introduction to Pattern Recognition, Chapter 2: Bayesian pattern classification
E395 - Pattern Recognton Solutons to Introducton to Pattern Recognton, Chapter : Bayesan pattern classfcaton Preface Ths document s a soluton manual for selected exercses from Introducton to Pattern Recognton
More informationUVA CS / Introduc8on to Machine Learning and Data Mining
UVA CS 4501-001 / 6501 007 Introduc8on to Machne Learnng and Data Mnng Lecture 11: Classfca8on wth Support Vector Machne (Revew + Prac8cal Gude) Yanjun Q / Jane Unversty of Vrgna Department of Computer
More informationAdditional Codes using Finite Difference Method. 1 HJB Equation for Consumption-Saving Problem Without Uncertainty
Addtonal Codes usng Fnte Dfference Method Benamn Moll 1 HJB Equaton for Consumpton-Savng Problem Wthout Uncertanty Before consderng the case wth stochastc ncome n http://www.prnceton.edu/~moll/ HACTproect/HACT_Numercal_Appendx.pdf,
More informationKernel Methods and SVMs
Statstcal Machne Learnng Notes 7 Instructor: Justn Domke Kernel Methods and SVMs Contents 1 Introducton 2 2 Kernel Rdge Regresson 2 3 The Kernel Trck 5 4 Support Vector Machnes 7 5 Examples 1 6 Kernel
More informationA 2D Bounded Linear Program (H,c) 2D Linear Programming
A 2D Bounded Lnear Program (H,c) h 3 v h 8 h 5 c h 4 h h 6 h 7 h 2 2D Lnear Programmng C s a polygonal regon, the ntersecton of n halfplanes. (H, c) s nfeasble, as C s empty. Feasble regon C s unbounded
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 16
STAT 39: MATHEMATICAL COMPUTATIONS I FALL 218 LECTURE 16 1 why teratve methods f we have a lnear system Ax = b where A s very, very large but s ether sparse or structured (eg, banded, Toepltz, banded plus
More informationThe General Nonlinear Constrained Optimization Problem
St back, relax, and enjoy the rde of your lfe as we explore the condtons that enable us to clmb to the top of a concave functon or descend to the bottom of a convex functon whle constraned wthn a closed
More information15 Lagrange Multipliers
15 The Method of s a powerful technque for constraned optmzaton. Whle t has applcatons far beyond machne learnng t was orgnally developed to solve physcs equatons), t s used for several ey dervatons n
More informationFinite Mixture Models and Expectation Maximization. Most slides are from: Dr. Mario Figueiredo, Dr. Anil Jain and Dr. Rong Jin
Fnte Mxture Models and Expectaton Maxmzaton Most sldes are from: Dr. Maro Fgueredo, Dr. Anl Jan and Dr. Rong Jn Recall: The Supervsed Learnng Problem Gven a set of n samples X {(x, y )},,,n Chapter 3 of
More informationLearning Theory: Lecture Notes
Learnng Theory: Lecture Notes Lecturer: Kamalka Chaudhur Scrbe: Qush Wang October 27, 2012 1 The Agnostc PAC Model Recall that one of the constrants of the PAC model s that the data dstrbuton has to be
More informationOnline Classification: Perceptron and Winnow
E0 370 Statstcal Learnng Theory Lecture 18 Nov 8, 011 Onlne Classfcaton: Perceptron and Wnnow Lecturer: Shvan Agarwal Scrbe: Shvan Agarwal 1 Introducton In ths lecture we wll start to study the onlne learnng
More informationMachine Learning. Support Vector Machines. Eric Xing. Lecture 4, August 12, Reading: Eric CMU,
Machne Learnng Support Vector Machnes Erc Xng Lecture 4 August 2 200 Readng: Erc Xng @ CMU 2006-200 Erc Xng @ CMU 2006-200 2 What s a good Decson Boundar? Wh e a have such boundares? Irregular dstrbuton
More informationThe Multiple Classical Linear Regression Model (CLRM): Specification and Assumptions. 1. Introduction
ECONOMICS 5* -- NOTE (Summary) ECON 5* -- NOTE The Multple Classcal Lnear Regresson Model (CLRM): Specfcaton and Assumptons. Introducton CLRM stands for the Classcal Lnear Regresson Model. The CLRM s also
More informationProseminar Optimierung II. Victor A. Kovtunenko SS 2012/2013: LV
Prosemnar Optmerung II Vctor A. Kovtunenko Insttute for Mathematcs and Scentfc Computng, Karl-Franzens Unversty of Graz, Henrchstr. 36, 8010 Graz, Austra; Lavrent ev Insttute of Hydrodynamcs, Sberan Dvson
More informationOn a direct solver for linear least squares problems
ISSN 2066-6594 Ann. Acad. Rom. Sc. Ser. Math. Appl. Vol. 8, No. 2/2016 On a drect solver for lnear least squares problems Constantn Popa Abstract The Null Space (NS) algorthm s a drect solver for lnear
More information3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X
Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number
More information6) Derivatives, gradients and Hessian matrices
30C00300 Mathematcal Methods for Economsts (6 cr) 6) Dervatves, gradents and Hessan matrces Smon & Blume chapters: 14, 15 Sldes by: Tmo Kuosmanen 1 Outlne Defnton of dervatve functon Dervatve notatons
More information