Support Vector Machine. Industrial AI Lab.

Transcription

1 Support Vector Machine Industrial AI Lab.

2 Classification (Linear) Autonomously figure out which category (or class) an unknown item should be categorized into Number of categories / classes Binary: 2 different classes Multiclass: more than 2 classes Feature The measurable parts that make up the unknown item (or the information you have available to categorize) 2

3 Distance from a Line 3

4 ω If p and q are on the decision line 4

5 d If x is on the line and x = d ) ) from the origin to the line) (where d is a normal distance 5

6 for any vector of x Distance from a Line: h 6

7 Distance from a Line: h 7

8 Distance from a Line: h Another method to find a distance between g x = 1 and g x = 1 8

9 Binary classification C / and C 0 Features Illustrative Example The coordinate of the unknown animal i in the zoo 9

10 Hyperplane Is it possible to distinguish between C / and C 0 by its coordinates on a map of the zoo? We need to find a separating hyperplane (or a line in 2D) 10

11 Data Generation for Classification 11

12 Data Generation for Classification 12

13 Decision Making Given: Hyperplane defined by ω and ω 3 Animals coordinates (or features) x Decision making: Find ω and ω 3 such that x given ω 4 x + ω 3 = 0 13

14 Decision Boundary or Band Find ω and ω 3 such that x given ω 4 x + ω 3 = 0 or Find ω and ω 3 such that x C / given ω 4 x + ω 3 > 1 and x C 0 given ω 4 x + ω 3 > 1 14

15 Classification Band 15

16 Optimization Formulation 1 n (= 2) features N belongs to C / in training set M belongs to C 0 in training set m = N + M data points in training set ω and ω 3 are the unknown variables 16

17 Optimization Formulation 1 17

18 CVXPY 1 18

19 CVXPY 1 19

20 CVXPY 1 20

21 Linear Classification: Outlier Note that in the real world, you may have noise, errors, or outliers that do not accurately represent the actual phenomena Linearly non- separable case 21

22 Outliers 22

23 Outliers No solutions (hyperplane) exist We have to allow some training examples to be misclassified! but we want their number to be minimized 23

24 Optimization Formulation 2 n (= 2) features N belongs to C / in training set M belongs to C 0 in training set m = N + M data points in training set For the non- separable case, we relax the above constraints Need slack variables u and v where all are positive 24

25 Optimization Formulation 2 The optimization problem for the non- separable case 25

26 Expressed in a Matrix Form 26

27 CVXPY 2 27

28 CVXPY 2 28

29 Further Improvement Notice that hyperplane is not as accurately represent the division due to the outlier Can we do better when there are noise data or outliers? Yes, but we need to look beyond linear programming Idea: large margin leads to good generalization on the test data 29

30 Maximize Margin Finally, it is Support Vector Machine (SVM) Distance (= margin) Minimize ω 0 to maximize the margin (closest samples from the decision line) Use gamma (γ) as a weighting between the followings: Bigger margin given robustness to outliers Hyperplane that has few (or no) errors 30

31 Support Vector Machine 31

32 Support Vector Machine 32

33 In a more compact form Support Vector Machine 33

34 Support Vector Machine 34

35 Classifying Non- linear Separable Data Consider the binary classification problem each example represented by a single feature x No linear separator exists for this data 35

36 Classifying Non- linear Separable Data Consider the binary classification problem each example represented by a single feature x No linear separator exists for this data Now map each example as x x, x 0 Data now becomes linearly separable in the new representation Linear in the new representation = nonlinear in the old representation 36

37 Classifying Non- linear Separable Data Let's look at another example Each example defined by a two features No linear separator exists for this data x = x /, x 0 Now map each example as x = x /, x 0 z = x / 0, 2x / x 0, x 0 0 Each example now has three features (derived from the old representation) 37

38 Classifying Non- linear Separable Data Data now becomes linear separable in the new representation 38

39 Kernel Often we want to capture nonlinear patterns in the data nonlinear regression: input and output relationship may not be linear nonlinear classification: classes may note be separable by a linear boundary Linear models (e.g. linear regression, linear SVM) are note just rich enough by mapping data to higher dimensions where it exhibits linear patterns apply the linear model in the new input feature space mapping = changing the feature representation Kernels: make linear model work in nonlinear settings 39

40 Nonlinear Classification 40

41 Classifying Non- linear Separable Data 41

42 Classifying Non- linear Separable Data 42

43 Classifying Non- linear Separable Data 43