INF Introduction to classifiction Anne Solberg Based on Chapter 2 ( ) in Duda and Hart: Pattern Classification

Transcription

1 INF Introduction to classifiction Anne Solberg Based on Chapter 1-6 in Duda and Hart: Pattern Classification INF

2 Introduction to classification One of the most challenging topics in image analsis is recognizing a specific object in an image To do that, several steps are normall used Some kind of segmentation can delimit the spatial etent of the foreground objects, then a set of features to describe the object characteristics are computed, before the recognition step that decides which object tpe this is The focus of the net three lectures is the recognition step The starting point is that we have a set of K features computed for an object These features can either describe the shape or the gra level/color/teture properties of the object Based on these features we need to decide what kind of an object this is Statistical classification is the process of estimating the probabilit that an object belongs to one of S object classes based on the observed value of K features Classification can be done both unsupervised or supervised In unsupervised classification the categories or classes are not known, and the classification process with be based on grouping similar objects together In supervised classification the categories or classes are known, and the classification will of consist of estimating the probabilit that the object belongs to each of the S object classes The object is assigned to the class with highest probabilit For supervised classifcation, training data is needed Training data consists of a set of objects with know class tpe, and the are used to estimate the parameters of the classifier Classification can be piel-based or region-based The performance of a classifier is normall computed as the accurac it gets when classifing a different set of objects with known class labels called the test set INF 4300

3 Assume that each object i in the scene is represented b a feature vector i If we have computed K features, i will be a vector with K components: i 1 i K i A classifier that is based on K features is called a multivariate classifier The K features and the objects in the training data set will define a K- dimensional feature space The training data set is a set of objects with known class labels As we saw last week, we can use a scatter plot to visualize class separation on data with known class labels if we have one or two features, but visualizing multidimensional space is difficult INF

4 In the scatter plot, each object in the training data set is plotted in K- dimensional space at a position relative to the value of the K features Each object is represented b a dot, and the color of the dot represents the class In this eample we have 4 classes visualize using features A scatter plot is a tool for visualizing features We will later look at other class separabilit measures INF

5 Concepts in classification In the following three lectures we will cover these topics related to classification: Training set Test set Classifier accurac/confusion matrices Computing the probabilit that an object belongs to a class Let each class be represented b a probabilit densit function In general man probabilit densities can be used but we use the multivariate normal distribution which is commonl used Baes rule Discriminant functions/decision boundaries Normal distribution, mean vector and covariance matrices knn classification Unsupervised classification/clustering INF

6 Introduction to classification Supervised classification is related to thresholding Divide the image into two classes: foreground and background Thresholding is a two-class classification problem based on a 1D feature vector The feature vector consist of onl the gre level f, How can we classif a feature vector of K shape features into correct character tpe? We will now stud multivariate classification theor where we use N features to determine if an object belongs to a set of K object classes Recommended additional reading: Pattern Classification, R Duda, P Hart and D Stork Chapter 1 Introduction Chapter Baesian Decision Theor, 1-6 See ~inf3300/www_docs/bilder/dudahart_chappdf and dudahart-appendipdf INF

7 Plan for this lecture: Eplain the relation between thresholding and classification with classes Background in probabilit theor Baes rule Classification with a Gaussian densit and a single feature Briefl: training and testing a classifier We go deeper into this in two weeks INF

8 From INF310: Thresholding Basic thresholding assigns all piels in the image to one of classes: foreground or background g, 0 if 1 if f f,, T T This can be seen as a -class classification problem based on a single feature, the gra level The classes are background and foreground, and the threshold T defines the border between them INF

9 Classification error for thresholding - Background - Foreground Threshold t In this region, foreground piels are misclassified as background In this region, background piels are misclassified as foreground INF

10 Classification error for thresholding We assume that bz is the normalized histogram for background bz and fz is the histogram for foreground The histograms are estimates of the probabilit distribution of the gra levels in the image Let F and B be the prior probabilities for background and foregroundb+f=1 The normalized histogram for the image is then given b p z B b z F f z The probabilit for misclassification given a treshold t is: E E B F t t t t f z dz b z dz INF

11 Find T that minimizes the error t E t F f z dz B b z dz de t 0 F dt f T t B b T Minimum error is achieved b setting T equal to the point where the probabilities for foreground and background are equal The locations in feature space where the probabilities are equal is called decision boundar in classification The goal of classification is to find the boundaries INF

12 Partitioning the feature space using thresholding 1 feature and 1 threshold Tresholding one feature gives a linear boundar separating the feature space INF

13 Partitioning the feature space using thresholding features and thresholds Tresholding two feature gives a rectangular boes in feature space INF

14 Can we find a line with better separation? We cannot isolate all classes using straight lines This problem is not linearl separable INF

15 The goal: partitioning the space with smooth non-linear boundaries INF

16 Distributions, standard deviation and variance A Gaussian distribution normal distribution is specified given the mean value and the variance : 1 p z Variance, standard deviation e INF

17 Two Gaussian distributions for a single feature Assume that bz and fz are Gaussian distributions, then p B F B F B F z B e F e B and F are the mean values for background and foreground B and F are the variance for background and foreground INF

18 The -class classification problem summarized Given two Gaussian distributions bz and fz The classes have prior probabilities F and B Ever piel should be assigned to the class that minimizes the classification error The classification error is minimized at the point where F fz = B bz What we will do now is to generalize to K classes and D features INF

19 How do we find the best border beteen K classes with features? We will find the theoretical answer and a geometrical interpretation of class means, variance, and the equivalent of a threshold INF

20 The goal of classification We estimate the decision boundaries based on training data Classification performance is alwas estimated on a separate test data set We tr to measure the generalization performance The classifier should perform well when classifing new samples Have lowest possible classification error We often face a tradeoff between classification error on the training set and generalization abilit when determining the compleit of the decision boundar INF

21 Probabilit theor - Appendi A4 Let be a discrete random variable that can assume an of a finite number of M different values The probabilit that belongs to class i is p i = Pr=i, i=1,m A probabilit distribution must sum to 1 and probabilities must be positive so p i 0 and M i1 p i 1 INF

22 Epected values - definition The epected value or mean of a random variable is: The variance or second order moment is: These will be estimated from training data where we know the true class labels: INF 4300 M i ip i P E 1 P u u E Var P E 1 1 k N i i i k k N i i i k k k k N N

23 Pairs of random variables Let and be random variables The joint probabilit of observing a pair of values =i,=j is p ij Alternativel we can define a joint probabilit distribution function P, for which, 0, P, P 1 The marginal distributions for and if we want to eliminate one of them is: P P P P,, INF

24 Statistical independence Epected values of two variables Variables and are statistical independent if and onl if Two variables are uncorrelated if Epected values of two variables: INF , P P P P E P E P E P E P E P f f E,,,,,,,, 0 Where in this course have ou seen similar formulas?

25 Feature Can ou sketch,,,, Feature 1 INF

26 Conditional probabilit If two variables are statisticall dependent, knowing the value of one of them lets us get a better estimate of the value of the other one The conditional probabilit of given is: Pr i, j Pr i j Pr j and for distributions : P P, P Eample: Threshold a page with dark tet on white background is the gre level of a piel, and is its class F or B If we consider which gre levels can have - we epect small values if is tet =F, and large values if is background =B INF

27 Epected values of M variables Using vector notation: E μ E P Σ μ μ T INF

28 Baesian decision theor A fundamental statistical approach to pattern classification Named after Thomas Baes , an english priest and matematician It combines prior knowledge about the problem with a probabilit distribution function The most central concept is Baes decision rule INF

29 Baes rule in general The equation: P P P P P P P P In words: likelihood prior posterior evidence are observations/feature vector, is the unknown class labels We want to find the most probable class given the observed feature vector To be eplained for theinf classification 4300 problem later :- 9

30 Mean vectors and covariance matrices in N dimensions If f is a n-dimensional feature vector, we can formulate its mean vector and covariance matri as: with n features, the mean vector will be of size 1n and or size nn The matri will be smmetric as kl = lk INF n n n n n nn n n n n n n n E E E E f f f f Σ μ

31 Baes rule for a classification problem Suppose we have J, j=1,j classes is the class label for a piel, and is the observed gra level or feature vector We can use Baes rule to find an epression for the class with the highest probabilit: p j P j P j p prior probabilit posterior probabilit likelihood normalizing factor For thresholding, P j is the prior probabilit for background or foreground If we don't have special knowledge that one of the classes occur more frequent than other classes, we set them equal for all classes P j =1/J, j=1,,,j Small p means a probabilit distribution Capital P means a probabilit scalar value between 0 and 1 INF

32 Baes rule eplained P j p P j p j p j is the probabilit densit function that models the likelihood for observing gra level if the piel belongs to class j Tpicall we assume a tpe of distribution, eg Gaussian, and the mean and covariance of that distribution is fitted to some data that we know belong to that class This fitting is called classifier training P j is the posterior probabilit that the piel actuall belongs to class j We will soon se that the the classifier that achieves the minimum error is a classifier that assigns each piel to the class j that has the highest posterior probabilit p is just a scaling factor that assures that the probabilities sum to 1 INF

33 Probabilit of error If we have classes, we make an error either if we decide 1 if the true class is if we decide if the true class is 1 If P 1 > P we have more belief that belongs to 1, and we decide 1 The probabilit of error is then: P error P 1 if P if we decide we decide 1 INF

34 Back to classification error for thresholding - Background - Foreground P error P error, d P error p d In this region, foreground piels are misclassified as background In this region, background piels are misclassified as foreground INF

35 Minimizing the error P error P error, d P error p d When we derived the optimal threshold, we showed that the minimum error was achieved for placing the threshold or decision border as we will call it now at the point where P 1 = P This is still valid INF

36 Baes decision rule In the class case, our goal of minimizing the error implies a decision rule: Decide ω 1 if Pω 1 >Pω ; otherwise ω For J classes, the rule analogusl etends to choose the class with maimum a posteriori probabilit The decision boundar is the border between classes i and j, simpl where Pω i =Pω j Eactl where the threshold was set in minimum error thresholding! INF

37 Baes classification with J classes How do we generalize: and D features To more the one feature at a time To J classes To consider loss functions that some errors are more costl than others INF

38 Baes rule with c classes and d features If we measure d features, will be a d-dimensional feature vector Let { 1,, C } be a set of c classes The posterior probabilit for class c is now computed as Still, we assign a piel with feature vector to the class that has the highest posterior probabilit: INF c j j j j j j P p p p P p P 1 i j P for all, P if Decide j 1 1

39 Discriminant functions The decision rule Decide1 if P 1 P j, for all j i can be written as assign to 1 if g i g j The classifier computes J discriminant functions g i and selects the class corresponding to the largest value of the discriminant function Since classification consists of choosing the class that has the largest value, a scaling of the discriminant function g i b fg i will not effect the decision if f is a monotonicall increasing function This can lead to simplifications as we will soon see INF

40 Equivalent discriminant functions The following choices of discriminant functions give equivalent decisions: The effect of the decision rules is to divide the feature space into c decision regions R 1,R c If g i >g j for all ji, then is in region R i The regions are separated b decision boundaries, surfaces in features space where the discriminant functions for two classes are equal INF ln ln i i i i i i i i i i P p g P p g p P p P g

41 Decision functions - two classes If we have onl two classes, assigning to 1 if g 1 >g is equivalent to using a single discriminant function: g = g 1 -g and decide 1 if g>0 The following functions are equivalent: g g P 1 p ln p 1 P INF

42 The Gaussian densit - univariate case a single feature To use a classifier we need to select a probabilit densit function p i The most commonl used probabilit densit is the normal Gaussian distribution: p 1 ep E with epected value or mean E and variance 1 p d p d INF

43 Training a univariate Gaussian classifier To be able to compute the value of the discriminant function, we need to have an estimate of j and j for each class j Assume that we know the true class labels for some piels and that this is given in a mask image Training the classifier then consists of computing j and j for all piels with class label j in the mask file The are computed from training data as: For all piels i with label k in the training mask, compute k k 1 N k 1 N k N k ii N k ii i i k INF

44 Classification with a univariate Gaussian Decide on values for the prior probabilities, P j If we have no prior information, assume that all classes are equall probable and P j =1/c Estimate j and j based on training data based on the formulae on the previous slide For class j=1, J, compute the discriminant function P p P j j j 1 j ep 1 j j P j Assign piel to the class with the highest value of P j The result after classification is an image with class labels corresponding to the most probable class for each piel We compute the classification error rate from an independent test mask INF

45 Eample: image and training masks The masks contain labels for the training data If a piel is not part of the training data, it will have label 0 A piel belonging to class k will have value k in the mask image We should have a similar mask for the test data INF

46 Estimating classification error A simple measure of classification accurac can be to count the percentage of correctl classified piels overall averaged for all classes, or per class If a piel has true class label k, it is correctl classified if j =k Normall we use different piels to train and test a classifier, so we have a disjoint training mask and test mask Estimate the classification error b classifing all piels in the test set and count the percentage of wrongl classified piels INF