JAI 97 XVI. JORNADA DE ATUALIZAÇÃO EM INFORMÁTICA JAI 97 XVI. JOURNEY OF ACTUALIZATION IN COMPUTER SCIENCE

Transcription

1 JAI 97 XVI. JORNADA DE ATUALIZAÇÃO EM INFORMÁTICA XVII. CONGRESSO DA SOCIEDADE BRASILEIRA DE COMPUTAÇÃO Brasília, DF- 997 Mini-Curso Reonheimento de Padrões Resumo: Este doumento apresenta uma introdução para a área de reonheimento de padrões om um destaque espeial para lassifiação de padrões. Em primeiro lugar motiva-se o problema e dão-se os termos básios usados, junto om três eemplos ilustrativos. Depois riam-se os fundamentos para aproimar o problema de lassifiação num onteto Bayesiano, introduzindo erro, rejeição e riso para fazer deisões. A parte prinipal do urso dedia-se aos modelos diversos de lassifiadores: Bayesiano baseado em modelos estatístios, estimador de Parzen, lassifiador do vizinho mais próimo, Funções de Disriminação Linear om um representante dos algoritmos de aprendizagem, a regra de delta, que é estendida para a regra de delta generalizada no pereptron multi-amadas, funções de base radiais e lassifiadores polinomiais. Apresenta-se a redução da dimensão do espaço das araterístias omo um importante passo de préproessamento em reonheimento de padrões, inluindo seleção de araterístias e etração de araterístias. Mostram-se estratégias de pesquisa e ritérios de seleção para a seleção de araterístias. A análise dos omponentes prinipais e a análise disriminativa linear são os eemplos para a etração de araterístias. O tema de estimativa de erro é a medição do desempenho do lassifiador. Utiliza-se o mapa de Sammon para a visualização de dados de alta dimensão. O teto feha om o algoritmo de aglomeração -means omo um representante de aprendizagem não-supervisionada. JAI 97 XVI. JOURNEY OF ACTUALIZATION IN COMPUTER SCIENCE XVII. CONGRESS OF THE BRAZILIAN COMPUTER SCIENCE SOCIETY Brasília, DF, Brazil Short Course Pattern Reognition Abstrat: This tet gives a short introdution into the field of automati pattern reognition, with a speial emphasis of pattern lassifiation. First the motivation and the neessary terms are given, together with three illustrative eamples. Then the groundwork is laid how to approah lassifiation in a Bayesian probabilisti framework, introduing error, rejetion and risk for deision making. The prinipal part deals with the different lassifier arhitetures, model-based parametri Bayesian lassifiers, Parzen windows, Nearest-Neighbor lassifiers, Linear Disriminant Funtions with one representative learning algorithm, the delta rule, whih is etended to the generalized delta rule in the multilayer pereptron, radial basis funtion networks and polynomial lassifiers. Dimensionality redution is presented as an important preproessing step in pattern reognition, onsisting of feature seletion and feature etration. Feature seletion searh strategies are outlined together with the seletion riteria. Feature etration omprises Prinipal Component Analysis and Linear Disriminant Analysis. The topi of error estimation is used to measure the performane of a lassifier. The Sammon map is used to visualize highdimensional data. The tet loses with the -means lustering algorithm as a representative of unsupervised learning. Thomas Walter Rauber Departamento de Informátia Universidade Federal do Espírito Santo Av. F. Ferrari, Vitória - ES, BRASIL Tel.: (+55)(27) Fa: (+55)(27) thomas@inf.ufes.br WWW-homepage:

2 INDEX MOTIVATION AND TERMINOLOGY.... Introdution....2 Patterns, Pattern Reognition, Features and Classifiation....3 Constraints Pattern lassifiation and higher-level information proessing Numerial features vs. symboli features Probabilisti framework Time stationary patterns vs. dynami patterns Basi Model of a Pattern Reognition System Nomenlature Eamples of Pattern Generating Proesses Eample Iris : Iris Flowers Eample Reator : Fault detetion in a hemial reator Eample Vision : Objet reognition in an industrial assembly task Further Reading Software Support PROBABILISTIC FRAMEWORK FOR PATTERN CLASSIFICATION Bayes Theorem Bayes Deision Rule For Minimum Error Classifiation Bayes Deision Rule For Maimum Likelihood Classifiation Bayes Deision Rule For Minimum Risk Classifiation Inferene and Deision Model Assumption and Parameter Estimation Model assumption Parameter estimation Assuming multivariate Gaussian distribution and estimating its parameters CLASSIFIER MODELS, SUPERVISED LEARNING AND PROBLEMS Model-based Parametri Classifiers Parzen Windows The -Nearest Neighbor Classifier The K-Nearest Neighbor Classifier The Nearest Prototype Classifier Linear Disriminant Funtions for Classifiation The Model Gradient Desent Learning and Least Mean Squared Error Generalized Linear Disriminant Funtions for Classifiation Universal Funtion Approimators Curse of Dimensionality and the Bias-Variane Dilemma Polynomial Classifier Radial Basis Funtion Networks Multilayer Pereptron Arhiteture Delta Rule and Generalized Delta Rule...29 i

3 4 DIMENSIONALITY REDUCTION BY FEATURE SELECTION AND EXTRACTION Feature Seletion Seletion riteria Searh strategies Limitations of Feature Seletion Feature Etration ERROR ESTIMATION HIGH-DIMENSIONAL DATA VISUALIZATION UNSUPERVISED LEARNING...38 BIBLIOGRAPHY ii

4 . Motivation and Terminology. Introdution Pattern reognition is one of the most hallenging topis when it omes to the simulation of the ognitive apabilities of a human being. Appliation eamples of pattern reognition are the identifiation of a person by its voie, fingerprints or the struture of the iris of the eye, the reognition of handwritten haraters, guiding an autonomous vehile by sensorial information, the supervision of proesses or mahinery and many other fields. Despite the variety of the appliation areas there do eist essential parts of the pattern reognition proess that are idential for all the eamples that were mentioned above. The following ourse material tries to give a slight glimpse into some tehniques that are neessary to build a pattern reognition system. It will be our goal to etrat those methods of pattern reognition that an universally be applied to most potential appliation areas. Naturally sine we want our system to be automati, the methods must be suitable to be eeuted by a mahine, speifially as a software implementation on a digital omputer. First of all we have to lay the fundament for the subsequent material. This means that we have to define the basi model of pattern reognition. Furthermore, in order to limit the sope of this tet, we will restrit our attention to the most prominent tehnique in pattern reognition, namely the automati data-driven lassifiation of patterns that are represented by a multidimensional ontinuous feature vetor into a finite set of lasses. Other areas of pattern reognition, like strutural desriptions, symboli reasoning, fuzzy approahes or regression problems are outside the sope of this tet. It an be stated that muh progress has been made in the last years in the area of automati pattern lassifiation, espeially due to the reborn interest in artifiial neural networks. Nevertheless a solid amount of knowledge about the field remains always up to date and it would be disadvantageous for the reader to omit important topis of pattern reognition tehniques that have be established by Bayes, Fisher or Duda & Hart, to mention only a few prominent names related to the field. The organization of the ourse will be as follows. First we will introdue the neessary terminology of pattern reognition and limit the material to the lassifiation of time-stationary, multidimensional ontinuous and labeled patterns. A short outline is given about the problems that arise and how they will be approahed. Then we will regard the at of pattern lassifiation as a universal funtion approimation in a probabilisti framework. Different lassifier arhitetures and assoiated learning tehniques will be presented, originating from statistial and artifiial neural network tehniques. Feature etration and feature seletion are presented as an indispensable part of pattern reognition. The evaluation of how well the generated lassifier will perform is the topi of error estimation. Also important is the visualization of high dimensional data points in two or three dimensions in order to make the intrinsi struture of the data aessible to a human observer. The ourse loses with the topi of unsupervised learning..2 Patterns, Pattern Reognition, Features and Classifiation What is a pattern? What is pattern reognition? The answers to these questions are intuitively very lear to a human being sine pattern reognition is onstantly eerised during most of its lifetime. When the reader is looking at this tet, the eyes apture the physial signal that represent the tet, then haraters are reognized, words, sentenes, and finally the semantis of the tet is etrated on the highest level of information pro-

5 essing. Pattern reognition is transforming information from the subsymboli level (signals) to the symboli level (meanings). One an observe that pattern reognition is also a proess of information redution. From the unountable amount of physial information that is onneted to eah atom on a piee of paper, only the visual blak-to-white transitions are interesting in order to read a tet. A harater is reognized by the reader based on ertain harateristis or features of the harater. In the eample of the harater reognition, the features are rather of symboli nature, like the eistene or absene of a stroke or the shape of the urves. One should note however that a urve of a harater ould also be represented numerially, like for instane traing the urve and measuring its urvature. For omputer analysis it is most desirable to have numerial ontinuous features at hand. Another advantage of the numerial representation of a pattern is the availability of mature mathematial tools to handle multivariate ontinuous data, like for instane statistis. The use of statistis is furthermore justified by the noisy and hene stohasti nature of the pattern generation soure. The same lass of objets is represented by a probability distribution. This approah enables the use of statistial inferene, i.e. to say to whih lass a ertain pattern belongs. This is the final step in the information redution sequene, namely the lassifiation of an objet that is represented by the feature vetor. In this tet we will not onsider the higher-level reasoning that an eventually follow the lassifiation proess. Referring to the eample above, after the reader has reognized a word he ould for instane detet a grammatial error in the sequene of isolated haraters. In this sense the lassifiation of a pattern would be embedded into a more omple framework of symboli reasoning..3 Constraints In order to define the topis of pattern reognition that are presented here, some further omments are given about the allowed ompleity of the problem. As it was mentioned before we will restrit ourselves to the most important aspets of pattern reognition..3. Pattern lassifiation and higher-level information proessing In the lassial tetbook of [Duda and Hart, 973] it is mentioned that for the desriptive approah the lassifiation of objets is not suffiient to solve a omple pattern reognition task. It would be rather important to obtain high-level desriptions of a given real world situation. For instane the desription of a sene inside a room is suh a highlevel desription. It would be neessary not only to lassify a ertain region of a digital image as a penil and another region as a notebook, but also to relate eah of the objets in the sene to eah other. Ever more omple reasoning an be imagined, for instane the spaial relationship of the objets in the sene. As it was mentioned before we will restrit the material of this ourse to the lassifiation of objets. In many ases of pratial pattern reognition problems the lassifiation an be seen as a stand-alone problem, for instane the reognition of fingerprints or handwritten numerals..3.2 Numerial features vs. symboli features If objets are desribed by ategorial (symboli, qualitative) features, like ugly, beautiful or aeptable, unaeptable, no geometrial relationship an be established between different patterns. The tools to lassify objets that are desribed by symboli features usually differ from those for the handling of numerial features. Here we will only onsider ontinuous numerial features R. An objet is desribed by more than one feature. Therefor the representation of an objet results in a D-dimensional fea- 2

6 ture vetor = (,, D ) T whih is equivalent to a point in a D-dimensional spae. This spae is alled pattern spae. A data point in a 2-D or 3-D pattern spae an be visualized diretly. For higher-dimensional pattern spaes appropriate mapping tehniques must be employed, for instane the Sammon map that is presented later..3.3 Probabilisti framework We will regard the pattern generating soure as a stohasti proess. This benefits us with the possibility to use a probabilisti framework to model the relation between features and lasses. Probabilisti tehniques are further reommended beause the data whih we are working with are noisy measurements and hene need a mathematial framework that an handle these random variations. Probabilities and probability densities will play an essential role in the presented tehniques. It beomes possible to use tehniques from statistis, artifiial neural networks and optimization to reate a lassifier and to adapt its variables to the problem at hand..3.4 Time stationary patterns vs. dynami patterns A further restrition that we impose on the nature of the patterns is that they do not hange their position in time. Hene ( t i ) = ( t j ) = for all times t i, t j. Often dynami information an be transformed to time stationary patterns by appropriate mappings, for instane the Fourier analysis of periodi signals an deliver non-hanging patterns in frequeny spae..4 Basi Model of a Pattern Reognition System Fig. shows a sequene of proessing steps that has been widely aepted as the model for automati pattern lassifiation. The model onsists of a Divide and Conquer approah to the ompleity problem. A sequene of dimensionality redution steps and information filters are applied to the raw data in order to ahieve the final lassifiation results. World Sensor Measurements Feature Etration Dimensionality redution FEATURES Classifier indution Classifier Classifiation Classes Fig. Classifiation model in pattern reognition A sensor is attahed to the signal generating proess whih ollets measurable physial data. A feature etrator onverts the raw signals to mahine proessable features, in our ase ontinuous numerial values. Many features ompose a multidimensional feature vetor. Sine the dimension of this feature vetor is sometimes too big, or sine irrelevant or redundant information is hiding within the features, a dimensionality redution step is performed. This means that fewer features are derived from the original feature set. An obvious advantage of reduing the number of features is that the lassifier based on the features is more effiient in terms of storage and proessing time requirements. The new feature set serves as input to a lassifier that ategorizes the real world objets represented by the features. We present tehniques that generate or adapt a lassifier on the basis of a training data set, i.e. we are learning a lassifier from eamples. The basi motivation to separate the lassifiation problem into a sequene of data redution steps is the need to handle the ompleity of the pattern reognition problem. 3

7 The goal of lassifiation goes hand in hand with data redution. Eah representation spae in the sequene ontains an order of magnitude less information than the anterior spae. From the huge volume of the information in the physial world, only a few bits remain whih represent the lasses. In this ourse we will fous on the proessing steps that follow after the features have been aquired, i.e. dimensionality redution and the definition and adaptation of the lassifier. Our basi goal will be the analysis of the features that desribe the given problem, the definition of the lassifier arhiteture, the adaptation of the free parameters of the lassifier and the analysis of the performane of the lassifier..5 Nomenlature We will now introdue the neessary terminology that is needed for the formal definition of our lassifiation tools. A real world objet is represented in the form of a D- dimensional vetor = (,, j,, D ) T of ontinuous features, i.e. R D. Eah objet belongs to one lass ω i from the set Ω = { ω,, ω i,, ω } of mutually elusive lasses. Patterns in the sense of pattern reognition are pairs (, ω) whih assoiate the feature vetor with its meaning, i.e. to whih lass it belongs. In a probabilisti framework we are interested in probabilities and probability densities whih an be used to model the probabilisti distribution of the pattern generating soure and draw onlusions about the lass membership of a ertain objet. In the net hapter we will refine these ideas about using probabilisti tehniques to approah the lassifiation problem. Our pattern lassifiation is based on a learning-from-eample approah. This means that the problem is desribed by a finite set of eample pairs n n (, ), eah pair onsisting of the feature vetor and the lass label. Eah lass ω i has n i data samples, suh that is the kth sample of lass with,, and is the jth value of. These eamples normally are the only available information about the problem at hand. We must therefor rely on them to build our pattern lassifier, i. e. define the struture of the lassifier. Furthermore, if we use a parametri statistial or neural model we must adapt the free parameters of the pattern lassifier. Also for this task the data samples an be used. A good lassifier generalizes the data well. This means that future unknown samples are lassified onforming to the true distribution of the data. It means also that the lassifier is not overfitted to the noise in the samples, i.e. we must find a good ompromise between the degrees of freedom and the fleibility of our lassifier models..6 Eamples of Pattern Generating Proesses As often, the best way to support the theory is to give some pratial eamples. It will beome lear that a pattern an represent a wide variety of real world objets or situations that have to be lassified. In Fig. 2 three pattern lassifiation eamples are resumed in terms of the possible lasses, the sensors that are used to aquire the information and the features that haraterize the real world objets or situations..6. Eample Iris : Iris Flowers In 935 E. Anderson olleted the measurements of the petal and sepal leaves of three different speies of the Iris of the family Iridaeae: Iris setosa, Iris versiolor and Iris virginia. One year later R. A. Fisher used the (modified) data to illustrate the onept of Linear Disriminant Funtions [Fisher, 936]. Four different features were inves- ω i { ω p p } p = ω ik i k = n i ikj ik 4

8 tigated: sepal length, sepal width, petal length and petal width. For eah of the three flower lasses 50 samples were olleted, totalling 50 samples. The features are all ontinuous. Sine then this data set has turned into a lassi in pattern reognition. The real world objets in this eample are speies of flowers. The number of lasses is = 3 with Ω = { Setosa, Virginia, Versiolor}. The sensor in this ase is of etremely simple nature, namely a devie that measures length, for instane a ruler. The measurements an diretly be used as the D = 4 features. The lassifiation task is to disriminate the flowers by the four attributes. The only information available to the lassifier designer is the data set of 50 eamples for eah of the 3 lasses. This means that all the knowledge about the problem must be etrated from the available data. The data set is available at the following address: ftp://ftp.is.ui.edu/pub/mahine-learning-databases/.6.2 Eample Reator : Fault detetion in a hemial reator From a quite different appliation area omes the net eample of pattern reognition. A liquid is poured into a stirred reator tank where a hemial reation takes plae. The finished produt is leaving the reator through an outflow pipe. The hemial reation produes heat so the temperature of the liquid must be ontrolled. Also the level in the tank must be kept within a ertain interval to insure the proper funtioning of the reation. The ontrol of the proess is not onsidered in more detail here sine it does not interest for the pattern reognition part of the problem. The important fat is that there are two features whih haraterize the proess: the temperature and the level of the liquid, hene D = 2. During normal operation only ertain values are allowed for these two features. For instane we have an abnormal or faulty situation if the temperature eeeds a ertain level or if the level falls below a ertain value. The lassifiation task is to distinguish between two lasses: normal funtioning and fault, hene = 2. We define that also in this ase the problem is speified by a number of eamples, let us assume n = 5000 onsisting of pairs of 2-D feature vetors and the respetive lass, for instane ( ( 60degrees, 3m) T, Normal), ( ( 59degrees, 3. m) T, Normal), ( ( 00degrees, 3m) T, Fault) ould be the first three learning eamples available..6.3 Eample Vision : Objet reognition in an industrial assembly task The final eample tries to ategorize 3-D objets by their 2-D silhouette whih an be one of = 3 lasses from the set Ω = { Retangle, Triangle, Cirle}. In this eample the sensor is more omple than in Iris. The features are not diretly available from the aquisition proess, and must be etrated (alulated) from the raw measurements, see Fig.. The visual information is aptured by a amera and the digital image is transferred to a image proessing faility, e.g. a frame grabber with proessing apability. Then the 2-D image is analyzed by image proessing algorithms (e.g. filtering, enhanement, segmentation) to finally alulate the D = 4 features length, width, area and perimeter. These features finally are the basis for the lassifiation of the different geometri shapes. We will return to some of these eamples in the subsequent parts of the ourse, espeially in the onsideration of the probability distribution of the pattern samples and the analysis of the features for the lassifiation task..7 Further Reading Many good tetbooks eist in the area of pattern reognition. Although reently there has been a number of publiations, espeially from the area of artifiial neural networks, the older books that lean themselves prinipally to statistial approahes did not loose muh of their atuality. [Duda and Hart, 973] for whih a seond edition [Duda et al., 5

9 996] is in preparation, is a must. Important is also [Fukunaga, 990]. Another eellent work for statistial pattern reognition that both emphasizes the theoretial bakground and provides pratial methods is [Devijver and Kittler, 982]. Easier to read is [Tou and Gonzalez, 974], [Shalkhoff, 992] and [Pao, 989] where the latter two give also onepts for syntati and neural lassifiation methods. More on statistial lassifiation an be found in [MLahlan, 992] and [Vapnik, 996]. The renewed interest in the area of artifiial neural networks has generated a number of interesting insights into the relation of pattern reognition and neural omputation, e.g. [Bishop, 995], [Ripley, 996], [Shürmann, 996]. Topis that are omitted here are syntati pattern reognition [Gonzalez and Thomason, 982] and mahine learning approahes applied to pattern reognition, e.g. [Breiman et al., 984]. For unsupervised learning (lustering) tehniques see e.g. [Anderberg, 973]. A good starting point on the internet is the page of the pattern reognition group of the University of Delft at the following address: Software Support The majority of the pattern reognition methods that are presented in this tet have been implemented in a highly portable C-oded program pakage alled Tooldiag. The program an be freely opied for researh purposes: PATTERN CLASSIFICATION TASK CLASSES SENSORS FEATURES IRIS FLOWERS 3 Flower speies - Iris Setosa - Iris Virginia - Iris Versiolor - Ruler 4 Shape harateristis - Sepal length - Sepal width - Petal length - Petal width CHEMICAL PROCESS Level [m] L 2 Operation states - Normal - Fault - Level meter - Thermometer 2 physial parameters - Level - Temperature Temperature [ o C] T Reation tank OBJECT RECOGNITION 3 Shapes (2-D) - Retangle - Triangle - Cirle Image proessing - Aquisition: Camera - Image proessing unit - Measuring algorithm Shape harateristis - Length - Width - Area - Perimeter Fig. 2 Eamples of pattern lassifiation problems 2. Probabilisti Framework for Pattern Classifiation The Bayes rule for deision making is a theoretial optimum of the performane of the lassifier beyond whih it is impossible to go. Pratial implementations of lassifiers an only try to reah this limit as lose as possible. Therefor the Bayes deision the- 6

10 ory is a good starting point to understand the priniples of pattern reognition, espeially pattern lassifiation as an at of deision making. All major tetbooks treat this subjet and should be onsulted for more details. At this point of the ourse the reader should refresh his or her knowledge of statistis, respetively aquire the neessary basis to understand the ideas, for instane [Mood et al.,974]. A lassifiation is made if a pattern is ategorized into one of the possible lasses. Let d ( ) be a deision rule that makes the deision to whih lass the pattern belongs, d: R D Ω. Note that d ( ) is defined as a funtion. Let us for now use the notation to epress the deision that has been lassified to belong to. d ( ) ω i = ω i Sometimes it is wiser for an automati lassifiation system not to deide at all. This means that we introdue the option of rejetion. The at of rejetion is easily integrated into lassifiation by introduing a new lass ω 0, the rejetion lass. The number of lasses inreases then to +, i.e. the new set of possible outomes of the lassifiation beomes Ω = { ω 0, ω,, ω }. The usual eample for the use of the rejet option is the diagnosis in a medial ontet. Often it is better to rejet the automati lassifiation of an illness and leave the final deision to a human epert in order to avoid damage to the patient s health. We an also assoiate a ost funtion to deision making in order to penalize wrong deisions that have a onsiderable negative effet. If it was deided that a pattern belongs to lass ω i, i = 0 by some rule, but in reality the pattern belongs to lass ω j, j =, then we will attribute a ost λ ( ω i ω j ) to that deision. The introdution of the ost ould provoke a deision that does not neessarily hoose the most probable lass. Nevertheless it is important to weigh one s deision in pattern reognition beause the onsequenes of that deision have to be taken into onsideration. In the Reator eample a ost funtion is quite appropriate to reflet the realities in proess supervision. If we have a fault with a low probability in a hemial proess and do not deide that we really have a fault and onsequently do not shut down the plant, we ould provoke an immense damage to the plant. Therefor λ ( Normal Fault) should have a high value. All mutual ost funtions among the lasses are resumed in the ost matri C = [ λ ( ω i ω j ) ]. The dimension of the matri is, respetively ( + ) if the rejetion option is allowed. One partiular problem of ost funtions is that they must be defined by a human being and therefor introdue a fair amount of heuristi knowledge into the deision making proess. In most ases a unit ost is assoiated with a lassifiation error and a zero ost is assoiated with a orret deision. In this ase the ost funtion beomes λ ( ω i ω j ) = δ ij, where δ ij is the Kroneker delta symbol. As an eample take the lassifiation of the ten digits. It surely makes no sense to assoiate a different ost for the onfusion of a for a 2 than for the onfusion of a 3 for a Bayes Theorem Let us onsider the pattern generating soure as a stohasti proess that is governed by probabilities P (. ) respetively by probability densities p (. ) in the ase of ontinuous variables. The soure produes a feature vetor in onjuntion with its meaning, i.e. their lass membership ω, as outlined in Fig. 3. Both are random variables. What is interesting to us are the stohasti variables involved in this proess. We shall onsider the feature vetor as a multidimensional ontinuous random variable and the lass membership ω as a disrete random variable. The probabilities respetively probability density funtions involved are: 7

11 p (, ω) : The joint probability density of a pattern having the values of feature vetor, and at the same time belonging to lass ω. p ( ) : The probability density funtion of a pattern having the values of feature vetor. This funtion tells us how the D-dimensional pattern spae is filled up with the patterns, regardless of their lass membership. P ( ω) : The a priori probability of lass ω. This value answers the question: How probable is it, without knowing anything about the feature vetor of a pattern that the pattern belongs to the lass ω. p ( ω) : The onditional probability density funtion of a pattern having the values of feature vetor, given that the pattern belongs to lass ω. P ( ω ) : The onditional probability of a pattern belonging to lass ω given the fat that the pattern has the values of feature vetor, also alled the a posteriori probability The latter value is eatly what we need for lassifiation. We know and want to infer the lass membership ω. If we knew the law that defines the pattern generation soure p (, ω), the lassifiation would be trivial. Basi laws from probability alulus establish the relationship among the values. The joint probability density for a partiular lass (, ) an be epressed in two different ways, using the produt rule: ω i p ω i p (, ω i ) = p ( ω i ) P ( ω i ) = P ( ω i ) p ( ) The Bayes theorem for the a posteriori probability an be derived diretly from () by further epanding p ( ) to p ( ) = p (, ω i ) = p ( ω : (2) i = i ) P ( ω i ) i = Bayes theorem: P ( ω i ) = p ( ω i ) P ( ω i ) ( p ( ω i ) P ( ω i ) ) (3) i = The Bayes theorem gives us the neessary tool to alulate the a posteriori probability for a given pattern belonging to a ertain lass. This is all we need for lassifiation in its basi form. Just aquire the feature vetor and then alulate the probability that this feature vetor belongs to a partiular lass. () p (, ω ) ω pattern = featurevetor lass Fig. 3 Pattern generating soure as a stohasti proess Eample : Calulating a posteriori probabilities Let us hoose a simple one-dimensional eample that illustrates the Bayes theorem of (3) to alulate a posteriori probabilities. Assume that in the Iris pattern reognition problem we ignore all but the feature petal width. Now we have feature vetors of dimension D =, and the feature vetor degenerates to a single real value whih represents petal width. Let us further assume that the a priori probabilities for the three flower speies were P ( Setosa) = 0.35, P ( Virginia) = 0.37 and P ( Versiolor) = The last assumption onerns the way how the values for petal width are distributed. Let us say that the onditional probability density funtion (pdf) is from the same family of density funtions for all three lasses. The most ommon type of pdf found in pattern reognition is the Gaussian (Normal) distribution. A onedimensional Gaussian pdf p ( ) of a random variable is a parametri funtion that is defined by two parameters, the mean µ and variane σ 2 (standard deviation σ). A more ompat, speial notation is N ( µ, σ 2 ). Gaussian probability density funtion of random variable : p ( ) = ep [ ( µ ) 2 ( 2σ 2 ) ] (4) σ 2π 8

12 Assume that we had the knowledge p ( PetalWidth ω i ) about the lass-onditional pdf for eah lass as the following rules p ( PetalWidth setosa) N ( 0.24, 0.0 ), p ( PetalWidth virginia) N ( 2.03, ) and p ( PetalWidth versiolor) N (.33, ). What is the a posteriori probability for instane for the partiular value =.5 in respet to eah of the three lasses? The value p (.5 ) of the unonditional pdf of the feature follows from (2) as p (.5 ) = p (.5 setosa) P ( setosa) + p (.5 virginia) P ( virginia) + p (.5 versiolor) P ( versiolor), i.e. p (.5 ) = We are now able to alulate the lass-onditional a posteriori probability for eah lass from the Bayes theorem (3): P ( setosa.5 ) = = 0.000, P ( virginia.5 ) = = 0.83 and P ( versiolor.5 ) = = One an observe that the a posteriori probabilities sum up to unity, as required for the sure event (the true lass is one of the three possible). The value of the unonditional pdf p ( ) ats as a normalizing fator whih ensures that the sum of all a posteriori probabilities satisfy this ondition of summing to unity. 2.2 Bayes Deision Rule For Minimum Error Classifiation The net logial step on the way to build a lassifier is to design a deision rule d ( ) whih says that the feature vetor belongs to lass ω i, i.e. d ( ) = ω i. An equivalent for deision rule is disriminant funtion. What do we want to ahieve during lassifiation? One possible obvious answer is that we want to deide in suh a manner that we ommit as less errors in the lassifiation as possible, i.e minimizing the average probability of error. We define an error for a given feature vetor, if we deide that belongs to a lass ω j although the a posteriori probability for another lass ω i is higher than for the lass ω j. Furthermore we define the probability of the error as the a posteriori probability of the wrongly deided lass: if we deide and ( ) > P ( ω j ), i, j, i j (5) P ( error ) = P ( ω j ) ω j P ω i To minimize the average error over the whole pattern spae P ( error) = p ( error, ) d = R D P ( error ) p ( ) d we simply have to minimize the probability of error P ( error ) for eah feature vetor. This is equivalent to hoosing that lass with the highest a posteriori probability for. We are now able to formulate the first lassifiation rule using Bayes theorem in a probabilisti ontet. Bayes Deision Rule for Minimum Error Classifiation: = if P ( ω i ) > P ( ω j ), j =, i j (7) d ( ) ω i This rule does not mean that we do not ommit any lassifiation errors. It just ensures that the error reahes a minimum value if we always deide that lass that possesses the highest a posteriori probability. If we allow rejetion we have to introdue a rejetion threshold. For the sake of simpliity it is defined as being onstant for all lasses, i.e. λ ( ω 0 ω j ) = λ r, j =. In this ase the value of λ r is the minimum a posteriori probability that is neessary to make a deision. It ats as a threshold value whih says: Unfortunately the probability that the pattern belongs to one of the possible lasses is so low (namely less or equal than λ r ) that the lassifiation result annot be aepted. The rule (7) beomes then: Bayes Deision Rule for Minimum Error Classifiation with rejetion option: = if P ( ω i ) > P ( ω j ), j =, i j, and d ( ) ω i d ( ) = ω 0 R D P ( ω i ) > λ r (6) otherwise (8) 9

13 2.3 Bayes Deision Rule For Maimum Likelihood Classifiation Eample 2: Minimum risk lassifiation Let us assume the ost matri C below in the Reator eample, allowing rejetion. We assoiate no ost with orret lassifiation. If a normal situation has been deteted and in reality the reator has a prob- If we observe the Bayes theorem of (3) we easily verify that the deision rule (7) is unaffeted by the division of the normalizing fator p ( ). The alternative deision rule then beomes Deide if > ( ), i =, i j (9) ω i p ( ω i ) P ( ω i ) p ( ω j ) P ω j The lass onditional pdf ( ) is also alled the likelihood of ω i with respet to, p ω i and the ratio ( ) alled the likelihood ratio. p ( ω i ) p ω j Sometimes there is no useful information about the a priori probabilities P ( ω) of the lasses available. Classifiation based solely on the likelihoods defines the Bayes Deision Rule for Maimum Likelihood Classifiation: if > ( ), j =, i j (0) d ( ) ω i = p ( ω i ) p ω j In the eample above this would mean that we based the flower lassifiation on the probability density funtions p ( PetalWidth flower), for instane: Deide setosa if p ( PetalWidth setosa) > p ( PetalWidth virginia), p ( PetalWidth setosa) > p ( PetalWidth versiolor). A rejetion option an readily be introdued into (0), but this ase is left out here. 2.4 Bayes Deision Rule For Minimum Risk Classifiation The most general ase for making deisions within the Bayesian framework is the deision rule that minimizes the risk of taking deisions. The risk is the epeted value of the osts of deisions that are made over the whole set of possible lasses and the whole (in our ase ontinuous) pattern spae. The epeted loss or onditional risk for deiding that belongs to is defined as ω i Conditional risk lassifying ω i : R i ( ) = λ ( ω i ω j ) P ( ω j ), i = 0 () j = Note that this formula also inludes the onditional risk R 0 ( ) of rejeting the lassifiation. Sine the deided lass ω i is a funtion of the deision rule we an formulate the onditional risk as a funtion of the deision funtion d ( ), suh that () an be equivalently be formulated as Conditional risk of making a deision: R ( ) = λ ( d ( ) ω j ) P ( ω j ) (2) j = The average risk over the whole pattern spae is the epeted value E { R ( ) } of the onditional risk R ( ) : Average risk: E { R ( ) } = R ( ) p ( ) d (3) An optimal rule d ( ) tries to minimize this average risk. The integral in (3) is minimized by hoosing always the minimum R ( ) for eah possible value. By observing () we see that this is easily ahieved by hoosing that onditional risk R i ( ) that has the smallest value for all lasses ω i, i = 0. We an now state the d ( ) ω i Bayes Deision Rule for Minimum Risk: if ( ) < R j ( ), j =, i j, i { 0 } (4) = R i This rule is the most general mehanism that lassifies a pattern into a lass, allowing the possibility for rejetion and the assoiation of osts to ertain deisions. R D 0

14 lem we say the ost is high, e.g. 000 in this eample. This reflets the high risk of a situation in whih the operation of the reator ontinues while a serious problem might damage the system. If in normal operation a fault has been deteted, we have the ost of unneessarily having to take ountermeasures to a fault that has not ourred, e.g. the osts of suspending the operation of the plant. We furthermore say that asking a human operator (i.e. rejeting automati lassifiation) osts 0 in the ase of a fault and 20 in ase of normal operation. True lass Normal Fault C = [ λ ( ω i ω j ) ] = Rejet Normal Fault Classified lass Let us further assume that we know the a priori probabilities of the two lasses as P ( Normal) = 0.95 and P ( Fault) = 0.05, i.e. a fault ours relatively seldom. Finally let us assume that the onditional probability density funtion for Normal p ( Normal) and Fault p ( Fault) were known and that the likelihood of the two lasses for the proess situation = ( 62degrees, 2.8 m) T had been alulated as p ( Normal) = 2.5 and p ( Fault) =.4. Using (2) on page 8 we alulate p ( ) = = The two a posteriori probabilities from the Bayes theorem (3) on page 8 follow as P ( Normal ) = = 0.97 and P ( Fault ) = = We an now alulate the onditional risk for eah of the three possible deisions Rejet, Normal and Fault, using (): R Rejet R Normal R Fault ( ) = R 0 ( ) = λ ( ω 0 ω ) P ( ω ) + λ ( ω 0 ω 2 ) P ( ω 2 ) = = 9.7 ( ) = R ( ) = λ ( ω ω ) P ( ω ) + λ ( ω ω 2 ) P ( ω 2 ) = = 30 ( ) = R 2 ( ) = λ ( ω 2 ω ) P ( ω ) + λ ( ω 2 ω 2 ) P ( ω 2 ) = = 97 Finally (7) deides that it is less risky to rejet the deision. Although the Normal situation is muh more probable, the high risk assoiated with a wrong deision reverts the result of the lassifiation. 2.5 Inferene and Deision From the onepts of the Bayesian deision rules that were presented above we an establish a model for Bayesian pattern lassifiation as depited Fig. 4. Inferene P ( ω j ) j = { } Deision making ω i = d ( ) i { 0 } Cost funtion Rejetion rule Fig. 4 Bayesian pattern lassifiation: Inferene and deision making We have to distinguish learly between the two stages of inferene and deision making. In the inferene part we alulate or estimate the a posteriori probabilities P ( ω j ) for all possible lasses. The seond part of deision making works on a logially higher level of weighting the onsequenes of a lassifiation result or rejeting a lassifiation. The introdution of a ost funtion and the definition of a rejetion rule pay attention to this. The deision making part finally determines the lass ω i that will be attributed to the pattern.

15 More generally in the inferene part we want to find disriminant funtions d j ( ) whih alulate a degree of membership for a lass ω j. Ideally this is the a posteriori probability P ( ω ), but not always it is possible to obtain this value diretly sine no statistial model is available to alulate probabilities. Let us assume that the inferene part of the lassifier delivers a disriminant funtion d j ( ) for eah of the possible lasses ω j and that d ma and d ma2 are the highest and seond highest values of the disriminant funtion. Let us furthermore assume that t is the j target vetor of the -out-of- oding of lass ω j and that d ( ) is the deision vetor of the deision rule (see setion 3.6.2). For the rejetion option we an define three different rules whih make the deision when a rejetion should be triggered: - The maimum value of the deision funtion falls short of a threshold d ma < τ - The two highest values of the disriminant funtion are too similar d ma d ma2 < - The value of the disriminant funtion is too far away from any plausible target value: 2 d > τ, j =,, ( ) t j In the subsequent part of this tet our main onern will be the inferene stage of Bayesian pattern lassifiation task. It will be our goal to find funtions whih quantify the probability, or more general the degree of membership, that a pattern belongs to a ertain lass. We must remember the very objetive that we have in mind, namely lassifying patterns into lasses. The probabilisti framework is not the only method for lassifiation. We an also partition the pattern spae diretly into deision regions, by defining boundaries between different lasses. Let us however first fous on the basi problems of datadriven pattern lassifiation in a probabilisti framework. 2.6 Model Assumption and Parameter Estimation The eamples of setion 2. and setion 2.4 lead us to the main problems linked to the diret alulus of the a posteriori probabilities in the Bayesian framework. First, how do we know that the features obey a ertain probability density funtion and seond where do we get the parameters of these distributions from? For instane in Eample on page 8 we ould ask: How do we know that the feature petal width is really normally distributed, i.e. PetalWidth N ( µ, σ 2 ) and what are the values of the two parameters that are neessary to haraterize the distribution, the mean µ and variane σ 2? Usually in a real world pattern reognition task the only available information are the eamples, the training set a.k.a. learning set. If we use the Bayesian framework to alulate a posteriori probabilities, we onsequently have to solve two often unsolvable problems. The first is the model assumption problem. We have to assume that the data obeys a ertain law of probability distributions. The seond problem is the parameter estimation problem. From the available data we have to estimate the true parameters of the probability distribution. Both problems are potential soures of error that an deteriorate the lassifiation performane of a system to suh an etend that the ahievable theoretial error rates are onsiderably surpassed. In order to illuminate the reader with the problems of assumption and estimation we will diret our attention to the knowledge that is normally available to the lassifier designer. As it already was mentioned earlier, the only information that is aessible are the training samples whih haraterize the lassifiation problem. These eamples are points in the multidimensional pattern spae. They are a finite set of values for the τ 2

16 unknown joint probability of the stohasti pattern soure of Fig. 3. For tutorial purposes usually only two of the D features are used sine it is not possible to visualize more that three dimensions diretly. In Fig. 5a only two of the four possible features are used to plot the position of the 50 eamples of the Iris data set. Sine for Reator only two features eist a priori, we an use all features in Fig. 5b. From the plot the observer an derive a lot of information that is surely helpful in making assumptions or hoosing a partiular (probabilisti) model of the lassifier. The dotted urves are boundaries that the observer would probably realize in this or a similar form as separators among the different lasses. The boundaries define regions whih divide the pattern spae into the lasses. It an be seen that the boundaries of the two speies Versiolor and Virginia overlap in the Iris eample. This means impliitly that the lasses are not totally separable and also that we will automatially ommit lassifiation errors. The overlap region defines a new region, for instane a rejetion region, sine the lass of a sample is quite ambiguous in that overlapping region. Petal width Iris using 2 of 4 features Iris-setosa Iris-versiolor Iris-virginia Petal length a.) Tank level Reator Normal Fault Temperature b.) Fig. 5 Training samples in the pattern spae for the Iris and Reator eample The real positions of the boundaries between deision regions are defined by the deision rule (disriminant funtion) d ( ). We attribute the pattern to that lass with the highest value of d ( ). Note that the use of disriminant funtions divides the pattern spae in mutually elusive regions (without overlap). Some eamples for the disriminant funtions are: Bayes minimum risk (negative): d i ( ) = R i ( ) (Eq. (7)) Bayes minimum error: d i ( ) = P ( ω ) (Eq. (7)) Bayes minimum error with rejetion: (Eq. (7)) Any monotonous funtions of the above, e.g. logp ( ω ), sine the deision boundaries are not hanged. Easier understanding or omputation an result from this approah. The nature of the deision boundaries are also different. We an assume a linear separator between Setosa and the resting two lasses, and nonlinear separators between all other lasses. The Reator eample has an interesting onstellation of the lasses and their boundaries. The Normal lass is situated within the Fault lass and the separator seems to be irular whih means that the boundary an be desribed by a quadrati funtion. If we look at the density distribution of the samples in Fig. 5 we an suppose a quite different situation of the modes of the probability density funtions. Remember that a 3

17 mode is defined as a maimum of a pdf. The pdf of the three flowers in Fig. 5a seem to be unimodal i.e. all samples of the same lass appear in the same luster, whereas the pdf of the Fault lass in Fig. 5b seems to be quite multimodal, i.e. there appears more than one luster Model assumption We have speulated about the distribution of the samples. The plot gave us an idea about what ould really be the nature of the probability density funtions of the lass. Sine we need the definition of the pdf, and sine we do not have it, we must assume it. This assumption should be in aordane with the real situation that haraterizes the pattern soure. In statistis there eist hypotheses tests whih theoretially ould be employed to verify to a ertain etent that the sample data belong to a ertain family of probability density distributions. In lassifiation pratie these tests do not make muh sense. This is beause the size of the training set is usually small. Espeially if the dimension D of the pattern spae is large we need a pratially impossibly high number of training samples to be able to make reliable statements about the statistial nature of the data. Therefor in pratie we go the other way. First we assume a ertain model, estimate its optimal parameters, look at the performane of the lassifier and ompare it to alternative lassifiers. If the result mathes our epetations we an say that the assumed distribution model is satisfatory Parameter estimation One the model has been defined its free variables must be defined. So to say we have defined the vehile whih will lead us to the desired lassifiation, but now we have to tune its mahine in order to provide best performane. The number of samples is limited. Therefore we have to rely on the data to estimate the parameters. Estimation is usually done by maimum-likelihood estimators, tehniques that try to minimize the approimation error between the real values of the parameters and its estimated version. Parameter estimation is a partiular ase of supervised learning in the ontet of pattern reognition. We use the labeled eamples to adapt the free variables of a given model. The goal is the optimize the internal variables of the model relative to a performane riterion Assuming multivariate Gaussian distribution and estimating its parameters The most ited eample in the pattern reognition literature for an assumed probability density funtion is the multivariate Gaussian distribution. Equation (4) on page 8 already defined a unidimensional Gaussian probability density funtion. If we use more than one variable the probability density funtion for a lass is given as p ( ω i ) = ( 2π) D 2 Σ 2 i where is the mean vetor of lass and is the ovariane matri of lass. The ompat notation for the multivariate Gaussian distribution is ep (5). The mean vetor is the multidimensional etension of the mean of one lass and the D D ovariane matri Σ i the multidimensional etension of the variane σ i, this time also inluding ovarianes σ i σ j between two features i and j. Furthermore we have to know the a priori probabilities P ( ω i ) of eah lass in order to alulate minimum error and minimum risk Bayesian lassifiers. ω i ( µ ) T ( Σ 2 i i ( µ ) ) i µ i ω i Σ i ω i D µ µ i i p ( ω i ) N ( µ, Σ i i ) 4