INFORMATION TRANSFER THROUGH CLASSIFIERS AND ITS RELATION TO PROBABILITY OF ERROR

Size: px

Start display at page:

Download "INFORMATION TRANSFER THROUGH CLASSIFIERS AND ITS RELATION TO PROBABILITY OF ERROR"

Buck Spencer
5 years ago
Views:

1 IFORMATIO TRAFER TROUG CLAIFIER AD IT RELATIO TO PROBABILITY OF ERROR Deniz Erdogmus, Jose C. Prini Comutational euroengineering Lab (CEL, University of Florida, Gainesville, FL 6 [deniz,rini]@nel.ufl.edu ABTRACT Fano s bound identifies a lower bound for the lassifiation error robability and indiates how the information transfer through lassifier affets its rformane. It was an imortant ste towards lining the information theory and attern reognition. In this ar, a family of lower bounds is derived using Renyi s entroy, whih yields Fano s lower bound as a sial ase. Using a different set of entroy orders, Renyi s definition also allows the onstrution a family of ur bounds for the robability of error. This is imossible using hannon s definition of entroy. Further analysis to obtain the tightest lower and ur bounds revealed the fat that Fano s bound is indeed the tightest lower bound, and the ur bounds beome tighter as the entroy order aroahes to one from below. umerial evaluations of the bounds are resented for three digital modulation shemes under AWG hannel.. ITRODUCTIO Fano's bound is imortant in that it identifies the lin between information transfer through a lassifier and its robability of mislassifiation []. It emloys hannon's definition of entroy to arrive at a lower bound for the error robability for a lassifier. owever, Fano's bound annot be utilized in evaluating lassifier rformane beause it is a lower bound for a quantity we wish to minimize []. Our urose in this ar is to resent a family of information theoretial lower and ur bounds for the error robability of lassifiers that enomass Fano's inequality as a sial ase. We derive our bounds using the unommon Renyi's definition of entroy rather than the widely reognized definition of hannon. Renyi's definition is a arametri family of entroy values with the hannon's entroy being the limit value when the arameter alha of the family aroahes to one []. In fat, it turns out that the existene of this arameter beomes useful in formulating an ur bound besides a lower bound for two disoint sets of values it an tae. Fano's inequality, orresonding to Renyi's entroy with arameter equal to one, then beomes a sial ase in the family of lower bounds. The onditional entroy of the lassifier outut given the inut an be regarded as the average information transfer through the lassifier, thus the version of the bounds whih inororates this quantity is signifiant in understanding the relationshi between the information transfer and mislassifiation robability. hannon's marginal entroy of a random variable is equal to the sum of the onditional entroy and mutual information [4]. The same equality is not valid for Renyi's definitions of marginal and onditional entroies and mutual information. owever, it is ossible to obtain lower bounds from Renyi's definitions of these information theoretial quantities by maing use of Jensen's inequality. Thus, while we ut the main emhasis on the lower and ur bounds, whih inororate the onditional entroy due to the above stated reasons, other versions emloying mutual information and oint entroy are also introdued. We also identify the values of the arameters to obtain the tightest lower and ur bounds in among the arametri family of bounds. In order to evaluate the usefulness of these bounds, we introdue a numerial ase study, whih demonstrates that the ur bound we have derived is as tight for a very wide range of lassifiers as for the one with the otimal onfusion matrix. Besides, we resent the numerial evaluations of our lower and ur bounds as well as the ommonly used union bounds for a PK and 6AM digital ommuniation sheme under AWG hannel in order to demonstrate the rformane of these bounds [5]. In a reent wor, we have utilized Jensen's inequality on Renyi's definition of onditional entroy, oint entroy, and mutual information to derive lower and ur bounds for the mislassifiation robability of the lassifier under onsideration [6]. It turns out that Fano's bound is still sial beause it is the tightest in the family of lower bounds. On the other hand, the bounds formulated from the onditional entroy reveals the relation between the amount of information transferred through the lassifier and its rformane. Finally, the examination of the ur bound exression reveals valuable insights to our understanding of how the robabilities in the onfusion matrix of a lassifier

2 should be distributed suh that its rformane is otimized. The organization of this ar is as follows. We first give a baground on the definitions of the information theoretial quantities used. ext, we revisit Fano s bound. In etion IV, we resent the lower and ur bounds for robability of error using Renyi s entroy definition. Following that, we rovide two numerial examles with analytial solutions, for PK, 4PAM, and 6AM ommuniation shemes. Finally, we summarize the results in the onlusions.. BACKGROUD DEFIITIO It is ossible to exress lower and ur bounds for the lassifiation error robability using mutual information, onditional entroy, and oint entroy. Therefore, we first give both the hannon s and Renyi s definitions for these quantities. Later in the following, we refer to the lasses at the inut and outut of the lassifier with the random variables M and W, restively. The random variable e is used to denote the events of erroneous and orret lassifiation with robabilities {P(e,-P(e}. hannon s Definitions: For a disrete random variable M, whose robability mass funtion (mf is given by, hannon s entroy is given by [7] { } m ( M m log m ( The oint entroy, mutual information, and onditional entroy an be defined based on the entroy as [,4,7] I ( M, W ( M, W ( M W ( M w w m, w log m, w m, w m, w log m w ( ( M w m w log m w ( and m,, w and m w are the oint mf and the onditional mf of M and W. The following rorty is satisfied by hannon s mutual information [,4,7]. I ( M, W ( M ( M W (4 Renyi s Definitions: Renyi s entroy for M is [] ( M log ( m (5 >0 is the entroy order. Aordingly, we obtain the mutual information and onditional entroy as [] ( M, W log ( m, w I ( M, W log ( W M ( W m log m ( W m ( m, w ( m ( w m ( w (6 (7 In braeting the robability of error from above and below, the entroy order will be useful as it hanges the onvexity of the funtion and allows the use of different forms of Jensen s inequality.. FAO BOUD Fano determined a lower bound to the robability of error for the lassifiation in disrete-symbol ommuniation systems []. The symbols are seleted from a disrete symbol set onsisting of elements with eah symbol m having a nown rior robability m. The onditional robability of deision being the th symbol when th symbol was sent is w m. Then, Fano s lower bound for the robability of lassifiation error an be written as ( W M ( e P ( e (8 log( A ommon modifiation in the attern reognition literature is to relae the denominator with log( to aommodate for -lass roblems. In addition, the identity in (4 is used to obtain a lower bound exressed in terms of the mutual information between the inut and the outut saes [8]. 4. BOUD WIT REYI ETROPY In this setion, we will rovide the exressions for the lower and ur bounds of the robability of error in lassifiation that emloy Renyi s definitions of the entroy and mutual information. As hannon s entroy

3 is the limit for Renyi s entroy when the order aroahes to one, the limit of the lower bound exressions we rovide is equal to Fano s bound. The detailed derivation roedure to obtain these bounds an be found in [6]. ere, it suffies to say that the derivation extensively emloys the Jensen s inequality for onvex and onave funtions, and the order of Renyi s entroy allows us to ontrol the onvexity of the exressions. In onlusion, with some wor, we obtain the following bounds for the error robability in terms of the onditional entroy of the outut given the inut sae of the lassifier. ( ( ( ( W M e β W M e Pe (, (9 log( min ( W e, m β < w m β ( W e, m log (0 β e m is the onditional entroy of the outut given that we mae an error when m is the atual lass. The numerator of the ur bound exression is always greater than the numerator of the lower bound as a rorty of Renyi s entroy with varying order. The denominator is an entroy with ( - terms, hene it is smaller than or equal to log( -. Thus, the ur bound is always greater than the lower bound. The bounds inororating the oint entroy and mutual information an be obtained by relaing the onditional entroy with γ ( W, M ( M, and ( W I γ ( W ; M, restively. In addition, in the ur bound with mutual information, the denominator is evaluated using hannon s entroy [6]. otie that, as for a given mf Renyi s entoy inreases as order goes to one from the above, the tightest lower bound I obtained with Fano s bound. Analyzing the effet of entroy order on the ur bound is not that trivial sine it aars both in the numerator and in the denominator. For that reason, we investigate a simlified examle to observe the behavior of the ur bound under variations in entroy order. Consider a three-lass roblem with the following onfusion matrix the i th entroy denotes the onditional robability w i m. ε ε P W M ε ε ( e ε ε Results resented below in Fig. assume equal lass riors, but exriments with different rior assignments showed that the onlusion remains the same; the ur β bound beomes tighter when the entroy order aroahes to one. One other rorty of the ur bound, whih is desirable, is exhibits the same level of tightness for a broad range of lassifiers, as, the lower bound tends to be loose for these. The following lots the lower and ur bounds with onditional entroy as a funtion of ε in the onfusion matrix. The overall robability of error is fixed to 0.. Figure. Bounds for different entroy orders Although we do not resent here any results using the mutual information and oint entroy bounds, exriments demonstrated that they rodue very lose values to those given by the onditional entroy bounds [6]. For Renyi s entroy, sine the identity in (4 is not satisfied, we do not get an exat equivalene, as in Fano s bound, the three bounds using three different quantities are exatly equal. Although Renyi s entroy offers a way to braet the robability of lassifiation error by adusting the order of entroy, the bounds obtained are not free f roblems. In some extreme ases, the lower bounds may beome negative (exet Fano s bound, whih beomes zero, and the ur bounds may blow u if the denominator aroahes to zero. Although, in most ratial ases this situation will not be enountered, it is ossible. 5. UMERICAL EXAMPLE As an examle, the information theoreti bounds are evaluated for a PK modulation sheme over an AWG hannel. The energy r transmitted bit is E b and the PD for the additive white Gaussian noise is 0 /. We an omute the exat exression for the onfusion matrix in terms of -funtions, with the assumtion of unorrelated noise in the in-hase and quadrature omonents, in this roblem.

4 P PK WM ( *( *( *( ( *( ( x E / *( ( *( *( *( ( (. The riors for symbols x b 0 are assumed equal, i.e. m / 4,,,, 4. Fig. shows the theoretial robability of symbol error and the lower and ur bounds for that. We note that, we ould obtain an arbitrarily tight ur bound by simly maing the entroy order aroah arbitrarily lose to one. This roess will not introdue any additional omutation, but numerial auray may beome an issue. ur bound blows for some values of R, as it is extremely tight for others. P(e and Bounds The Probability of Error and Bounds for 4PAM 0-0 P(e Fano Lower(alha Ur(alha E b / 0 (db Figure. P s and its bounds for 4PAM Finally, we evaluate the bounds for a 6AM sheme the signal onstellation onsists of 6 lasses uniformly distributed on a square area in two dimensions. Again, with the assumtion of unorrelated white Gaussian noise in the orthogonal diretions, we an evaluate the exat onfusion matrix, hene the exat robability of lassifiation error and the bounds. The results are summarized below in Fig The Probability of Error and Bounds for 6AM Figure. P s and its bounds for PK As a seond examle, we evaluate the bounds for a 4PAM modulation sheme over an AWG hannel. This is an examle to those roblemati situations that may our. The four lasses in 4PAM are loated on a line, equally searated with the R value indiating the ratio of the distane between means of lasses to the variane of the Gaussian distributions entered at these means. As the R inreases, the denominator aroahes to zero beause the mf whose entroy is evaluated aroahes to a δ-distribution. In terms of the bit energy and noise ower, we an write the onfusion matrix as 4 P PAM W M ( The riors for symbols are again assumed equal. Fig. shows the theoretial robability of symbol error and the lower and ur bounds for this ase. ote that the P(e and Bounds P(e Fano Lower(alha.005 Ur(alha E s / 0 (db Figure 4. P s and its bounds for 6AM In this setion, we resented three simle examles, from digital ommuniations, whih an be framed as a attern reognition roblem, and whose analytial solutions an be easily alulated. In PK, two of the wrong lasses are always loated equidistant to the atual lass, therefore, the denominator of the ur bound is the entroy of a mf with at least two terms, hene as R inreases, the ur bound is still tight. On the other hand, the 4PAM examle, designed to illustrate that unstability of the ur bound is ossible,

5 has wrong lasses for whih the mf among these may aroah a δ-distribution very fast for ertain R values. In turn, the denominator of the ur bound may beome arbitrarily small ausing the bound to diverge. The 6AM examle demonstrates that the ur bound may loose auray for the same hoie of entroy order when the number of lasses is high. This is due to the denominator exression, whih basially dends on the robabilities of the losest lasses. As the number of lasses inreases, the number of farther neighbors inreases. Therefore, the rformane degrades. 6. COCLUIO Fano s bound is a well-nown result that rovides an insight to how robability of lassifiation error is lined to the information transfer through a lassifier. It is derived from hannon s definition of entroy, whih is a sial ase of Renyi s definition, and it is only a lower bound for a quantity we wish to minimize. Insired by the wor of Fano, we have derived a family of lower and ur bounds for the robability of error starting from Renyi s entroy, the entroy order identifies if the exression is a lower or an ur bound. Thus, we were able to exloit this rorty of Renyi s entroy to aquire more information about the robability of error. Interestingly enough, Fano s bound, orresonding to Renyi s entroy of order one, turned out to be the tightest of the lower bounds, and the ur bounds beame arbitrarily tight as the entroy order aroahed one from below. To demonstrate the rformane of the bounds in ation, analytial solutions for PK, 4PAM, and 6AM digital ommuniation shemes with AWG were evaluated. The results indiated that the bounds are useful in braeting the robability of error in realisti situations. Although not illustrated here, it is ossible to obtain estimates of the bounds by emloying various nonarametri estimates for the mfs that are required in the omutation. The simlest of these estimators is the samle-ount method. Our simulations have showed that with a reasonably small number of samles (around 500, the bounds for PK an be estimated with a small variane. Alternatively, neural networs an be trained to rodue estimates of the desired onditional robabilities or nonarametri df estimation methods lie Parzen windowing an be emloyed to obtain df estimates, whih an then be integrated over the aroriate regions in the outut sae to yield estimates of the required onditional robabilities. As a final remar, in ratie, it is ossible to obtain an estimate of the robability of error with the information that is required to obtain an estimate of the bounds. evertheless, the bounds an still be informative and may be used as onfirmation arameters for these estimates. Anowledgments: This wor was suorted by the F grant EC REFERECE [] R.M. Fano, Transmission of Information: A tatistial Theory of Communiations, ew Yor: MIT Press & John Wiley & ons, In. 96. [] K. Fuunaga, An Introdution to tatistial Pattern Reognition, Aademi Press, ew Yor, Y, 97. [] A. Renyi, Probability Theory, ew Yor: Amerian Elsevier Publishing Comany In., 970. [4] T. Cover, J. Thomas, Elements of Information Theory, John Wiley, ew Yor, 99. [5] J.G. Proais, Digital Communiations, rd ed., Y: MGraw ill, 995. [6] D.Edogmus, J.C. Prini, Information Theoretial Lower and Ur Bounds for Error Probability of Classifiers, submitted to IEEE.Trans. on Pattern Analysis and Mahine Intelligene, De [7] C.E. hannon, A Mathematial Theory of Communiations, Bell ystems Teh. J., vol 7,.79-4,6-656, 948. [8] K. Torola, W.M. Cambell, Mutual Information in Learning Feature Transformations, Proeedings of the International Conferene on Mahine Learning, tanford, CA, UA, 000.

CSC321: 2011 Introduction to Neural Networks and Machine Learning. Lecture 11: Bayesian learning continued. Geoffrey Hinton

CSC321: 2011 Introduction to Neural Networks and Machine Learning. Lecture 11: Bayesian learning continued. Geoffrey Hinton CSC31: 011 Introdution to Neural Networks and Mahine Learning Leture 11: Bayesian learning ontinued Geoffrey Hinton Bayes Theorem, Prior robability of weight vetor Posterior robability of weight vetor