10.5 Unsupervised Bayesian Learning

Transcription

1 The Bayes Classifier Maximum-likelihood methods: Li Yu Hongda Mao Joan Wang parameter vetor is a fixed but unknown value Bayes methods: parameter vetor is a random variable with known prior distribution 10.5 Unsupervised Bayesian Learning The Bayes Classifier Bayes Formula: The Bayes Classifier information from samples unknown Assumptions: Based on the assumptions, we have 1. Classes number is known. 2. Prior probabilities for eah lass are known, k = 1,...,. 3. Forms of the lass-onditional probability densities 4. Part of our knowledge about is from a known prior density. are known. and 5. Rest knowledge about is from the set of samples D. x is independent of the samples lass has nothing to do with distribution of The Bayes Classifier The basi equations for unsupervised Bayesian learning The Posterior Density : Estimate of over is obtained by averaging The likelihood yielded by the samples: The task at hand now is to estimate sample set D. from the The Posterior Density in reursive form: 1

2 The relation between Bayesian and the M-L solutions Again, the posterior density: This equation emphasizes the relation between the Bayesian and the maximum-likelihood solutions. If is essentially uniform over the region peaks, then peaks at the same plae. 1. The only signifiant peak ours at 2. The peak is very sharp We an get: The relation between Bayesian and the M-L solutions ont d Conlusions: The use of the maximum-likelihood estimate as if it were the true value of in designing the Bayes lassifier. Disussions: If there are a large amounts of data, maximum-likelihood and Bayes methods will agree (or nearly agree). If there are only a small amounts of data, there exist some small problems the approximations are poor. Supervised and unsupervised learning Differenes: Lak of identifiability ( main differene) With supervised learning, it merely means that instead of obtaining a unique parameter vetor we obtain an equivalene lass of parameter vetors. With unsupervised learning, it may ause serious problems. When an t be determined uniquely, the mixture an t be deomposed into its true omponents. Computational omplexity Supervised learning: the possibility of finding suffiient statistis allows solutions that are analytially pleasing and omputationally feasible. Unsupervised learning: there is no way to avoid the fat that the samples are obtained from a mixture density. Supervised and unsupervised learning ont d From the above equation, it is not easy for us to find a simple exat solutions for Again, the likelihood n p(d θ) = [ p(x k w j,θ j ) p(w j )] k=1 j=1 1. p(d θ) is the sum of n produts of omponent densities. Eah term in this sum an be interpreted as the joint probability of obtaining the samples x 1, x n.. 2. If the omponent densities do not overlap, thus as varies, only one term in the mixture density is nonzero. Supervised and unsupervised learning ont d Another way to ompare supervised and unsupervised learning : Example 1: Unsupervised learning of Gaussian Data Consider the one-dimensional, two-omponent mixture with, P(w 1 ) and P(w 2 ) are known. Substitute the mixture density for Let p(w 1 )=1 and all the other prior probabilities are zero. Unsupervised We seek the mean of the seond omponent. Suppose that the prior density is uniform from a to b, Then after one observation (x=x 1 ) we an get. Supervised 2

3 Example 1 ont d Example 1 ont d Consider a seond sample x 2 Disussions: Unfortunately, the primary thing we learn from this expressios already Compliated when n=2. With n samples there will be 2 n terms, as a result, the omputational ost will be very heavy. Example 1 ont d Example 1 ont d So it is possible to use the following relation. Disussion: One of the main differenes between the Bayesian and the maximum-likelihood approahes to unsupervised learning is the presene of the prior density Deision-Direted Approximation Why: both maximum-likelihood and the bayesian methods have high omputational requirements. Solutions: beause the differene between supervised and unsupervised learning is the presene of labels, it is natural to propose the following: Use prior information to train a lassifier. Label new data with this lassifier. Use the new labeled samples to train a new (supervised) lassifier. This approah is known as the deision-direted approah [1] to unsupervised learning. Obvious limitations inlude: If the initial lassifier is not reasonably good, the proess an diverge. The tails of the distribution tend not to be modeled well this way, whih results in signifiant overlap between the omponent densities. In pratie, this approah works well beause it is easy to leverage previous work for the initial lassifier. Also, it is less omputationally expensive than the pure Bayesian unsupervised learning approah. [1] Classial adaptive algorithms (LMS, RLS, CMA, deision direted) seen as reursive strutures by P. Duhamel, M. Montazeri and K. Hilal Funtions For Clustering 3

4 Funtions for Clustering Purpose: measures the lustering quality of any partition of the data. Suppose: a set D of n samples X1, X2, Xs lassified into lusters D1, D2, D. Samples in the same luster are more similar than samples in different lusters. Finds a the partition that optimizes the riterion funtion The Sum-of-Squared-Error Simplest and most widely used one: Sum-of-squared errors: is the number of samples in is the mean of the samples is the number of lusters The optimal partitioning is defined as one that minimizes (minimum variane) The Sum-of-Squared-Error Problem: Work well when lusters form ompat louds Fail when there are great differenes in the number of samples in different lusters. Related Criteria Rewrite the riterion funtion J e = x m i 2 m i = 1 x D i x D i x J e = 1 2 s i s i = 1 2 x x' 2 x D i x' D i S i is the average squared distane between points in the i-th luster (a similarity funtion) Related Criteria - Continued Satter Criteria Another type of Funtions The similarity funtion an be replaed by other appropriate similarity funtions e.g. the average, the median or even the maximum distane between points in a luster Definition of Satter Matrix Satter matrix for the i-th luster S i = (x m i )(x m i ) t x D i Within-luster satter matrix S W = S i Between-luster satter matrix S B = (m i m)(m i m) t as m i = 1 n x D i x (Cluster mean) m = 1 n (Total mean) m i 4

5 10/21/08 Total Satter Matrix ST = m= (x m)(x m)t x D 1 mi n Trae (Total mean) Is the sum of within-luster satter matrix and between-luster satter matrix Not Sum of diagonal elements dependent on the formation of lusters Only How to measure a satter matrix? as Invariant Determinant of the matrix Appropriate Appropriate funtions funtionsof of eigenvalues eigenvalues dependent on all samples Determinant Trae Minimize Minimize Or Determinant the sum of diagonal elements of Sw - tr[sw ] J d = SW = tr[sb ] maximize the determinant of SW - Sw S i Interesting fat: SB is not hosen beause it will beome singular if the number of lusters is less than or equal to the dimensionality tr[sw ] = tr[si ] = x mi 2 = J e x Di Minimizing Jd is similar to minimizing Je, but not neessarily the same Invariant Criteria Maximize d Trae tr[sw 1SB ] = λi d Or minimize 1 1 λi J f = tr[st 1SW ] = Whereas λi are eigenvalues of Sum of diagonal elements Determinant Invariant Determinant of the matrix Appropriate funtions of eigenvalues SW 1SB Eigenvalue a salar value for linear transformation that only hanges the eigenvetor s length but not diretion Are they the same? 5

6 Trae riterion (Tend to equal the number of points in eah luster) Determinant riterion (Tend to form large and small lusters) Invariant riterion (subtle) Figure from R. O. Duda, P. E. Hart, Pattern lassifiation, A Comparison of Cluster Validity Criteria For a Mixture of Normal Distributed Data [2] Clustering experiment based on 21 different riteria for simulated Gaussian data sets Conlusion: the most reliable riteria among the ones that they tested were: (1) The trae average density riterion (trae of fuzzy ovariane matrix) (2) The Steinberg±Zeitouni riterion [3] (3) The modified trae riterion ( tr[s W ]/ ). [2] A. Geva et al, "A omparison of luster validity riteria for a mixture of normal distributed data", Pattern Reognition Letters, [3] Y. Steinberg and O. Zeitouni, On tests for normality, IEEE Trans. Inform. Theory, Vol. 38, Questions? Thank you 6