Robust Parameter Estimation For Mixture Model

Transcription

1 Robust Parameter Estimatio For Mixture Model Saldju Tadjudi Netom Systems, I Nordhoff Street Chatsworth, CA Phoe (818) Fax (818) David A. Ladgrebe Shool of Eletrial ad Computer Egieerig Purdue Uiversity West Lafayette, IN Phoe (765) Fax (765) IEEE. Persoal use of this material is permitted. However, permissio to reprit/republish this material for advertisig or promotioal purposes or for reatig ew olletive works for resale or redistributio to servers or lists, or to reuse ay opyrighted ompoet of this work i other works must be obtaied from the IEEE. This work origially appeared I the IEEE Trasatios o Geosiee ad Remote Sesig, Vol. 38, No. 1, pp , Jauary ABSTRACT I patter reogitio, whe the ratio of the umber of traiig samples to the dimesioality is small, parameter estimates beome highly variable, ausig the deterioratio of lassifiatio performae. This problem has beome more prevalet i remote sesig with the emergee of a ew geeratio of sesors with as may as several hudred spetral bads. While the ew sesor tehology provides higher spetral ad spatial resolutio, eablig a greater umber of spetrally separable lasses to be idetified, the eeded labeled samples for desigig the lassifier remai diffiult ad expesive to aquire. Better parameter estimates a be obtaied by exploitig a large umber of ulabeled samples i additio to traiig samples usig the expetatio maximizatio algorithm uder the mixture model. However, the estimatio method is sesitive to the presee of statistial outliers. I remote sesig data, misellaeous lasses with few samples are ofte diffiult to idetify ad may ostitute statistial outliers. Therefore, we propose to use a robust parameter estimatio method for the mixture model. The proposed method assigs full weight to traiig samples, but automatially gives redued weight to ulabeled samples. Experimetal results show that the robust method prevets performae deterioratio due to statistial outliers i the data as ompared to the estimates obtaied from the diret EM approah. Work leadig to this paper was supported i part by NASA uder Grat NAG ad the Army Researh Offie uder Grat DAAH

2 INTRODUCTION I a mixture model, data are assumed to osist of two or more distributios mixed i varyig proportios. For remote sesig appliatios, it is a ommo pratie to osider several "spetral sublasses" withi eah "iformatio lass" or groud over type. Eah of suh spetral sublasses is osidered to be multivariately ormally distributed ad lassifiatio is the performed with respet to the spetral sublasses. Uder this model, we a regard remote sesig data as a mixture model fitted with ormally distributed ompoets. To estimate the model parameters i a mixture, a ommo approah is to apply the expetatio maximizatio (EM) algorithm, whih is a iterative method for umerially approximatig the maximum likelihood (ML) estimates of the parameters i a mixture model. Alteratively, it a be viewed as a estimatio problem ivolvig iomplete data i whih eah ulabeled observatio is regarded as missig a label of its origi [1]. I [2], the EM algorithm has bee studied ad applied to remote sesig data. It was show that by assumig a mixture model ad usig both traiig samples ad ulabeled samples i obtaiig the lass distributio estimates, the lassifiatio performae a be improved. Also, the Hughes pheomeo [3] a be delayed to a higher dimesioality ad hee more features a be used to obtai better performae. I additio, the parameter estimates represet the true lass distributios more aurately. However, the urepreseted pixel lasses have bee dealt with by rejetio usig a hisquare threshold. This method a be viewed as a hard deisio. Ufortuately, a suitable threshold value is diffiult to selet ad is usually arbitrary. Cosequetly, "useful" pixels might be rejeted as outliers. We propose to use a robust method to estimate the mea vetor ad ovariae matrix for lassifyig multispetral data uder the mixture model. This approah assigs full weight to the traiig samples, but automatially gives redued weight to ulabeled samples. Therefore, it avoids the risk of rejetig useful pixels while still limitig the ifluee of outliers i obtaiig the ML estimates of the parameters. I the ext setio, the EM algorithm is reviewed ad disussed. EXPECTATION MAXIMIZATION ALGORITHM The Expetatio Maximizatio (EM) algorithm is a iterative method for umerially approximatig the maximum likelihood (ML) estimates of the parameters i a mixture model. Alteratively, it a be viewed as a estimatio problem ivolvig 2

3 iomplete data i whih eah ulabeled observatio i the mixture is regarded as missig its label. Uder the mixture model, the distributio of the data x R p is give as: ( ) = α i f x;φ L f i x; φ i i=1 ( ) where α 1,K,α L are the mixig proportios, f i is the ompoet desity parameterized by φ i ad L is the total umber of ompoets. The mixture desity f is the parameterized by Φ= ( α 1,K,α L,φ 1,K,φ L ). Uder the iomplete data formulatio, eah ulabeled sample x is osidered as the labeled sample y with its lass origi missig. Therefore, we a deote y = ( x,i) where i =1LL idiates the sample origi. Let g( x Φ) be the probability desity futio (pdf) of the iomplete data x = ( x 1,K,x ) ad f ( y Φ) be the pdf of the ompletely labeled data y = ( y 1,K,y ). The maximum likelihood estimatio the ivolves the maximizatio of the log likelihood of the iomplete data L( Φ) = logg( x Φ). The estimatio is ompliated by the fat that the sample origi is missig. Hee, the EM algorithm uses the relatioship betwee f ( y Φ) ad g( x Φ) to maximize the iomplete data log-likelihood L( Φ) = logg( x Φ). Usig a iterative approah, the EM algorithm obtais the maximum likelihood estimates by startig with a iitial estimate Φ 0 ad repeatig the followig two steps at eah iteratio: E-Step) Determie Q ΦΦ ( ) = E { log f ( y Φ ) x, Φ } M-Step) Choose Φ = argmaxq( ΦΦ ) The ext ad urret values of the parameters are deoted by the supersripts ad respetively. The algorithm begis with a iitial estimate. It has bee show that uder some relatively geeral oditios the iteratio overges to ML estimates, at least loally. Sie the overgee is oly guarateed to a loal maximum, the algorithm usually must be repeated from various iitial poits. However, the traiig samples, if available, a provide good iitial estimates. ( ) are the m i traiig samples from lass i. Also, there Assume that y = y 1,K,y mi are L Gaussia lasses ad a total of ulabeled samples deoted by x = ( x 1,K,x ). The 3

4 parameter set Φ the otais all the prior probabilities, mea vetors ad ovariae matries. The EM algorithm a the be expressed as the followig iterative equatios [4]: E-Step: τ ij = τ i ( x j φ i ) = α i f i x j φ i L ( ) α t f t x j φ t where τ ij is the posterior probability that x j belogs to lass i. t=1 ( ) (1) M-Step: α i = τ ij / (2) m i µ i = m i y ij m i ( )( y ij µ i ) T τ ij x j τ ij y ij µ i τ ij x j µ i Σ i = m i τ ij ( )( x j µ i ) T (3) (4) There are several fators affetig the overgee of the EM algorithm to the maximum likelihood estimates. First of all, the seletio of traiig samples as iitial estimates a affet the overgee to a great extet. I this work, the traiig set is assumed to provide a good iitial estimate. Aother fator that affets the performae of the EM algorithm is the presee of statistial outliers. Assume that the umber of ompoets have bee deided ad give by the traiig set. Statistial outliers are defied as those observatios that are substatially differet from the distributios of the mixture ompoets. As idiated by Eq. (1) through Eq. (4), the EM algorithm assigs eah observatio to oe of the ompoets with the sample s posterior probability as its weight. Eve though a outlyig sample is iosistet with the distributios of all the defied ompoets, it may still have a large posterior probability for oe or more of the ompoets. As a result, the iteratio overges to erroeous solutios. The problem of outliers is ot uommo i pratial appliatios. I remote sesig, a see usually otais pixels of ukow origi whih form "iformatio 4

5 oise". For example, i a agriultural area, there ould be pixels belogig to houses, trees or rural roads. The statistial distributios of these pixels may be sigifiatly differet from those of traiig lasses ad ostitute statistial outliers. Ufortuately, these outlyig pixels are usually sattered throughout the see ad are small i umber. Cosequetly, idetifyig these pixels ould be a tedious task. A ommo approah to elimiate those pixels i the EM algorithm is to apply a hi-square threshold test. I other words, pixels whose distaes are greater tha the threshold value are osidered as outliers ad are subsequetly exluded from updatig the estimates. The hi-square threshold T α for a give probability α is defied as the squared distae betwee the sample x R p ad the mea vetor for lass i based o the hi-square distributio as show i the followig [11]: { ( ) T Σ 1 i ( x m i ) T α } = α. Pr x x m i The thresholdig approah a be regarded as performig a hard deisio to elimiate outlyig samples before iitiatig the EM algorithm. A suitable threshold value is ofte diffiult to selet ad is usually arbitrary. Cosequetly, "useful" pixels might be rejeted as statistial outliers. I partiular, as dimesioality ireases, most pixels might be osidered as outliers. A alterative would be to assig a differet weight to eah pixel ad use all available ulabeled pixels for updatig the statistis. This method a be regarded as applyig a soft deisio. I the ext setio, the robust EM equatios will be disussed ad modified to proess remote sesig data. ROBUST ESTIMATION The robust estimatio of model parameters was first developed as Huber [5] proposed a theory of robust estimatio of a loatio parameter usig M-estimates i a omixture otext. It was later exteded to the multivariate ase by takig a elliptially symmetri desity ad the assoiatig it with a otamiated ormal desity [6]. Campbell [7] derived the M-estimates for the mixture desity ad obtaied a EM-like algorithm but with a weight futio assiged to eah pixel as a measure of typiality. The outlier problem i remote sesig has bee addressed i [8]. The author proposed a modified M- estimatio of the parameters to deal with the situatio whe the traiig samples of a ertai iformatio lass otai samples of other lasses. This is typial for a mixture model. The modified M-estimates were show to be robust with respet to the otamiatio i the traiig samples as ompared to the least-square estimates. However, the use of ulabeled samples i updatig statistis was ot addressed. This setio will desribe the method of 5

6 robust EM algorithm followig the disussio i [7], ad adaptig the approah for remote sesig data. The EM algorithm first estimates the posterior probabilities of eah sample belogig to eah of the ompoet distributios, ad the omputes the parameter estimates usig these posterior probabilities as weights. With this approah, eah sample is assumed to ome from oe of the ompoet distributios, eve though it may greatly differ from all ompoets. The robust estimatio attempts to irumvet this problem by iludig the typiality of a sample with respet to the ompoet desities i updatig the estimates i the EM algorithm. To iorporate a measure of typiality i the parameter estimatio of the mixture desity, eah ompoet desity f i x µ i, Σ i ( ) for x R p is assumed to be a member of the family of p -dimesioal elliptially symmetri desities with mea vetor µ i ad ovariae matrix Σ i [7]: 2 T 1 where δ = ( x µ ) Σ ( x µ ) i some symmetri futio ρ( δ i ): i i i 12 Σ i fs { δ i ( x; µ i,σ i )}. Typially, f S δ i f S ( δ i ) = exp ρ δ i ( ) is assumed to be the expoetial of { ( )}. The, the likelihood parameter estimatio for these ompoet desities a be obtaied by applyig the expetatio ad maximizatio steps. Deotig the urret ad future parameter values by the supersripts "" ad "", the iterative equatios are derived as [7]: α i = τ ij / µ i = τ ij w ij x j τ ij w ij Σ i = τ ij w ij x j µ i ( )( x j µ i ) T τ ij 6

7 ( ) δ ij is the weight futio ad ψ ( δ ij ) = ρ ( δ ij ) is the first derivative of ( ). To limit the ifluee of large atypial samples, the ovariae estimator is modified where w ij = ψ δ ij ρ δ ij to be: ( )( x j µ i ) T τ ij Σ i = τ ij w 2 ij x j µ i w 2 ij. The weight futio has bee hose to be ψ ( s) s where s = δ ij. A popular hoie of ψ ( s) is the Huber's ψ -futio that is defied by ψ ( s) = ψ ( s) where for s > 0 ψ s ( ) = s 0 s k ( p) 1 ( p) s > k 1 ( p) k 1 for a appropriate hoie of the "tuig" ostat k 1 ( p), whih is a futio of the dimesioality p. This seletio of ψ s ρ( s) = ( ) gives: 1 2 s2 0 s k 1 ( p) k 1 ( p)s 1. 2 k 2 1 ( p) s > k 1 ( p) The value of the tuig ostat is a futio of dimesioality. It also depeds o the amout of otamiatio i the data that is usually ot kow. Sie the traiig samples are represetative of the lasses, it is desirable that they are give more emphasis i the updates of the estimates. Therefore, i the proposed approah, the traiig samples are assiged uit weight. To do so, the value of k 1 ( p) is defied to be k 1 ( d ij ) ( p) = max ˆ where d ˆ 2 ij = ( y ij µ i ) T Σ 1 i ( y ij µ i ) ad y ij is the traiig sample j from lass i. I other words, the tuig ostat is seleted suh that the traiig samples are give uit weight ad the weights for the ulabeled samples are iversely proportioal to the square root of their distaes to the lass mea. Therefore, the weight assiged to eah sample a be expressed as: w ij = 1 max ˆ d ij max( d ˆ ij ) ( d ij ) d ij max( d ˆ ij ) < d ij < 7

8 where d 2 ij = ( x j µ i ) T Σ 1 i ( x j µ i ) is the squared distae of ulabeled samples x j. The iterative equatios for the mea ad ovariae estimates a the be expressed as: Σ i = m i µ i = m i ( y ij µ i ) y ij µ i y ij m i ( ) T τ ij w ij x j τ ij w ij m i τ ij w ij 2 τ ij w 2 ij ( x j µ i )( x j µ i ) T. EXPERIMENTAL RESULTS I the followig experimets, we ompare the performae of quadrati lassifiers usig the parameters estimated from traiig samples aloe (ML), the EM algorithm (EM) ad the proposed robust algorithm (REM). Experimets 1 through 4 are performed usig a portio of a AVIRIS data set take over NW Idiaa's Idia Pie test site i Jue The see otais four iformatio lasses: or-o till, soybea-o till, soybea-mi till ad grass. By visual ispetio of the image, the list of these groud over types is assumed to be exhaustive. A total of 20 haels from the water absorptio ad oisy bads ( , , 220) are removed from the origial 220 spetral haels, leavig 200 spetral features for the experimets. The test data ad the groud truth map are show i Fig. 1. The umber of labeled samples i eah lass is show i Table 1. Due to the limited labeled samples, we selet the umber of spetral haels at 10, 20, 50, 67 ad 100. These haels are seleted by samplig the spetral rage at fixed iterval. The traiig samples are radomly seleted ad the remaiig labeled samples are used for testig. The algorithms are repeated for 10 iteratios ad the lassifiatio is performed usig the Gaussia maximum likelihood lassifier. The maximum likelihood (ML) method usig oly the traiig samples to estimate the parameters is deoted as ML i the followig experimets. 8

9 Class Names No. of Labeled Samples Cor-o till 910 Soybea-o till 638 Soybea-mi till 1421 Grass 618 Table 1. Class Desriptio for AVIRIS Data. Fig. 1. A portio of AVIRIS Data ad Groud Truth Map (Origial i Color). Experimet 1 The first experimet is iteded to ompare EM ad REM without outliers i the data. To obtai data without outliers, we geerate sytheti data usig the statistis omputed from the labeled samples of the four lasses. A total of 2000 test samples per lass is geerated, 500 of whih are used as the traiig samples. Sie the traiig samples are seleted at radom, the experimet is repeated 5 times ad the mea lassifiatio auray is reorded. The mea auray is show i Fig. 2. 9

10 Auray (%) Number of Dimesios ML EM REM Fig. 2. Mea Auray for Experimet 1 with 500 Traiig Samples ad 1500 Test Samples. The results show that whe o outliers are preset i the data, the EM ad REM algorithms have similar performae ad both result i a better performae tha the maximum likelihood lassifier usig the traiig samples aloe. Sie there are may desig samples available, the best performae is obtaied at 200 features. Experimet 2 I this experimet, the sytheti data from the Experimet 1 is used with the exeptio that oly 250 traiig samples are seleted for eah lass. The umber of test samples is kept at Agai, o outliers are preset i the data. The results are show i Fig

11 Auray (%) Number of Dimesios ML EM REM Fig. 3. Mea Auray for Experimet 2 with 250 Traiig Samples ad 1500 Test Samples. Sie fewer traiig samples are used, the performae of the maximum likelihood lassifier (ML) usig the traiig samples aloe deteriorates. The delie is partiularly obvious at higher dimesioality. Compared to the previous experimet, the auray has dropped 7% at 200 features. However, whe ulabeled samples are used for the mixture model, the performae remais stable eve whe the umber of traiig samples delies. The results agai show that whe o outliers are preset i the data, the EM ad REM algorithms have omparable performae ad both ahieve better lassifiatio auray tha the ML lassifier without usig additioal ulabeled samples. Experimet 3 The previous experimet is repeated with oly 400 test samples geerated for eah lass. The umber of traiig samples per lass is 250. Agai, o outliers are preset i the data. The results are show i Fig. 4. Compared to the results from two previous experimets i whih may more ulabeled samples were used, the lassifiatio results 11

12 for all three methods deteriorate i this experimet. This deterioratio is maifested as the Hughes pheomeo. Hee, the likelihood parameter estimatio for the mixture model is show to be affeted by the umber of ulabeled samples relative to dimesioality. Speifially, it implies that 650 samples are still iadequate to haraterize these 200- dimesioal Gaussia distributios. The results agai idiate that without outliers, the EM ad REM algorithms have omparable performae ad both have better lassifiatio auray tha the ML lassifier without usig additioal ulabeled samples Auray (%) Number of Dimesios ML EM REM Fig. 4. Mea Auray for Experimet 3 with 250 Traiig Samples ad 400 Test Samples. 12

13 Experimet 4 This experimet is oduted usig the real samples from the data. Agai, sie all four lasses are represeted by the traiig samples, the lasses are assumed to be exhaustive. As idiated i Table 1, the umber of labeled samples is small. To retai eough test samples, oly about 200 traiig samples are hose for eah lass. Due to the limited labeled sample size, to obtai reasoably good iitial estimates for omparig the EM ad REM algorithms, the umber of spetral haels are seleted at 10, 20, 50, 67 ad 100. These spetral features are agai hose by samplig the spetral haels at fixed itervals. Fig. 5 shows the lassifiatio results at the seleted dimesios Auray (%) Number of Dimesios ML EM REM Fig. 5. Auray for Experimet 4 usig AVIRIS Data. The results show that the REM algorithm performs better tha the ML ad EM methods. This demostrates that although it is assumed that the see otais o outliers, there are some outlyig pixels that were ot idetified. This further justifies the motivatio 13

14 of usig a robust parameter estimatio method for the mixture model. The results also show that all methods exhibit the Hughes pheomeo. As disussed previously, the delie i performae at high dimesioality is aused by the limited umber of ulabeled samples available i the image. Fig. 6. Flightlie C1 Image ad Groud Truth Map (Origial i Color). 14

15 Experimet 5 This experimet is oduted usig a data set desigated Flightlie C1 (FLC1), whih is a 12-bad multispetral data take over Tippeaoe Couty, Idiaa by the M7 saer [10] i Jue, The data ad the groud truth map are show i Fig. 6. The traiig fields are marked i the groud truth map. The umber of labeled samples ad traiig samples i eah lass is show i Table 2. The parameters are estimated usig the traiig samples aloe, the EM algorithm with various threshold settigs, ad the REM algorithm. For the EM algorithm, two hi-square threshold values (1% ad 5%) are applied for ompariso. The lassifiatio results are plotted i Fig. 7. Class Names No. of Labeled Samples No. of Traiig Samples Alfalfa Bare Soil Cor Oats Red Clover Rye Soybeas Wheat Wheat Ukow Table 2. Class Desriptio for Flightlie C1 Data Auray (%) ML REM EM EM (1%) EM (5%) Fig. 7. Classifiatio Results for Flightlie C1 Data. 15

16 The etire Flightlie C1 image otais lasses with a few pixels suh as rural roads, farmstead ad water that are ot iluded i the traiig set. There may be other ukow lasses that are ot idetified i the groud truth iformatio. Therefore, it is highly likely that statistial outliers are preset i the image. This is ofirmed by experimetal results. The performae of the EM algorithm is sigifiatly lower tha those of ML, REM ad EM with thresholdig. Agai, the experimet demostrates that REM has similar performae as EM with thresholdig, but without the eed of settig a threshold. CONCLUSION I this paper, a robust method for parameter estimatio uder the mixture model (REM) is proposed ad implemeted for lassifyig multispetral data. This work is motivated by the fat that a multispetral image data set usually otais pixels of ukow lasses whih a be time-osumig to idetify. These pixels of ukow origi may have desity distributios quite differet from the traiig lasses ad ostitute statistial outliers. Without a list of exhaustive lasses for the mixture model, the expetatio maximizatio (EM) algorithm a overge to erroeous solutios due to the presee of statistial outliers. This problem eessitates a robust versio of the EM algorithm that iludes a measure of typiality for eah sample. The experimetal results have show that the proposed robust method performs better tha the parameter estimatio methods usig the traiig samples aloe (ML) ad the EM algorithm i the presee of outliers. Whe o outliers are preset, the EM ad REM have similar performaes ad both are better tha the ML approah. Speifially, whe there are may ulabeled samples available, the EM ad REM algorithms a mitigate the Hughes pheomeo sie they utilize ulabeled samples i additio to the traiig samples. Whe the umber of ulabeled samples are limited, both EM ad REM methods exhibit the Hughes pheomeo, but still ahieve better lassifiatio auray tha the ML approah at lower dimesioality. Despite the promisig results, the proposed REM algorithm has several limitatios. Sie the weight futio i the REM algorithm is based o lass statistis, the iitial parameter estimates are importat i determiig the overgee. I partiular, a good ovariae estimate requires a suffiiet umber of traiig samples. Whe the umber of traiig samples is lose to or less tha the dimesioality, the ovariae estimate beomes poor or sigular ad the EM or REM algorithm aot be applied. This eessitates the use of a ovariae estimatio method for limited traiig samples. Further details of this algorithm a be foud i [9]. 16

17 REFERENCES [1] A.P. Dempster, N.M. Laird, D.B. Rubi, "Maximum likelihood estimatio from iomplete data via EM algorithm," J. R. Statist. So., Vol. B39, pp. 1-38, [2] B. Shahshahai ad D. A. Ladgrebe, Classifiatio of Multi-spetral Data by Joit Supervised-Usupervised Learig, Purdue Uiversity, West Lafayette, IN, TR-EE 94-1, Jauary [3] G.F. Hughes, "O the mea auray of statistial patter reogizers," IEEE Tras. Iform. Theory., Vol. IT-14, pp , [4] R.A. Reder, H.F. Walker, "Mixture desities, maximum likelihood ad the EM algorithm," SIAM Review, Vol. 26, No. 2, pp , [5] P.J. Huber, "Robust estimatio of a loatio parameter," A. Math. Statist., Vol. 35, pp , [6] R.A. Maroa, "Robust M-estimators of multivariate loatio ad satter," A. Statist., Vol. 4, pp , [7] N.A. Cambell, "Mixture models ad atypial values," Math. Geol., Vol. 16, pp , [8] Y. Jhug, Bayesia Cotextual Classifiatio of Noise-Cotamiated Multi-variate Images, Ph.D. Dissertatio, Shool of Eletrial Egieerig, Purdue Uiversity, [9] S. Tadjudi, Classifiatio of High Dimesioal Data with Limited Traiig Samples, Ph.D. Thesis, Shool of Eletrial ad Computer Egieerig, Purdue Uiversity, 1998 (123 pages). Available from [10] Swai, P. H. ad S. M. Davis, eds., Remote Sesig: The Quatitative Approah, MGraw Hill, 1978, Chapt. 2. [11] Hoel, P. G., Port, S. C. ad Stoe, C. J., Itrodutio to Statistial Theory, Houghto Miffli Co., 1971, Chapter 3. 17