HYPERSPECTRAL IMAGE CLASSIFICATIO BASED O A FAST BREGMA SPARSE MULTI OMIAL LOGISTIC REGRESSIO ALGORITHM. Portugal - (jun,

Transcription

1 HYPERSPECRAL IMAGE CLASSIFICAIO BASED O A FAS BREGMA SPARSE MULI OMIAL LOGISIC REGRESSIO ALGORIHM J. Li a, *, J. Bioucas-Dias a, Anonio Plaza b a Insiuo de elecomunicações, Insiuo Superior écnico, , Lisbon Porugal - (un, bioucas@lx.i.p b Deparmen of echnology of Compuers and Communicaions, Universiy of Exremadura E-007 Cáceres, Spain - aplaza@unex.es KEY WORDS: Sparse Mulinomial Logisic Regression (SMLR, hyperspecral classificaion, Bregman ieraion ABSRAC: he Sparse Mulinomial Logisic Regression (SMLR mehod inroduced in (Krishnapuram, 005 is among he sae-of-he-ar in supervised learning. However is applicaion o large daases, such as hyperspecral imagery is sill a raher challenging ask from he compuaional poin of view, someimes even impossible o perform. In his paper, he Bregman ieraion-based SMLR mehod (Bregman-SMLR recenly inroduced in (Bioucas-Dias, 008 is applied o hyperspecral daa classificaion problems. he Bregman mehod allows replacing a difficul, non-smooh convex problem wih a sequence of quadraic plus diagonal l-l problems which are very easy o solve (Bioucas-Dias, 008. Compared wih he SMLR algorihm, he reducion of compuaional complexiy is on he order of d(m- 3 (d is he number of feaures, and m is he number of classes. he effeciveness of he proposed mehod is evaluaed wih simulaed daa ses and a real AVIRIS image. Resuls are presened and compared wih ohers obained by sae-of-he-ar supervised algorihms.. I RODUCIO he sparse mulinomial logisic regression (SMLR mehod inroduced in (Krishnapuram, 005 is among he sae-of-he-ar in supervised learning. he core of he SMLR is he soluion of a wo-erm opimizaion problem: one erm is he logisic regression and he oher is a Laplacian prior which enforces sparseness, hus conrolling he machine complexiy. However, he SMLR applicaion o large daases, such as hyperspecral imagery, is sill a raher challenging ask from he compuaional poin of view, being someimes even impossible o perform. his is because SMLR has he complexiy of he ieraive reweighed leas squares (IRLS algorihm for maximum likelihood esimaion of feaure weighs. o lighen he SMLR compuaional burden, a fas sparse mulinomial logisic regression (FSMLR was inroduced in (Borges, 006 o implemen an ieraive scheme (based on he block Gauss-Seidel mehod o compue he feaure weighs of he decision funcion. he compuaional gain wih respec o he SMLR algorihm is of he order of he number of classes. he FSMLR algorihm is hus well-suied o hyperspecral daa ses wih a large number of classes. * Corresponding auhor. his work was suppored by Marie Curie Gran MES-C from he European Commission.

2 However, when dealing wih classificaion problems wih large raining ses resuling, for example, from kernel-based regression, he FSMLR mehod is sill very complex in compuaional erms. In his paper, he Bregman ieraion-based SMLR mehod (Bregman-SMLR recenly inroduced in (Bioucas-Dias, 008 is applied o hyperspecral daa classificaion problems. he Bregman mehod allows replacing a difficul, non-smooh convex problem wih a sequence of quadraic plus diagonal l-l problems which are very easy o solve. If d is he number of feaures and m is he number of classes, he complexiy of he Bregman-SMLR mehod is O(d, which is in conras wih he O((d(m- 3 figure of SMLR. As a resul, he reducion of compuaional complexiy is on he order of d(m- 3. In order o illusrae he effeciveness of he Bregman-SMLR mehod, we apply i o simulaed daa ses and real AVIRIS hyperspecral image and compare he obained resuls wih hose provided by he FSMLR, he suppor vecor machines (SVMs, and he linear discriminan analysis (LDA (Camps-Valls, 005 in erms of he following aspecs: overall accuracy; compuaional cos; 3 robusness o noise; and 4 number of he raining samples required.. MEHOD he SMLR used here is, basically, he algorihm inroduced in Krishnapuram e Al. (Krishnapuram, 005. he Bregman-SMLR solves he same opimizaion problem, bu uses he augmened Lagrangian framework. In his secion, we briefly review he SMLR he Bregman-SMLR mehods.. Sparse Mulinomial Logisic Regression (SMLR d n d Le x= [ x,..., xn] R, xi R be a vecor of ( ( m observed feaures, and y= [ y,..., y ] a - of-m encoding of he classes (n is he number of samples, d is he number of feaures, and m is he number of classes. he goal of classificaion is o esimae y given x. Suppose ( i is he feaure-weigh vecor corresponding o class i; hen, according o he mulinomial logisic regression model, he probabiliy ha a given sample x belongs o class i is given as follows: p y = x = ( i (, ( i exp ( h( x m ( exp ( h( x = ( where, h( x = [ h ( x,..., h ( ] l x is a vecor of l fixed funcions of he inpu. Usual choices for h(x are linear maps h( x [,,,...,, ] i = xi xi n, where xi, means he h componen (band of x i and kernels i h( x = [, K( x, x,..., K( x, x ], where K(, is some symmeric kernel funcion. In his paper, we only consider kernels of he radial basis funcion (RBF class.; for he nonlinear mapping guaranees ha he ransformed samples are more likely o be linearly separable. he SMLR mehod uses he Maximum A poseriori (MAP mehod o esimae he componens of from he raining se: MAP = arg max L( [ l p ] = arg max ( log ( ( where l( is he log-likelihood funcion, l y x x n m m ( ( ( i i i = log exp( = i= i= (3 and p( is a Laplacian prior on, which means ha p( exp( λ, where λ is a regularizaion parameer conrolling he degree of sparseness of. According o he bound MAP opimizaion approach (Lange, 000, he loglikelihood funcion l( can be opimized by ieraively maximizing a surrogae funcion Q, such ha: n

3 = arg max Q( log p( (4 While he log-likelihood funcion is concave, he surrogae funcion Q( ' can be deermined by using a bound on he Hessian H of he log-likelihood. Le B be a negaive definie marix such ha H ( B is posiive semi-definie, i.e., H ( B for any. A valid surrogae funcion is (B o&& hning, 99, Q( = ( g( B B = (5 n B I / m x x (6 where is he Kronecher marix produc, = [,,...,], and n ' ( = ( ( = g y p x (7 where y = y, y,..., y and ' ( ( ( m ( ( m ( (,..., ( p = p p. Wih he inclusion of a Laplacian prior, he obecive funcion becomes L( = l( λ = Q( λ he esimaes of are hen given by: MAP= arg max L( = arg max Q( λ (8 (9 = arg max ( g( B B λ = arg min ( g( B B λ I is no easy o minimize (9 direcly in closed form. A line of aack is o replace he l norm wih a lower quadraic bound in order o ge a surrogae funcion o ieraively opimize he log-prior. his leads he updae funcion = λ Λ ( B ( B g( Where Λ = diag { } ( d,..., ( m (0 ( Numerically, (0 is equivalen o solve (Krishnapuram, 005: λ =Γ ( Γ B Γ I Γ ( B g( where ( { } d ( m,..., Γ =diag and sands for he ih value of vecor ( i ( (3. Now i is possible o esimae he MAP mulinomial logisic regression wih a Laplacian prior by using he classical IRLS mehod. And he complexiy is he same as he IRLS algorihm for ML esimaion.. Bregman-SMLR he compuaional cos involved in solving he linear sysem presened in ( is O((dm 3, which is prohibiive when dealing wih large daases, eiher wih large number of feaures, or wih a very large raining daase. Recenly, a Bregman ieraion based sparse mulinomial logisic regression (Bregman-SMLR was inroduced by J. Bioucas (Bioucas-Dias, 008, which made possible o deal efficienly wih large daa ses. In his secion, we briefly review he Bregman-SMLR algorihm. In expression (9, suppose ha ν =. hen, we can replace he problem wih he following one:, ν = arg min ( g( B ( ν, s..: = ν B λ ν (4

4 (4 can be ieraively solved (Bioucas-Dias, 008 as follows:, ν = arg min ( g( B ( b ν, β B λ ν ν b = b ( ν (5 he above minimizaion is sill a difficul problem. However, he minimizaions wih respec o eiher or ν are very easy o compue. Exploiing his fac he Bregman- SMLR ieraive scheme proposed in (Bioucas- Dias, 008 is a follows: β ν β ν = arg min λ ν ν b ν, b = b ( ν = arg min ( g( B B b ν, (6 he firs sep leads o he updae funcion of : = ( B β I ( g( B β ( ν b ( (7 he second sep amouns o apply he sof shrinkage funcion o updae eν : ν = sofshrink, λ / β ( b (8 3. Simulaed hyperspecral images Simulaed daases are used o es he proposed mehod in comparison wih he FSMLR algorihm, which demonsraes high qualiy in supervised hyperspecral classificaion as shown in (Borges, 006. he size of he simulaed images is (00 00 is he spaial size of he simulae images, and 4 is he number of specral bands., and 0 classes. he feaurures are Gausssian vecors wih means seleced from he USGS (Clark, 007 library. And covariance marix σ I, where I is he ideniy marix. he parameer σ deermines he signal-o-noise raio (SNR, SNR 0log E[ x x] / E[ nσ ] (n is he as 0( number of samples. Figure, op, shows he overall accuracy (OA as a funcion of SNR using 00 raining samples (% of he whole image.he remaining samples are used for validaion. In his case, he Bregman-SMLR algorihm ouperforms he FSMLR. Figure, middle, shows OA resuls as a funcion of he number of raining samples wih SNR se o 5. Boh algorihms obain similar OA. Figure, boom, shows he compuaional cos as a funcion of he number of raining samples. As expeced, he Bregman-SMLR is much faser han FSMLR. Where B and g( are given in (6 and (7, respecively. According o (6, B is fixed, so he par of ( B β I in (7 doesn need o be updaed during he ieraions. I can be compued before hand, which grealy lighen he compuaional complexiy, leading o a cos of O(d. 3. EXPERIME RESUL In his secion, experimenal resuls will be presened. In he firs par, simulaed daa ses are performed o analysis he compuaional cos, robusness o noise and limied raining samples. In he second par shows he resuls obained from real hyperspecral imagery.

5 algorihm. Considering he compuaional coss a funcion of he raining se size, he Bregman- SMLR achieves much beer performance han he FSMLR mehod. For 50% of he raining se, he FSMLR needs seconds while i us needs seconds of he Bregman- SMLR. Bregman- FSMLR SVMs LDA SMLR 9.3% 90.5% ~9% 9.08% able. Comparison of he proposed mehod wih he resuls from (Camps-Valls, 005; Borges, 006 on a real daase. Figure. Resuls on simulaed daa ses. Each value was obained from 00 Mone Carlo runs. 3. Experimens on real hyperspecral daa Experimens are also carried ou using an AVIRIS specromeer image aken over norhwes Indianas Indian Pine es sie in June 99 (Landgrebe, 99. I conains pixels and 0 bands. Noisy bands in number of 0, namely due o waer absorpion, were discarded during he experimens. he ground ruh daa image conains 6 classes, 7 of which were discarded for insufficien number of raining samples. he remaining 9 classes were used o generae a se of 4757 raining samples, wih random pariion, and 4588 es samples. able compares he OA resuls wih sae-of-he-ar supervised algorihms. he Bregman-SMLR obained much beer resuls han LDA, similar or comparable resuls o FSMLR and SVMs. Figure. shows he compuaional cos as a funcion of he number of raining samples in comparison wih FSMLR Figure. Resuls on a real daase 4. CO CLUSIO S In his paper, a fas Bregman sparse mulinomial logisic regression algorihm (Bregman-SMLR is applied o hyperspecral imagery. Compared wih he SMLR algorihm, i is much faser and more efficien.. he performance of he proposed approach was evaluaed by using simulaed daa ses and real AVIRIS hyperspecral imagery. he resuls obained show high qualiy in supervised hyperspecral classificaion in erms of overall accuracy, robusness o noise, low complexiy and limied raining samples. ACK OWLEDGEME he auhors would like o hank Prof. Landgrebe (Purde Universiy, USA for providing he AVIRIS daa.

6 REFERE CES Bioucas-Dias, J., 008. Bregman-SMLR: A fas sparse logisic regression algorihm. echnical Repor, Insiuo Superior écnico, ULisbon. Borges, J. and Bioucas-Dias, J., 006. Fas Sparse Mulinomial Regression Applied o Hyperspecral Daa. Inernaional Conference on Image Analysis and ICIAR. B o&& hning, D., 99. Mulinomial logisic regression algorihm. Annals of he Insiue of Sa-isical Mahemaics, 44, pp Camps-Valls, G. and Bruzzone, L., 005. Kernel-based mehods for hyperspecral image classificaion. IEEE ransacions on Geoscience and Remoe Sensing, 43(6, pp Clark, R. N., Swayze, G. A., Wise, R., Livo, E., Hoefen,., Kokaly, R. And Suley S.J., 007. USGS digial specral library splib06a. U.S. Geological Survey, Digial Daa Series 3,. Krishnapuram, B., Carin, L., Figueiredo, M. and Haremink, A., 005. Sparse Mulinomial Logisic Regression: Fas Algorihms and Generalizaion Bounds. IEEE ransacions on Paern Analysis and Machine Inelligence PAMI, 7(6, pp Landgrebe, D., 99. AVIRIS NW Indiana's Indian pine. Url: hp://cobweb.ecn.purdue. edu/biehl/mulispec/ Lange, K., Huner, D. and Yang, I., 000. Opimizing ransfer using surrogae obecive funcions. Journal of Compuaional and Graphical Saisics, 9, pp. -59.