day month year documentname/initials 1

Transcription

1 ECE 599/692 Dp Larning Lctur 10 Rgularizd AE and Cas Studis Hairong Qi, Gonzalz Family Profssor Elctrical Enginring and Computr Scinc Univrsity of Tnnss, Knovill Outlin Lctur 9: Points crossd Gnral structur of AE Unsuprvisd Gnrativ modl? Th rprsntativ powr Basic structur of a linar autoncodr Dnoising autoncodr (DAE) AE in solving ovrfitting problm Lctur 10: Rgularizd AE and cas studis Lctur 11: VAE lading to GAN 2 Th prformanc of machin larning mthods is havily dpndnt on th choic of data rprsntation (or faturs) on which thy ar applid. much of th actual ffort in dploying machin larning algorithms gos into th dsign of prprocssing piplins and data transformations that rsult in a rprsntation of th data that can support ffctiv machin larning. Such fatur nginring is important but labor-intnsiv and highlights th waknss of currnt larning algorithms: thir inability to tract and organiz th discriminativ information from th data. [Bngio:2014] 3 day month yar documntnam/initials 1

2 Diffrnt approachs Probabilistic modls Capturs th postrior distribution of th undrlying planatory factors givn obsrvations Rconstruction-basd algorithms AE Gomtrically motivatd manifold-larning approachs 4 Issus Rquir strong assumptions of th structur in th data Mak svr approimations, lading to suboptimal modls Rly on computationally pnsiv infrnc procdurs lik MCMC 5 Basic structur of AE y z W 2 W 1 X y = f 1 (W 1 + b 1) z = g 2 (W 2y + b 2) } 1 = {W 1, b 1}, 2 = {W 2, b 2} X 1, 2 = arg min E q(x) [L H(X, g 2 (f 1 (X)))] 1, 2 6 day month yar documntnam/initials 2

3 Rgularizd AE Spars AE z y L( 1, 2 )=kx g 2 (f 1 (X))k s h(f 1 (X)) + r kw 1 k 1 7 Rgularizd AE Dnoising AE? [DAE:2008] X f y g 0 LH(, z) qd z 1, 2 = arg min E q(x, X) [L H(X, g 2 (f 1 ( X)))] { L 1, 2 } 0 q D ( ) g 0(f ( )) Th wll-known link btwn training with nois and rgularization 8 Application Eampl: AE in Spctral Unmiing 9 day month yar documntnam/initials 3

4 Spctral Unmiing Mathmatic formulation of HSI miing: = As + n : snsor radout A: sourc matri s: abundanc vctor n: masuring nois [Bannon, Natur Photonics 2009] Constraints Sum-to-on Nonngativ Unsuprvisd unmiing MVC-NMF GDME c minimiz f 0 ( s ) = å s j ln s j j =1 c subjct to h0 ( s ) = 1T s - 1 = 0, hi ( s ) = å Aij s j - i = 0, i = 1,..., N j =1 [1] L. Miao, H. Qi, Endmmbr traction from highly mid data using minimum volum constraind non-ngativ matri factorization, IEEE Transactions on Goscinc and Rmot Snsing, 45(3): , March (Highst Impact Papr Award) [2] L. Miao, H. Qi, H. Szu, A maimum ntropyapproach to unsuprvisd mid pil dcomposition, IEEE Transactions on Imag Procssing, 16(4): , April Unmiing and DL? Dp Larning [Ciznicki,SPIE 2012] Spctral Unmiing A good marriag? [CVPR 2012 Tutorial by Honglak L] AutoEncodr (AE) How to formulat th unmiing problm in th framwork of AutoEncodr? 1. What is ndmmbr? 2. What is abundanc? 3. How to incorporat th constraints? 4. What ar th constraints? X = AS vs. ^X = As(W1X) Part-basd AE assums Tid wight: W1 = AT Nonngativ wight: [1]Chn, Minmin, t al. "Marginalizd dnoising autoncodrs for domain adaptation." arxiv prprint arxiv: (2012). [2] Lmm, Andr, Rné Fli Rinhart, and Jochn Jakob Stil. "Efficint onlin larning of a non-ngativ spars autoncodr." ESANN day month yar documntnam/initials 4

5 AutoEncodr Cascad Marginalizd Dnoising AutoEncodr (mda) Non-ngativ Spars AutoEncodr (NNSA) Rui Guo, Wi Wang, Hairong Qi, Hyprspctral imag unmiing using cascadd autoncodr, IEEE Workshop on Hyprspctral Imag and Signal Procssing: Evolution in Rmot Snsor (WHISPERS), Tokyo, Japan, Jun 2-5, (Bst Papr Award) Discussion on th Ntwork Structur On tid wight X = AS = As(W 1 X) = W 1T s(w 1 X) That is, S = s(w 1 X) (a) (b) Discussion on th Ntwork Structur (cont d) On th nonngativ constraint X = AS = As(W 1 X) Assum s is linar: AW 1 =I day month yar documntnam/initials 5

6 Proposd Algorithm T 16 Proposd Algorithm Objctiv function: Th dnoising constraint: multilayr scald marginalizd dnosing autoncodr (mda): W n Nonngativity: Sum-to-on: day month yar documntnam/initials 6