Updte Rules for Weghted Non-negtve FH*G Fctorzton Peter Peers Phlp Dutré Report CW 440, Aprl 006 Ktholeke Unverstet Leuven Deprtment of Computer Scence Celestjnenln 00A B-3001 Heverlee (Belgum)
Updte Rules for Weghted Non-negtve FH*G Fctorzton Peter Peers Phlp Dutré Report CW 440, Aprl 006 Deprtment of Computer Scence, K.U.Leuven Abstrct In ths report the dervton nd prove of convergence of weghted non-negtve fctorzton of the form FH*G s dscussed. The dervton nd proof s bsed on Lee nd Seung s orgnl dervton of Non-negtve Mtrx Fctorzton. Ths form s prtculrly suted to fctor mtrces exhbtng bnd-dgonl structure wth horzontl nd vertcl dscontnutes. Keywords : Non-negtve Mtrx Fctorzton.
Updte Rules for Weghted Non-negtve FH*G Fctorzton Peter Peers nd Phlp Dutré 1 Introducton Ths document s ntended to support []: A Compct Fctored Representton for Heterogeneous Subsurfce Sctterng. However, the derved fctorzton cn be used n other more generl contexts. More specflly, n ths techncl report we dcuss the dervton nd proof of the convergence of the followng non-negtve fctorzton form: R d = F H G. The notton ndctes component-wse mtrx multplcton: c j = (A B) j = j j. We wll follow the proof of Lee nd Seung [1] closely, nd mke necessry djustments where needed. Ths prtculr fctorzton form s well suted for compctly representng bnd-dgonl mtrces wth horzontl, nd vertcl structured dscontnutes. Furthermore we wll nclude weghtng mtrx W tht cn be used to plce more or less confdence n prtculr elements of the orgnl mtrx R d. Updte Rules for the Fctorzton Usng the followng updte rules ensures non-negtve fctorzton of R d : h µ h µ (F T (W G R d )) µ (F T (W G (F H G))) µ, (1) f f ((W G R d )H T ) (W G (F H G))H T ). () 1
3 Proof of Convergence We follow smlr pproch s n [1] to prove the convergence of these updte rules. We would lke to prove tht the updte rules (1) nd () mnmze the followng cost functon: F(h) = 1 (w (r (F h g ))), (3) where r, g, w nd h re correspondng columns n respectvely R d, G, W nd H. As n [1] we wll defne n uxlry functon G(h, h t ) whch dheres to the followng: In ths cse we cn updte h by: G(h, h) = F(h), (4) G(h, h t ) F(h). (5) h t+1 = rg mn h G(h, h t ). (6) If t, then h t h, thus t s suffcent to prove tht G(h, h t ) s n uxlry functon for the cost functon (3) n order to prove convergence. G cn be freely chosen, s long s the condtons (4), nd (5) re met. Keepng the updte rules (1), nd () n mnd, we defne G(h, h t ) by: G(h, h t ) = F(h t ) (h h t ) T F(h t ) + 1 (h ht ) T K(h t )(h h t ), (7) where K s dgonl mtrx defned by: K b (h t ) = b(f T (F h t g w )) h t. (8) It s obvous tht f h = h t n formul (7) condton (4) s met. Thus, we only need to prove the second condton (5) n order to prove tht G(h, h t ) s vld uxlry functon. Frst rewrte F(h) usng Tylor expnson. Note tht F(h) s second degree polynoml, nd thus the Tylor expnson only hs 3 terms: F(h) = F(h t ) + (h h t ) T F(h t ) + 1 (h ht ) T F(h t )(h h t ). (9) Note tht:
F(h) = ( 1 (w (r (F h g ))) ) = ( 1 (w r w s g (F h ) + (w (F h g )) )) nd = w r g F + 1 w g ( F F bh h b ) b = F T (s g w ) + (F T (F h g w )), F(h) = w g F F b b = F T (F g w ). Subtrctng the Tylor expnson (9) from the uxlry functon (7) results n n equvlent equton for the second condton condton (5): 0 (h h t ) T [K(h t ) F(h t )](h h t ). (10) Ths nequlty cn be proven by consderng the followng mtrx M(h t ): Then for ny vector v we hve: M b (h t ) = h t (K(h t ) F(h t ))h t b. v T Mv = b v M b v b = b v v t ( b (F T (F g w )) b F T (F g w )) b v b h t b = (vh t h t b(f T (F g w )) b v h t (F T (F g w )) b h t v b ) b = (F T (F g w )) b h t h t b[v v v b ] b = b = b 0. (F T (F g w )) b h t h t b[ 1 v + 1 v b v v b ] (F T (F g w )) b h t h t 1 b (v v b ) 3
Thus v T Mv s non-negtve for ny v snce F, g, w nd h re non-negtve. To understnd the step from the frst to the second equton consder: v h t K(h t )h t bv b = b b = v h t b (F T (F h t g w )) h t h t bv b v h t (F T (F h t g w )) = = v h t ( v h t ( b F ( F b h t bg w )) b F F b g w ) h t b = b v h t h t b( F F b g w ) = b v h t h t b(f T (F g w )) b. Thus, snce v T Mv s non-negtve for ny v, we hve proven (10), from whch follows tht G(h, h t ) s n uxlry functon for F. 4 Dervton of the Updte Rules We cn now derve the multplctve updte rules by computng: h t = rg mn h G(h, h t ), thus, by tkng the dervtve of G n terms of h, nd settng ths equl to 0: Rewrte ths s: 0 = G(h, ht ) h t+1 = F(h t ) + (h t+1 h t )K(h t ). nd thus: F(h t ) = (h t+1 h t )K(h t ) h t+1 = h 1 F(h t )K 1 (h t ). h t+1 = h t + F T (r g w ) (F T (F h t g w )) (F T (F h t g w )) h t = h t (F T (r g w ) (F T (F h t g w. )) 4
Resultng n the updte rule for H. A smlr result cn be obtned by nterchngng F nd H to obtn n updte rule for F. Note, tht the updte rules gurntee convergence, but not necessry to globl mnmum. Ths s well-known problem wth the multplctve updte rules for non-negtve mtrx fctorzton. 5 Concluson We hve derved nd proved the convergence of the updte rules of weghted mtrx fctorzton R d = F H G, by trvlly extendng the dervton nd proof of the orgnl non-negtve mtrx fctorzton pper of Lee nd Seung [1]. References [1] Dnel D. Lee nd H. Sebstn Seung. Algorthms for non-negtve mtrx fctorzton. In NIPS, pges 556 56, 000. [] Peter Peers, Krl vom Berge, Wojcech Mtusk, Rv Rmmoorth, Json Lwrence, Szymon Rusnkewcz, nd Phlp Dutré. A comct fctored representton of heterogeneous subsurfce sctterng. ACM Trnsctons on Grphcs, 006. 5