A Riemannian Limited-Memory BFGS Algorithm for Computing the Matrix Geometric Mean

Size: px

Start display at page:

Download "A Riemannian Limited-Memory BFGS Algorithm for Computing the Matrix Geometric Mean"

Norma Arnold
5 years ago
Views:

1 A Remannan Lmted-Memory BFGS Algorthm for Computng the Matrx Geometrc Mean Xnru Yuan 1, Wen Huang 2, P.-A. Absl 2 and Kyle A. Gallvan 1 1 Florda State Unversty 2 Unversté Catholque de Louvan July 3, 2016 A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 1

2 Symmetrc Postve Defnte (SPD) Matrx Defnton A symmetrc matrx A s called postve defnte f x T Ax > 0 for all nonzero vector x R n. 2 2 SPD matrx v λv u λu u λu 3 3 SPD matrx w λw v λv A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 2

3 Possble Applcatons of SPD Matrces Dffuson tensors n medcal magng A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 3

4 Motvaton of Averagng SPD Matrces Subtas n nterpolaton schemes Example: retreve a tensor feld between measured tensor values at sparse ponts [PFA06] 1 Fgure : Intal tensor measurements Fgure : Retreved tensor feld after a certan nterpolaton scheme 1 X. Pennec, P. Fllard and N. Ayache, A Remannan framewor for tensor computng, Internatonal Journal of Computer Vson, 2006 A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 4

5 Averagng Schemes: from Scalars to Matrces Let A 1,..., A K be SPD matrces. A =1 Not approprate n many practcal applcatons Generalzed arthmetc mean: 1 K K A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 5

6 Averagng Schemes: from Scalars to Matrces Let A 1,..., A K be SPD matrces. A =1 Not approprate n many practcal applcatons Generalzed arthmetc mean: 1 K K A A+B 2 B det A = 50 det( A+B 2 ) = det B = 50 A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 5

7 Averagng Schemes: from Scalars to Matrces Let A 1,..., A K be SPD matrces. A =1 Not approprate n many practcal applcatons Generalzed arthmetc mean: 1 K K A A+B 2 B det A = 50 det( A+B 2 ) = det B = 50 Generalzed geometrc mean: (A 1 A K ) 1/K Not approprate due to non-commutatvty How to defne a matrx geometrc mean? A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 5

8 Desred Propertes of a Matrx Geometrc Mean The desred propertes are gven n the ALM lst 2, some of whch are: Invarance under permutaton: G(A π(1),..., A π(k) ) = G(A 1,..., A K ) wth π a permutaton of (1,..., K) 2 T. Ando, C.-K. L, and R. Mathas, Geometrc means, Lnear Algebra and Its Applcatons, 385: , 2004 A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 6

9 Desred Propertes of a Matrx Geometrc Mean The desred propertes are gven n the ALM lst 2, some of whch are: Invarance under permutaton: G(A π(1),..., A π(k) ) = G(A 1,..., A K ) wth π a permutaton of (1,..., K) Consstency wth scalars: f A 1,..., A K commute, then G(A 1,..., A K ) = (A 1,..., A K ) 1/K 2 T. Ando, C.-K. L, and R. Mathas, Geometrc means, Lnear Algebra and Its Applcatons, 385: , 2004 A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 6

10 Desred Propertes of a Matrx Geometrc Mean The desred propertes are gven n the ALM lst 2, some of whch are: Invarance under permutaton: G(A π(1),..., A π(k) ) = G(A 1,..., A K ) wth π a permutaton of (1,..., K) Consstency wth scalars: f A 1,..., A K commute, then G(A 1,..., A K ) = (A 1,..., A K ) 1/K Invarance under nverson: G(A 1,..., A K ) 1 = G(A 1 1,..., A 1 K ) 2 T. Ando, C.-K. L, and R. Mathas, Geometrc means, Lnear Algebra and Its Applcatons, 385: , 2004 A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 6

11 Desred Propertes of a Matrx Geometrc Mean The desred propertes are gven n the ALM lst 2, some of whch are: Invarance under permutaton: G(A π(1),..., A π(k) ) = G(A 1,..., A K ) wth π a permutaton of (1,..., K) Consstency wth scalars: f A 1,..., A K commute, then G(A 1,..., A K ) = (A 1,..., A K ) 1/K Invarance under nverson: G(A 1,..., A K ) 1 = G(A 1 1,..., A 1 K ) Determnant equalty: det(g(a 1,..., A K )) = (det(a 1 ) det(a K )) 1/K 2 T. Ando, C.-K. L, and R. Mathas, Geometrc means, Lnear Algebra and Its Applcatons, 385: , 2004 A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 6

12 Geometrc Mean of SPD Matrces A well-nown mean on the manfold of SPD matrces s the Karcher mean [Kar77]: G(A 1,..., G K ) = arg mn X S n K K dst 2 (X, A ), where dst(x, Y ) = log(x 1/2 YX 1/2 ) F s the dstance under the Remannan metrc =1 g(η X, ξ X ) = trace(η X X 1 ξ X X 1 ). It has been shown recently [LL11] that the Karcher mean satsfes all the geometrc propertes n the ALM lst. A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 7

13 Remannan Manfolds A Remannan manfold M s a smooth set wth a smoothly-varyng nner product on the tangent spaces. A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 8

14 Manfold of SPD Matrces Let S n ++ be the manfold of SPD matrces Tangent space at X S n ++: T X S n ++ = {S R n n : S = S T } Dmenson: d = n(n + 1)/2 Remannan metrc: g(η X, ξ X ) = trace(η X X 1 ξ X X 1 ) A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 9

15 Algorthms G(A 1,..., G K ) = arg mn X S n K K dst 2 (X, A ), =1 Remannan steepest descent [RA11] Remannan steepest descent, conjugate gradent, BFGS, and trust regon Newton methods [JVV12] Rchardson-le teraton [BI13] Lmted-memory Remannan BFGS method [YHAG16] A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 10

16 Karcher Mean of SPD Matrces Hemsttchng phenomenon for steepest descent well-condtoned Hessan ll-condtoned Hessan Small condton number fast convergence Large condton number slow convergence A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 11

17 Karcher Mean of SPD Matrces Condtonng of the Remannan Hessan of F at the mnmzer: For the cost functon F (X ) = 1 2K K =1 dst2 (A, X ), we have Hess F (X )[ X, X ] 1 X ln(max κ ), 2 where κ s the condton number of A. If max κ = 10 10, then 1 + ln(max κ ) A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 12

18 Optmzaton Algorthm: from Eucldean to Remannan Update formula: x +1 = x + α η, where η s the search drecton and α s the stepsze. Search drecton: η = B 1 grad f (x ). A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 13

19 Optmzaton Algorthm: from Eucldean to Remannan Update formula: x +1 = x + α η, where η s the search drecton and α s the stepsze. Search drecton: η = B 1 grad f (x ). x x + α η R x (α η ) Optmzaton on a Manfold A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 13

20 Retracton Defnton A retracton s a mappng R from T M to M satsfyng the followng: R s contnuously dfferentable R x (0) = x DR x (0)(η) = η Eucldean Remannan x +1 = x + α η x +1 = R x (α η ) A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 14

21 Remannan Optmzaton Algorthm Eucldean Remannan x +1 = x + α η x +1 = R x (α η ) Steepest Descent: η = grad f (x ) Newton s Method: Quas-Newton Method: η = Hess f (x ) 1 grad f (x ) η = B 1 grad f (x ) A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 15

22 Remannan Gradent and Hessan Remannan gradent: grad F (X ) = 1 K K =1 A 1/2 log(a 1/2 XA 1/2 )A 1/2 X 1/2 Remannan Hessan: Hess F (X )[ξ X ] = 1 2K + 1 K K =1 ξ X log(a 1 X ) 1 2K K =1 K =1 X D(log)(A 1 X )[A 1 ξ X ] log(xa 1 )ξ X A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 16

23 Rchardson-le Iteraton [BI13] Update formula: X +1 = X αx 1/2 K =1 log(x 1/2 A 1 X 1/2 )X 1/2 Stepsze: α = 2/ K =1 κ + 1 κ 1 log κ, where κ s the condton number of matrx X 1/2 A 1 X 1/2 Local lnear convergence s proved A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 17

24 Rchardson-le Iteraton, Ctd The Rchardson-le teraton can be consdered from the vew of Remannan optmzaton algorthm X +1 = R X (αη ) Search drecton: η = grad F (X ) Choce of retracton: R X (η) = X + η Stepsze α satsfes the Wolfe condton A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 18

25 BFGS Quas-Newton Algorthm: from Eucldean to Remannan Search drecton: η = B 1 grad f (x ) B update: B +1 = B B s s T B s T B s + y y T y T s, where s = x +1 x, and y = grad f (x +1 ) grad f (x ). A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 19

26 BFGS Quas-Newton Algorthm: from Eucldean to Remannan Search drecton: η = B 1 grad f (x ) B update: B +1 = B B s s T B s T B s + y y T y T s, where s = x +1 x, and y = grad f (x +1 ) grad f (x ). On a manfold: replaced by R 1 x (x +1 ) on dfferent tangent spaces A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 19

27 Vector Transport Vector Transport Transport a tangent vector from one tangent space to another T ηx ξ x, denotes transport of ξ x to tangent space of R x (η x ) A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 20

28 Remannan BFGS (RBFGS) Algorthm Update formula: x +1 = R x (α η ) wth η = B 1 grad f (x ) B update [HGA15]: B +1 = B B s ( B s ) + y y ( B s ) s y s, where s = T α η α η, y = β 1 grad f (x +1 ) T α η grad f (x ), and B = T α η B Tα 1 η. A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 21

29 Remannan BFGS (RBFGS) Algorthm Update formula: x +1 = R x (α η ) wth η = B 1 grad f (x ) B update [HGA15]: B +1 = B B s ( B s ) + y y ( B s ) s y s, where s = T α η α η, y = β 1 grad f (x +1 ) T α η grad f (x ), and B = T α η B Tα 1 η. Stores and transports B 1 as a dense matrx Requres excessve computaton tme and storage space for large-scale problem A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 21

30 Lmted-memory RBFGS (LRBFGS) Remannan BFGS: B +1 = B B s ( B s ) + y y ( B s ) s y s, where s = T α η α η, y = β 1 grad f (x +1 ) T α η grad f (x ), and B = T α η B Tα 1 η Lmted-memory Remannan BFGS: Stores only the m most recent s and y Transports those vectors to the new tangent space rather than the entre matrx B 1 Computatonal and storage complexty depends upon m A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 22

31 Lmted-memory RBFGS (LRBFGS), Ctd B 1 update [HGA15]: B 1 = Ṽ Ṽ 1... Ṽ m H 0 +1Ṽ m... Ṽ 1 Ṽ + ρ m Ṽ Ṽ 1... Ṽ m+1s (+1) + + ρ s (+1) s (+1), s (+1) m m Ṽ m+1... Ṽ 1 Ṽ where Ṽ = d ρ y (+1) s (+1), ρ = g(y 1,s ), H +1 0 = g(s,y ) g(y,y ) d, and s (+1) represents a tangent vector n T x+1 M gven by transportng s, and lewse for y (+1) Avods expensve matrx operatons Requres less memory A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 23

32 Complexty Comparson RBFGS Lmted-memory RBFGS Acton Complexty Acton Complexty get B 1 +1 from B 1 O(d 2 ) - - B 1 η O(d 2 ) B 1 η O(md) d denotes the dmenson of SPD manfold A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 24

33 Algorthm Comparson Not sure where to put ths table Algorthm Remannan steepest descent Remannan BFGS Remannan lmted-memory BFGS Remannan Newton s method Convergence global, lnear global(convex), superlnear global(convex), faster than lnear global, quadratc A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 25

34 Outlne Bacground of the SPD Karcher mean computaton A bref descrpton of the lmted-memory RBFGS (LRBFGS) method Apply the LRBFGS to the SPD Karcher mean computaton Practcal mplementaton Experments on varous data sets A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 26

35 Implementatons Cost functon: F (X ) = 1 2K Remannan gradent: = 1 2K K dst 2 (A, X ) =1 K log(a 1/2 =1 XA 1/2 ) 2 F grad F (X ) = 1 K K =1 A 1/2 log(a 1/2 XA 1/2 )A 1/2 X 1/2 The domnant computaton tme s on the functon evaluaton A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 27

36 Implementatons Effcent representatons of tangent vectors, see [YHAG16] Retracton Exponental mappng: Exp X (ξ X ) = X 1/2 exp(x 1/2 ξ X X 1/2 )X 1/2 Vector transport Parallel translaton: T pηx (ξ X ) = Qξ X Q T, where Q = X 1/2 exp( X 1/2 η X X 1/2 2 )X 1/2 A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 28

37 Implementatons Effcent numercal representatons of tangent vectors Retracton Exponental mappng: Exp X (ξ X ) = X 1/2 exp(x 1/2 ξ X X 1/2 )X 1/2 Second order retracton [JVV12]: R X (ξ X ) = X + ξ X ξ X X 1 ξ X Vector transport Parallel translaton: T pηx (ξ X ) = Qξ X Q T, where Q = X 1/2 exp( X 1/2 η X X 1/2 2 )X 1/2 A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 29

38 Implementatons Effcent numercal representatons of tangent vectors Retracton Exponental mappng: Exp X (ξ X ) = X 1/2 exp(x 1/2 ξ X X 1/2 )X 1/2 Second order retracton: R X (ξ X ) = X + ξ X ξ X X 1 ξ X Vector transport Parallel translaton: T pηx (ξ X ) = Qξ X Q T, where Q = X 1/2 exp( X 1/2 η X X 1/2 2 )X 1/2 Vector transport by parallelzaton 3 : essentally an dentty 3 W. Huang, P.-A. Absl, and K. A. Gallvan, A Remannan symmetrc ran-one trust-regon method, Mathematcal Programmng, 2014 A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 30

39 Numercal Results I: K = 100, 3 3, d = 6, m = 2 1 κ(a ) RSD 10 2 RSD 10 0 RCG LRBFGS 10 0 RCG LRBFGS 10-2 RL Iteraton 10-2 RL Iteraton dst(µ,x) dst(µ,xt) teratons tme (s) 10 3 κ(a ) RSD 10 2 RSD 10 0 RCG LRBFGS 10 0 RCG LRBFGS RL Iteraton RL Iteraton dst(µ,x) dst(µ,xt) teratons tme (s) Fgure : Evoluton of averaged dstance between current terate and the exact Karcher mean wth respect to tme and teratons A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 31

40 Numercal Results II: K = 30, , d = 5050, m = 2 1 κ(a ) RSD RCG LRBFGS RL Iteraton RSD RCG LRBFGS RL Iteraton dst(µ, Xt ) dst(µ, X ) teratons tme (s) 104 κ(a ) RSD RCG LRBFGS RL Iteraton dst(µ, Xt ) dst(µ, X ) RSD RCG LRBFGS RL Iteraton teratons tme (s) Fgure : Evoluton of averaged dstance between current terate and the exact Karcher mean wth respect to tme and teratons A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 32

41 Summary Analyze the condtonng of the Hessan of the Karcher cost functon Apply a Remannan verson of lmted-memory BFGS method to computng the SPD Karcher mean Present effcent mplementatons Compare wth the state-of-the-art methods The proposed LRBFGS method s seen to be the method of choce for computng the SPD Karcher mean when the data matrces ncrease n sze or number. A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 33

42 Implementaton n Matlab F (X ) = 1 2K = 1 2K = 1 2K K log(a 1/2 =1 XA 1/2 ) 2 F K log(a 1/2 LL T A 1/2 =1 K log[(a 1/2 =1 L)(A 1/2 ) 2 F L) T ] 2 F Let S = A 1/2 L, and SS T = QUQ T, where QQ T = I, and U s an upper trangle matrx whose dagonal elements are the egenvalues. Then F (X ) = 1 2K = 1 2K K log(ss T ) 2 F =1 K Q log(dag(u))q T 2 F =1 A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 34

43 References I D. A. Bn and B. Iannazzo. Computng the Karcher mean of symmetrc postve defnte matrces. Lnear Algebra and ts Applcatons, 438(4): , W. Huang, K. A. Gallvan, and P.-A. Absl. A Broyden class of quas-newton methods for Remannan optmzaton. SIAM Journal on Optmzaton, 25(3): , B. Jeurs, R. Vandebrl, and B. Vandereycen. A survey and comparson of contemporary algorthms for computng the matrx geometrc mean. Electronc Transactons on Numercal Analyss, 39: , H. Karcher. Remannan center of mass and mollfer smoothng. Communcatons on Pure and Appled Mathematcs, A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 35

44 References II Jmme Lawson and Yongdo Lm. Monotonc propertes of the least squares mean. Mathematsche Annalen, 351(2): , Xaver Pennec, Perre Fllard, and Ncholas Ayache. A remannan framewor for tensor computng. Internatonal Journal of Computer Vson, 66(1):41 66, Q. Rentmeesters and P.-A. Absl. Algorthm comparson for Karcher mean computaton of rotaton matrces and dffuson tensors. In 19th European Sgnal Processng Conference, pages , Aug Xnru Yuan, Wen Huang, P.-A. Absl, and Kyle A. Gallvan. A Remannan lmted-memory BFGS algorthm for computng the matrx geometrc mean. In ICCS 2016, A LRBFGS Algorthm for Computng the Matrx Geometrc Mean 36

Vector Norms. Chapter 7 Iterative Techniques in Matrix Algebra. Cauchy-Bunyakovsky-Schwarz Inequality for Sums. Distances. Convergence.

Vector Norms. Chapter 7 Iterative Techniques in Matrix Algebra. Cauchy-Bunyakovsky-Schwarz Inequality for Sums. Distances. Convergence. Vector Norms Chapter 7 Iteratve Technques n Matrx Algebra Per-Olof Persson persson@berkeley.edu Department of Mathematcs Unversty of Calforna, Berkeley Math 128B Numercal Analyss Defnton A vector norm