The Robust Low Rank Matrix Factorization in the Presence of. Zhang Qiang

Size: px

Start display at page:

Download "The Robust Low Rank Matrix Factorization in the Presence of. Zhang Qiang"

Samuel Craig
5 years ago
Views:

1 he Robust Low Rank Matrix Factorization in the Presence of Missing data and Outliers Zhang Qiang

2 Content he definition of matrix factorization and its application he methods for matrix factorization in the presence of missing data he L Wiberg Algorithm for matrix factorization in the presence pese ceof missing data a and doutliers he Huber Wiberg Algorithm for matrix factorization

3 he definition of matrix factorization and its application i In linear algebra, matrix factorization, also called matrix decomposition, matrix approximation, is a factorization of a matrix ti into some canonical lform. here are many different matrix decompositions, such as LU, Cholesky, QR, SVD, which are utilized amonga a particular class of problems. In many vision problems, such as Structure from Motion(SFM), motion estimation, object recognition, and object tracking, matrix factorization is the decomposition of measurements or observation data into the product of two low rank matrices, one of which is a lower dimensional subspace within the original high dimension.

4 he definition of matrix factorization and its application Matrix factorization of M M d n = U V dn dk nk Where d is the dimension of the input data space; n is the number of input data items; k is the dimension of the linear subspace. he k columns of the matrix U are the basis of the linear subspace. In SFM, M includes the observed points in dff different images, U is the stake of all camera matrices, V is the matrix of space points.

5 he definition of matrix factorization and its application SVD for matrixfactorization When the observation matrix is the matrix without missing any element, the common method is SVD decomposition. he procedure as follow SVD ) ; M = U D V d n d d d n n n SVD ˆ M d n = U d kdk kvn k U dk U d d D k k 2), where includes the first k columns of, is a diagonal matrix with the first k diagonal elements of D dn, and V nk includes the first k columns of V n n ; SVD 3) ˆ ˆ M, where. d n= Ud kv Vˆ n k nk = Dkk Vnk SVD decomposition is equivalent to least squares minimization, e.g. min M UV.In computer vision, we F directly use SVD decompositionto to compute affine reconstruction.

6 he methods for matrix factorization in the presence of missing data When the observation matrix is the matrix with missing elements, SVD decomposition doesn't work. We need change the form of originalproblem problem. he followingequations are equivalent. M m = = U V dn dk nk ij UV il jl l m = GUv ( ) m= FVu () where m = [ m,,, m2, L, m d ] O and is the row of u muv,, i i i G ( U ) M, U, V. u d u = O O u d F ( V ) v M v n = O v M v n

7 he methods for matrix factorization in the presence of missing data So each element of M is equal to inner product of the rows from matrices U and V, which creates a set of equations. If there are missing elements in M, we just ignore these corresponding equations. For example, m u u2 m2 u u2 v m 3 u u 2 m m2 m3 u u2 m2 u2 u v2 22 vv2 v3 m2 m22 m23 u2 u 22 m v2 = 22 = u2 u 22 v2 v22 v 32 v22 m3 m32 m33 u3 u32 m23 u2 u22 v m3 u3 u 32 3 v 32 m 32 u3 u 32 m 33 u 3 u32

8 he methods for matrix factorization in the presence of missing data So each element of M is equal to inner product of the rows from matrices U and V, which creates a set of equations. If there are missing elements in M, we just ignore these corresponding equations. For example,? u u2 m2 u u2 v m 3 u u 2? m2 m3 u u2 m2 u2 u v2 22 vv2 v 3 m2 m22 m23 u2 u 22 m v2 = 22 = u2 u 22 v2 v22 v 32 v22 m3? m33 u3 u32 m23 u2 u22 m v 3 u3 u 32 3 v 32? u3 u 32 m33 u3 u32

9 he methods for matrix factorization in the presence of missing data So each element of M is equal to inner product of the rows from matrices U and V, which creates a set of equations. If there are missing elements in M, we just ignore these corresponding equations. For example, m2 u u2 v m3 u u2 v? m2 m3 u u2 m 2 u2 u 22 vvv 2 3 v m2 m22 m23 = u2 u22 m22 = u2 u22 v2 v22 v32 v m3? m33 u3 u32 m23 u2 u22 v m3 u3 u32 m33 u3 u v 32 hroughout the remainder of this ppt, we also use original notation for modified vectors or matrices

10 he methods for matrix factorization in the presence of missing data Alternated Least Squares Algorithm he optimization problem is min Φ( U, V ) U, V 2 where Φ( UV, ) = M UV. 2 o find the minimum of Φ( UV, ), we search for a solution to the equations Φ/ = u Φ/ = v 0.Using the notations mentioned above, these are expressed as Φ/ u FV ( ) ( FV ( ) u m) 0 = = Φ/ v GU ( ) ( GU ( ) v m) 0 AKAYUKI OKAANI AND KOICHIRO DEGUCHI. On the Wiberg Algorithm for Matrix Factorization in the Presence of Missing Components.IJCV,2007.

11 he methods for matrix factorization in the presence of missing data Alternated Least Squares Algorithm When considering the two equations independently, solutions are given by ˆ = ( ( ) ( )) ( ) u F V F V F V m ˆ = ( ( ) ( )) ( ) v G U G U G U m he ALS algorithm updates u from v,and v from u in an alternative manner, starting from some initial values (0) (0) of u, v.

12 he methods for matrix factorization in the presence of missing data Gauss Newton Algorithm Defining x= u, v, we rewrite Φ( UV, ) as Φ( x) = f f 2 where f = Fu m = Gv m. Here, we will find a solution x to Φ( x)/ x= 0.he Newton s algorithm seeks a solution by iteratively updating x as x+ Vx x. Vx is computed as a 2 2 solution to Φ( x)/ + x ( Φ( x)/ x )Δx= 0. Substituting f, the equation is represented as 2 2 ( f / x) f+ (( f / x )( f / x) + ( f / x ) f )Δx = 0

13 he methods for matrix factorization in the presence of missing data the Wiberg Algorithm Basic idea: Deriving a Newton based algorithm for the rewritten problem achieves btt better algorithms in terms of computational complexity etc. than deriving one for the originalproblem problem. Sowe rewrite the function Φ( U, V ) intoa function Ψ( v) of only v as follow. Ψ( v) = Φ( uˆ ( v), v) ˆ( ) ( ) uv ( ) = ( F F) F m Wiberg,. Computation of principal components when data are missing. In Proceedings of the Second Symposium of Computational Statistics.,Berlin.,976.

14 he methods for matrix factorization in the presence of missing data the Wiberg Algorithm It is clear that the new minimization problem yields the same solution as the original ii problem, since v minimizing i i i Ψ( v) together with u= uv ˆ( ) minimizes Φ( uv,). hen, minimizationof of Ψ by Gauss Newton algorithm yields the Wiberg algorithm. he function Ψ( v) can be written as Ψ( v) = g g 2 where g = Fuˆ( () v m. By Gauss Newton algorithm, Δv is obtained by solving equation 2 2 ( g / v ) g + (( g / v )( g / v ) + ( g / v ) g )Δ v = 0

15 he methods for matrix factorization in the presence of missing data the Wiberg Algorithm Substituting F, G, the equation mentioned above is GQQGΔv GQQy= 0 F F F F ( QF = I F F F) F

16 he methods for matrix factorization in the presence of missing data Experiment As test data, matrices of with rank r = 4 are used. Each component is randomly generated according to y ij = uv i j +ε ij,where u are all random variables generated v ij, ij according to a normal density N(0,),the noise ε ij according to N(0,0.05). he missing components are also randomly chosen in the matrix. In this experiment, we test three algorithms, Wiberg, LM, ALS. And each simulation repeats 0different matrices, andeach matrix30trials trials.

17 he methods for matrix factorization in the presence of missing data 300 Wiberg 50 LM 300 ALS Wiberg 60 LM 60 ALS Results for synthetic data with 30% missing components. Iteration count(up), Residue(down)

18 he methods for matrix factorization in the presence of missing data Wiberg LM ALS Wiberg LM ALS Results for synthetic data with 50% missing components. Iteration count(up), Residue(down).

19 he L Wiberg Algorithm for matrix factorization in the presence of missing data and outliers Which is the better estimator,l norm or L2 norm, for scale parameter of a normal distribution? /2 2 L norm: d L2 norm: n = xi x sn= ( xi x) n n Eddington advocated dthe use of the former: his is contrary to the advice of most textbook; but it can be shown to be true. Fisher pointed out that for normal observations s n is about 2% moreefficient efficient than d. d n Huber, P J.. Robust Statistics. John Wiley & Sons,98. Eddington, A.S. Stellar Movements and the structure of the universe. Macmillan,London, 94. Fisher, R.A. A mathematical examination of the methods of determining the accuracy of an observation by the mean error and the mean square error. Monthly Not. Roy. Astron. Soc., 80,920.

20 he L Wiberg Algorithm for matrix factorization Huber(his paper Robust Estimation of a Location Parameter (964)formed the first basis for a theory of robust estimation): just 2 bad observations in 000 suffice to offset the 2% advantage of the mean square error. ε ARE ARE ARE = Asymptotic Efficiency( d )/ Asymptotic Efficiency( s ) n n ε Huber, P.J. Robust statistical procedures. Regional Conference Series in Applied Mathematics No. 27, SIAM, Philadelphia, Penn, 979.

21 he L Wiberg Algorithm for matrix factorization Fit a line to 0 given data points. he two data points on upper right are outliers.

22 he L Wiberg Algorithm for matrix factorization Efficient Computation of Robust Low Rank Matrix Approximations in the Presence of Missing Data using the L Norm. CVPR, 200. Best Paper Anders Eriksson, Research Staff Anton van den Hengel, Professor School of Computer Science University of Adelaide, Australia

23 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm he original problem M = U V dn dk kn where M is known, U, V is unknown. d n d k k n the optimization problem with L nrom is min W e ( M UV ) UV, where W is a indicator function such that wij is if mij is known, and 0 otherwise. e is Hadamard product.

24 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm is equal to delete the missing data, because We ( M UV)? m2 m3 uv + uv 2 2 uv 2 + uv 2 22 uv 3 + uv 2 23 m2 m22 m23 u2v + u22v2 u2v2 + u22v22 u2v3 + u22v23 m3? m 33 uv 3 uv 32 2 uv 3 2 uv uv 3 3 uv m2 m3 0 uv2 + u2v22 uv3 + u2v23 m2 m22 m23 u2v + u22v2 u2v2 + u22v22 u2v3 + u22v23 m 3 0 m 33 uv 3 uv uv 3 3 uv

25 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm Fixing U and V, it is possible to rewrite the optimization problem as vˆ( U ) = argmin wm wg( U) v () v uv ˆ( ) = argmin wm wfv ( ) u (2) u where v and u is the vectors of V and U with column nd nk nd kd expansion,. GU ( ) = I U R, FV ( ) = V I R n d

26 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm As L2 Wiberg algorithm, firstly we should obtain the optimal solution of v with hypothesis that t u is fixed. hen, substituting btit v(u) into the second equation, we obtain the minimization problem as follow. min We M We UV% ( U) U min wm wg( U) v% ( U) u

27 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm By aylor expansion, m+ m+ wg( U ) v% ( U ) wg U v % U + wg U v % U u u u m m m m m+ m ( ) ( ) ( ( ( ) ( ))/ )( ) he minimization problem mentioned above is rewritten as min wm ( wg( U) v% ( U) + ( ( wg( U) v% ( U))/ u ) δ) u

28 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm ( wg ( U ) v % ( U ))/ u = wg( U )( v% ( U )/ u ) + wf( V% ( U )) Unfortunately, vu %( ) have not a easily differentiable, closed form solution.

29 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm So it is important to show how the optimal solution v% to be obtained. he equivalent tformulation of () is min [0,0,,0] v +,,, t s v v t s v + + v wg( U ) wg( U ) I v wm st.. I = ( ) ( ) wm wg U wg U I t b AU ( ) s + v, v, t, s 0 + v, v R, t R, s R kn dn 2 dn

30 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm simplex method A standard form of linear program is min x R n c x st.. Ax= b x 0 he solution form of simplex method is xb B b x = N 0 where B is the basis matrix. From A to B,there is a transformation matrix such that B= AQ. Q

31 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm Supposing the result of minimization of () is there is a matrix such that Q 2 x x N 0 B Bb = v + v xb = Q2 t x N s and a matrix = ( I I 0) such that v. nk nk + v %= v t s

32 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm So the optimal solution of () as follow then, vu %( ) ( A( UQ ) ) = Q2 0 b (( AU ( ) Q ) b) / u vu % ( )/ u = Q2, 0 (( QAU ( )) b)/ u = ( Q AU ( ) b)/ u Q AU b vec A U ( ( ) ) ( ( )) = vec( A( U)) u where vec(a) stands for the vector A with column expansion.

33 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm ( Q AU ( ) b) vec( A( U) ) = ( b I2kn 3dn)( I Q ) + vec ( A ( U )) vec ( A ( U )) = ( b I )( I Q )( AU ( ) AU ( ) ) = 2kn+ 3dn (( AU ( ) b ) Q AU ( ) ) = (( QQ ( AU ( ) b)) Q AU ( ) )

34 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm wg( U) wg( U) Idn vec ( I2dn ) vec( A( U)) wg( U) wg( U) Idn = u u wg ( U ) w vec( ) ( Ink ) vec( ( II Id ) u ( II 2 Id ) u ( II nk Id ) u ) wg( U) w L = u u II I d w II2 Id = ( Ink ) w M IInk Id kn II R where n stands for a matrix whose vectorization is. e n

35 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm After solving vu %()/ u, we introduce it into the minimization problem min wm ( wgu ( ) v %( U) + ( ( wgu ( ) v %( U))/ u ) δ). o obtain the u optimal solution of u%, the linear program to solve as follow

36 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm δ min [0 0 0] δ δ, t t s st δ ( ( wg ( U ) v % ( U ))/ u ) ( ( wg ( U ) v % ( U ))/ u ) I wm wgu ( ) v ( U) I δ + % = ( ( wg( U) v( U))/ u) ( ( wg( U) v( U))/ u) I t wm wg( U) v( U) % % % s δ μ +, dk dn δ δ R, t R, s 2 dn + R + δ, δ, ts, 0 andu% = u+ δ *

37 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm input : U, > η > η > 0andc > 0 2 k = 0; repeat Computethe Jacobian of wg( U) v% ( U) using functions in page 32 page 34; Solvethe subproblemin page Let gain = fu fu δ % fu % fu δ * ( k) ( k+ k) * ; ( k) ( k+ k) * 36 to obtainδk ;

38 he L Wiberg Algorithm for matrix factorization he L Wiberg Algorithm if gain > η * k + = k + k; U U δ end if gain η then μ = μ /2; end if gain η 2 then μ = 2 μ ; end k = k + ; until convergence. η = / 4, η = 3 / 4 2

39 he Huber Wiberg Algorithm for matrix factorization Huber Distribution 0.4 Huber Normal ( ε ) x exp( ep( ), x / σ c 2 2πσ 2σ fx ( / σ) = 2 ( ε) c cx 0.06 exp( ), x/ σ > c 2πσ 2 σ

40 he Huber Wiberg Algorithm for matrix factorization Huber M estimator Huber L2-norm L-norm Ax x = b i ( Ax b ) min ρ ( ) i 2 t, t γ where ρ () t = 2 2 γ t γ, t > γ

41 he Huber Wiberg Algorithm for matrix factorization Fit a line to 0 given data points. he two data points on upper right are outliers.

42 he Huber Wiberg Algorithm for matrix factorization Huber Wiberg Algorithm At first, we assume that the γ in Huber M estimator is known. Ui Using an alternate t form, the Huber M estimator t can be given by: ( Ax b ) min ρ ( ) x i 2 min z + γ Ax b z xz, 2 i 2 Qifa Ke and akeo Kanade.Robust L Norm Factorization in the Presence of Outliers and Missing Data by Alternative Convex Programming. g y g g In Conference on Computer Vision and Pattern Recognition, Washington, USA, O. L. Mangasarian and D. R. Musicant. Robust linear and support vector regression. IEEE rans. on PAMI, 22(9): , 2000.

43 he Huber Wiberg Algorithm for matrix factorization Huber Wiberg Algorithm he optimization problem min ρ( wij ( mij UV i j )) UV, i, j 2 min Z + γ W( M UV) Z UV,, Z min z + γ w( m G( U) v) z uvz,, 2 2 where w is a matrix that deletes unknown elements. his optimal problem is a convex optimization.

44 he Huber Wiberg Algorithm for matrix factorization Huber Wiberg Algorithm By aylor expansion, wg U % wg U v U + wg U v U u u u m+ m+ ( ) v( U ) m m m m m+ m ( ) % ( ) ( ( ( ) % ( ))/ )( ) he minimization problem mentioned above is rewritten as min z wm ( wg( U) v( U) ( ( wg ( U) v( U))/ u ) ) z uvz,, where 2 ( wg( U) v% ( U))/ u = wg( U)( v% ( U)/ u ) + wf( V %( U)) 2 + γ % + % δ 2

45 he Huber Wiberg Algorithm for matrix factorization Huber Wiberg Algorithm Assuming that U is valued, 0 v v min [ v, t, z ] 0 t + [0,,0] t uvz,, γ I z c z 4243 H wg( U ) I I v wm s.. t wg U I I t wm ( ) 0 I 0 z which is a quadratic programming problem. A b

46 he Huber Wiberg Algorithm for matrix factorization Huber Wiberg Algorithm quadratic programming A quadratic programming with constraints is min x Hx + c x 2 st.. Ax b he solution must satisfy KK condition, so the form of solution is H A x c Ω % = AΩ 0 λ b Ω where AΩx= bω is active constraint. here is a transformation matrix ti Q such that t A = Q Ab, = Qb. QΩ Ω Ω Ω Ω

47 he Huber Wiberg Algorithm for matrix factorization Huber Wiberg Algorithm + x% H ( Q A) c x Ω % + + K d v QK d % λ ( Q A) 0 QΩ b Ω λ where Q = K ( I 0) kn + v% K d vec ( K ) vec ( A ) Q u vec( K ) vec( A) u d = Q ( d K I)(( K K) ( K K) )(( I K ) + ( K I) ) + ( d ( K K) ) II I d 0kn+ 2dn w I II2 I Q d Ω ( I w ) nk M I 0 I ( I QΩ ) IInk I d 0 kn+ 2dn 0 ( k n where is transposition matrix for which vec A ) = vec( A), IIn R stands for a matrix whose vectorization is e n.

48 he Huber Wiberg Algorithm for matrix factorization Huber Wiberg Algorithm After solving vu %( )/ u, we have the new minimization problem 2 min z +γ t uvz,, 2 2 δ ( wg( U) v% ( U))/ u ) I I wm wg( U) v( U) st.. % t I I wm+ wg U v U ( wg( U) v% ( U))/ u ) ( ( )%( ) z t 0, δ μ * and u% = u +δ

49 hanks!any Questions?

Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data using the L 1 Norm

Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data using the L 1 Norm Anders Eriksson, Anton van den Hengel School of Computer Science University of Adelaide,