Application of Numerical Algebraic Geometry to Geometric Data Analysis

Transcription

1 Application of Numerical Algebraic Geometry to Geometric Data Analysis Daniel Bates 1 Brent Davis 1 Chris Peterson 1 Michael Kirby 1 Justin Marks 2 1 Colorado State University - Fort Collins, CO 2 Air Force Institute of Technology - Wright Patterson Air Force Base, OH August 2, 2013

2 Definitions Definition: Grassmann manifold Let the Grassmann manifold Gr(n, p) denote the set of all p-dimensional subspaces of R n. Definition: Elements in Gr(n, p) A point [M] Gr(n, p) is = to an equivilence class of full rank n p orthonormal matrices that have the same column space as M. Definition: Principal Angles Two subspaces [X] and [Y ] of R n have p principal angles θ 1 θ 2 θ p π 2 where p = min{dim [X], dim [Y ]}. The principal angles between [X] and [Y ] are the inverse cosine of the singular values of the matrix X T Y.

3 Examples Example: Angles between lines Consider the x-axis and y-axis in Gr(2, 1) represented by the unit vectors e T 1 = (1, 0)T and e T 2 = (0, 1)T respectively. (1, 0) T (0, 1) = 0. SVD(0) = T θ 1 = cos 1 (0) = π 2

4 Examples Example: Angle between hyperplanes in R 3 Consider two matrices X , Y θ 1 0 θ In fact for any two distinct [X], [Y ] Gr(3, 2): θ 1 = 0 since the two subspaces intersect in a line.

5 Big Picture and Problems Fundamental Problem in Geometric Data Analysis Given a cluster of points {[Y 1 ],..., [Y k ]} in some Grassmann manifold(s) how do we assign a mean representative to a cluster of points using principal angles?

6 Big Picture and Problems Problems 1 The cluster of points might be in Gr(n, 1) Gr(n, 2) Gr(n, n 1) a disjoint union of Grassmann manifolds of various dimensions. 2 Geometric assumptions need to be made so local gradient-based methods work properly. 3 There could be more than one mean representative and even an infinite number of them.

7 Section 2 Problem Statement

8 Problem Statement We address these issues using a mean representative based on the cosine of the principal angles. Problem Statement {V 1,..., V k } : subspaces of R n. Y i : fixed n d i orthonormal matrices such that V i = [Y i ]. L : one dimensional subspace of R n θ(l, V i ) : the principal angle between L and V i. Find L such that it maximizes the function: F (L) = k cos θ(l, V i ) i=1

9 Geometric Examples Example 1 Consider the three standard coordinate planes in R 3. The special structure of the coordinate axes will produce four distinct lines L 1,..., L 4 to all give the same optimal solution.

10 Geometric Examples Example 2 Consider three randomly chosen planes in R 3. The generic behavior of three hyperplanes will produce one distinct line L that gives the optimal solution.

11 Geometric Examples Example 3 Consider the xy-plane and the z-axis in R 3 The line L is non-unique. There is an entire cone of lines L.

12 Section 3 Reformulating the Problem

13 Step 1 Main Idea The problem can be reformulated as a completely algebraic optimization problem. We break the reformulation into two main parts. Step 1 L = span of some unit length vector l. cos θ(l, V i ) is the singular value of l T Y i. l T Y i is the length of the projection of l onto V i. proj Vi l is a vector which makes the smallest angles with l that is in V i.

14 Step 1 Step 1 (continued) Therefore, max L subject to is equivilent to k cos θ(l, V i ) i=1 L R n is a one-dimensional vector space max l k proj Vi l i=1 subject to l T l = 1.

15 Step 1 Step 1 (continued) Since proj Vi l = cos θ(l, V i ) = l T v i for some unit length vector v i V i the problem: max l k proj Vi l i=1 subject to l T l = 1. is equivilent to finding an l to the optimization problem: max l,v i subject to k l T v i i=1 l T l = 1, v T i v i = 1 and v i V i

16 Step 2 We need to introduce numerical data for V i which is given in the form of orthonormal matrices Y i such that V i = [Y i ]. Step 2: Introduce data Y i v i [Y i ] v i = Y i α i for some coefficient vector α i. vi T v i = αi T Y i T Y i α i = αi T α i = 1 since Y i orthonormal Now we have: max l,α i after factoring out l T. l T k Y i α i i=1 subject to l T l = 1, α T i α i = 1 for 1 i k.

17 Step 2 Key Fact: The unit length vector l can be chosen independently of the choice of α i. To Maximize: l should point in the direction of k i=1 Y iα i. Reformulation 2 Find the α i s that optimizes max α i k 2 Y i α i i=1 subject to α T i α i = 1 then set v = k Y i α i and recover l = v/ v and produce L. i=1 We call L the max-length-vector-line of best fit to a collection of subspaces {V 1,..., V k }.

18 Geometry of max-length-vector-line of best fit Geometry behind max-length-vector-line of best fit Red vectors rotate in their subspaces Black vector represents the max-length-vector-line of best fit Maximize the length of the black vector(s).

19 Section 4 Karush-Kuhn-Tucker

20 KKT conditions The constraints αi T α i = 1 form a compact set. k 2 Y i α i continuous maximum obtained. i=1 Local solutions found at Karush-Kuhn-Tucker (KKT) points. Set α T = (α1 T,..., αt k ). The KKT conditions are k 2 α Y i α i + i=1 k λ i α (αi T α i 1) = 0 i=1 α T i α i = 1

21 KKT equations KKT equations The KKT polynomial equations can be written compactly as: where ( ) [Y 1 Y k ] [Y 1 Y k ] + diag(λ d 1 1,..., λd k k ) α = 0 α T i α i = 1 denotes the block outer product of the matrices Y i. diag(λ d 1 1,..., λd k k ) is a diagonal matrix with diagonal elements λ 1,..., λ k each repeated d i times. Geometrically, we are almost solving a symmetric eigenvalue problem on S d 1 S d k.

22 Section 5 Implementation

23 Main Algorithm Main Algorithm To solve max L k cos θ(l, V i ) i=1 1 Find all solutions to ( ) [Y 1 Y k ] [Y 1 Y k ] + diag(λ d 1 1,..., λd k k ) α = 0 2 Using the α i s compute v = k Y i α i. i=1 3 Find the largest v and set l = v v to recover L. α T i α i = 1

24 Main Algorithm Implementing Main Algorithm 1 Use Bertini as a blackbox to approximate all solutions and find real solutions. 2 Standard MATLAB routines 3 Standard MATLAB routines

25 Main Algorithm

26 Example Consider five subspaces {[Y 1 ],..., [Y 5 ]} R 10 of dimension 4, 3, 3, 2, 2, respectively. Y 1 = Y 2 = Y 3 = Y 4 = Y 5 =

27 Example Using the formulation: ([Y 1 Y 2 Y 3 Y 4 Y 5 ] [Y 1 Y 2 Y 3 Y 4 Y 5 ] + diag(λ 4 1, λ 3 2, λ 3 3, λ 2 4, λ 2 5)) α = 0 α α α α14 2 = 1 α α α23 2 = 1 α α α33 2 = 1 α α42 2 = 1 α α52 2 = 1

28 Example Results: Bertini was capable of finding 172 possible real solutions in terms of α T = (α T 1, αt 2,..., αt 5 ) For these paticular matrices, l = ± which was produced by a vector v of length

29 Example Parallel implementation of Bertini v GHz Xeon 5650 compute nodes on the CentOS 6.4 operating system. Using the regeneration routine, Bertini tracked 14, 866 paths in approximately 52 seconds. Among the 14, 866 paths, 3552 of them were successful for which 172 produces real solutions. Post-processing of the data to compute v and l was done in serial in negligible time.

30 Future Work A few notes and next steps: The ambient dimension n can be made arbitrarily large. Find applications outside of math such as image classification problems. Relax to a symmetric eigenvalue problem and use parameter homotopies for faster solving. Find an entire flag of max-length-vector-lines of best fit Compute new mean representatives similiar to this method. Thank you for listening!