Back to the Roots: Solving Polynomial Systems with Numerical Linear Algebra Tools

Transcription

1 Back to the Roots: Solving Polynomial Systems with Numerical Linear Algebra Tools Bart De Moor Katholieke Universiteit Leuven Department of Electrical Engineering ESAT/SCD 1 / 56

2 Outline 1 Introduction 2 History 3 Linear Algebra 4 Multivariate Polynomials 5 Applications 6 Conclusions 2 / 56

3 Why Linear Algebra? System Identification: PEM LTI models Non-convex optimization Considered solved early nineties Linear Algebra approach Subspace methods 3 / 56

4 Why Linear Algebra? Nonlinear regression, modelling and clustering Most regression, modelling and clustering problems are nonlinear when formulated in the input data space This requires nonlinear nonconvex optimization algorithms Linear Algebra approach Least Squares Support Vector Machines Kernel trick = projection of input data to a high-dimensional feature space Regression, modelling, clustering problem becomes a large scale linear algebra problem (set of linear equations, eigenvalue problem) 4 / 56

5 Why Linear Algebra? Nonlinear Polynomial Optimization Polynomial object function + polynomial constraints Non-convex Computer Algebra, Homotopy methods, Numerical Optimization Considered solved by mathematics community Linear Algebra Approach Linear Polynomial Algebra 5 / 56

6 Research on Three Levels Conceptual/Geometric Level Polynomial system solving is an eigenvalue problem! Row and Column Spaces: Ideal/Variety Row space/kernel of M, ranks and dimensions, nullspaces and orthogonality Geometrical: intersection of subspaces, angles between subspaces, Grassmann s theorem,... Numerical Linear Algebra Level Eigenvalue decompositions, SVDs,... Solving systems of equations (consistency, nb sols) QR decomposition and Gram-Schmidt algorithm Numerical Algorithms Level Modified Gram-Schmidt (numerical stability), GS from back to front Exploiting sparsity and Toeplitz structure (computational complexity O(n 2 ) vs O(n 3 )), FFT-like computations and convolutions,... Power method to find smallest eigenvalue (= minimizer of polynomial optimization problem) 6 / 56

7 Four instances of polynomial rooting problems p(λ) = det(a λi) = 0 (x 1)(x 3)(x 2) = 0 (x 2)(x 3) = 0 x 2 + 3y 2 15 = 0 y 3x 3 2x x 2 = 0 min x,y x 2 + y 2 s. t. y x 2 + 2x 1 = 0 7 / 56

8 Outline 1 Introduction 2 History 3 Linear Algebra 4 Multivariate Polynomials 5 Applications 6 Conclusions 8 / 56

9 Solving Polynomial Systems: a long and rich history... Diophantus (c200-c284) Arithmetica Al-Khwarizmi (c780-c850) Zhu Shijie (c1260-c1320) Jade Mirror of the Four Unknowns Pierre de Fermat (c ) René Descartes ( ) Isaac Newton ( ) Gottfried Wilhelm Leibniz ( ) 9 / 56

10 ... leading to Algebraic Geometry Etienne Bézout ( ) Jean-Victor Poncelet ( ) August Ferdinand Möbius ( ) Evariste Galois ( ) Arthur Cayley ( ) Leopold Kronecker ( ) Edmond Laguerre ( ) James Joseph Sylvester ( ) Francis Sowerby Macaulay ( ) David Hilbert ( ) 10 / 56

11 So Far: Emphasis on Symbolic Methods Computational Algebraic Geometry Emphasis on symbolic manipulations Computer algebra Huge body of literature in Algebraic Geometry Computational tools: Gröbner Bases (next slide) Wolfgang Gröbner ( ) Bruno Buchberger 11 / 56

12 So Far: Emphasis on Symbolic Methods Example: Gröbner basis Input system: x 2 y + 4xy 5y + 3 = 0 x 2 + 4xy + 8y 4x 10 = 0 Generates simpler but equivalent system (same roots) Symbolic eliminations and reductions Monomial ordering (e.g., lexicographic) Exponential complexity Numerical issues! Coefficients become very large Gröbner Basis: 9 126y + 647y 2 624y y 4 = y 6432y y x = 0 12 / 56

13 Outline 1 Introduction 2 History 3 Linear Algebra 4 Multivariate Polynomials 5 Applications 6 Conclusions 13 / 56

14 Homogeneous Linear Equations A p q X q (q r) = 0 p (q r) C(A T ) C(X) rank(a) = r dim N(A) = q r = rank(x) A = [ ] [ S U 1 U X = V 2 ] [ V T 1 V T 2 ] James Joseph Sylvester 14 / 56

15 Homogeneous Linear Equations A p q X q (q r) = 0 p (q r) Reorder columns of A and partition p q p (q r) p r A = [ ] A 1 A 2 rank(a 2 ) = r (A 2 full column rank) Reorder rows of X and partition accordingly [ ] [ q r ] X A1 A 1 2 X 2 q r r = 0 rank(a 2 ) = r rank(x 1 ) = q r 15 / 56

16 Dependent and Independent Variables [ ] [ q r ] X A1 A 1 2 X 2 X 1 : independent variables X 2 : dependent variables q r r = 0 X 2 = A 2 A 1 X 1 A 1 = 1 A 2 X 2 X 1 Number of different ways of choosing r linearly independent columns out of q columns (upper bound): ( ) q q! = q r (q r)! r! 16 / 56

17 Grassmann s Dimension Theorem A p q X q (q r A ) = 0 p (q r A ) and B p t Y t (t r B ) = 0 p (t r B ) What is the nullspace of [ A B ]? [ q r A t r B? ] X 0? [ A B ] 0 Y? = 0 Let rank([ A B ]) = r AB (q r A ) + (t r B )+? = (p + t) r AB? = r A + r B r AB 17 / 56

18 Grassmann s Dimension Theorem [ A B ] [ q r A t r B r A +r B r AB ] X 0 Z 1 0 Y Z 2 Intersection between column space of A and B: = 0 AZ 1 = BZ 2 A B #A #(A B) #B rab ra rb Hermann Grassmann ra + rb rab #(A B)=#A + #B #(A B) 18 / 56

19 Univariate Polynomials and Linear Algebra Characteristic Polynomial The eigenvalues of A are the roots of p(λ) = det(a λi) = 0 Companion Matrix Solving q(x) = 7x 3 2x 2 5x + 1 = 0 leads to /7 5/7 2/7 1 x x 2 = x 1 x x 2 19 / 56

20 Univariate Polynomials and Linear Algebra Consider the univariate equation x 3 + a 1 x 2 + a 2 x + a 3 = 0, having three distinct roots x 1, x 2 and x a 3 a 2 a a 3 a 2 a a 3 a 2 a x 1 x 2 x 3 x 2 1 x 2 2 x 2 3 x 3 1 x 3 2 x 3 3 x 4 1 x 4 2 x 4 3 x 5 1 x 5 2 x = Homogeneous linear system Rectangular Vandermonde corank = 3 Observability matrix-like Realization theory! 20 / 56

21 Two Univariate Polynomials Consider x 3 + a 1 x 2 + a 2 x + a 3 = 0 x 2 + b 1 x + b 2 = 0 Build the Sylvester Matrix: a 1 a 2 a a 1 a 2 a 3 1 b 1 b b 1 b b 1 b x x 2 x 3 x = 0 Row Space Ideal =union of ideals =multiply rows with powers of x Null Space Variety =intersection of null spaces Corank of Sylvester matrix = number of common zeros null space = intersection of null spaces of two Sylvester matrices common roots follow from realization theory in null space notice double Toeplitz-structure of Sylvester matrix 21 / 56

22 Two Univariate Polynomials Sylvester Resultant Consider two polynomials f(x) and g(x): f(x) = x 3 6x x 6 = (x 1)(x 2)(x 3) g(x) = x 2 + 5x 6 = (x 2)(x 3) Common roots iff S(f, g) = S(f, g) = det James Joseph Sylvester 22 / 56

23 Two Univariate Polynomials The corank of the Sylvester matrix is 2! Sylvester s construction can be understood from 1 x x 2 x 3 x 4 f(x) = x f(x) = x 1 x 2 g(x) = x 2 1 x 2 2 x g(x) = x 3 x 2 1 x 3 = 0 2 g(x) = x 4 1 x 4 2 where x 1 = 2 and x 2 = 3 are the common roots of f and g 23 / 56

24 Two Univariate Polynomials The vectors in the canonical kernel K obey a shift structure : 1 x x x 2 x = x 2 x 3 x 3 x 4 The canonical kernel K is not available directly, instead we compute Z, for which ZV = K. We now have S 1 KD = S 2 K S 1 ZV D = S 2 ZV leading to the generalized eigenvalue problem (S 2 Z)V = (S 1 Z)V D 24 / 56

25 Outline 1 Introduction 2 History 3 Linear Algebra 4 Multivariate Polynomials 5 Applications 6 Conclusions 25 / 56

26 Null space based Root-finding Consider j p(x, y) = x 2 + 3y 2 15 = 0 q(x, y) = y 3x 3 2x x 2 = 0 Fix a monomial order, e.g., 1 < x < y < x 2 < xy < y 2 < x 3 < x 2 y <... Construct M: write the system in matrix-vector notation: 1 x y x 2 2 xy y 2 x 3 x 2 y xy 2 y 3 3 p(x, y) q(x, y) x p(x, y) y p(x, y) / 56

27 Null space based Root-finding Continue to enlarge M: { p(x, y) = x 2 + 3y 2 15 = 0 q(x, y) = y 3x 3 2x x 2 = 0 it # form 1 x y x 2 xy y 2 x 3 x 2 y xy 2 y 3 x 4 x 3 yx 2 y 2 xy 3 y 4 x 5 x 4 yx 3 y 2 x 2 y 3 xy 4 y 5 p d = 3 xp yp q x 2 p xyp d = 4 y 2 p xq yq x 3 p x 2 yp xy 2 p d = 5 y 3 p x 2 q xyq y 2 q # rows grows faster than # cols overdetermined system rank deficient by construction! 27 / 56

28 Null space based Root-finding Coefficient matrix M: M = " Solutions generate vectors in kernel of M: Mk = 0 Number of solutions s follows from corank # Canonical nullspace K built from s solutions (x i, y i): x 1 x 2... x s y 1 y 2... y s x 2 1 x x 2 s x 1 y 1 x 2 y 2... x sy s y1 2 y ys 2 x 3 1 x x 3 s x 2 1 y 1 x 2 2 y 2... x 2 sy s x 1 y1 2 x 2 y x sys 2 y1 3 y ys 3 x 4 1 x x 4 4 x 3 1 y 1 x 3 2 y 2... x 3 sy s x 2 1 y2 1 x 2 2 y x 2 sys 2 x 1 y1 3 x 2 y x sys 3 y1 4 y ys / 56

29 Null space based Root-finding Choose s linear independent rows in K S 1K This corresponds to finding linear dependent columns in M x 1 x 2... x s y 1 y 2... y s x 2 1 x x 2 s x 1 y 1 x 2 y 2... x sy s y1 2 y ys 2 x 3 1 x x 3 s x 2 1 y 1 x 2 2 y 2... x 2 sy s x 1 y1 2 x 2 y x sys 2 y1 3 y ys 3 x 4 1 x x 4 4 x 3 1 y 1 x 3 2 y 2... x 3 sy s x 2 1 y2 1 x 2 2 y x 2 sys 2 x 1 y1 3 x 2 y x sys 3 y1 4 y ys / 56

30 Null space based Root-finding Shift property in monomial basis [ ] x = [ ] 1 x y x 2 xy y 2 1 x y x 2 xy y 2 y = [ [ Finding the x-roots: let D = diag(x 1, x 2,..., x s ), then S 1 KD = S 2 K, where S 1 and S 2 select rows from K wrt. shift property Reminiscent of Realization Theory ] ] 1 x y x 2 xy y 2 1 x y x 2 xy y 2 30 / 56

31 Null space based Root-finding Nullspace of M Find a basis for the nullspace of M using an SVD: [ ] [ M = Σ1 0 W T ] = [ X Y ] Z T Hence, We have MZ = 0 S 1 KD = S 2 K However, K is not known, instead a basis Z is computed as ZV = K Which leads to (S 2 Z)V = (S 1 Z)V D 31 / 56

32 Null space based Root-finding Algorithm 1 Fix a monomial ordering scheme 2 Construct coefficient matrix M 3 Compute basis for nullspace of M, Z 4 Find s linear independent rows in Z 5 Choose shift function, e.g., x 6 Write down shift relation in monomial basis k for the chosen shift function using row selection matrices S 1 and S 2 7 The construction of above gives rise to a generalized eigenvalue problem (S 2 Z)V = (S 1 Z)V D of which the eigenvalues correspond to the, e.g., x-solutions of the system of polynomial equations. 8 Reconstruct canonical kernel K = ZV 32 / 56

33 Data-driven Root-finding Data-Driven root-finding Dual version of Kernel-based root-finding All operations are done on coefficient matrix M Find linear dependent columns of M instead of linear independent rows of K (corank) Write down eigenvalue problem in terms of partitiong of M Allows sparse representation of M Rank-revealing QR instead of SVD 33 / 56

34 Data-driven Root-finding { p(x, y) = x 2 + 3y 2 15 = 0 q(x, y) = y 3x 3 2x x 2 = 0 Finding linear dependent columns of M 1 x y x 2 xy y 2 x 3 x 2 y xy 2 y 3... p xp yp q x 2 p 15 xyp 15 y 2 p 15 xq yq x 3 p 15 x 2 yp 15 xy 2 p 15 y 3 p 15 x 2 q xyq y 2 q / 56

35 Data-driven Root-finding { p(x, y) = x 2 + 3y 2 15 = 0 q(x, y) = y 3x 3 2x x 2 = 0 Writing down the eigenvalue problem in terms of a re-ordered partitioning of M all linear dependent columns of M corresponding with monomials of the lowest possible degree are grouped in M 1 M = [ ] = [M 1 M 2 ] [M 1 M 2 ] [ K1 K 2 ] = 0 ( : Moore-Penrose pseudoinverse) K 2 = M 2 M 1 K 1 35 / 56

36 Data-driven Root-finding { p(x, y) = x 2 + 3y 2 15 = 0 q(x, y) = y 3x 3 2x x 2 = 0 Writing down the eigenvalue problem in terms of a partitioning of M x 1 0 [ ] K 1... K1 = S x K 2 0 x s K 1 x x s [ = S x I tcr M 2 M 1 ] K 1 36 / 56

37 Complications There are 3 kinds of roots: 1 Roots in zero 2 Finite nonzero roots 3 Roots at infinity Applying Grassmann s Dimension theorem on the Kernel allows to write the following partitioning [ ] X1 0 X [M 1 M 2 ] 2 = 0 0 Y 1 Y 2 X 1 corresponds with the roots in zero (multiplicities included!) Y 1 corresponds with the roots at infinity (multiplicities included!) [X 2 ; Y 2 ] corresponds with the finite nonzero roots (multiplicities included!) 37 / 56

38 Complications Roots at infinity: univariate case 0 x 2 + x 2 = 0 transform x 1 X X(1 2X) = 0 1 affine root x = 2 (X = 1 2 ) 1 root at infinity x = (X = 0) Roots at infinity: multivariate case j (x 2)y = 0 y 3 = 0 transform x X T, y Y T 1 affine root (2,3,1) (T = 1) 1 root at infinity (1,0,0) (T = 0) j XY 2Y T = 0 Y 3T = 0 38 / 56

39 Multiplicities General Canonical null space K Multiplicities of roots multiplicity structure of kernel K Partial derivatives j1 j 2...j s j x 1 1 x j x js s 1 j 1!j 2!... j s! needed to describe extra columns of K Currently investigating technicalities j 1+j j s x j 1 1 x j x js s Possibility of trading in multiplicities for extra equations (Radical Ideal) 39 / 56

40 Multiplicities Univariate case f(x) = (x 1) 3 = 0 triple root in x = 1 : f (1) = 0 and f (1) = 0 f [ ] x 1 0 x 2 2x 1 x 3 3x 2 3x = 0 or 0 f 1 f 2 f x x 2 x 3 = 0 40 / 56

41 Multiplicities Multivariate case Polynomial system in 2 unknowns (x, y) with 1 affine root z 1 = (x 1, y 1 ) with multiplicity 3: [ 00 z1 10 z1 01 z1 ] 1 root z 2 = (x 2, y 2 ) at infinity: 00 z2 M matrix of degree 4 41 / 56

42 Multiplicities Canonical Kernel K K = 00 z1 10 z1 01 z1 00 z x y x 2 1 2x x 1 y 1 y 1 x 1 0 y y x 4 1 4x x 3 1 y 1 3x 2 1 y 1 x x 2 1 y2 1 2x 1 y1 2 2x 2 1 y 1 0 x 1 y1 3 y1 3 3x 1 y1 2 0 y y / 56

43 Polynomial Optimization Polynomial Optimization Problems If A 1 b = xb and A 2 b = yb then (A A 2 2)b = (x 2 + y 2 )b. (choose any polynomial objective function as an eigenvalue!) Polynomial optimization problems with a polynomial objective function and polynomial constraints can always be written as eigenvalue problems where we search for the minimal eigenvalue! Convexification of polynomial optimization problems 43 / 56

44 Outline 1 Introduction 2 History 3 Linear Algebra 4 Multivariate Polynomials 5 Applications 6 Conclusions 44 / 56

45 System Identification: Prediction Error Methods PEM System identification e H(q) Measured data {u k, y k } N k=1 Model structure u G(q) y y k = G(q)u k + H(q)e k Output prediction ŷ k = H 1 (q)g(q)u k + (1 H 1 )y k Model classes: ARX, ARMAX, OE, BJ A(q)y k = B(q)/F (q)u k +C(q)/D(q)e k Class ARX ARMAX OE BJ Polynomials A(q), B(q) A(q), B(q), C(q) B(q), F (q) B(q), C(q), D(q), F (q) 45 / 56

46 System Identification: Prediction Error Methods Minimize the prediction errors y ŷ, where ŷ k = H 1 (q)g(q)u k + (1 H 1 )y k, subject to the model equations Example ARMAX identification: G(q) = B(q)/A(q) and H(q) = C(q)/A(q), where A(q) = 1 + aq 1, B(q) = bq 1, C(q) = 1 + cq 1, N = 5 min ŷ,a,b,c (y 1 ŷ 1) (y 5 ŷ 5) 2 s. t. ŷ 5 cŷ 4 bu 4 (c a)y 4 = 0, ŷ 4 cŷ 3 bu 3 (c a)y 3 = 0, ŷ 3 cŷ 2 bu 2 (c a)y 2 = 0, ŷ 2 cŷ 1 bu 1 (c a)y 1 = 0, 46 / 56

47 Structured Total Least Squares Static Linear Modeling Dynamical Linear Modeling Rank deficiency minimization problem: min ˆ A b 2 F, s. t. (A + A)v = b + b, v T v = 1 Singular Value Decomposition: find (u, σ, v) which minimizes σ 2 Let M = ˆ A b 8 Mv = uσ >< M T u = vσ v >: T v = 1 u T u = 1 Rank deficiency minimization problem: min ˆ A b 2 F, s. t. (A + A)v = b + b, v T v = 1 ˆ A b structured Riemannian SVD: find (u, τ, v) which minimizes τ 2 8 >< >: Mv = D vuτ M T u = D uvτ v T v = 1 u T D vu = 1 (= v T D uv) 47 / 56

48 Structured Total Least Squares 3 min v τ 2 = v T M T Dv 1 Mv s. t. v T v = 1. phi STLS Hankel cost function TLS/SVD soln STSL/RiSVD/invit steps STLS/RiSVD/invit soln STLS/RiSVD/EIG global min STLS/RiSVD/EIG extrema theta method TLS/SVD STLS inv. it. STLS eig v v v τ global solution? no no yes 48 / 56

49 Maximum Likelihood Estimation CpG Islands genomic regions that contain a high frequency of sites where a cytosine (C) base is followed by a guanine (G) rare because of methylation of the C base hence CpG islands indicate functionality Given observed sequence of DNA: CTCACGTGATGAGAGCATTCTCAGA CCGTGACGCGTGTAGCAGCGGCTCA Problem Decide whether the observed sequence came from a CpG island 49 / 56

50 Maximum Likelihood Estimation The model 4-dimensional state space [m] = {A, C, G, T} Mixture model of 3 distributions on [m] 1 : CG rich DNA 2 : CG poor DNA 3 : CG neutral DNA Each distribution is characterised by probabilities of observing base A,C,G or T Table: Probabilities for each of the distributions (Durbin; Pachter & Sturmfels) DNA Type A C G T CG rich CG poor CG neutral / 56

51 Maximum Likelihood Estimation The probabilities of observing each of the bases A to T are given by p(a) = 0.10 θ θ p(c) = θ θ p(g) = θ θ p(t ) = 0.09 θ θ θ i is probability to sample from distribution i (θ 1 + θ 2 + θ 3 = 1) Maximum Likelihood Estimate: ( ˆθ 1, ˆθ 2, ˆθ 3) = arg max θ l(θ) where the log-likelihood l(θ) is given by l(θ) = 11 logp(a) + 14 logp(c) + 15 logp(g) + 10 logp(t ) Need to solve the following polynomial system 8 l(θ) < θ 1 = P 4 u i p(i) i=1 p(i) θ 1 = 0 : l(θ) θ 2 = P 4 u i p(i) i=1 p(i) θ 2 = 0 51 / 56

52 Maximum Likelihood Estimation Solving the Polynomial System corank(m) = 9 Reconstructed Kernel K = θ 1 θ 2 θ1 2 θ 1 θ 2.. θ i s are probabilities: 0 θ i 1 Could have introduced slack variables to impose this constraint! Only solution that satisfies this constraint is ˆθ = (0.52, 0.22, 0.26) 52 / 56

53 And Many More Applications are found in Polynomial Optimization Problems Structured Total Least Squares Model order reduction Analyzing identifiability nonlinear model structures Robotics: kinematic problems Computational Biology: conformation of molecules Algebraic Statistics Signal Processing / 56

54 Outline 1 Introduction 2 History 3 Linear Algebra 4 Multivariate Polynomials 5 Applications 6 Conclusions 54 / 56

55 Conclusions Finding roots of multivariate polynomials is linear algebra and realization theory! Finding minimizing zero of a polynomial optimization problem is extremal eigenvalue problem (Numerical) linear algebra/systems theory version of results in algebraic geometry/symbolic algebra (Gröbner bases, resultants, rings, ideals, varieties,... ) These relations in principle convexify /linearize many problems Algebraic geometry System identification (PEM) Numerical linear algebra (STLS, affine EVP Ax = xλ + a, etc.) Multilinear algebra (tensor least squares approximation problems) Algebraic statistics (HMM, Bayesian networks, discrete probabilities) Differential algebra (Glad/Ljung) Convexification occurs by projecting up to higher dimensional vector space (difficult in low number of dimensions; easy in high number of dimensions: an eigenvalue problem) Many challenges remain: Efficient construction of the eigenvalue problem - exploiting sparseness and structure Algorithms to find the minimizing solution directly (inverse power method) / 56

56 Questions? Bart De Moor ( ) Kim Batselier ( ) Philippe Dreesen ( ) At the end of the day, the only thing we really understand, is linear algebra. 56 / 56