The Chasm at Depth Four, and Tensor Rank: Old results, new insights

Transcription

1 The Chasm at Depth Four, and Tensor Rank: Old results, new insights Suryajith Chillara 1 Joint work with Mrinal Kumar, Ramprasad Saptharishi and V. Vinay June 12, Research partly supported by TCS PhD fellowship. Part of this work was done while visiting Tel Aviv University, hosted by Amir Shpilka.

2 Arithmetic circuits Definition An Arithmetic circuit Φ over the field and the set of variables X = (x 1,x 2,...,x n ) is a directed acyclic graph as follows. Every gate with in-degree 0 is called an input gate and is labelled either by a variable or a field element from. Every other gate is labelled by either (product gate) or +(sum gate). The size of Φ is the number of edges present in it. The depth of Φ is the maximal depth of a gate in it.

3 Arithmetic circuit f (x 1,x 2,x 3 ) x 1 x 2 x 3

4 Arithmetic formulas Definition An arithmetic formula is an arithmetic circuit where the fan-out of every gate is at most 1.

5 Arithmetic formulas Definition An arithmetic formula is an arithmetic circuit where the fan-out of every gate is at most 1. Homogeneous formulas are formulas where the polynomial computed at each node is a homogeneous polynomial.

6 Arithmetic formula x 1 1 x 2 1 x n 1 1 x n Figure: The arithmetic circuit for f (x) = x i = (1 + x i ) S [n] i S i S

7 Best known size lower bounds for arithmetic formulas Kalorkoti [Kal85] proved a quadratic size lower bound against a very simple polynomial f (x 1,x 2,...,x n,y 1,...,y n ) = i [n] j [n] xj i y j using transcendence degree as a complexity measure.

8 Best known size lower bounds for arithmetic formulas Kalorkoti [Kal85] proved a quadratic size lower bound against a very simple polynomial f (x 1,x 2,...,x n,y 1,...,y n ) = i [n] j [n] xj i y j using transcendence degree as a complexity measure. He also proved a lower bound of Ω(n 3 ) against Det n.

9 Best known size lower bounds for arithmetic formulas Kalorkoti [Kal85] proved a quadratic size lower bound against a very simple polynomial f (x 1,x 2,...,x n,y 1,...,y n ) = i [n] j [n] xj i y j using transcendence degree as a complexity measure. He also proved a lower bound of Ω(n 3 ) against Det n. We in [CM15] proved a lower bound of Ω(dn 3 ) against IMM n,d.

10 Set-multilinear polynomials Let X = X 1 X d be a partition of the variables.

11 Set-multilinear polynomials Let X = X 1 X d be a partition of the variables. Definition (Set-multilinear polynomials) A polynomial f (X) is said to be set-multilinear with respect to the above partition if every monomial M in f satisfies var(m) X i = 1 for all i [d].

12 Set-multilinear polynomials Let X = X 1 X d be a partition of the variables. Definition (Set-multilinear polynomials) A polynomial f (X) is said to be set-multilinear with respect to the above partition if every monomial M in f satisfies var(m) X i = 1 for all i [d]. A set-multilinear formula is an arithmetic formula where the polynomial computed at every node is a set-multilinear polynomial.

13 Tensors Definition (Tensor) A tensor T is a map of the form T : V 1 V d where each V i is a vector space over, of dimension say m i. The parameter d is called the order of the tensor, and we say that the shape of T is [m 1 ] [m d ].

14 Tensors as polynomials Observation For any tensor T of shape [m 1 ] [m d ], we can associate a set-multilinear polynomial f (X) where X = X 1 X d and X i = x i1,...,x imi as follows. f (X) = 1 i j m j j [d] T (i 1,...,i d ) x 1i1 x did.

15 Tensor rank A tensor T 0 : [n] d is said to be of rank 1 if there exist d vectors v 1,,v d such that T 0 = v 1 v d.

16 Tensor rank A tensor T 0 : [n] d is said to be of rank 1 if there exist d vectors v 1,,v d such that T 0 = v 1 v d. The tensor rank of a tensor T : [n] d is the minimal r such that there exist r rank-1 tensors T 1,...,T r such that T = T T r.

17 Tensor rank A tensor T 0 : [n] d is said to be of rank 1 if there exist d vectors v 1,,v d such that T 0 = v 1 v d. The tensor rank of a tensor T : [n] d is the minimal r such that there exist r rank-1 tensors T 1,...,T r such that T = T T r. The tensor rank of T : [n] d is at most n d 1.

18 Tensors of high rank Lemma (Folklore) A generic tensor T : [n] d has a rank of at least nd 1 d.

19 Tensors of high rank Lemma (Folklore) A generic tensor T : [n] d has a rank of at least nd 1 d. Proof is via a simple argument involving algebraic independence.

20 Connections between formula size and tensor rank Theorem ([Raz10]) Let Φ be a formula of size s n c computing a set-multilinear polynomial f with respect to the partition X = X 1 X d. If d = O(logn/loglogn), then trk(f ) n d(1 1/exp(O(c))).

21 Connections between formula size and tensor rank Theorem ([Raz10]) Let Φ be a formula of size s n c computing a set-multilinear polynomial f with respect to the partition X = X 1 X d. If d = O(logn/loglogn), then trk(f ) n d(1 1/exp(O(c))). Theorem (This work) Let Φ be a formula of size s n c computing a set-multilinear polynomial f with respect to the partition X = X 1 X d. If d = O(logn), then trk(f ) n d(1 1/exp(O(c))).

22 An improvement Theorem (This work) Let Φ be a homogeneous formula of size s n c computing a set-multilinear polynomial f with respect to the partition X = X 1 X d. If logd = o(logn), then trk(f ) n d(1 1/exp(O(c))).

23 Overview of Raz s work Theorem (Step 1) Let Φ be a formula of size poly(n) computing an n-variate homogeneous polynomial f of degree d = O(logn). Then, there is a homogeneous formula Φ that also computes f of size at most poly(n).

24 Overview of Raz s work Theorem (Step 1) Let Φ be a formula of size poly(n) computing an n-variate homogeneous polynomial f of degree d = O(logn). Then, there is a homogeneous formula Φ that also computes f of size at most poly(n). Theorem (Step 2) Suppose d = O logn loglogn. If Φ is a formula of size s = poly(n) that computes a set-multilinear polynomial f, then there is a set-multilinear formula of poly(s) size that computes f as well.

25 Overview of Raz s work Theorem (Step 1) Let Φ be a formula of size poly(n) computing an n-variate homogeneous polynomial f of degree d = O(logn). Then, there is a homogeneous formula Φ that also computes f of size at most poly(n). Theorem (Step 2) Suppose d = O logn loglogn. If Φ is a formula of size s = poly(n) that computes a set-multilinear polynomial f, then there is a set-multilinear formula of poly(s) size that computes f as well. Theorem (Step 3) Let Φ be a set-multilinear formula of size s n c computing a polynomial f. Then trk(f ) < nd. n d/exp(o(c))

26 Overview of this result Theorem ([Raz10]) Let Φ be a formula of size poly(n) computing an n-variate homogeneous polynomial f of degree d = O(logn). Then, there is a homogeneous formula Φ that also computes f of size at most poly(n).

27 Overview of this result Theorem ([Raz10]) Let Φ be a formula of size poly(n) computing an n-variate homogeneous polynomial f of degree d = O(logn). Then, there is a homogeneous formula Φ that also computes f of size at most poly(n). Theorem (This work) Let Φ be a homogeneous formula of size s n c computing a set-multilinear polynomial f with respect to the partition X = X 1 X d. If logd = o(logn), then trk(f ) n d(1 1/exp(O(c))).

28 Properties of Tensor rank Sub-additivity: For T,T 1,T 2 : [n] d, if T = T 1 + T 2 then trk(t ) trk(t 1 ) + trk(t 2 ).

29 Properties of Tensor rank Sub-additivity: For T,T 1,T 2 : [n] d, if T = T 1 + T 2 then trk(t ) trk(t 1 ) + trk(t 2 ). Sub-multiplicativity: For T : [n] d, T 1 : [n] d 1 and T 2 : [n] d d 1, if T = T1 T 2 then trk(t ) trk(t 1 ) trk(t 2 ).

30 An example Let f be a set-multilinear polynomial over X 1 X d such that X i = n for all i.

31 An example Let f be a set-multilinear polynomial over X 1 X d such that X i = n for all i. The tensor rank is at most n d 1.

32 An example Let f be a set-multilinear polynomial over X 1 X d such that X i = n for all i. The tensor rank is at most n d 1. But if also know that f = f 1 f 2 where the product respects set-multilinearity, then trk(f ) trk(f 1 ) trk(f 2 ) n d 1 1 n d d 1 1 = n d 2 where d 1 is the degree of f 1.

33 Crux of the arguments Crux If a set multilinear polynomial that corresponds to an explicit tensor can be expressed as a summation over a few summands each of which has large number of factors, then we can get non-trivial bounds on the tensor rank.

34 Structure of homogeneous formulas Theorem (This work) Let f be a homogeneous n-variate degree d polynomial computed by a size s homogeneous formula. Then for any 0 < t d, f can be equivalently computed by a homogeneous ΣΠ [a] ΣΠ [t] formula of top fan-in s 10(d/t) where a > 1 d 10 t logt.

35 Structure of homogeneous formulas Theorem (This work) Let f be a homogeneous n-variate degree d polynomial computed by a size s homogeneous formula. Then for any 0 < t d, f can be equivalently computed by a homogeneous ΣΠ [a] ΣΠ [t] formula of top fan-in s 10(d/t) where a > 1 d 10 t logt. That is, f can be expressed as s 10(d/t) f = Q i1 Q i2...q ia i=1 where the degree of each Q ij is at most t.

36 Structure of homogeneous formulas Theorem (This work) Let f be a homogeneous n-variate degree d polynomial computed by a size s homogeneous formula. Then for any 0 < t d, f can be equivalently computed by a homogeneous ΣΠ [a] ΣΠ [t] formula of top fan-in s 10(d/t) where a > 1 d 10 t logt. That is, f can be expressed as s 10(d/t) f = Q i1 Q i2...q ia i=1 where the degree of each Q ij is at most t. It is important to note that this product is not set-multilinear.

37 Workaround Consider a term T = Q 1 Q 2...Q a from the above expression.

38 Workaround Consider a term T = Q 1 Q 2...Q a from the above expression. Let us extract out the set-multilinear computations from the product (denoted by SML(T )).

39 Workaround Consider a term T = Q 1 Q 2...Q a from the above expression. Let us extract out the set-multilinear computations from the product (denoted by SML(T )). For a partition S 1 S 2 S a of [d], the corresponding set-multilinear component is Q 1,S1 Q 2,S2 Q a,sa where Q i,si is the sum of set-multilinear monomials in Q i over the parts X j : j S i.

40 Upper bound SML(T ) = trk(sml(t )) S 1 S a =[d] S i =d i S 1 S a =[d] S i =d i trk(f ) n 10c(d/t)+d a Q 1,S1 Q a,sa n d a n d a d d 1 d 2 d a d d 1 d 2 d a

41 Upper bound SML(T ) = trk(sml(t )) S 1 S a =[d] S i =d i S 1 S a =[d] S i =d i trk(f ) n 10c(d/t)+d a Q 1,S1 Q a,sa n d a n d a d d 1 d 2 d a d d 1 d 2 d a We fix t to 2 110c and logd = o(logn), and get that trk(f ) n d(1 1/exp(c)).

42 Log product decomposition for homogeneous formulas Theorem ([HY11]) Let f be an n-variate degree d polynomial computed by a size s homogeneous formula. Then, f can be expressed as where f = s f i1 f i2 f ir (1) i=1 1. the expression is homogeneous, 2. for each i,j, we have 1 j 3 d deg(fij ) 2 j 3 d and r = Θ(logd), 3. each f ij is also computed by homogeneous formulas of size at most s.

43 Proof of depth reduction for homogeneous formulas We start with equation (1) which is easily seen to be a homogeneous ΣΠΣΠ [2d/3] circuit with top fan-in s: f = s f i1 f i2 f ir i=1

44 Proof of depth reduction for homogeneous formulas We start with equation (1) which is easily seen to be a homogeneous ΣΠΣΠ [2d/3] circuit with top fan-in s: f = s f i1 f i2 f ir i=1 1. For each summand f i1...f ir in the RHS, pick the gate f ij with largest degree (if there is a tie, pick the one with smaller index j). If f ij has degree more than t, expand that f ij in-place using (1).

45 Proof of depth reduction for homogeneous formulas We start with equation (1) which is easily seen to be a homogeneous ΣΠΣΠ [2d/3] circuit with top fan-in s: f = s f i1 f i2 f ir i=1 1. For each summand f i1...f ir in the RHS, pick the gate f ij with largest degree (if there is a tie, pick the one with smaller index j). If f ij has degree more than t, expand that f ij in-place using (1). 2. Repeat this process until all f ij s on the RHS have degree at most t.

46 Depth reduction to depth four g s g 11 g 12...g 1r s g s1 g s2...g sr s g 111 g g 11r g 12...g 1r g 1s1 g 1s2...g 1sr g 12 g 1r g s11 g s12...g s1r g s2...g sr g ss1 g ss2 g ssr g s2 g sr Figure: Depth reduction analysis

47 Analysis of the iterative procedure 1. Consider a potential function the total degree of all the big factors i.e., factors of degree > t.

48 Analysis of the iterative procedure 1. Consider a potential function the total degree of all the big factors i.e., factors of degree > t. 2. Geometric progression of degrees = total degree of small factors in a summand is at most 3t. Total degree of big terms is at least d 3t.

49 Analysis of the iterative procedure 1. Consider a potential function the total degree of all the big factors i.e., factors of degree > t. 2. Geometric progression of degrees = total degree of small factors in a summand is at most 3t. Total degree of big terms is at least d 3t. 3. Potential drops by at most 3t per iteration. = at least d/3t iterations are needed.

50 Analysis of the iterative procedure 1. Consider a potential function the total degree of all the big factors i.e., factors of degree > t. 2. Geometric progression of degrees = total degree of small factors in a summand is at most 3t. Total degree of big terms is at least d 3t. 3. Potential drops by at most 3t per iteration. = at least d/3t iterations are needed. 4. Every expansion introduces at least (log 3 t) non-trivial terms. Thus, a > d 3log3t logt.

51 Thank you! 2 2 The theme of these slides is based on the the theme made available at undercba licence.

52