6 Matrix Concentration Bounds

Transcription

1 6 Matix Concentation Bounds Concentation bounds ae inequalities that bound pobabilities of deviations by a andom vaiable fom some value, often its mean. Infomally, they show the pobability that a andom vaiable deviates fom its expectation is small. A basic example of andom vaiables being concentated aound the mean is stated by law of lage numbes that says unde mild conditions, sum of independent andom vaiables is close to thei expectation with a high pobability. Concentation bounds, which ae also known as tail bounds, ae a majo tool in analyzing aveage behaviou of algoithms, e.g. estimating the failue pobability o establishing high pobability bounds on unning time and space usage. In this lectue, we give an intoduction to some of concentation bounds on andom vaiables and andom matices. 6.1 Basic Tail Bounds If X is a andom vaiable and a 2 R is a eal value, then P[X a] and P[X apple a] ae called uppe tail and lowe tail of the distibution on X, espectively. Concentation bounds might bound one of the tails o both. We stat with the most basic yet fundamental tail bound, called as Makov s Inequality. Theoem (Makov s Inequality). Let X be a non-negative andom vaiable. Then fo all a>0 P(X Poof. Define an indicato andom vaiable I a = And that completes the poof. a) apple E[X] a ( 1 if X a 0 othewise. Note in both cases X ai a, theefoe E[X] a E[I a ] = a P(X a) Makov s inequality is the best possible tail bound one can get if all he knows is non-negativity of andom vaiable and its expectation, yet it is often too weak to yeild useful esults. Tail bounds can be impoved if moe infomation about X, e.g. its vaiance, is available. Below, we state a significantly stonge bound that employs exta statistics. Theoem (Chebyshev s Inequality). Fo any andom vaiable X, and any a>0 P ( X E[X] a) apple Va[X] a 2 Poof. Let s apply Makov s inequality on andom vaiable X =(Y E[Y ]) 2 and scala a = b 2, P (Y E[Y ]) 2 b 2 apple E (Y E[Y ]) 2 Note that P (Y E[Y ]) 2 b 2 = P ( Y E[Y ] b), and E (Y E[Y ]) 2 = Va(Y ) is the definition of vaiance. Theefoe P ( Y E[Y ] b) apple Va(Y ) b 2 b 2

2 The use of Chebyshev s inequality indicates that with pobability lage than 1 1/a 2, a andom vaiable X will fall within a times the standad deviation aound E[X]. Fo example, 75% of the times a andom value falls in the inteval [E[X] 2Va(X), E[X]+2Va(X)]. Both Makov s and Chebyshev s inequalities povide polynomially decaying bounds in amount of deviation (i.e. a in the fomula). Moe inteesting ae concentation bounds in which deviation pobabilities decay exponentially in the distance fom mean. Theoem (Chenoff-Hoeffding Inequality). Conside a set of independent andom vaiables {X 1,X 2,,X }. If we know each andom vaiable is bounded as a i apple X i apple b i with i = b i a i, then fo M = P X i and any paamete 2 (0, 1/2) 2 2 P ( M E[M] >) apple 2exp P Although Chenoff-Hoeffding inequality povides a stonge exponentially deceasing bound, it equies andom vaiables to be independent a condition that neithe the Makov s no the Chebyshev s inequalities equie. Befoe we move on to concentation bounds fo matices, we state a concete vesion of Chenoff bound that has an analogous in matices. This vesion bounds the tail distibution of a sum of independent 0, 1 andom vaiables that ae not necessaily distibuted identically. Theoem (Chenoff s Inequality). Let {X 1,X 2,,X } be a sequence of independent 0, 1 andom vaiables. Define M = P X i and µ = E[M]. Then fo any >0: e µ P (M (1 + )µ) apple (1 + ) Matix Concentation Bounds In applications, it is common that a andom matix can be expessed as a sum of independent andom matices[1]. Fo example, the covaiance of X 2 R n d can be witten as X T X = P n xt i x i whee x i denotes i-th ow of X. In this section, we state two common bounds on andom matices[1] Matix Chenoff Bound Chenoff s Inequality has an analogous in matix setting; the 0, 1 andom vaiables tanslate to positivesemidefinite andom matices which ae unifomly bounded on thei eigenvalues. Theoem (Matix Chenoff bound). Let {X 1,X 2,,X } be a finite sequence of independent d d dimensional andom matices such that each X i is positive semi-definite (X i 0) and is uppe bounded on eigenvalues, i.e. max(x i ) apple R fo a positive constant R. Define µ min = min ( P E[X i]) and µ max = max ( P E[X i]), then fo any 2 [0, 1] And fo any 0 P P min max E! X i min! X i! apple apple (1 )µ min apple d.! X i 2 i e (1 ) µ min R log d! apple apple (1 + )µ max apple d. e (1 + ) 1+ µ min /R µ max/r

3 E max! X i apple 1.8 µ max + R log d Hee, we state an example to demonstate the application of Matix Chenoff bound. Example [Random Submatix]. [1] Conside a fix matix C =[c 1,c 2,,c n ] 2 R d n. Let s constuct a submatix Z 2 R d n C by andomly picking columns of C with pobability s/n. If column c i is selected, we set i-th column of Z to c i (i.e. z i = c i ), othewise z i =0. We ae inteested in finding the expected lagest singula value of Z and expected smallest singula values of Z. Solution: Let s define a andom vaiable i fo each column c i. If c i is selected, we set i =1and othewise we set it to zeo. Theefoe i is a 0, 1 andom vaiable as following ( 1 with pobability s/n i = 0 othewise Putting them in a diagonal matix =diag( 1, 2,, n) 2 R n n allows us to wite Z as Z = C. In ode to be able to use the Matix Chenoff bound, we need to convet singula values to eigenvalues. Note that singula values of Z ae elated to eigenvalues of ZZ T as i(z) 2 = i (ZZ T ); theefoe if we conside singula values and eigen values indexed in descending ode, we can bound lagest singula value of Z as and smallest singula value as 1(Z) 2 = 1 (ZZ T ) d(z) 2 = d (ZZ T ) Next, we wite ZZ T in tems of sum of andom matices that satisfy conditions of theoem 6.2.1: If we denote each summand as X i, ZZ T = ZZ T =(C )(C ) T = C 2 C T = C C T = ic i c T i = X i E[ZZ T ]= E[X i ]= s n ic i c T i c i c T i = s n CCT Note that X i s ae independent, positive semi-definite, and bounded as max(x i ) applekc i k 2. So the maximum eigenvalue of all summands is bounded as R = max kc ik 2. The mean paametes satisfy 1appleiapplen and µ max = n (E[ZZ T ]) = s n 1 (C) 2 µ min = d (E[ZZ T ]) = s n d (C) 2 Applying Matix Chenoff inequality we obtain E 1(Z) 2 = E d(zz T ) apple 1.8 ( s n ) 1(C) 2 + max 1appleiapplen kc ik 2 log d E d(z) 2 = E d(zz T ) 0.6 ( s n ) d(c) 2 max 1appleiapplen kc ik 2 log d As this bound shows andom matix Z gets a shae of the spectum of C in popotion to the numbe of columns it picks. In othe wods, matix C has n columns which andom matix Z includes about s of them in expectation, and the bound is showing each singula value i (Z) 2 of Z inheits an s/n shae of i(c) 2 fo all 1 apple i apple d.

4 6.2.2 Matix Benstein Inequality Benstein Inequality dops the condition of positive semi-definitness and concens a sum of zeo-mean bounded andom matices. Theoem (Matix Benstein Inequality). Let {X 1,X 2,,X } be a set of independent d 1 d 2 dimensional andom matices with E[X i ]=0and P kx i k 2 apple R fo a positive constant R and fo all X i s. Define vaiance paamete as = max E(X ixi T ), P E(XT i X i), then fo any t 0 t 2 /2 P X i t apple (d 1 + d 2 )exp + Rt/3 E X i apple p 2 log(d 1 + d 2 )+ R 3 log(d 1 + d 2 ) Example [Randomize Matix Multiplication]. [1] Conside matix B 2 R d1 n with unit nom columns (i.e. kb i k =1, 81 apple i apple n) and matix C 2 R n d 2 with unit nom ows (i.e. kc i k =1, 81 apple i apple n). We ae inteested in computing an appoximation to the poduct BC 2 R d 1 d 2 using a andom sample of columns of B and ows of C. solution: Fist note that if we constuct a andom vaiable S =(nb j c j ) 2 R d 1 d 2 whee j 2 [1,n] is sampled unifomly at andom, then S will be an unbiased estimato fo the poduct BC. This is tue because: E[S] =E nb j c j = n (b k c k P(k = j)) = n (b k c k ( 1 n )) = (b k c k )=BC k=1 Although S is an unbiased estimato fo BC, it has a lage vaiance. In ode to educe the vaiance we compute m independent copy of S and aveage them as Z = 1 P m m S i.now Z is a bette appoximation fo BC. The eo we ae inteested to compute is m X e = E [kz BCk 2 ]=E ( 1 m S 1 i m BC) 2 Theefoe each summand is X i = 1 m (S i BC). Note that summands ae independent and E[X i ]=0. Befoe we bound spectal nom of summands, we use symmetization technique to say E [kz BCk] apple 2 m m E X i S i Whee i ae independenet Rademache andom vaiables, independent of S i s. Theefoe k=1 kx i k 2 applek i S i kappleks i k = nkb j kkc i k = n Theefoe the uppe bound R on spectal nom of any summand is n. Now we compute the vaiance in two steps: E[S i Si T ]= n 2 (b j c j )(b j c j ) T P(j = i) = n 2 kc j k 2 b j b T j = n 2 BB T And E[S T i S i ]= n 2 (b j c j ) T (b j c j )P(j = i) = k=1 n 2 kb j k 2 c jt c j = n 2 C T C

5 Theefoe we obtain = max{kmn BB T k, kmn C T Ck} = mn max{kbk 2, kck 2 }. Applying the Benstein Bound E [kz BCk] apple 2 m m E X i S i apple 2 p 2mn log(d1 + d 2 ) max{kbk 2, kck 2 } + n m 3 log(d 1 + d 2 ) = 8n m log(d 1 + d 2 ) max{kbk 2, kck 2 } + 2n 3m log(d 1 + d 2 ) In ode to make sense out of this inequality, let s take a look at numeic ank of B and C. As we know, numeic ank of a matix is defined as nank(a) =kak 2 F /kak2 2. Since B and C has unit columns and unit ows, espectively then kbk 2 F = kck2 F = n. Then the geometic mean of numeic anks will be µ = p s n nank(b) nank(c) = 2 n kbk 2 = 2 kck2 2 kbk 2 kck 2 Conveting the eo bound to a elative scale we obtain E [kz BCk] kbk 2 kck 2 8n apple m log(d 1 + d 2 ) max{kbk 2, kck 2 } + 2 kbk 2 kck 2 3 8n apple m log(d d 2 ) min{kbk 2, kck 2 } s n 8 = kbk 2 kck 2 m log(d 1 + d 2 ) 8µn apple m log(d 1 + d 2 )+ 2 µ log(d 1 + d 2 ) 3 m n log(d 1 + d 2 ) mkbk 2 kck 2 µ log(d 1 + d 2 ) m kbk 2 kck 2 min 2 {kbk 2, kck 2 } µ log(d 1 + d 2 ) m Note last inequality follows since min 2 {kbk 2, kck 2 } =min{kbk 2 2, kck2 2 } and 1 applekbk 2, kck 2 apple n. Setting the elative eo to be less than, the numbe of samples (m) will be m = ( 2 µn log(d 1 +d 2 )). This shows the numbe of samples we need to appoximate poduct of two matices is popotional to the geometic mean of thei numeic ank. And that makes sense because if two matices of high ank, it means thee ae many distinct diections in thei spectum, theefoe we need to pick many samples to get a good appoximation to thei poduct.

6

7 Bibliogaphy [1] Joel A Topp. Use-fiendly tail bounds fo sums of andom matices. Foundations of Computational Mathematics, 12(4): ,