7. Algorithms for Massive Data Problems

Transcription

1 July 4, Mave Data Samlng on the fly 7 Algorthm for Mave Data Problem Mave Data, Samlng h chater deal wth mave data roblem where the nut data (a grah, a matrx or ome other object) too large to be tored n random acce memory One model for uch roblem the treamng model, where the data can be een only once In the treamng model, the natural technque to deal wth the mave data amlng Samlng done ``on the fly, e, a each ece of data een, baed on a con to, one decde whether to nclude the data n the amle ycally, the robablty of ncludng the data ont n the amle may deend on t value Model allowng multle ae through the data are alo ueful; but the number of ae need to be mall We alway aume that Random Acce Memory (RAM) lmted, o entre data cannot be tored n RAM o ntroduce the bac flavor of amlng on the fly, conder the followng rmtve rom a tream of n otve real number a, a, an, draw a amle element a o that the robablty of cng an element roortonal to t value It eay to ee that the followng amlng method wor Uon eeng a, a, a, ee trac of the um a = a+ a + + a and a amle aj, j drawn wth robablty roortonal to t value On readng a +, relace the current a+ amle by a + wth robablty and udate a a+ a 7 Setch of a large matrx + We wll ee how to fnd a good -norm aroxmaton of an m n matrx A ( m, n large) whch beng read from external memory he aroxmaton cont of a amle of column of A, a amle of r row of A and a r multler matrx n between he chematc dagram : A Samle column Mult ler Samled row he crucal ont that the amle of row and column have to drawn accordng to a ecal robablty dtrbuton roortonal to the quared length of the row or column One may recall that f we do the SVD of A and tae only the to ngular vector, we would get a mlar cture; but the SVD tae more tme to comute, requre all of A to be tored n RAM, and

2 July 4, Mave Data Samlng on the fly doe not have the roerty that the row and column are drectly from A However, the SVD doe yeld the bet -norm aroxmaton We wll how error bound for our aroxmaton, but they wll be weaer We brefly touch uon two motvaton for uch a etch Suoe A the document-term matrx of a large collecton of document We are to ``read the collecton at the outet and tore a etch o that later, when a query (rereented by a vector wth one entry er term) arrve, we can fnd t mlarty to each document n the collecton Smlarty defned a dot roduct Wth the cture above, t clear that we can do the matrx-vector roduct of the rght hand de wth a query n tme Orn ( + r+ m) whch would be lnear f r, O() o bound error for th roce, we need to how that the dfference between A and the rght hand de ha mall -norm Recall that the -norm B of a matrx B max{ x B x: x = } A econd motvaton from Recommendaton Sytem Here A would be a cutomer-roduct matrx whoe (, j) th entry the reference of cutomer for roduct j he objectve to collect a few amle entre of A and baed on them, get an aroxmaton to A o that we can mae future recommendaton Our reult would ay that a few amled row of A (all reference of a few cutomer) and a few amled column (all cutomer reference for a few roduct) are enough to gve u an aroxmaton to A rovded we can hyotheze that the amle were drawn accordng to length-quared dtrbuton We frt tacle a mler roblem that of multlyng two matrce he matrx multlcaton wll alo ue length quared amlng and a buldng bloc to the etch 7 Matrx multlcaton ung amlng Suoe A an m nmatrx and B an n matrx and the roduct AB dered We now how how to ue amlng to get an aroxmate roduct fater than the tradtonal A :, denote the th A :, a B,: be multlcaton Let ( ) the th row of B B (,:) column of A ( ) a n matrx hen t eay to ee that n AB = A( :, ) B(,: ) = x m matrx Let ( ) An obvou ue of amlng ugget telf Samle ome value for and comute A( :, ) B(,:) for the amled and ue ther um (caled utably) a the etmate of AB It turn out that non-unform amlng robablte wll be ueful So, we wll defne a random varable that tae on value n {,, n} Let denote the robablty that aume the value ; o the are non-negatve and um to Defne an aocated random matrx varable

3 July 4, Mave Data Samlng on the fly 3 X = A( :, ) B(,: ) Let E( X) denote the entry-we exectaton = n E ( X) = Prob( = ) A( :, ) B(,: ) = A( :, ) B(,: ) = AB h exlan the calng by all t entre n X Defne the varance of X a the um of the varance of ( X ) Var x E x a b m Var = ( ) ( ) j j j = j= j j We can mlfy the lat term by exchangng the order of ummaton to get ( ) :,,: Var X a b = A( ) B( ) j j Now, f we mae roortonal to ( ) we get a mle bound for Var ( X ): A :,, n other word ( ) A :, = A ( X ) A B,: = A B Var ( ) Ung robablte roortonal to length quared of column of A turn out to be ueful n other context a well and amlng accordng to them called ``length-quared amlng A uual, to reduce the varance, do ndeendent tral to c,,, hen fnd the average of the correondng X h A( :, ) B(,: ) A( :, ) B(,: ) A( :, ) B(,:) whch can alo be rereented comactly a ADD B, where D n ``column elector matrx wth D j, j = for j =,, and all other entre of D are zero he j dagonal element of DD j j All other element of

4 July 4, Mave Data Samlng on the fly 4 DD are zero hu, wth column j of B ADD B cont of the weghted um of the roduct of column j of A Lemma 7: Suoe A an m nmatrx and B an n matrx he roduct AB can be etmated by ADD B, where D n and each column of D ha recely one non-zero entry; D ced to be the non-zero entry n column wth robablty ced, t et to / roortonal to A and f he error bounded by E ( ADD B AB ) A B Proof: ang the average of d amle dvde the varance by ndng ADD B n the order (( )( )) AD D B can be done n Omn (( + n+ m)) tme by traghtforward multlcaton Note that f O() and m = n =, then th tme quadratc n n whch O(number of entre) An mortant ecal cae the multlcaton AA hen n fact length-quared dtrbuton can be hown to be the otmal dtrbuton he varance of X defned above for th cae : 4 E( Xj ) ( EXj ) = A AB, j j he econd term ndeendent of the he frt term mnmzed when the are roortonal to A, hence length-quared dtrbuton mnmze the varance of th etmator WHY? 7 Aroxmatng a matrx wth a amle of row and column Now, conder the followng mortant roblem Suoe a large matrx A read from external memory and a etch of t et for further comutaton A mle-mnded etch dea would be to ee a random amle of row and a random amle of column of the matrx If we do unform random amlng, th can fal n general Conder the cae where very few row/column of A are non-zero A mall amle wll m them and we wll only ee zero h toy examle can be made more ubtle by mang a few row have very hgh (abolute value) entre comared to the ret We need to mae the amlng robablte deend on the ze of the entre If we do length-quared amlng of both column and row, then from thee, we can get an aroxmaton to A wth a bound on the aroxmaton error Note that the amlng can be acheved n two ae through the matrx, the frt to comute runnng um of the length-quared of each row and column and the econd a to draw the amle Only the

5 July 4, Mave Data Samlng on the fly 5 amled row and column (whch much maller than the whole) need to be tored n RAM for further comutaton he algorthm tart wth: () Pc column of A ung length-quared amlng Let C (for column) be the m matrx of the ced column caled by/ rob of ced column, e, C = ADwhere D a column elector matrx a n the Lemma 7 () Pc r row of A ung length-quared amlng (here length are row length) Let R (for row) be the r n matrx of the ced row caled by / r rob of ced row Wrte R = DA, where D a r m``row elector matrx We wll aroxmate A by CUR, where U a rmultler matrx comuted from C and R Recall the chematc cture at the begnnng of Secton 7 Before tatng recely what U, we gve ome ntuton A frt dea would be to aroxmate A by ADD I, where I the n ndentty matrx By Lemma 7, the roduct ADD I aroxmate AI = A Note that we would not multly A and I by amlng; the ue of Lemma 7 mly a roof devce here Lemma 7 tell u that n E A ADD I A I A =, whch not very nteretng nce we want <<n Intead of ung the dentty matrx whch ntroduced the factor of n, we wll ue an ``dentty-le matrx baed on R HE OLLOWING NEEDS WORK Lemma 7: If RR nvertble, then R ( RR ) R act a the dentty matrx on the row ace of R Ie, for every vector x of the form x = R y (th defne the row ace of R ), we have R ( RR ) R x= x Proof: or x = R y we have R ( RR ) Rx= R ( RR ) RR y = R y = x Now, we are ready to defne U ()Let U = D R ( RR ) (aumng (v) Lemma 73: Let U = D R ( RR ) (aumng RR nvertble) RR nvertble) hen CUR ADD R ( RR ) R = and ( ) E A CUR A /4 rovded r 3/ r

6 July 4, Mave Data Samlng on the fly Proof: he frt aerton obvou Let R ( RR ) R W = hen CUR = ADD W E( A CUR ) E( A AW ) E( AW ADD W ) Lemma 7 whch tell u ( ) ( ) + he econd term bounded by E AW ADD W E AW ADD W A W (nce for any random varabley, ( ) E Y E( Y ) ) rom Lemma 7, W act le the dentty on a r dmenonal ace and W z = 0 for any z erendcular to the row ace of R So, whch we aw (Lemma 7) wa the um of t quared ngular value, r and hence r E( AW ADD W ) A A /4 r W, Conder the frt term E A AW We recall A AW = Max ( A AW) x / x If x attan th maxmum, x mut be orthogonal to the row ace of R, nce by Lemma 9, for any vector n the row ace of R, W act le the dentty or x orthogonal to row of R, we have ( A AW) = A = A A = ( A A R R) A A A D DA A A A D DA x x x x x x We wll ue Lemma 7 agan to bound th, nce we are cng column of E A A A DD A A whch r A (ame a row of A ) n cng R Indeed, Lemma 7 tell that ( ) mle that E A AW A /4 r rovng the Lemma 7 requency moment of data tream Introducton An mortant cla of roblem concern the frequency moment of data tream Here a data tream a, a, an of length n cont of ymbol a from an alhabet of m oble ymbol whch for convenence we denote a {,, m} hroughout th ecton, nm,, and a wll have thee meanng and (for ymbol) wll denote a generc element of {,, m} he frequency f of the ymbol the number of occurrence of n the tream or a non-negatve nteger, the th frequency moment of the tream

7 July 4, Mave Data Samlng on the fly 7 m = f Note that the = 0 frequency moment correond to the number of dtnct ymbol occurrng n the tream he frt frequency moment jut n, the length of the trng he econd frequency moment f ueful n comutng the varance of the tream n f n f = m m m m In the lmt a become large, f /, the frequency of the mot frequent element() We wll decrbe amlng baed algorthm to comute thee quantte for treamng data hortly But frt a note on the motvaton for thee varou roblem he dentty and frequency of the the mot frequent tem or more generally, tem whoe frequency exceed a fracton of n clearly mortant n many alcaton If the tem are acet on a networ wth ource detnaton addree, the hgh frequency tem dentfy the heavy bandwdth uer If the data urchae record n a uermaret, the hgh frequency tem are the bet-ellng tem he number of dtnct ymbol could be a frt te n the mortant ta of dulcate elmnaton he econd moment (and varance) are ueful n networng a well a databae alcaton and other Large amount of networ-log data are generated by router that can record for all the meage ang through them, the ource addre, detnaton addre, and the number of acet h mave data cannot be ealy orted or aggregated nto total for each ource/detnaton But t mortant to ee f there a ew f ome oular ourcedetnaton ar have a lot of traffc for whch the varance the natural meaure 7 Number of dtnct element n a data tream Conder a equence a, a,, an of n element, each a an nteger n the range to m where n and m are very large Suoe we wh to determne the number of dtnct a n the equence Each a mght rereent a credt card number extracted from a equence of credt card tranacton and we wh to determne how many dtnct credt card account there are he model a data tream where ymbol are een one at a tme We frt how that any determntc algorthm that determne the number of dtnct element exactly mut ue at leat m bt of memory Lower bound on memory for exact determntc algorthm

8 July 4, Mave Data Samlng on the fly 8 Suoe we have een the frt mymbol he et of dtnct ymbol een o far could be any of the m ubet of {,,,m} Each ubet mut reult n a dfferent tate for our algorthm and hence m bt of memory are requred o ee th, uoe frt that two ubet of dtnct ymbol of dfferent ze lead to the ame nternal tate hen our algorthm would roduce the ame count of the number of dtnct ymbol for both nut, clearly an error for one of the nut equence If two equence wth the ame number of dtnct element but dfferent ubet lead to the ame tate, then on next eeng a ymbol that aeared n one equence but not the other would reult n ubet of dfferent ze and thu requre dfferent tate Algorthm for the Number of dtnct element Let aa an a,,, m he number of dtnct element can be etmated wth O(log m) ace Let S {,,, m} be the et of element that aear n the equence Suoe that the element of S were elected unformly at random from be a equence of element where each { } {,,, m} ubject to ther total number S Let mn denote the mnmum element of S Knowng the mnmum element of S allow u to etmate the ze of S he element of S m artton the et {,,, m} nto S + ubet each of ze aroxmately hu, the S + m mnmum element of S hould have value cloe to Solvng m m mn = yeld S = S + S + mn Snce we can determne mn, th gve u an etmate of S he above analy requred that the element of S were ced unformly at random from {,,, m} h generally not the cae when we have a equence aa an of element from {,,, m} Clearly f the element of S were obtaned by electng the S mallet element of {,,, m}, the above technque would gve the wrong anwer If the element are not ced unformly at random, can we etmate the number of dtnct element? he way to olve th roblem to ue a hah functon h where { } { } h:,,, m 0,,,, M DO WE NEED WO DIEREN SIZE SES m AND M? I NO WE COULD AVOID USING A CAPIAL OR AN INEGER o count the number of dtnct element n the nut, count the number of element n the maed et ha ( ), ha ( ),, the ont beng that ha ( ), ha ( ), behave le a random ubet and o the above heurtc argument ung the mnmum to etmate the number of element may aly If we needed ha ( ), ha ( ), to be comletely ndeendent, the ace needed to tore the hah functon would too hgh ortunately, only -way ndeendence needed We recall the formal defnton below But frt recall that a hah functon alway choen from a famly of

9 July 4, Mave Data Samlng on the fly 9 hah functon (at random) and hrae le ``robablty of collon refer to the robablty n th choce Unveral Hah uncton he et of hah functon H = { h h: {,,, m} { 0,,,, M } } -unveral f for all x and y n {,,, m}, x y, and for all z and w n { 0,,,, M } ( ) = ( ) = = Prob h x z and h y w M for a randomly choen h he concet of a -unveral famly of hah functon that gven x, h( x ) equally lely to be any element of { 0,,,, M } ndeendent and for x y, h( x) and ( ) h y are We now gve an examle of a -unveral famly of hah functon or mlcty let M be a rme or each ar of nteger a and b n the range [0,M-], defne a hah functon ab ( ) = + mod( ) h x ax b M o tore the hah functon h ab, we need only tore the two nteger a and b and th requre only O(log M ) ace o ee that the famly -unveral note that h(x)=z and h(y)=w f and only f x a z = mod( M ) y b w x If x y, the matrx nvertble modulo M and there only one oluton for a and b y hu, for a and b choen unformly at random, the robablty of the equaton holdng exactly M Analy of dtnct element countng algorthm Let b, b,, bd be the dtnct value that aear n the nut hen { ( ), ( ), ( d )} S = h b h b h b a et of d random and -way ndeendent value from the et M { 0,,,, M } We now how that mn element n the nut, where mn=mn(s) a good etmate for d, the number of dtnct

10 July 4, Mave Data Samlng on the fly 0 d M Lemma 73: Aume M>00d Wth robablty at leat /3, d, where mn the mn mallet element of S M Proof: rt, we how that Prob > d < mn Prob M Prob mn M d Prob h( b ) M > mn = < = < d d M f h( b ) < d or =,,, d defne the ndcator varable z = 0 otherwe and let d z = z If hb ( ) choen randomly from{ 0,,,, M }, then = Prob[ z = ] hu, E( z ) = d and E( z ) = Now d M M Prob > d Prob h( b ) Prob( z ) Prob z E( z) mn = < = = d By Marov nequalty ( ) Prob z E z M d nally, we how that Prob < < mn M d M M Prob < = Prob mn Prob > h d = ( b ) > d mn or =,,, d defne the ndcator varable and let y d = y = = y Now Prob( ) d M 0 f h( b ) > d otherwe y = =, E( y ) =, and E( y) d or -way ndeendent random varable, the varance of ther um the um of ther varance So Var ( y) = dvar ( y ) urther, t eay to ee that Var( y ) E ( y ), o we have Var( y) E ( y) Now by the Chebychev nequalty, t follow that

11 July 4, Mave Data Samlng on the fly M d M M Prob < = Prob mn Prob ( ) mn > d = h b > d Var( y) = Prob( y = 0) Prob y E( y) E( y) ( E y ) ( ) E( y) M Snce mn > d wth robablty at mot / and M d mn < wth robablty at mot /, d M mn d wth robablty at leat /3 7 Countng the number of occurrence of a gven element Conder a trng of 0 and of length n n whch we wh to count the number of occurrence of Clearly f we had log n bt of memory we could ee trac of the exact number of However, we can aroxmate the number wth only log log n bt Let m be the number of that occur n the equence Kee a value uch that aroxmately the number of occurrence m Storng requre only log log n bt of memory he algorthm wor a follow Start wth =0 or each occurrence of a, add one to wth robablty/ At the end of the trng, the quantty the etmate of m o obtan a con that come down head wth robablty/, fl a far con, one that come down head wth robablty ½, tme and reort head f the far con come down head n all fl Gven, on average t wll tae one before ncremented hu, the exected number of to roduce the current value of = 73 Countng requent element he Majorty and requent Algorthm rt conder the very mle roblem of n eole votng here are m canddate,{,,, m} We want to determne f one canddate get a majorty vote and f o who ormally, we are gven a tream of nteger a, a,, an, each a belongng to {,,, m} and want to determne whether there ome {,,, m} whch occur more than n/ tme and f o whch It eay to ee that to olve the roblem on treamng data (read once only) exactly wth a determntc algorthm, requre Ω( n) ace Suoe n even and the frt n/ tem are all dtnct and the lat n/ tem are dentcal After readng the frt n/ tem, we need to remember exactly whch element of {,, m} have occurred If for two dfferent et of element occurrng n the frt half of the tream, the content of the memory are the ame, then a mtae would occur f the econd half cont of

12 July 4, Mave Data Samlng on the fly an element n one et, but not the other hu, log Ω ( n), are needed m n / bt of memory, whch f m>n he followng a mle low-ace algorthm whch alway fnd the majorty vote f there one If there no majorty, the outut may be arbtrary hat, there may be ``fale otve, but no ``fale negatve Majorty Algorthm: Store a and ntalzed a counter to one or each ubequent a, f a the ame a the currently tored tem, ncrement the counter by one If t dffer, and the counter zero, then tore a and et the counter to one; otherwe, decrement the counter by one o analyze the algorthm, t convenent to vew the decrement counter te a ``elmnatng two tem, the new one and the one whch caued the lat ncrement n the counter It eay to ee that f there a majorty element, t mut be tored at the end If not, each occurrence of wa elmnated; but each uch elmnaton alo caue another tem to be elmnated and o for a majorty tem not to be tored at the end, we mut have elmnated more than n tem, a contradcton Next we modfy the above algorthm o that not jut the majorty, but alo tem wth frequency above ome threhold are detected We wll alo enure (aroxmately) that there are no fale otve a well a no fale negatve Indeed the algorthm below wll fnd the frequency (number of occurrence) of each element of {,, m} to wthn an addtve term of O ( log n) ace by eeng counter ntead of jut n + ung Algorthm requent: Mantan a lt of tem beng counted Intally the lt emty or each tem, f t the ame a ome tem on the lt, ncrement t counter by one Ele, f the lt ha le than tem, add to the lt the new tem wth t counter et to 0ne Otherwe, decrement each of the current counter by one Delete an element from the lt f t count become zero heorem 7: At the end of Algorthm requent, for each {,, m}, t counter on the lt at leat the number of occurrence of n the tream mnu n/(+) In artcular f ome doe not occur on the lt, t counter zero and the theorem aert that t occur fewer than n/(+) tme n the tream Proof: Vew each decrement counter te a elmnatng ome tem An tem can be elmnated n one of two way Ether t the current a beng read and there are already ymbol on the lt n whch cae, t and other tem are beng elmnated he econd way when the tem

13 July 4, Mave Data Samlng on the fly 3 had caued an ncrement n an extng counter earler and that counter beng decremented now In th cae too, other tem are beng multaneouly elmnated hu, the elmnaton of each occurrence of an {,, m} really the elmnaton of + tem hu, no more than n/(+) occurrence of any ymbol can be elmnated Now, t clear that f an tem not elmnated, then t mut tll be on the lt at the end h rove the theorem heorem 7 mle that we can comute the true relatve frequency (number of occurrence / n) n of every {,, m} to wthn an addtve term of (hn of why thee no + contradcton n the fact that we can do t for EVERY element) Examle: 74 he Second Moment h ecton focue on comutng the econd moment, {,, m} Here m = f, of a tream wth ymbol from f denote the number of occurrence of ymbol or m, et each x to ± ndeendently wth robablty / Mantan a um by addng x to the um each tme the ymbol occur n the tream At the end the um wll equal m = x f Now m E x f = 0 = Here the exectaton over the choong of the x Alo nce the ndeendence of m m m x = x + x t t = = = = / t = E f E f E x f f f, x tell u that ( ) 0 E x x = for =/ t hu t

14 July 4, Mave Data Samlng on the fly 4 A = x f an etmator of f We now comute the varance of th etmator 4 Var ( a) E x f = E xx x x f f f f t u v t u v, t, uv, m he frt nequalty becaue the varance at mot the econd moment and the econd equalty by exanon In the econd um, nce the x are ndeendent, f any one of, u, t, or v dtnct from the other, then the exectaton of the whole term zero Note that th doe not need the full ower of mutual ndeendence of all the x, t only need what called 4-way ndeendence, that any four of the x are mutually ndeendent Contnung the calculaton, we need to deal only wth term of the form x xt for t =/ and term of the form x 4 hu, Var ( a) E x xt f ft + E t x f = f f + f = / t = / t 3 f + f = 4( E( a) ) he varance can be reduced by a factor of r by tang the average of r ndeendent tral Wth r ndeendent tral the varance would be at mot 4 ( E ( a) ), o to acheve relatve error ε n r the etmate of f, we need only O( / ε ) ndeendent tral We wll brefly dcu the ndeendent tral here, o a to undertand exactly the amount of ndeendence needed Intead of comutng A a the runnng um x f for one random vector x = ( x, x, x m ), we ndeendently generate r m-vector comute r runnng um x f, x f, x f r 3 r x, x, x,, x at the outet and r = f a = f ar = f Our etmate Let a x, x,, x he varance of th etmator ( a + a + + a r ) r

15 July 4, Mave Data Samlng on the fly 5 ( a+ a + + ar ) = ( Var a + Var a + ) = Var( a), r r r Var ( ) ( ) where we have aumed that the a, a,, ar are mutually ndeendent Now we comute the varance of a a we have done for the varance of a Note that th calculaton aume only 4- way ndeendence between the coordnate of x We wll ummarze the aumton here for future reference: o get an etmate of have r = O( / ε ) vector t () ( ) 0 f E x = for all and t wthn relatve error ε wth robablty at leat 09999, t uffce to r x, x,, x, each wth m coordnate of ± wth () r x, x,, x are mutually ndeendent h mean that for any r vector ± coordnate, r r Pr ( X v ; X v ; X v ) = = = = mr v, v r v wth () Any four coordnate of x are ndeendent Same for 3 r x, x,, x he only drawbac wth the algorthm a we have decrbed t o far that we need to ee the r r vector x, x,, x n memory o that we can do the runnng um h too aceexenve We want to do the roblem n ace deendent uon the logarthm of m, not m telf If ε n Ω (), then r n O (), o t not the r whch the roblem; t the m In the next ecton, we wll ee that we can do the comutaton n O(log m) ace by ung eudo-random vector x, x, ntead of truly random one he eudo-random vector wll atfy (), () and () and o they wll uffce h eudo-randomne and lmted ndeendence have dee connecton, o we wll go nto the connecton a well Error-Correctng code, olynomal nterolaton and lmted-way ndeendence Conder the roblem of generatng a random m-vector x of ± o that any ubet of four coordnate mutually ndeendent, e, for any dtnct, t, u, and v n{,, m} and any a, b, c, and d n {-, +}, Pr( x = ax ; t = bx ; u = cx ; v = d) =

16 July 4, Mave Data Samlng on the fly We wll ee that uch a vector may be generated from a (truly) random ``eed of only O(log m) mutually ndeendent bt he frt fact needed for th that for any, there a fnte feld wth exactly element, each of whch can be rereented wth bt and arthmetc oeraton n the feld can be carred out n O ( ) tme each We aume th here Here, wll be the celng of log m We alo aume another bac fact about olynomal nterolaton whch ay that a olynomal of degree at mot three unquely determned by t value (over any feld ) at four ont More recely, for any 4 dtnct ont a, a, a3, a4 and any 4 (obly not dtnct) value b, b, b 3, b 4, there a unque 3 olynomal f ( x) = f0 + fx+ fx + f3x of degree at mot three o that wth all comutaton done over, we have: f ( a) = b; f( a) = b; f( a3) = b3; f( a4) = b4 Now our defnton of the eudo-random vector X wth 4-way ndeendence mle: Chooe f0, f, f, f3ndeendently and unformly at random from or =,, m, let x be 3 the leadng bt of the bt rereentaton of f () = f + f + f + f Lemma 74: he X defned above ha 4-way ndeendence 0 3 Proof: Let,t,u, and v be any 4 coordnate of x and let α, βγδ,, {,} here are exactly element of whoe leadng bt α and mlarly for β, γ, and δ So, there are exactly 4( ) 4-tule of element b, b, b3, b4 o that leadng bt of b α ; the leadng bt of b β ; the leadng bt of b3 γ and the leadng bt of b4 δ or each uch b, b, b3, b 4, there recely one olynomal f o that f ( ) = b; f( t) = b ; f( u) = b ; f( v) = b a we aw above So, the robablty that we have x 3 4 = α; x = β; x = γ; x = δ t u v recely 4( ) 4( ) total number of 4 f = = a aerted he lemma tate how to get one vector X wth 4-way ndeendence We needed r = O( / ε ) of them Alo they all mut be mutually ndeendent But th eay, jut chooe r olynomal at the outet

17 July 4, Mave Data Samlng on the fly 7 o mlement the algorthm wth low ace, tore only the olynomal n memory h requre 4 = O(log m) bt er olynomal for a total of O(log m/ ε) bt When a ymbol n the tream read, comute, r x x, x But note that x jut the leadng bt of the frt olynomal evaluated at ; th calculaton n O(log m) tme hu, we reeatedly comute the x from the ``eed, namely the coeffcent of the olynomal h dea of olynomal nterolaton alo ued n everal context Error-correctng code an mortant examle Say we wh to tranmt n bt over a channel whch may ntroduce noe One can ntroduce redundancy nto the tranmon o that ome channel error can be corrected A mle way to do th to vew the n bt to be tranmtted a coeffcent of a olynomal f ( x ) of degree n Now tranmt f evaluated at ont,, 3, n+ m At the recevng end, any n correct value wll uffce to recontruct the olynomal and the true meage So u to m error can be tolerated But even f the number of error at mot m, t not a mle matter to now whch value are corruted We do not elaborate on th here Exerce Exerce 7: Gven a tream of n otve real number a, a, an, uon eeng a, a, a ee trac of the um a = a+ a + + a and a amle aj, j drawn wth robablty roortonal to a+ t value On readng a +, wth robablty relace the current amle wth a + and a+ a+ udate a Prove that the algorthm elect an a from the tream wth the robablty of cng an element beng roortonal to t value Exerce 7: Gven a tream of ymbol,,, n, gve an algorthm that wll elect one ymbol unformly at random from the tream How much memory doe your algorthm requre? Exerce 73: How would one c a random word from a very large boo where the robablty of cng a word roortonal to the number of occurrence of the word n the boo Exerce 74: or the treamng model gve an algorthm to draw ndeendent amle each wth the robablty roortonal to t value Jutfy that your algorthm wor correctly

18 July 4, Mave Data Samlng on the fly 8 Exerce 75: Suoe we want to c a row of a matrx at random where the robablty of cng row roortonal to the um of quare of the entre of that row How would we do th n the treamng model? Do not aume that the element of the matrx are gven n row order () Do the roblem when the matrx gven n column order () Do the roblem when the matrx rereented n are notaton: t jut reented a a lt of trle (, j, a j ), n arbtrary order Exerce 7: Suoe A,B are two matrce, then how that AB A( :, ) B(,: ) n = Exerce 77: Generate two 00 by 00 matrce A and B wth nteger value between and 00 Comute the roduct AB both drectly and by amlng Plot the dfference n L norm between the reult a a functon of the number of amle Exerce 77: Show that ADD B exactly A( :, ) B(,: ) A( :, ) B(,: ) A( :, ) B(,:) = Exerce 78: Suoe a, a,, am are non-negatve real Show that the mnmum of the contrant x 0; x = attaned when x are roortonal to a a m = x ubject to Exerce 79: Show that for a -unveral hah famly Pr ( hx ( ) z) x {,, m} and z {0,,, M} = = for all M + Exerce70: Let be a rme A et of hah functon H = { h { 0,,, } { 0,,, } } { 0,,, } the Prob ( ), ( ), ( ) 3-unvreal? 3-unveral f for all u,v,w,x,y, and z n ( h x u h y v h z w) = = = equal I the et of hah functon H Exerce 70: Gve an examle of a et of hah functon that not -unveral Exerce 70: (a) What the varance of the above method of countng the number of occurrence of a wth log log n memory? (b) Can the algorthm be terated to ue only log log log n memory? What haen to the varance?

19 July 4, Mave Data Samlng on the fly 9 Exerce 7: Conder a con that come down head wth robablty a Prove that the exected number of fl before a head occur /a Exerce 7: Randomly generate a trng xx xn of 0 0 and wth robablty ½ of x beng a Count the number of one n the trng and alo etmate the number of one by the aroxmate countng algorthm Reeat the roce for =/4, /8, and / How cloe the aroxmaton? Exerce 73: Contruct an examle n whch the majorty algorthm gve a fale otve, e, tore a non-majorty element at the end Exerce 74: Contruct examle where the frequent algorthm n fact doe a badly a n the theorem, e, t ``undercount ome tem by n/(+) Exerce 75: Recall bac Stattc on how an average of ndeendent tral cut down varance and comlete the argument for relatve error ε etmate of f Exerce 7: Let be a feld Prove that for any 4 dtnct ont a, a, a3, a4 and any 4 (obly not dtnct) value b, b, b3, b4, there a unque olynomal 3 f ( x) = f0 + fx+ fx + f3x of degree at mot three o that wth all comutaton done over, we have: f ( a) = b; f( a) = b; f( a3) = b3; f( a4) = b4