1. Universal v.s. non-universal: know the source distribution or not.

Transcription

1 28. Radom umber geerators Let s play the followig game: Give a stream of Ber( p) bits, with ukow p, we wat to tur them ito pure radom bits, i.e., idepedet fair coi flips Ber( / 2 ). Our goal is to fid a uiversal way to extract the most umber of bits. I 95 vo Neuma proposed the followig scheme: Divide the stream ito pairs of bits, output if, output if, otherwise do othig ad move to the ext pair. Sice both ad occur with probability pq (where q = p), regardless of the value of p, we obtai fair coi flips at the output. To measure the efficiecy of vo Neuma s scheme, ote that, o average, we have 2 bits i ad 2pq bits out. So the efficiecy (rate) is pq. The questio is: Ca we do better? Several variatios:. Uiversal v.s. o-uiversal: kow the source distributio or ot. 2. Exact v.s. approximately fair coi flips: i terms of total variatio or Kullback-Leibler divergece We oly focus o the uiversal geeratio of exactly fair cois. 28. Setup Recall from Lecture 6 that {, } = k {, } k = {,,,,,... } deotes the set of all fiitelegth biary strigs, where deotes the empty strig. For ay x {, }, let l( x) deote the legth of x. Let s first itroduce the defiitio of radom umber geerator formally. If the iput vector is X, deote the output (variable-legth) vector by Y {, }. The the desired property of Y is the followig: Coditioed o the legth of Y beig k, Y is uiformly distributed o {, } k. Defiitio 28. (Extractor). We say Ψ {, } {, } a extractor if. Ψ(x) is a prefix of Ψ( y) if x is a prefix of y. 2. i.i.d. For ay ad ay p (, ), if X Ber( p ), the Ψ( X ) Ber( / 2) k coditioed o l( Ψ( X )) = k. The rate of Ψ is r Ψ (p) = lim sup E[ l( Ψ( X ))] i.i.d., X Ber(p). Note that the vo Neuma scheme above defies a valid extractor Ψ vn (with ΨvN( x 2+ ) = Ψ vn ( x 2 )), whose rate is r vn ( p) = pq. 285

2 28.2 Coverse No extractor has a rate higher tha the biary etropy fuctio. The proof is simply data processig iequality for etropy ad the coverse holds eve if the extractor is allowed to be o-uiversal (depedig o p). Theorem 28.. For ay extractor Ψ ad ay p (, ), rψ(p) h(p) = p log 2 p + q log 2. q Proof. Let L = Ψ(X ). The h( p) = H( X ) H(Ψ( X )) = H(Ψ(X ) L) + H(L) H(Ψ(X ) L) = E [L] bits, where the step follows from the assumptio o Ψ that Ψ( X ) is uiform over {, } k coditioed o L = k. The rate of vo Neuma extractor ad the etropy boud are plotted below. Next we preset two extractors, due to Elias [Eli72] ad Peres [Per92] respectively, that attai the biary etropy fuctio. (More precisely, both ideas costruct a sequece of extractors whose rate approaches the etropy boud). bit rate r vn 2 p 28.3 Elias costructio of RNG from lossless compressors The ituitio behid Elias scheme is the followig:. For iid X, the probability of each strig oly depeds its type, i.e., the umber of s. Therefore coditioed o the umber of oes, X is uiformly distributed (over the type class). This observatio holds uiversally for ay p. 2. Give a uiformly distributed radom variable o some fiite set, we ca easily tur it ito variable-legth fair coi flips. For example, if U is uiform over {, 2, 3 }, we ca map, 2 ad 3. Lemma 28.. Give U uiformly distributed o [ M ], there exists f [ M] {, } such that coditioed o l( f( U)) = k, f( U) is uiformly over {, } k. Moreover, log 2 M 4 E[l(f(U))] log 2 M bits. 286

3 Proof. We defied f by partitioig [ M] ito subsets whose cardialities are powers of two, ad assig elemets i each subset to biary strigs of that legth. Formally, deote the biary expasio of M by M = i m i 2 i =, where the most sigificat bit m = ad = log 2 M +. Those o-zero t i m a partitio M j i i s defies [ ] = j j= M j, where M i = 2. Map the elemets of M j to {, }. To prove the boud o the exp ected legth, the upper boud follows from the same etropy argumet log 2 M = H( U) H(f(U )) H(f(U) l( f( U))) = E[l( f(u))], ad the lower boud follows from E[l(f(U))] = m i 2 i i = M i= M i= where the last step follows from log 2 M +. m i 2 i ( i) 2 M i= 2 i ( i) 2+ M 4, Elias extractor. Let w( x ) defie the Hammig weight (umber of oes) of a biary strig. Let T k = {x {, } w(x ) = k} defie the Hammig sphere of radius k. For each k, we apply the fuctio f from Lemma 28. to each T k. This defies a mappig Ψ E, { } {, } ad the we exted it to Ψ E {, } {, } by applyig the mappig per -bit block ad discard the last icomplete block. The it is clear that the rate is give by E l Ψ [ ( E)( X )]. By Lemma 28., we have E log ( 4 w(x ) ) E l [ (Ψ E)(X )] E log ( w(x ) ) Usig Stirlig s expasio (see, e.g., [Ash65, Lemma 4.7.]), we have 2h (p) 8pq ( k ) 2h(p) where 2πpq p = q = k/ (, ) ad hece E[ l( Ψ E )( X )] = h( p) + O( log ). Therefore the extractio rate approaches h( p) as Peres iterated vo Neuma s scheme Mai idea: Recycle the bits throw away i vo Neuma s scheme ad iterate. What did vo Neuma s extractor discard: (a) bits from equal pairs. (b) locatio of the distict pairs. To achieve the etropy boud, we eed to extract the radomess out of these two parts as well. First some otatios: Give x 2 2, let k = l( ΨvN( x )) deote the umber of cosecutive distict bit-pairs. Let m <... < m k deote the locatios such that x 2mj x 2mj. Let i <... < i k deote the locatios such that x 2ij = x 2ij. y j = x 2mj, v j = x 2ij, u j = x 2j x 2j +. Here y k are the bits that vo Neuma s scheme outputs ad both v k ad u are discarded. Note that u is importat because it ecodes the locatio of the y k ad cotais a lot of iformatio. Therefore vo Neuma s scheme ca be improved if we ca extract the radomess out of both v k ad u. Peres extractor: For each t N, recursively defie a extractor Ψ t as follows: Set Ψ to be vo Neuma s extractor Ψ, i.e., Ψ (x 2+ ) = Ψ (x 2 ) = y k vn. Defie Ψ t by Ψ t (x 2 ) = Ψ t (x 2+ ) = (Ψ (x 2 ), Ψ t (u ), Ψ t (v k )). 287

4 Example: Iput x = of legth 2 = 2. Output recursively: ( )()() ()()()() ()() Note that the bits that eter ito the iteratio are loger iid. To compute the rate of Ψ t, it is coveiet to itroduce the otio of exchageability. We say X are exchageable if the joit distributio is ivariat uder permutatio, that is, P X,...,X = P Xπ(),...,X π() for ay permutatio π o [ ]. I particular, if X i s are biary, the X are exchageable if ad oly if the joit distributio oly depeds o the Hammig weight, i.e., P X =x = p ( w ( x )). Examples: X is iid Ber( p ); X is uiform over the Hammig sphere T k. Lemma 28.2 (Ψ t preserves exchagebility). Let X 2 be exchageable ad L = Ψ( X 2 ). The coi.i.d. ditioed o L = k, Y k, U ad V k are idepedet ad exchageable. Furthermore, Y k Ber( 2) ad U is uiform over T k. Proof. If suffices to show that y, y {, } k, u, u T k ad v, v {, } k such that w( v) = w( v ), P[ Y k = y, U = u, V k = v L = k] = P[ Y k = y, U = u, V k = v L = k ]. Note that X 2 ad the triple ( Y k, U, V k 2 ) are i oe-to-oe correspodece of each other (to recostruct X, simply read the k distict pairs from Y ad fill them accordig to the locatios oes i U ad fill the remaiig equal pairs from V ). Fially, ote that u, y, v ad y, u, v correspod to two iput strigs x ad x of idetical Hammig weight (w( x) = 2k + 2w( v )) ad hece of idetical probability due to the exchageability of X 2. [Examples: ( y, u, v) = (,, ) x = ( ), ( y, u, v) = (,, ) x = ( ).] Computig the margials, we coclude that both Y k ad U are uiform over their respective support set. Lemma 28.3 (Ψ is a extractor). Let X 2 2 i.i.d. t be exchageable. The Ψt( X ) Ber( / 2) coditioed o l( Ψ t ( X 2 )) = m. Proof. Note that Ψ (X 2 m t ) {, }. It is equivalet to show that for all s {, } m, P[ Ψ t (X 2 ) = s m ] = 2 m P[ l( Ψt( X 2 )) = m ]. Proceed by iductio o t. The base case of t = follows from Lemma 28.2 (the distributio of the Y part). Assume Ψ t is a extractor. Recall that Ψt X 2 k Ψ X 2 ( ) = ( ( ), Ψ t ( U ), Ψ t ( V )) ad write the legth as L = L + L 2 + L 3, where L 2 L 3 L by Lemma The P[ Ψ t ( X 2 ) = m = k= Lemma = m iductio s m ] m P[Ψ t (X 2 ) = s L = k] P[ L = k] mk 28.2 P[L = k ]P[ Y k = s k L = k]p[ψ (U ) = s k +r t k+ L = k]p[ψ t (V k ) = s m k+r+ L = k] k= r= m mk = P[L = k]2 k 2 r P[L 2 = r L = k]2 ( r) P[L 3 = m k r L = k] k= r= = 2 m P[L = m]. If X 2 is iid Ber(p), the V k is iid Ber(p 2 /(p 2 + q 2 )), sice L Biom(, 2pq) ad P[Y k = y, U = u, V k = v L = k] = 2 k ( k ) ( p2 ) m ( q2 ) km, where m = w(v). I geeral we caot say much more tha the fact p 2 +q 2 p 2 + q 2 that V k is exchageable. 288 mk

5 i.i.d. Next we compute the rate of Ψ t. Let X 2 Ber(p). The by SLLN, coverges a.s. to pq. Assume that 2 l(ψ t(x 2 )) a.s. r t (p). The L l 2 (Ψ t(x 2 )) = l(ψ t(u )) + 2 l(ψ t(v L )). 2 l(ψ (X 2 )) L 2 Note that U i.i.d. Ber(2pq), V L i.i.d. L Ber(p 2 /(p 2 + q 2 a.s. )) ad L. The the iductio hypothesis implies that l a 2 (Ψ t(u )).s. r t (2pq) ad 2(L l(ψ ) t(v L )) r a.s. t (p 2 /(p 2 + q 2 )). We obtai the recursio: r t (p) = pq + 2 r t(2pq) + p2 + q 2 r t ( p2 T r t p, (28.) 2 p 2 q 2 ) ( )( ) + where the operator T maps a cotiuous fuctio o [, ] to aother. Note that f g poitwise the T f T g. The it ca be show that r t coverges mootoically from below to the fixed-poit of T, which turs out to be exactly the biary etropy fuctio h. Istead of directly verifyig i.i.d. T h = h, ext we give a simple proof: Cosider X, X 2 Ber( p ). The 2h( p) = H( X, X 2 ) = H( X X 2, X ) = H(X X 2 ) + H(X X X 2 ) = h(p 2 + q 2 ) + 2pqh( 2 ) + (p2 + q 2 )h( p2 ). p 2 +q 2 The covergece of r t to h are show i Fig Figure 28.: Rate fuctio r t for t =, 4, versus the biary etropy fuctio Beroulli factory Give a stream of Ber(p) bits with ukow p, for what kid of fuctio f [, ] [, ] ca we simulate iid bits from Ber( f( p )). Our discussio above deals with f( p) 2. The most famous example is whether we ca simulate Ber( 2p) from Ber( p ), i.e., f( p) = 2p. Keae ad O Brie [KO94] showed that all f that ca be simulated are either costats or polyomially bouded away from or : for all < p <, mi{f( p), f( p)} { p, p} for some. I particular, doublig the bias is impossible. The above result deals with what f(p) ca be simulated i priciple. What type of computatioal devices are eeded for such as task? Note that sice r ( p) is quadratic i p, all rate fuctios r t that arise from the iteratio (28.) are ratioal fuctios (ratios of polyomials), covergig to the biary etropy fuctio as Fig. 28. shows. It turs out that for ay ratioal fuctio f that satisfies < f < o (, ), we ca geerate idepedet Ber(f(p)) from Ber(p) usig either of the followig schemes [MP5]: 289

6 . Fiite-state machie (FSM): iitial state (red), itermediate states (white) ad fial states (blue, output or the reset to iitial state). 2. Block simulatio: let A k, A be disjoit subsets of {, }. For each k-bit segmet, output if fallig i A or if fallig i A. If either, discard ad move to the ext segmet. The block size is at most the degree of the deomiator polyomial of f. The ext table gives examples of these two realizatios: Goal Block simulatio FSM f(p) = /2 A = ; A = f(p) = 2pq A =, ; A =, p3 f(p) = A p 3 +q 3 = ; A = Exercise: How to geerate f(p) = /3? It turs out that the oly type of f that ca be simulated usig either FSM or block simulatio is ratioal fuctio. For f( p) = p, which satisfies Keae-O Brie s characterizatio, it caot be simulated by FSM or block simulatio, but it ca be simulated by pushdow automata (PDA), which are FSM operatig with a stack [MP5]. What is the optimal Beroulli factory with the best rate is uclear. Clearly, a coverse is the h(p) etropy boud, which ca trivial (bigger tha oe). h( f(p)) be 28.6 Related problems Geerate samples from a give distributio The problem of how to tur pure bits ito samples of a give distributio P is i a way the opposite directio of what we have bee cosiderig so far. This ca be doe via Kuth-Yao s tree algorithm: 29

7 Startig at the root, flip a fair coi for each edge ad move dow the tree util reachig a leaf ode ad outputtig the symbol. Let L deote the umber of flips, which is a radom variable. The H( P ) E[ L] H( P ) + 2bits. Examples: To geerate P = [/2, /4, /4] o {a, b, c}, use the fiite tree: E[L] =.5. a b c To geerate P = [/3, 2/3] o {a, b} (ote that 2/3 = ifiite tree: E[ L] = 2 (geometric distributio)...., /3 =....), use the a b a Approximate radom umber geerator The goal is to desig f X {, } k s.t. f(x ) is close to fair coi flips i distributio i certai distaces (TV or KL). Oe formulatio is that D( P f(x ) Uiform) = o(k). Ituitios: The coectio to lossless data compressio is as follows: A good compressor squeezes out all the redudacy of the source. Therefore its output should be close to pure bits, otherwise we ca compress it furthermore. So good lossless compressors should act like good approximate radom umber geerators. 29

8 MIT OpeCourseWare Iformatio Theory Sprig 26 For iformatio about citig these materials or our Terms of Use, visit: