We are interested in the problem sending messages over a noisy channel. channel noise is behave nicely.

Transcription

1 Chapte 31 Shao s theoem By Saiel Ha-Peled, Novembe 28, Vesio: 0.1 This has bee a ovel about some people who wee puished etiely too much fo what they did. They wated to have a good time, but they wee like childe playig i the steet; they could see oe afte aothe of them beig killed - u ove, maimed, destoyed - but they cotiued to play ayhow. We eally all wee vey happy fo a while, sittig aoud ot toilig but just bullshittig ad playig, but it was fo such a teible bief time, ad the the puishmet was beyod belief; eve whe we could see it, we could ot believe it. A Scae Dakly, Philip K. Dick Codig: Shao s Theoem We ae iteested i the poblem sedig messages ove a oisy chael. chael oise is behave icely. We will assume that the Defiitio The iput to a biay symmetic chael with paamete p is a sequece of bits x 1, x 2,..., ad the output is a sequece of bits y 1, y 2,..., such that P[x i = y i ] = 1 p idepedetly fo each i. Taslatio: Evey bit tasmitted have the same pobability to be flipped by the chael. The questio is how much ifomatio ca we sed o the chael with this level of oise. Natually, a chael would have some capacity costaits (say, at most 4,000 bits pe secod ca be set o the chael, ad the questio is how to sed the lagest amout of ifomatio, so that the eceive ca ecove the oigial ifomatio set. Now, its impotat to ealize that hadlig oise is uavoidable i the eal wold. Futhemoe, thee ae tadeoffs betwee chael capacity ad oise levels (i.e., we might be able to sed cosideably moe bits o the chael but the pobability of flippig [i.e., p] might be much lage. I desigig a commuicatio potocol ove this chael, we eed to figue out whee is the optimal choice as fa as the amout of ifomatio set. Defiitio A (k, ecodig fuctio Ec : {0, 1} k {0, 1} takes as iput a sequece of k bits ad outputs a sequece of bits. A (k, decodig fuctio Dec : {0, 1} {0, 1} k takes as iput a sequece of bits ad outputs a sequece of k bits. Thus, the sede would use the ecodig fuctio to sed its message, ad the eceive would use the tasmitted stig (with the oise i it, to ecove the oigial message. Thus, the sede stats with a message with k bits, it blow it up to bits, usig the ecodig fuctio (to get some obustess 1 This wok is licesed ude the Ceative Commos Attibutio-Nocommecial 3.0 Licese. To view a copy of this licese, visit o sed a lette to Ceative Commos, 171 Secod Steet, Suite 300, Sa Facisco, Califoia, 94105, USA. 1

2 to oise, it sed it ove the (oisy chael to the eceive. The eceive takes the give (oisy message with bits, ad use the decodig fuctio to ecove the oigial k bits of the message. Natually, we would like k to be as lage as possible (fo a fixed, so that we ca sed as much ifomatio as possible o the chael. The followig celebated esult of Shao 2 i 1948 states exactly how much ifomatio ca be set o such a chael. Theoem (Shao s theoem. Fo a biay symmetic chael with paamete p < 1/2 ad fo ay costats δ, γ > 0, whee is sufficietly lage, the followig holds: (i Fo a k (1 H(p δ thee exists (k, ecodig ad decodig fuctios such that the pobability the eceive fails to obtai the coect message is at most γ fo evey possible k-bit iput messages. (ii Thee ae o (k, ecodig ad decodig fuctios with k (1 H(p+δ such that the pobability of decodig coectly is at least γ fo a k-bit iput message chose uifomly at adom Ituitio behid Shao s theoem Let assume the sedes has set a stig S = s 1 s 2... s. The eceive got a stig T = t 1 t 2... t, whee p = P[t i s i ], fo all i. I paticula, let U be the Hammig distace betwee S ad T; that is, U = i[ si t i ]. Ude ou assumptios E[U] = p, ad U is a biomial vaiable. By Cheoff iequality, we kow that U [ (1 δp, (1 + δp ] with high pobability, whee δ is some tiy costat. So lets assume this ideed happes. This meas that T is i a ig R ceteed at S, with ie adius (1 δp ad oute adius (1 + δp. This ig has (1+δp i=(1 δp ( i ( 2 (1 + δp α = 2 2 H((1+δp. Let us pick as may igs as possible i the hypecube so that they ae disjoit: R 1,..., R κ. If somehow magically, evey wod i the hypecube would be coveed, the we could use all the possible 2 codewods, the the umbe of igs κ we would pick would be at least κ 2 R H((1+δp 2(1 H((1+δp. I paticula, coside all possible stigs of legth k such that 2 k κ. We map the ith stig i {0, 1} k to the cete C i of the ith ig R i. Assumig that whe we sed C i, the eceive gets a stig i R i, the the decodig is easy - fid the ig R i cotaiig the eceived stig, take its cete stig C i, ad output the oigial stig it was mapped to. Now, obseve that as desied What is wog with the above? k = log κ = (1 H((1 + δp (1 H(p, The poblem is that we ca ot fid such a lage set of disjoit igs. The easo is that whe you pack igs (o balls you ae goig to have wasted spaces aoud. To ovecome this, we would allow igs to ovelap somewhat. That makes thigs cosideably moe ivolved. The details follow. 2 Claude Elwood Shao (Apil 30, Febuay 24, 2001, a Ameica electical egiee ad mathematicia, has bee called the fathe of ifomatio theoy. 2

3 31.2. Poof of Shao s theoem The poof is ot had, but equies some cae, ad we will beak it ito pats How to ecode ad decode efficietly The scheme Ou scheme would be simple. Pick k (1 H(p δ. Fo ay umbe i = 0,..., K = 2 k+1 1, adomly geeate a biay stig Y i made out of bits, each oe chose idepedetly ad uifomly. Let Y 0,...,Y K deote these code wods. Hee, we have K = 2 (1 H(p δ. Fo each of these codewods we will compute the pobability that if we sed this codewod, the eceive would fail. Let X 0,..., X K, whee K = 2 k 1, be the K codewods with the lowest pobability to fail. We assig these wods to the 2 k messages we eed to ecode i a abitay fashio. The decodig of a message w is doe by goig ove all the codewods, ad fidig all the codewods that ae i (Hammig distace i the age [p(1 ε, p(1 + ε] fom w. If thee is oly a sigle wod X i with this popety, we etu i as the decoded wod. Othewise, if thee ae o such wods o thee is moe tha oe wod, the decode stops ad epot a eo The poof Ituitio. Let S i be all the biay stigs (of legth such that if the eceive gets this wod, it would deciphe it to be i (hee ae still usig the exteded codewod Y 0,...,Y K. Note, that if we emove some codewods fom cosideatio, the set S i just iceases i size. Let W i be the pobability that X i was set, but it was ot decipheed coectly. Fomally, let deote the eceived wod. We have that W i = P[ eceived whe X i was set]. S i To boud this quatity, let (x, y deote the Hammig distace betwee the biay stigs x ad y. Clealy, if x was set the pobability that y was eceived is As such, we have w(x, y = p (x,y (1 p (x,y. P[ eceived whe X i was set] = w(x i,. Let S i, be a idicato vaiable which is 1 if S i. We have that W i = P[ eceived whe X i was set] = w(x i, = S i, w(x i,. S i S i The value of W i is a adom vaiable of ou choice of Y 0,...,Y K. As such, its atual to ask what is the expected value of W i. Coside the ig R( = { x } (1 εp (x, (1 + εp, whee ε > 0 is a small eough costat. Suppose, that the code wod Y i was set, ad was eceived. The decode etu i if Y i is the oly codewod that falls iside R(. 3

4 Lemma Give that Y i was set, ad was eceived ad futhemoe R(Y i, the the pobability of the decode failig, is τ = P S i R(Yi γ 8, whee γ is the paamete of Theoem Poof: The decode fails hee, oly if R( cotais some othe codewod Y j (j i i it. As such, τ = P S i R(Yi P [ Y j R(, fo ay j i ] P [ Y j R( ]. Now, we emid the eade that the Y j s ae geeated by pickig each bit adomly ad idepedetly, with pobability 1/2. As such, we have P [ Y j R( ] = (1+εp m=(1 εp ( m 2 2 (, (1 + εp sice (1 + εp < 1/2 (fo ε sufficietly small, ad as such the last biomial coefficiet i this summatio is the lagest. By Coollay (i, we have P [ Y j R( ] ( 2 (1 + εp 2 2H((1+εp = 2 (H((1+εp 1. As such, we have τ = P S i R(Yi P [ Y j R( ] K P[Y 1 R(] 2 k+1 2 (H((1+εp 1 ( 2 (1 H(p δ+1 2 (H((1+εp 1 2 H((1+εp H(p δ +1 sice k (1 H(p δ. Now, we choose ε to be a small eough costat, so that the quatity H((1 + εp H(p δ is equal to some (absolute egative (costat, say β, whee β > 0. The, τ 2 β+1, ad choosig lage eough, we ca make τ smalle tha γ/2, as desied. As such, we just poved that τ = P S i R(Yi γ 2. Lemma We have, that R(Y i w(y i, γ/8, whee γ is the paamete of Theoem Poof: This quatity, is the pobability of sedig Y i whe evey bit is flipped with pobability p, ad eceivig a stig such that moe tha εp bits whee flipped. But this quatity ca be bouded usig the Cheoff iequality. Let Z = (Y i,, ad obseve that E[Z] = p, ad it is the sum of idepedet idicato vaiables. As such R(Y i sice ε is a costat, ad fo sufficietly lage. w(y i, = P[ Z E[Z] > εp] 2 exp ( ε2 4 p 4 < γ 4,

5 Lemma Fo ay i, we have µ = E[W i ] γ/4, whee γ is the paamete of Theoem Poof: By lieaity of expectatios, we have [ ] [ ] µ = E[W i ] = E S i, w(y i, = E S i, w(y i, sice S i, is a idicato vaiable. Settig, τ = P µ = = R(Y i R(Y i R(Y i ] = E [S i, w(y i, = P[x S i ]w(y i,, P[x S i ]w(y i, + S i R(Yi R(Y i P x S i R(Yi w(y i, + τ w(y i, + R(Y i P[x S i ]w(y i, w(y i, τ + R(Y i R(Y i ad sice w(y i, = 1, we get P[x S i ]w(y i, w(y i, γ 4 + γ 4 = γ 2. Now, the eceive got (whe we set Y i, ad it would miss ecode it oly if (i is outside of R(Y i, o R( cotais some othe codewod Y j (j i i it. As such, τ = P S i R(Yi P [ Y j R(, fo ay j i ] P [ Y j R( ]. Now, we emid the eade that the Y j s ae geeated by pickig each bit adomly ad idepedetly, with pobability 1/2. As such, we have P [ Y j R( ] = (1+εp m=(1 εp ( m 2 2 (, (1 + εp sice (1 + εp < 1/2 (fo ε sufficietly small, ad as such the last biomial coefficiet i this summatio is the lagest. By Coollay (i, we have P [ Y j R( ] ( 2 (1 + εp 2 2H((1+εp = 2 (H((1+εp 1. As such, we have τ = P S i R(Yi P [ Y j R( ]. K P[Y 1 R(] 2 k+1 2 (H((1+εp 1 ( 2 (1 H(p δ+1+(h((1+εp 1 2 H((1+εp H(p δ sice k (1 H(p δ. Now, we choose ε to be a small eough costat, so that the quatity H((1 + εp H(p δ is egative (costat. The, choosig lage eough, we ca make τ smalle tha γ/2, as desied. As such, we just poved that τ = P S i R(Yi γ

6 I the followig, we eed the followig tivial (but supisigly deep obsevatio. Obsevatio Fo a adom vaiable X, if E[X] ψ, the thee exists a evet i the pobability space, that assigs X a value µ. This holds, sice E[X] is just the aveage of X ove the pobability space. As such, thee must be a evet i the uivese whee the value of X does ot exceed its aveage value. The above obsevatio is oe of the mai tools i a poweful techique to povig vaious claims i mathematics, kow as the pobabilistic method. Lemma Fo the codewods X 0,..., X K, the pobability of failue i ecoveig them whe sedig them ove the oisy chael is at most γ. Poof: We just poved that whe usig Y 0,...,Y K, the expected pobability of failue whe sedig Y i, is E[W i ] γ 2, whee K = 2 k+1 1. As such, the expected total pobability of failue is E K W i = K E[W i ] γ 2 2k+1 = γ2 k, by Lemma (hee we ae usig the facts that all the adom vaiables we have ae symmetic ad behave i the same way. As such, by Obsevatio , thee exist a choice of Y i s, such that K W i 2 k γ. Now, we use a simila agumet used i povig Makov s iequality. Ideed, the W i ae always positive, ad it ca ot be that 2 k of them have value lage tha γ, because i the summatio, we will get that K W i > 2 k γ. Which is a cotadictio. As such, thee ae 2 k codewods with failue pobability smalle tha γ. We set ou 2 k codewod to be these wods. Sice we picked oly a subset of the codewods fo ou code, the pobability of failue fo each codewod shiks, ad is at most γ. Lemma cocludes the poof of the costuctive pat of Shao s theoem Lowe boud o the message size We omit the poof of this pat. 6

7 31.3. Fom pevious lectues Lemma Suppose that q is itege i the age [0, ]. The 2H(q + 1 Coollay We have: ( q 2 H(q. Lemma ca be exteded to hadle o-itege values of q. This is staightfowad, ad we omit the easy details. (i q [0, 1/2] ( q 2 H(q. (ii q [1/2, 1] ( q 2 H(q. (iii q [1/2, 1] 2H(q +1 ( q. (iv q [0, 1/2] 2 H(q +1 ( q. Theoem Suppose that the value of a adom vaiable X is chose uifomly at adom fom the iteges {0,..., m 1}. The thee is a extactio fuctio fo X that outputs o aveage at least lg m 1 = H(X 1 idepedet ad ubiased bits Bibliogaphical Notes The pesetatio hee follows [MU05, Sec. 9.1-Sec 9.3]. Bibliogaphy [MU05] M. Mitzemache ad U. Upfal. Pobability ad Computig adomized algoithms ad pobabilistic aalysis. Cambidge,