The Renyi Redundancy of Generalized Huffman Codes

Transcription

1 242 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 34, NO. 5, SEPTEMBER 988 The Renyi Redundancy of Genealized Huffan Codes ANSELM C. BLUMER, MEMBER, IEEE, AND ROBERT J. McELIECE, FELLOW, IEEE Abstact -If optiality is easued by aveage codewod length, Huffan's algoith gives optial codes, and the edundancy can be easued as the diffeence between the aveage codewod length and Shannon's entopy. If the objective function is eplaced by an exponentially weighted aveage, then a siple odification of Huffan's algoith gives optial codes. The edundancy can now be easued as the diffeence between this new aveage and Renyi's genealization of Shannon's entopy. By deceasing soe of the codewod lengths in a Shannon code, the uppe bound on the edundancy given in the standad poof of the noiseless souce coding theoe is ipoved. The lowe bound is ipoved by andoizing between codewod lengths, allowing linea pogaing techniques to be used on an intege pogaing poble. These bounds ae shown to be asyptotically equal, poviding a new poof of Kicevski's esults on the edundancy of Huffan codes. These esults ae genealized to the Renyi case and ae elated to Gallage's bound on the edundancy of Huffan codes. I N PREVIOUS WORK 96, Renyi [2] poposed that the Shannon entopy could be genealized to s+ ( ) H.( p) =-s-log p}l<s+l) ' s > 0, which appoaches the Shannon entopy ass~ o+. In 965, Capbell [] showed that just as the Shannon entopy is a lowe bound on the aveage codewod length of a uniquely decodable code, the Renyi entopy is a lowe bound on the exponentially weighted aveage codewod length Also, ~log ( P;2sl;)' s > 0. ( li -log L p) 5 ) ' = L P;l; s--+0+ S i= i= Manuscipt eceived Mach 8, 986. This wok was suppoted in pat by the Joint Sevices Electonics Poga unde Contact N C with the Univesity of Illinois, Ubana-Chapaign, and in pat by the National Science Foundation unde Gant IST to the Univesity of Denve, CO. This wok was patially pesented at the IEEE Intenational Syposia on Infoation Theoy, Santa Monica, CA, Januay 982, and Les Acs, Fance, June 982. It also foed pat of a dissetation subitted to the Depatent of Matheatics, Univesity of Illinois, Ubana-Chapaign, in patial fulfillent of the equieents fo the Ph.D. degee. A. C. Blue is with the Depatent of Copute Science, Tufts Univesity, Medfod, MA R. J. McEliece is with the Depatent of Electical Engineeing, 6-8 Califonia Institute of Technology, Pasadena, CA 925. IEEE Log Nube We define the Renyi edundancy of a code as _ ( Rs( p, I)= -log L P;2 5 ) ' - Hs(p). s i=l (Note: It will be assued that the code alphabet is binay, though genealization is not difficult. As a consequence, "log" will always ean the base 2 logaith; the natual logaith is denoted by "ln.") Hu [5], Hublet [6], and Pake [] have obseved that a siple genealization of Huffan's algoith solves the poble of finding a uniquely decodable code which iniizes R s ( p, i). In Huffan's algoith, each new node is assigned the weight P; +pi' whee P; and p ae the lowest weights on available nodes. In the genealized algoith, the new node is assigned the weight 2 5 (p; + p). Note that if s > 0 this diffes fo the usual Huffan algoith in that the oot will not have weight. To suaize, the genealized Huffan algoith finds the optial solution of the following nonlinea intege pogaing poble. Given p = (p, Pz,, P) with P; > 0, I:;"=lPi =, and s ~ 0, find i = (/, 2,..,_ l) with positive intege coponents to iniize Rs(P, I) subject to E 2-', ~L i= (KM) Call the value of this optial solution R s< p ). The constaint (KM), known as the Kaft-McMillan inequality, is a necessay and sufficient condition fo the existence of a uniquely decodable code with codewod lengths ;. Equality holds if setting I;= -log P; gives integal lengths. In any case, the inequality is satisfied by letting A code with these codewod lengths is known as a Shannon code. Fo s = 0 the existence of such a code shows [0] that the edundancy is in [0, ). In [], Capbell genealized this by choosing I= [log (. i!<s+ ))- - -log P ' J=; s+ ' which gives the following j88j $ IEEE

2 BLUMER AND MCELIECE: RENYI REDUNDANCY OF GENERALIZED HUFFMAN CODES 243 Theoe : 0 s Rs(P) < fo s ~ 0. Although Theoe shows that Shannon coding is always within bit of the optiu on the aveage, individual codewods can be uch longe than necessay. Fo exaple, when coding a two-lette alphabet with two codewods, each codewod should be bit. If one of the lettes has abitaily sall pobability, the Shannon codewod fo that lette is abitaily long. The eainde of this pape is devoted to showing how to get a bette uppe bound on the edundancy of optial (Huffan) codes by shotening these long codewods, and how to get a bette lowe bound by using the idea of andoizing codewods. Randoizing will not esult in codes which can be used in pactice, but it will enable us to obtain uch bette bounds on the edundancy. APPLICATIONS In 968, Jelinek [7] showed that coding with espect to the Renyi edundancy is useful when souce sybols ae poduced at a fixed ate and code sybols ae tansitted at a high fixed ate. In this case, the instantaneous ate at which code sybols ae poduced depends on the length of the cuent codewod. Fo long codewods, this ate will be highe than the aveage ate at which code sybols ae poduced. Excess code sybols ust be stoed tepoaily in a finite buffe. This buffe ay still oveflow if an unusually long sequence of low pobability souce sybols is encoded. This poble can be educed by shotening the lengths of the long codewods. Miniizing the lengths of the longest codewods esults in a code with unifo lengths (coesponding to s = oo ), which in ost cases will not have a good aveage ate. Jelinek [7] shows how to pick s to solve this poble and gives bounds on the pobability of buffe oveflow based on s. Siila consideations apply to the constuction of optial seach tees [8]. Each intenal node in such a tee coesponds to a decision ade duing the seach. An ite is found when a leaf is eached. The ites coespond to the souce sybols in Huffan coding, and Huffan's algoith constucts the tee which iniizes the aveage seach tie. The seach tie fo an ite is popotional to the path length fo the oot to the leaf coesponding to that ite, which is equivalent to the codewod length above. If thee is a equieent that seaches be copleted within a cetain tie afte they ae equested, then this tie liit coesponds to the buffe length above. Jelinek's analysis then shows that the genealized Huffan algoith can be used to educe the pobability that this liit is exceeded, and to obtain a bound on this pobability. Capbell [2] has shown that the lengths given by the genealized Huffan algoith aise in a natual way fo geoetic consideations when intepolating between the distibution p and the unifo distibution along cuves in a Rieannian geoety which coespond to staight lines in Euclidean geoety. Pake [] povides a list of othe possible applications and efeences. LOWER BOUND The 0 lowe bound on Rs(P) was obtained by elaxing the intege estiction on the I; and letting I;= f; =log c~/y<s+ll)- s: log P; Setting I;= f ~ gives a solution to (KM) and leads to the uppe bound Rs(P) <. Given the pobability distibution p and the paaete s, it will be useful to define a new pobability distibution p based on the optial (but not necessaily integal) lengths f;: p}/(s+l) L py(s+l) j~l Let t; denote the factional pat of - f; and let n; denote the diffeence n;=l;-ff;l =;-f;-t;. Thus n; + t; gives the discepancy between a solution I; and the optial solution ~. Using the fact that /J;2 =, (KM) ay be ewitten as an aveage with espect to the pobability distibution p: - I: 2- =- [fj;2,-t,=- [fj;2-n,-t,~o. If s > 0, the objective function ay also be ewitten as R,(p,l)=-log _ ( ) LP;2sl, ---log s+ ( ) LPV(s+l) s s =-log ( L p}/(s+l)pjf(s+l)2sl,2s(l,-l,) ) s -~log[( :. p}/(s+l))( :. p}/(s+l))s] t~l t~l Since ( Is) log x is a stictly inceasing function of x, this objective function ay be eplaced by If s = 0, we define v,(-p, i, n) = I: /J;2s(n,+t,)_. V 0 (p,i,n)=r 0 (p,l)= LP;i;- LP;log ;~ P; () (2) (3a) = L P;( I; -I~)= L p;(n; + t;), {3b)

3 244 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 34, NO.5, SEPTEMBER 988 since P; = P; in this case. We have tansfoed the oiginal intege pogaing poblet:n into the following. Poble : Given a pobability vecto p and s ~ 0, copute p and i by () and (2). Then find integes n; satisfying (KM) and iniizing T-~(p, i, n) given by (3). The optial value of this poga is 2sR,(p) if s > 0, and R 0 (p) if s = 0. Inequality (KM) ay be futhe ewitten as EP[l-2-N-T] ~ 0, whee P denotes expectation with espect to the pobability distibution p and T and N ae ando vaiables. T is defined so that P(T=t)= L P; i:,= and N is defined siilaly. We can also ewite _ { P(2s(N+T)], V,(p,t,n)= Efi[N+T], fo s > 0 fo s = 0 This notation suggests the following odification to the above poble. Poble 2: Given s ~ 0 and a ando vaiable T with values in [0, ) and discete pobability distibution p, find an intege-valued ando vaiable N satisfying and iniizing fi[2s(n+t)] o, if s=o, iniizing P[N + T]. Let L 5 ( p) be defined so that 2sL,(p) is the value of the inial solution to the above poble. If s = 0, let this value be L 0 ( p). Theoe 2: 0::::;; L 5 (p)::::;; Rs(P) fo s ~ 0. Poof: Evey feasible solution to Poble coesponds to a feasible solution to Poble 2, since T can be defined as above, and N can be defined by P(N=n,T=t)= Since this holds fo any feasible solution to Poble 2, taking s ties the logaith of both sides gives the left-hand inequality in the s > 0 case. UPPER BOUND The pevious section showed that coputing Ls( p) povides a lowe bound to R 5 (p). Since the value of any feasible solution to Poble povides an uppe bound to R 5 ( p), and Shannon coding povides a feasible solution with value exceeding the lowe bound by less than, we had lowe and uppe bounds showing that the edundancy is in [0, ). The feasible solution to Poble coesponding to Shannon coding is n = 0. The following algoith ipoves this solution by changing soe coponents of n to -. The value U,(p) of this solution will povide an ipoved uppe bound to the edundancy. As with the bound deived fo Shannon coding, the diffeence between the uppe and lowe bounds will be estiated. Algoith ) Given Ps and s ~ 0, copute p, i using () and (2) and c =-.L ft;2-l;_ i=l 2) Set n = 0. 3) Repeat the following pai of steps, in ode of deceasing t;, until step 3a causes C to becoe negative (step 3b is skipped when this happens); a) decease C by p)- b) eplace the coesponding n; by -. 4) Copute On the othe hand, not all feasible solutions to Poble 2 coespond to feasible solutions to Poble. Fo exaple, any N with 0 < P( N = n) < in P; fo soe n cannot coespond to a feasible solution to Poble. Whee thee ae coesponding feasible solutions, they have the sae value. It follows that 2sL,(ft)::::;; 2sR,(p) and L 0 ( p) ::::;; R 0 (p), poving the ight-hand inequality. Any feasible solution to Poble 2 satisfies P[2-N-T] ::::;;, so by Jensen's inequality Taking logaiths gives 2EP[-N-T] :S; P[2 -N-T] :S;. Efi(N + T] ~ 0 which poves the s = 0 case. Applying Jensen's inequality again and using this last inequality, if s > 0 if s = 0. Afte Step 3b, C is the value of the left side of (KM) fo the cuent solution. The algoith will stop in o fewe steps, since t; < fo all i, and so -2 LP;';=- Lft;2 - i=l i = <0. The eason that the algoith poceeds in ode of deceasing t; is the following. The change in (KM) esulting fo eplacing n; by n; - is - ft;2 -n,-l,, while the change in the objective function, V,(p,i,n) is- P;(-2-s)2s(n,+l,) (o if s = 0,- P;). Thus the coponent of n which gives the geatest decease in the objective function pe decease in (KM) is the coponent with the lagest value of n; + t;.

4 BLUMER AND MCELIECE: RENYI REDUNDANCY OF GENERALIZED HUFFMAN CODES 245 The following exaple, with s = 2, shows that this algaith will not always yield optial solutions to Poble. P; fi,, f, Optial/, 69 l Note that / 3 = 3 in the optial solution, which coesponds to n 3 =, so any algoith which can each an optial solution ust be able to conside positive values fo n;. The following algoith finds L 5 (p), as shown by Theoe 3 (to follow). Algoith 2 (2). ) Given s z 0 and p, copute p and i using () and 2) Find the sallest t E [0, ) satisfying the constaint o I: pi2-t, + I: p;2-t, :::;. 3) Find the lagest be [0, ) satisfying the constaint 4) If s = 0, let z( x) = x; othewise, let z( x) = 2sx. Let 5) Let W,(p), L - - s ( p ) - { -:; log ( W, ( p )), if s = 0 if s > 0. The only diffeence between this algoith and the pevious one is the exta andoization allowed when ti = t. Theoe 3: W,(p) is the iniu value of Poble 2. Poof" Let Pin= P(T= ti, N = n). Poble 2 can be viewed as a linea pogaing poble with the Pin as vaiables, as follows: Poble 2 (Restated): Given s z 0 and p, find p and i using () and (2). If s > 0, find Pin to iniize L~ Ln E zp;n2s(n+ t,) o, if s = 0, to iniize I::"~ LnEZPin(n + ti) subject to the constaints and L L Pin2-n-t, = nez The fist constaint is (KM) with equality holding. Inequality cannot hold in an optial solution, since soe Pin could then be educed by soe t: > 0 while Pi n- was inceased by t:. This esults in a eduction' of t:(l-2-s)2s(n+t,) in the objective function fo the s > 0 case, and a eduction of t: fo the s = 0 case. t: ust be chosen so that the incease t:2-n-t, in (KM) will not violate that inequality. The poof of the theoe now poceeds by constucting the dual poga and finding solutions to both the oiginal poga and the dual poga with the sae value. The dual poga is as follows. Find q 0, q,, q to axiize q 0 + L;"~ q;p;, subject to the constaints Fo any feasible solutions to these linea pogas, the following inequalities show that the value of the dual poga is at ost the value of the oiginal poga: qo + L q,pi = qo L L P;n2-n-t, + L qi L Pin i ~ n E Z = L L Pin(qo2-n-t, + qi) i ~ n E Z :::; L L Pin2s(n+t;)' i~ nez if s > 0 :::; L L Pin(n+t;), ifs=o. i ~ n E Z Thus, if feasible solutions with the sae value can be found fo both the oiginal and the dual pogas, this coon value ust be the optial value. Let t and b be chosen by Algoith 2, and let pi, bpi, Pin= (-b)pi, { 0, if ti < t, n = 0, o t; > t, n = - if ti = t, n = - if ti = t, n = 0 in all othe cases. By constuction of t and b, this is clealy a feasible solution to the oiginal poga with the appopiate value. If s > 0, let qo= -(-2-s)2(s+)t fo ti:::; t fot;zt.

5 246 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 34, NO. 5, SEPTEMBER 988 Siilaly, if s = 0, let qo= -2' q= ' ti- q 0 2- ', ( ti- qo2-t,, fo i s foti~ t. Note that, in both cases, the altenate expessions fo qi give the sae value when i =. It eains to be shown that these foulas give adissible solutions to the dual poga and that these solutions have the appopiate values. Fo s > 0, the value of the dual poga is qo+ I: Piqi+ I: Piqi+ I: Piqi t,<t t,=t t;>t t,<t t,=t +qo[- I: Pi2-' -(-b) I: pi2-' f; < ( t, =t -b I: P;2-t,_ I: Pi2-t,] t;=t tl>t The facto ultiplying q 0 above is zeo by choice of and b, leaving only the fist fou tes, which ae equal to ~(p), as desied. Siilaly, fo s = 0, the value of the dual poga is qo+ I: piqi+ 2: Piqi+ 2: Piqi t;<t t,=t l;>t +qo[- I: P;2-' -(-b) I: P;2-' ;< t;=t - b I: P;2-t,- I: P;2-t,]. l; = l t, > ( All that eains is to show that these solutions ae adissible. If s > 0, this equies showing that qi S 2s(n+t,) _ qo2 -n-t, = 2s(n+t,) + (- 2 -s)2st2 -(n+t,-t) fo i =, 2,, and all integes n. Call this last quantity gis(n), and let Agis(n) = gis(n +)- gis(n) = (2s -)2s(n+t,) _ {-2 -s)2s2 -(n+ l+t,-t) = 2s'(-2-s)[2s(n+l+t,-t) -2-(n+l+t,-t)]. Siilaly, fo s = 0 it is necessay to show that qi s n + t;- qo2-n-t, = n + ; +2-(n+t,-t) fo i =, 2,, and all integes n. Call this last quantity gi 0 ( n ), and let Ag; 0 ( n) = g;o(n + )- g; 0 (n) = -2 -(n+l +t,-)_ Fo ti < t, Agis(n) is negative fo n s - and positive fo n ~ 0, so the iniu value of gi,( n) is gi,(o) = qi. Fo ti ~ t, Agis(n) is negative fo n s -2 and nonnegative fo n ~ -, so in this case the iniu value of gis(n) is gi,( -) = qi again, as desied. Theoe 4: 0 s Ls(P) s Rs(P) s U,(p) <. Poof" The only pat that eains to be poved is the last inequality, which follows fo the fact that the uppe bound U,(jj) is an ipoveent ove that obtained fo Shannon coding. The following theoe bounds the diffeence [!, ( p) Ls( p). Theoe 5: 0 s Us(p)- Ls(P) < ax Pi fo s = 0 ::; i :s; nt (-2-s) < ax pi2st, fo s > 0. s ln2 l,;i,; Poof: In case ( s = 0), the diffeence between U 0 ( p ), which is the esult of Algoith, and L 0 ( p ), which is the esult of Algoith 2, is less than Pi fo the coponent which caused C to becoe negative in Algoith. In case 2 (s > 0), the diffeence between 2sU,(p) and J.V,(jj) = 2sL,(p) is less than (- 2 -s) p)s". fo the sae eason as in case. Theefoe, 2sU,{p) Us ( P)- Ls ( p) = -;log J.V, ( p) J.V,{jj)+(-2-s)pi2' < -log-----,--, s J.V,(p) s -log( + (- 2 -s) pi2st,) s since»',( p) = 2sL,(p) ~ 2 =. The conclusion now follows fo the fact that y = log ( + x) is convex, and theefoe lies below its tangent line at (0, 0). APPLICATION TO MEMORYLESS SOURCES One case whee the peceding theoe is paticulaly useful is fixed-to-vaiable-length (FV) coding of a eoyless souce (also known as block-to-vaiable-length o BV coding). Suppose that a eoyless souce has an output alphabet of size, with pobabilities p, p 2,, P with all Pi> 0 and 'f.~ pi =. If bi denotes the nube of ties that the ith lette occus in a block B fo this souce, then the pobability of this block is P(B)=TipY s( ax pif< i=l lst,; and so the pobabilities of the block appoach zeo unifoly as n ~ oo. The s = 0 case of the above theoe now applies, to show that U,(pn)- Ls(pn) ~ 0 as n ~ oo, whee

6 BLUMER AND MCELIECE: RENYI REDCNDANCY OF GENERALIZED HUFFMAN CODES 247 is the pobability distibution on blocks of length n fo this souce. The s > 0 case can be illustated by exaining the binay eoyless souce with p = p and P2 p fo 0 < p <. In this case P(B) Pk=pk(l-p k (-pl~p if the fist lette occus k ties in the block. Thee ae { Z) blocks having this pobability, so k/(s+l) (-p;<s+l) lpp ( ) (pl/(s+ll+( p)lj(s+l))" Lea : Pk* (/)+ O(p ) fo soe p E (0, ). Poof Let S = e 2 "if, then S, S 2, S 3,, s-l ae the oots of j(x) X- + X X+, SO f(si) {, 0, Pk* can be ewitten as P* k ifj=o(od) othewise "f ~t(s'-k)c)cp')'(i-p')"' ~0 n - =- I: I: su-k)j( ~ )( p')' (- p')" I 0 j~o - n ( ) - I: s-kj I: ~ (p'si)'(l p')"_, j~o ~0 To siplify this, let ~ Thus p' pl/(s+ ) pl/(s+) + (l- p )/(s+l) pk (- p')n( ~/p' (p')k( p'-k, which again appoaches zeo unifoly as n ~ oo. Fo a souce with oe output lettes, the only diffeence is that the ultinoial theoe ust be used instead of the binoial theoe to siplify the denoinato. Thus we have the following. Theoe 6: LsCpn)- U,( fi") ~ 0 as n ~ oo eoyless souce. fo any It is possible to copute an asyptotic foula fo the iniu edundancy R 5 (p ) of a binay eoyless souce by coputing L..("pn). By Theoe 6, this ust appoach the iniu edundancy of the souce. We will need the n + factional pats t k = {log Pi} { n log ( - p') + k log ( ~' p' ) }. Suppose that log( p'/( p')) is ational with denoinato when witten in lowest tes, so tk=tk' ifandonlyifk=k'(od) (the iational case will not be teated hee). In this case, steps 2-5 of Algoith 2 depend only on Pt I: Pk. I: (;,)cp')k'ct p'-k k:,.~tk k'=k() and not on the individual Pk. The following lea by Raus [8), shows that Pk* ~ / as n ~ oo. - =-+ :s-kj(p's +-p'(. ~ Let p ax.,,_ I p's i + - p'l, and the esult follows. Let t * < fi < < t;:, be the factional pats with duplications oitted. The values t and b coputed by Algoith 2 can be cobined into a single nube t* E (0, *] by choosing t* so that it ounds up to the index fo which tt t, and so that {- t*} =b. Let P,* be the total pobability ass at ti*, i.e., P/ L Pi. j: t = t * If g epesents the left half of the constaint in step 3 of Algoith 2, expessed in tes of Pi* tt, and t*, then g is a continuous function of these vaiables, and a stictly inceasing function of t*. Fo exaple, the following chat shows what happens as t* goes fo 5- E: to 5 + E:. The tes in g which change ae those ultiplying Ps* and P 6 *. Let c, c 2, c 3, and c 4 be the coefficients of P 5 *2~!, P 5 *2 - ~, P 6 *2- t, and P 6 *2 - l, espectively: t* b 5 ( ts* -< 0 5 ts* ,- t6* ( 0 - (, Siilaly, the objective function in step 4 is a continuous function of P *, tt, and t*. It follows that L 5 (pn) is a continuous function of the P *, and it follows fo the above lea that using Pk* = / will give a esult which is asyptotically tue.

7 248 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 34, NO.5, SEPTEMBER 988 Now suppose that ' = tt. In othe wods, ' = ~ { n log ( - p') + k log ( ~' p' ) } = -{nlog(l- p')}. Then t;* = ' +((i -)/) foi =,2,,. If t = t: = ' + ( (a - ) I ), the constaint in step 3 of Algoith 2 is (using P;* = lj) a- - L -2--(kj) + L -2--(kj) k=o k=a -l -l =- L 2--(kj) + _ L 2--(kj) k=o This is equivalent to so k=a 2 --(aj) 2((-a)/)- = = (l-2-/) < (2/_l) - a~ - 'lj- log{(2 / -)), a= - 'lj- log((2ll -)) leads to the best choice fo t. Let so that 8=-{ -l+'lj+log{(2 /_l))} a-8=l-'lj-og((2 /_l)). The next step is to find b, as in step 4 of Algoith 2: a-2 L l-b -2--(kj) ((a-)/) k =O If equality holds, then b ((a-l)j) + L -2--(kj) L 2 --(k/) k=a k=a 2((-a)/)- b ((a-)/) ~. (2 /_l) b = 2+((a-)/)---- = --- 2/- 2/-. Now, if s = 0, Ls('pn) can be evaluated diectly. Let a-2( k) l-b( a-l) Lo( pn) = L - ' ' + -- k=o +~('lj+ a-l-l)+ f,l ~('lj+~-l) k=a (-) b -a =j = 'lj ; 2/ [ + 8- 'lj- log ( (2ll -))] - =--log{(2 / -))+-.,-,-- 2 (2 /_l) :--;--;--..,- (2/ -) ' which agees with [9, eq. (6)). If s > 0, b sf+(a-)/)-l] + L -2S(+(k/)-) k=a a-2 = (-2-s) L -2S(+(k/)) k =0 - b + (-2-s) --2s[+((a-)/)] - + L -2s(+(k/)-) k=o 2s + (- 2 -s)-(- b )2s(a-)/ 2s-s 2s ')_si -l = ( 2 -s) 2s[+((a-)/)] [ + - b ] 2sj -l _ (-2-s) ( 2 f_2 8 ) / - 2sj -l s(2/ -) s whee the last equality follows fo the fact that a -l 8 = log{(2 / -))+ ' + --.

8 BLUMER AND MCELIECE: RENYI REDUNDANCY OF GENERALIZED HUFFMAN CODES 249 Thus ( 2 /-2 8 ) = -log(-2-s)+ -log + -- s S 2 / - 2sj log ( ( 2 ' - )). An inteesting application of the peceding esults is in the case of a binay eoyless souce with pobabilities such that p' = j(2 + ) fo soe intege > 0. In this case, log(p'/(- p')) =- is an intege, so = and all of the factional pats tk ae identical. Thus the lowe bounds can be calculated exactly, athe than appoxiately as in the analysis using Raus' lea. Plugging = into the foulas fo L 0 ("pn) and Ls(pn) obtained above gives and Also, and = { n log (- p')} = { n log ( 2 :: ) } fo s > 0. Since log(2/(2 + )) is iational, Konecke's theoe [4] says that the set of values taken on by 8 as n inceases is dense in (0, ), and ax L 0 ("pn) = +log loge -loge= o /) E (0,) a constant which appeas in the following theoe due to Gallage [3]. Theoe 7: Let p be the pobability of the ost likely lette fo a finite discete souce. The edundancy of the Huffan code fo this souce is at ost p + o. The above eaks povide a sequence of exaples showing that o is the sallest possible value fo this theoe. CONCLUSION The standad poof of the noiseless souce coding theoe shows that the edundancy is between 0 and. Capbell [] has genealized this to the Renyi edundancy. The uppe bound was obtained by the feasible solution known as Shannon coding. This pape has ipoved this bound by shotening soe of the codewods in the Shannon code while still etaining feasibility. The lowe bound was ipoved by andoizing between codewod lengths. This tansfoed the intege pogaing poble of iniizing the (exponentially weighted) aveage codewod length subject to the Kaft-McMillan inequality into a linea pogaing poble. The optial feasible solution to this poble povided a lowe bound but did not esult in an ipleentable code. In the case of eoyless souces, these bounds wee shown to appoach each othe asyptotically, poviding a new poof and genealization of Kicevski's esults [9]. In the case of a binay eoyless souce with s = 0, this povided a sequence of exaples showing that the constant in Gallage's theoe [3] is the best possible. REFERENCES [) L. L. Capbell, "A coding theoe and Renyi's entopy," Info. Cont., vol. 8, pp , 965. [2), "The elation between infoation theoy and the diffeential geoety appoach to statistics," Info. Sci., vol. 35, pp , 985. [3) R. G. Gallage, "Vaiations on a thee by Huffan," IEEE Tans. Info. Theoy, vol. IT-24, no. 6, pp , Nov [4) G. H. Hady and E. M. Wight, An Intoduction to the Theoy of Nubes, 4th ed. Oxfod, England: Claendon, 97. [5] T. C. Hu, D. J. Kleitan, and J. T. Taaki, "Binay tees optial unde vaious citeia," SIAM J. Appl. Math., vol. 37, no. 2, pp , Oct [6) P. A. Hublet, "Genealization of Huffan coding to iniize the pobability of buffe oveflow," IEEE Tans. Info. Theoy, vol. IT-27, no. 2, pp , Ma. 98. [7) F. Jelinek, "Buffe oveflow in vaiable length coding of fixed ate souces," IEEE Tans. Info. Theoy, vol. IT-4, pp , May 968. [8) D. E. Knuth, "The at of copute pogaing," in Fundaental Algoiths, vol. I, 2nd ed. Reading, MA: Addison-Wesley, 973. [9) R. E. Kicevski, "The block length necessay to obtain a given edundancy," Sou. Math. Dokl., vol. 7, no. 6, pp , 966. [0) R. J. McEliece, The Theoy of Infoation and Coding, 2nd pinting with evisions. Reading, MA: Addison-Wesley, 979. [) D. S. Pake, J., "Conditions fo the optiality of the Huffan algoith," SIAM J. Coput., vol. 9, no. 3, pp , Aug [2) A. Renyi, "On easues of entopy and infoation," in Poc. 4th Bek. Syp. Math. Stat. Pob., vol., 96, pp