Performance of Optimal Data Shaping Codes

Transcription

1 Perormance o Optma Data Shapng Codes Y Lu, Penge Huang, and Pau H. Sege Eectrca and Computer Engneerng Dept., Unersty o Caorna, San Dego, La Joa, CA 9093 U.S.A {y333, pehuang, psege}@ucsd.edu arx: [cs.it] 8 Apr 07 Abstract Data shapng s a codng technque that has been proposed to ncrease the etme o ash memory deces. Seera data shapng codes hae been descrbed n recent work, ncudng endurance codes [] and drect shapng codes or structured data [8], [4], [5]. In ths paper, we study normaton-theoretc propertes o a genera cass o data shapng codes and proe a separaton theorem statng that optma data shapng can be acheed by the concatenaton o optma ossess compresson wth optma endurance codng. We aso determne the expanson actor that mnmzes the tota wear cost. Fnay, we anayze the perormance o drect shapng codes and estabsh a condton or ther optmaty. I. INTRODUCTION NAND ash memory has become a ubqutous data storage technoogy. It uses rectanguar arrays, or bocks o oatnggate transstors (commony reerred to as ces) to store normaton. The ash memory ces graduay wear out wth repeated wrtng and erasng, reerred to as program/erase (P/E) cycng. The damage caused by P/E cycng s dependent on the programmed ce ee. For exampe, n SLC ash memory, each ce has two possbe ees, erased and programmed, represented by and 0, respectey. Storng n a ce causes ess damage, or wear, than storng 0. More generay, n mutee ash memores, the ce wear s an ncreasng uncton o the threshod otage assocated wth the programmed ce ees. Seera codng technques hae been proposed to ncrease the etme o ash memory dece by reducng the tota cost wear to the ash dece. Endurance codng, ntended or statstca shapng o random data, was proposed n []. The objecte s to reduce wear by conertng unconstraned data to encoded sequences wth a prescrbed dstrbuton o ce ees. For a gen ceee wear-cost mode and a speced target code rate, the optma dstrbuton o ees that mnmzes the aerage wearcost was determned anaytcay. Theoretcay acheabe gans or SLC and 4-ee MLC deces were aso computed. Adapte endurance codng (AEC) or structured source data, whch uses a concatenaton o ossess data compresson wth endurance codng, was aso proposed. The perormance o enumerate endurance codes that reduce the aerage racton o 0 symbos was eauated emprcay on SLC deces, as we as on MLC deces (apped to both pages). Resuts o a dece-ee smuaton o AEC on SLC ash were aso presented. A reazaton o endurance codng was presented n [3]. In [8], ow-compexty, rate-, drect shapng codes or structured data were proposed or use on SLC ash memory. Durng encodng, a rate- mappng dctonary s adaptey generated to ecenty transorm a structured data sequence drecty nto a sequence that nduces ess ce wear. Ths code constructon was extended to an MLC drect shapng code n [4], [5]. Computer smuaton was used to eauate the perormance o a drect shapng code on Engsh anguage text. Error propagaton propertes o drect shapng codes were aso anayzed. In ths paper, we study the perormance o a genera cass o data shapng codes. In Secton II, we use known propertes o word-aued sources to determne the symbo occurrence probabty o shapng codeword sequences. In Secton III, we deeop a theoretca bound on the trade-o between the rate and the aerage wear cost o a shapng code. A consequence o the anayss s a separaton theorem, whch states that optma shapng can be acheed by a concatenaton o ossess compresson and endurance codng. The expanson actor that mnmze tota wear cost s aso determned. We then specaze the anayss to SLC drect shapng codes. We show that they are n genera sub-optma as rate- shapng codes, and prode necessary and sucent source condtons or ther optmaty. II. INFORMATION-THEORETIC PRELIMINARIES A. Basc Mode Frst, we x some notaton. Let X = X X..., where X X or a, be an..d source wth aphabet X = {α,...,α u }. We use X to denote the sze o the aphabet, x to denote the ength o sequence x and use P(x ) to denote the probabty o any nte sequence x. Let Y = {β,..., β } be an aphabet and Y be the set o a nte sequences oer Y, ncudng the nu strng λ o ength 0. Each β s assocated wth a wear cost U. Wthout oss o generaty, we assume that U U... U. We use a cost ector U = [U, U,..., U ] to represent the wear cost assocated wth aphabet Y. A genera shapng code s dened as a prex-ree mappng φ : X q Y whch maps a ength-q data sequence x q to a arabe ength code sequence y. We use Y to denote the process φ(x q ), where X q s the ector process X q, Xq q+,.... The entropy rate o the process Y s H(Y) = m n H(Y Y... Y n ). () We denote the ength o a codeword φ(x q ) by L(φ(xq )) and the expected ength o codewords generated by ength-q source sequences s gen by E(L) = P(x q )L(φ(xq )). () x q X q The expanson actor s dened as the rato o the codeword expected ength to the ength o the nput source sequence, namey = E[L]/q. (3) n Remark. Endurance codes and drect shapng codes can be treated as speca cases o ths genera cass o shapng codes. Endurance codes are used when the source has a unorm

2 dstrbuton, wth entropy rate H(X) = og X. A ength-m drect shapng code s a shapng code wth q =, =, where both X and Y hae aphabet sze m. The par X and φ orm a word aued source, as dened n [7]. The oowng theorem, proed n [7], ges the entropy rate o the shapng code φ(x q ). Theorem. For a prex-ree shapng code Y = φ(x q ) such that H(X q ) < and E(L) <, the entropy rate o the encoder output satses H(Y) = H(Xq ) E(L) = qh(x) E(L). (4) B. Asymptotc Symbo Occurrence Probabty For smpcty, and wthout oss o generaty, we assume q =. The mappng s φ : X Y. Let y denote the rst symbos o φ(x). We assume the wear cost s ndependent and addte, so the wear cost o sequence y can be expressed as W(y ) = N (y )U (5) = where N (y ) stands or the number o occurrences o β n sequence y. The wear cost per code symbo s thereore N (y )U /. Let Q(y ) = Pr{Y = y } (6) denote the probabty dstrbuton o Y. The expected wear cost per symbo o a ength- shapng code sequence s W = = y Y Q(y )W(y )/ = Q(y )N (y )U /. y Y The asymptotc expected wear cost per symbo, or aerage wear cost o a data shapng code s A(φ(X)) = m W. (8) Let P = m y Q(y )N (y E(N (Y )/ = m )) be the asymptotc probabty o occurrence o symbo β. Ceary, P =. Then the aerage cost o a data shapng code can be expressed as (7) (9) A(φ(X)) = P U. (0) In the rest o ths subsecton, we w show how to cacuate P. Dene the prex operator π as y n π = y n or 0 < n and y n π = λ or n. Let π{y } denote the set o a the prexes o a sequence y. We denote by G φ (y ) the set o a sequences x X such that y s a prex o φ(x ) but not o φ(x π). That s, G φ (y ) = {x X y π{φ(x )} φ(x π) < } () and the dstrbuton o y can be expressed as Q(y ) = P(x ). () x G φ (y ) We dene by M the mnmum ength o sequence x n such that φ(x n ) and et S M be the ength o φ(x M ). Note that S M < S M. (3) Accordng to [7], the random arabe M satses the property o beng a stoppng rue or the sequence o..d. random arabe {φ(x )}. Wad s equaty [9] then mpes that E(N (φ(x M ))) = E(N (φ(x)))e(m ). (4) Remark. Gen a nonnegate-aued uncton, et F = (X ) and assume E(F) <. We shoud pont out that E(F M ) s not aways equa to E(F). For detas, see [7]. The oowng two emmas were proed n [7]. Lemma. Gen a nonnegate-aued uncton, et F = (X ). I E(F) <, then E(F M ) m = 0. (5) Lemma 3. I E[L] <, then E[M m ] = E(L). (6) Usng these resuts, we dere a emma whch tes us how to cacuate the asymptotc occurrence probabty o encoder output process Y. Lemma 4. For a shapng code φ : X q Y, the occurrence probabty P o codeword sequence s gen by P = Pr(Ŷ = β ) = E(N (φ(x q ))) E(L). (7) Proo: See Appendx A. C. Lower Bound on Symbo Dstrbuton Entropy The probabty dstrbuton o symbos determnes the aerage wear cost o the shapng code. To nd the mnmum possbe aerage wear cost, we rst study some propertes o the dstrbuton {P }. For conenence, dene the random arabe Ŷ by Pr(Ŷ = β ) = P. (8) The oowng emma ges a ower bound on ts entropy, H(Ŷ). Lemma 5. The entropy H(Ŷ) s ower bounded by the entropy rate o the shapng code sequence,.e., Proo: The entropy rate o Y s and we hae H(Ŷ) H(Y). (9) H(Y) = m H(Y ) (0) H(Y ) H(Y ). () =

3 Random arabe Y has probabty dstrbuton Pr(Y = β j ) = Q(y ). () y :y =β j We use P j to denote Pr(Y = β j ). Then H(Y ) can be expressed as H(Y ) = P j og P j. (3) j= Snce the entropy uncton s a concae uncton, we hae, by Jensen s nequaty, = H(Y ) = = j= P j og P j j= ( P j ) og ( (4) P j ) = H(Y ), where the random arabe Y has probabty dstrbuton Notce that = P j = = Pr(Y = β j ) = y :y =β j Q(y ) = y P j. (5) = Q(y ). (6) :y =β j For a specc ength- sequence y, the quantty Q(y ) appears N j (y ) tmes n the summaton n (6). Thereore, P j = N j (y )Q(y ) = y = E(N j(y )). Ths mpes that Y conerges n dstrbuton to Ŷ and (7) m H(Y ) = H(Ŷ). (8) Combnng equatons (0), (), (4), and (8), we hae H(Y) = m H(Y ) m n H(Y ) m H(Y ) = H(Ŷ). = (9) Remark 3. From the proo, we can concude that H(Ŷ) equas H(Y), then the oowng two condtons must hod. (a) H(Y ) = = H(Y ) or any ;.e., the random arabes Y are mutuay ndependent. (b) = H(Y ) = H(Y ) or any ;.e., the random arabes Y are dentcay dstrbuted. These two condtons, n turn, mpy that Y Y... acts ke an..d sequence generated by Ŷ. Exampe. Consder the unormy dstrbuted source X and the shapng code dened by the mappng {00 000, 0 00, 0 0, }. The occurrence probabty o symbo 0 s /3 and H(Ŷ) = 3 og 3 3 og (30) 3 The entropy rate o the shapng code sequence s H(Y) = H(X ) E(L) = = (3).5 We see that H(Y) < H(Ŷ). III. OPTIMAL SHAPING CODES A. Cost Mnmzng Probabty Dstrbuton Gen a data shapng code φ : X q Y, assume that ater nq source symbos are encoded, the codeword sequence s φ(x nq ). As n (7), (8), (9), we dene the expected wear cost per source symbo, or tota wear cost o a data shapng code as T(φ(X q )) = E(N (φ(x nq )))U nq = E(L) q E(N (φ(x q )))U E(L) = E(N (φ(x q )))U q = P U. = (3) To ncrease the etme o a ash memory dece, we want to nd the data shapng code that can mnmze the tota wear cost. Ths s equaent to song the oowng optmzaton probem. mnmze P, P U = subject to H(Ŷ) H(Y) = H(X) P =. (33) In order to do that, we rst proe a ower bound on the mnmum aerage wear cost acheabe wth a xed expanson actor. Theorem 6. Gen the dstrbuton P o source words and a cost ector U, the aerage cost o a shapng code φ : X q Y wth expanson actor s ower bounded by P U, wth P = N µu, where N s a normazaton constant such that P = and µ s a poste constant such that P og P = H(X)/. Proo: From Theorem and Lemma 5, we see that, or a shapng code φ wth expanson actor, the oowng nequaty hods: H(Ŷ) H(Y) = qh(x) E(L) = H(X). (34) To cacuate the mnmum possbe aerage cost, we must soe the optmzaton probem. mnmze P subject to P U H(Ŷ) H(X) P =. (35)

4 We dde ths optmzaton probem nto two parts. Frst, we x H(Ŷ) and nd the optma symbo occurrence probabtes. Then we nd the optma H(Ŷ) to mnmze the aerage wear cost. The optmzaton probem then becomes mnmze H(Ŷ) subject to mnmze P P U H(Ŷ) H(X) P =. (36) I we x H(Ŷ), we can soe the optmzaton probem by usng the method o Lagrange mutpers. The souton s P = N µu (37) where N = µu s a normazaton constant and µ s a non-negate constant such that H(Ŷ) = P og P. (38) Note that µ = 0 and ony H(Ŷ) = og Y. For smpcty, et h denote H(Ŷ). Then µ and N are unctons o h, whch we denote by N de = N(h) and µ de = µ(h), respectey. Let C(h) = cost, gen that h H(X) Then U N(h) µ(h)u. From (38), we see that be the mnmum C(h) = h og N. (39) µ dc dh = µ( N n dn dh ) (h og dµ N) dh µ, (40) where, as s easy ered, dn dh = U µu dµ dh Combnng (40) and (4), we get dµ n = N n C. (4) dh dc dh = µ + µc + og N h dµ µ dh. (4) dc Snce µc + og N = h = H(Ŷ), we deduce that dh = µ > 0. Thereore, the mnmum cost s acheed when h = H(Ŷ) = H(X). Remark 4. I the source X has a unorm dstrbuton, then µ satses P og P = og X. Thus, we recoer the resut n [] characterzng endurance codes wth mnmum aerage wear cost. Remark 5. When the mnmum aerage wear cost s acheed, we hae H(Ŷ) = H(Y). Thus, the codeword sequence ooks ke an..d sequence generated by Ŷ (see Remark 3). Usng Theorem 6, we can cacuate the tota wear cost as a uncton o the expanson actor. Fg. shows the tota wear cost cure or a quaternary source and code aphabet, a unormy dstrbuted source X, and cost ector U = [,, 3, 4]. There s an optma aue o and correspondng mnmum tota wear cost. Fg.. Tota wear cost ersus or random source wth U = [,, 3, 4] B. Separaton Theorem We now proe a separaton theorem or data shapng codes. Ths resut shows that the ower bound on the aerage wear cost can be acheed by combnng optma ossess compresson wth optma endurance codng. From ths resut and Theorem 6, we w then determne the expanson actor that achees the mnmum tota wear cost. Theorem 7. For a gen expanson actor, the mnmum aerage wear cost can be acheed by a concatenaton o an optma ossess compresson code wth an optma endurance code. Proo: Fx the expanson actor. Consder a source X wth aphabet X. Accordng to Shannon, the optma asymptotc compresson rate g s gen by g = H(X) og X. (43) From [7] we know that the probabty dstrbuton o the compressed sequences tends to a unorm dstrbuton. I we now appy endurance codng to the compressed sequence, the expanson actor o the endurance code s = g = og X. (44) H(X) Mnmzaton o the aerage cost transates to song the oowng optmzaton probem: mnmze P P U subject to H(Ŷ) = og X P =. = H(X) (45) Comparng (45) to (36), we concude that the mnmum cost can be acheed by a shapng code that rst appes optma ossess compresson to the source data, oowed by optma endurance codng. We now use the separaton theorem to determne the mnma acheabe tota wear cost o a data shapng code.

5 Theorem 8. Let P be the source dstrbuton and et U be a cost ector. I U = 0, then the mnmum tota wear cost o a shapng code φ : X q Y s gen by P U, where P = µu, µ s a poste constant seected such that µu =, and the expanson actor s H(X) = P og. (46) P I U = 0, then the tota cost s a decreasng uncton o. Proo: The proo s summarzed beow. See Appendx B or detaed cacuatons. The separaton theorem mpes that, or a source X, we can rst appy an optma ossess compresson to get a unorm source X. Now we consder the optma when endurance codng s apped to X. The tota wear cost s (P U ). To mnmze ths, we must soe the optmzaton probem: mnmze P, P U subject to H(Ŷ) = og X P =. (47) For xed, the symbo occurrence probabty correspondng to the mnmum aerage cost s where µ s a constant such that P = N µu, (48) = P og P og X and N s the normazaton actor (49) N = µu. (50) Treatng as a uncton o µ, and cacuatng d /dµ, we see that (µ) s a monotone ncreasng uncton o µ, wth (0) =. The optmzaton probem s then equaent to: mnmze F(µ) = og X U µu µu µu + N og N subject to µ 0. (5) Cacuatng df/dµ, we see that ts sgn s the negate o the sgn o og N. I U = 0, then N = µu >, mpyng that F(µ) s a monotone decreasng uncton on [0, ). I U > 0, then when µ = 0, we hae N = Y, so og N > 0. Snce dn/dµ < 0 or a µ, we concude that F w decrease as µ ncreases, reachng a mnmum at N =. Beyond that pont, dµ df > 0. Thus, the correspondng expanson actor that achees the mnmum tota cost s og = X P og P (5) where P = µu, and µ s a poste constant satsyng µu =. The expanson actor o the optma shapng code s thereore = H(X) og X = H(X) P og. (53) P A smar but weaker erson o Theorem 8 can be ound n []. Remark 6. I we ony appy optma ossess compresson to the source X, the code sequence has a unorm dstrbuton. Thereore, we hae µ = 0 and N = Y >. Ths mpes that smpy appyng compresson to the source data s not the best way to reduce the tota wear cost. C. Perormance o Drect Shapng Codes Drect shapng codes [8] hae parameters q =, =. Suppose the nput X and output Y both hae aphabet sze m, and the codewords hae cost ector U. The oowng theorem characterzes the mnmum aerage wear cost. Theorem 9. Gen the dstrbuton P and cost ector U, the mnmum aerage wear cost acheabe wth an SLC drect shapng code s P U. Proo: The proo oows rom the strong aw o arge numbers. Detas are gen n Appendx C. In genera, a drect shapng code w be suboptma or expanson actor =. From Theorem 6, we can determne the exact optmaty condtons. Coroary 0 SLC drect shapng codes are optma wth respect to aerage wear cost and ony where µ s a constant such that P =. P = µu, (54) IV. CONCLUSION In ths paper, we studed normaton-theoretc propertes and perormance mts o a genera cass o data shapng codes. We determned the asymptotc symbo occurrence probabty dstrbuton, and used t to determne the mnmum acheabe aerage wear cost or a gen expanson actor. We then proed a separaton theorem statng that optma data shapng can be acheed by a concatenaton o optma ossess compresson and optma endurance codng. Usng these resuts, we determned the mnmum tota wear cost and optma expanson actor or a data shapng code. As an appcaton, we showed that drect shapng codes are n genera suboptma, and gae the condtons under whch optmaty can be acheed. ACKNOWLEDGMENT Ths work was supported by NSF Grant CCF

6 REFERENCES [] I. Csszár and J. Körner, Inormaton Theory: Codng Theorems or Dscrete Memoryess Systems. Cambrdge Unersty Press, 0. [] A. Jagmohan, M. Franceschn, L. A. Lastras-Montaño, and J. Kards, Adapte endurance codng or NAND ash, n Proc. IEEE Gobecom Workshops, Mam, FL, Dec. 6, 00, pp [3] J. L, K. Zhao, J. Ma and T. Zhang, True-damage-aware enumerate codng or mprong NAND ash memory endurance, IEEE Trans. Very Large Scae Integr. (VLSI) Syst., o. 3, no. 6, pp , Jun. 05. [4] Y. Lu and P. H. Sege, Shapng codes or structured data, 7th Annua Non-Voate Memores Workshop, La Joa, CA, Mar. 6 8, 06. [5] Y. Lu and P. H. Sege, Shapng codes or structured data, n Proc. IEEE Gobecom, Washngton, D.C., Dec. 4-8, 06, pp. 5. [6] Y. Lu, P. Huang and P. H. Sege. Perormance o optma data shapng codes, [7] N. Mkhko and H. Morta, On the AEP o word-aued sources, IEEE Trans. In. Theory, o. 46, no. 3, pp. 6 0, May 000. [8] E. Sharon, et a., Data shapng or mprong endurance and reabty n sub-0nm NAND, presented at Fash Memory Summt, Santa Cara, CA, Aug. 4 7, 04. [9] A. Wad, Sequenta Anayss. Courer Corporaton, 973. APPENDIX A PROOF OF LEMMA 4 Proo: In ths proo, we w assume q =. Frst, we eauate the expectaton o sequence o random arabe {N (φ(x M ))} =. Combnng emma 4 wth equaton (4), we hae m E(N (φ(x M Smary, we hae m E(N (φ(x M ))) ))) = m E(N (φ(x)))e(m ) = E(N (φ(x))) m E(M ) = E(N (φ(x))) E(L). = m E(N (φ(x M )) N (φ(x M ))) = E(N (φ(x))) = E(N (φ(x))) E(L), E(L) m E(N (φ(x M ))) (55) (56) where m E(N (φ(x M ))) = 0 oows rom emma. By denton, E(N (φ(x M ))) = M P(x )N (φ(x M )). (57) y x M G φ(y ) Snce S M >, we hae N (φ(x M )) N (y ) and E(N (y )) can be bounded as oows E(N (Y )) = N (y )Q(y ) y = N (y ) P(x y x M G φ(y ) M ) N (φ(x M )) P(x y x M G φ(y ) = y x M M ) M P(x )N (φ(x M )) G φ(y ) = E(N (φ(x M ))). (58) Smary, N (φ(x M )) N (φ(y )) and E(N (Y )) s ower bounded by E(N (Y )) = N (y ) M P(x ) y x M G φ(y ) M P(x )N (φ(x M )) y x M G φ(y ) = E(N (φ(x M ))). Equaton (58) and (59) mpy that m sup m n E(N (Y )) m n E(N (φ(x M ))) = E(N (φ(x))) E(L) E(N (Y )) m sup E(N (φ(x M ))) Thus we proe emma = E(N (φ(x))) E(L). (59) (60) (6) P(Ŷ = β ) = m E(N (Y )) = E(N (φ(x))) E(L). (6) APPENDIX B PROOF OF THEOREM 8 Proo: The separaton theorem mpes that, or a source X, we can rst appy an optma ossess compresson to get a unorm source X. Now we consder the optma when endurance codng s apped to X. The tota wear cost s (P U ). To mnmze ths, we must soe the optmzaton probem: mnmze P, P U subject to H(Ŷ) = og X P =. (63) For xed, the margna dstrbuton correspondng to mnmum aerage cost s P = N µu, (64)

7 where µ s a constant such that = P og P og X and N s the normazaton actor (65) N = µu. (66) Treatng as a uncton o µ, and cacuatng d /dµ, we can see that (µ) s a monotone ncreasng uncton o µ, wth (0) =. Speccay, repacng the P n (65) by (64) ges N og = X ( µu µu ) + N og N. (67) The derate o wth respect to µ s d dµ = µ n < j µ(u+uj) (U U j ) [( µu µu ) + N og N], (68) whch s easy seen to be poste or µ > 0. The optmzaton probem s then equaent to: mnmze F(µ) = og X U µu µu µu + N og N subject to µ 0 (69) Cacuatng df/dµ, we see that ts sgn s the negate o the sgn o og N. Speccay, we nd that df dµ = n og X og N N U µu ( U µu ) ( µu µu + N og N). Obsere that N =, j U µu ( U µu ) U µ(u +U j ) = (U + U j U U j ) µ(u +U j ) < j = (U U j ) µ(u +U j ) > 0. < j Thereore, (70) U U U U j µ(u +U j ) < j (7) df dµ = n og X og N < j(u U j ) µ(u+uj) ( µu µu + N og N). (7) The camed reatonshp between the sgn o df/dµ and the sgn o og N s then edent. It oows that U = 0, then N = µu >, mpyng that F(µ) s a monotone decreasng uncton on [0, ). On the other hand, U > 0, then when µ = 0, we hae N = Y, so og N > 0. Snce dn/dµ < 0 or a µ, we concude that F w decrease as µ ncreases, reachng a mnmum at N =. Beyond that pont, df dµ > 0. Thus, the correspondng expanson actor that achees the mnmum tota cost s og = X P og P (73) where P = µu, and µ s a poste constant satsyng µu =. The expanson actor o the optma shapng code s thereore = H(X) og X = H(X) P og. (74) P APPENDIX C PROOF OF THEOREM 9 Proo: We make use o the notaton and termnoogy o [5]. For any two symbos w and w j n the aphabet X, we assume wthout oss o generaty that P > P j. Consder a sequence o..d random arabes X j j,... wth P(X, X j = ) = P, P(X j = ) = P j and P(X j = 0) = P P j. The random arabe X j k corresponds to the change n dstance between symbo and j at tme k. The expected aue o X j s µ j = P P j. Dene random arabe S j t = t k= X j k, and note that S j t > 0 means n (t) > n j (t). By the strong aw o arge numbers, S j j a.s. t /t µ 0. (75) Ths means that, or any ɛ > 0, {S j t /t} 0 s wthn ɛ o P P j > 0 or a but ntey many aues o t, wth probabty. Ths means that, or any symbos > j, n (t) > n j (t) and, thereore, the dctonary s stabe, or a but ntey many aues o t, wth probabty. Snce we are cacuatng the aerage cost, ths means we ony need to consder the contrbutons at tmes when the dctonary s stabe. Now, consder a sequence o..d random arabes Y, Y,... wth P(Y = k ) = P k. The tota cost ater t steps s the random arabe W t = = t Y. By the strong aw o arge numbers, W t /t m k= P k U k a.s. 0 (76) So the aerage cost s m k= P k U k. It can be shown that ths concuson hods een when symbos w and w j hae equa probabtes,.e., when P = P j.