Optimum Probability Distribution for Minimum Redundancy of Source Coding

Transcription

1 Aled Mathematcs, 04, 5, Publshed Ole Jauary 04 (htt:// htt://dx.do.org/0.436/am Otmum Probablty strbuto for Mmum Redudacy of Source Codg Om Parkash, Pryaka Kakkar eartmet of Mathematcs, Guru Naak ev Uversty, Amrtsar, Ida Emal: Receved October 8, 03; revsed November 8, 03; acceted November 5, 03 Coyrght 04 Om Parkash, Pryaka Kakkar. Ths s a oe access artcle dstrbuted uder the Creatve Commos Attrbuto cese, whch ermts urestrcted use, dstrbuto, ad reroducto ay medum, rovded the orgal work s roerly cted. I accordace of the Creatve Commos Attrbuto cese all Coyrghts 04 are reserved for SCIRP ad the ower of the tellectual roerty Om Parkash, Pryaka Kakkar. All Coyrght 04 are guarded by law ad by SCIRP as a guarda. ABSTRACT I the reset commucato, we have obtaed the otmum robablty dstrbuto wth whch the messages should be delvered so that the average redudacy of the source s mmzed. Here, we have take the case of varous geeralzed mea codeword legths. Moreover, the uer boud to these codeword legths has bee foud for the case of Huffma ecodg. KEYWORS Mea Codeword egth; Uquely echerable Code; Kraft s Iequalty; Etroy; Otmum Probablty strbuto; Escort strbuto; Source Codg. Itroducto Ay message that brgs a secfcato a roblem whch volves a certa degree of ucertaty s called formato ad t was Shao [] who amed ths measure of formato as etroy. I codg theory, the oeratoal role of etroy comes from the source codg theorem whch states that f H s the etroy of the source letters for a dscrete memoryless source, the the sequece of source oututs caot be rereseted by a bary sequece usg fewer tha H bary dgts er source dgt o the average, but t ca be rereseted by a bary sequece usg as close to H bary dgts er source dgt o the average as desred. To be clearer, let us cosder the dscrete source S that emts symbols x, x,, x wth robablty dstrbuto,,,. The am of source codg s to ecode the source usg a alhabet of sze, that s, to ma each symbol x to a codeword c of legth l exressed usg the letters of the alhabet. It s kow that f the set of legths l satsfes Kraft s [] equalty l (.) the there exsts a uquely decodable code wth these legths, whch meas that ay sequece cc c ca be decoded uambguously to a sequece of symbols x x x. I ths resect, Shao [] roved the frst oseless codg theorem for the uquely decherable code the form of followg equalty H log s a Shao s etroy ad H H (.) l s the mea codeword legth. ater, Cambell [3] ad Kaur [4] roved the source codg theorems for ther ow exoetated mea co-

2 O. PARKASH, P. KAKKAR 97 deword legth the form of followg equaltes ad resectvely, s Cambell s [3] mea codeword legth, s Kaur s [4] mea codeword legth ad R R (.3) R R (.4) l log,, 0 l log,, 0 R log,, 0 s Rey s [5] measure of etroy. Recetly, Parkash ad Kakkar [6] troduced two mea codeword legths gve by ad l, log, 0, 0, l, 0 Further, the authors rovded two source codg theorems whch show that for all uquely decherable, satsfy the relato: codes, the mea codeword legths ad (.5) (.6) E, E (.7) ad K K (.8) resectvely E log s a Kaur s [4] two arameter addtve measure of etroy ad

3 98 O. PARKASH, P. KAKKAR K log, 0 s measure of etroy develoed by Parkash ad Kakkar [6]. Ths s to emhasze that the etre lterature of source codg theorems, oe ca observe that the mea codeword legth s lower bouded by the etroy of the source ad t ca ever be less tha the etroy of the source but ca be made closer to t. Ths heomeo rovdes the dea of absolute redudacy whch s the umber of bts used to trasmt a message mus the umber of bts of actual formato the message, that s, the mea codeword legth mus the etroy of the source. The obectve of the reset commucato s to mmze ths redudacy order to crease the effcecy of the source ecodg. For ths urose we have made use of the cocet of escort dstrbuto as follows: If,,, s the orgal dstrbuto, the ts escort dstrbuto s gve by P P, P,, P P for some arameter 0. May researchers cludg Harte [7], Bercher [8,9], Beck ad Schloegl [0] etc. used ths dstrbuto ther resectve fdgs. The am of the reset aer s to obta the otmum robablty dstrbuto wth whch the source should delver messages order to mmze the absolute redudacy. To obta our goal, we have take to cosderato the above metoed geeralzed mea codeword legths. Moreover, the uer boud to these codeword legths has bee foud for Huffma [4] ecodg.. Otmum Probablty strbuto to Mmze Absolute Redudacy et us assume that for dscrete source S that emts symbols x, x,, x wth robablty dstrbuto,,,, the codewords c havg legths l,,,,, have bee obtaed usg some ecodg rocedure o oseless chael. Further, we assume that etroy of the source s E ad average codeword legth s,. Sce from (.7), we have E,, therefore, the average redudacy of the source code s gve by P l f,,,, E log log fp, P,, P ad,,, l f P P P P. I order to mmze the average redudacy, we resort to the followg theorem: Theorem : The otmum robablty dstrbuto that mmzes the absolute redudacy f,,, of the source wth etroy E ad the mea codeword legth, by Proof: To mmze the redudacy, we eed to mmze (.) s the escort dstrbuto, gve l P,,,, (.) l

4 O. PARKASH, P. KAKKAR 99 subect to the costrat f,,, log fp, P,, P (.3) whch s equvalet to extremz-,,, s mmum or maxmum wll deed uo To rove ths, we frst of all, fd the extremum of log f P, P,, P g f P P P ad the use the fact that f,,, the value of arameter. So, order to extremze f P P P,,, 0 s agrage s multler. Now ettg 0,,,,, we get P Substtutg (.6) (.4), we get P (.4), we cosder the agraga gve by l P P P P P l (.5) l (.6) Substtutg (.7) (.6), we get the result (.). Now, P l l l l l P l (.7) (.8) We see that Also, 0 P for 0 ad So, f P, P,, P Thus, log f P, P,, P observg the fucto f 0 P for. 0, PP has mmum value for 0 ad maxmum for. has mmum value for 0 ad maxmum for ad cosequetly,,,,, we see that t has mmum value for 0,. Thus, the mmum value s gve by l f,,, log. (.9) m

5 00 O. PARKASH, P. KAKKAR Aga, the ecessary codto for the costructo of uquely decherable codes s gve by Therefore, from (.9), we have NOTE: It s to be oted that Huffma ecodg, we have Therefore, for ths case, (.) becomes l f,,, 0. m m (.0) f,,, 0 f the source s Huffma [] ecoded sce for the l. (.) l P,,,,. (.) Smlarly, f we cosder the codeword legth whch satsfes the relato K absolute redudacy of the source code ths case s gve by P log l,,,,,, g K g P P P ad g P, P,, P Pl Plog P., the the Theorem. The otmum robablty dstrbuto that mmzes the absolute redudacy g,,, of the source wth etroy K ad mea codeword legth s the escort dstrbuto, gve by l P,,,, (.3) l Proof: We wll fd the extremum of g whch s equvalet to extremzg g P P P subect to costrat,,,,,, P (.4) et us cosder the agraga gve by gp, P,, P P (.5) 0 s a agrage s multler. For a extremum, let 0,,,,, that s, P Usg (.4), we get l log,,,, log,,,, l e e (.6) l e,,,, (.7)

6 O. PARKASH, P. KAKKAR 0 ad Substtutg (.7) (.6), we get (.3). Also, l 0 l l P log e l PP 0, So, g P P P,,, reaches ts mmum value whe l P,,,, ad s gve by l m l g P, P,, P log that s, l g,,, log m Note: Aga ths case also, f the source s Huffma [] ecoded, the the robabltes are gve by l P,,,,. Next, we wll fd the uer boud o the codeword legths, ad whe the source s Huffma ecoded., satsfes the followg equalty Theorem 3. The exoetated codeword legth, log (.8) f the source s ecoded usg Huffma rocedure., ca be wrtte the followg form Proof: The exoetated codeword legth, log P l P (.9) P. Cosderg (.), (.9) becomes l h l l l l, log log,,, (.0) l l hl l l,,,. We eed to fd the extremum of, subect to costrat Huffma Procedure). l (as the source s ecoded usg

7 0 O. PARKASH, P. KAKKAR For ths urose, we frst of all, fd the extremum of log hl, l,, l l ad the use the fact that, hl, l,, rameter. So, we cosder the agraga gve by 0 s a agrage s multler l Put x,,,,, (.) becomes ettg A 0,,,,, we get x whch s equvalet to extremzg s mmum or maxmum deedg uo the value of a- l Ahl, l,, l (.) A x x x (.),,,, Now, x gves Usg (.3) (.), we get that s, l log,,,, Now, A x (.3) x,,,, x. We see that Also, A x 0 for 0 ad A 0 x A PP for. 0,. So, hl, l,, l has mmum value for 0 Therefore, log hl, l,, l has mmum value for 0 quetly, observg the exoetated mea codeword legth, 0,. Thus, the maxmum value s gve by ad maxmum for. ad maxmum for ad cose-, we see that t has maxmum value for max, log. Theorem 4. The mea codeword legth s uer bouded by log, that s, f the source s ecoded usg Huffma rocedure. log (.4)

8 O. PARKASH, P. KAKKAR 03 Proof: The exoetated codeword legth ca be wrtte the followg form l l (.5) Pl l as P. We eed to fd the extremum of subect to costrat Huffma Procedure). So, we cosder the agraga gve by l (as the source s ecoded usg l B (.6) 0 s a agrage s multler. ettg B 0,,,,, we get l l log e. (.7) Sce l, we have Substtute (.8) (.7), we get Now, Also, log. e l log,,,,. B l l log log e 0. (.8) So, the mea codeword legth B PP 0,. has maxmum value whe l log,,,,, ad s gve by max log. Note-I: For the case of Cambell s codeword legth, we have from (.3), R redudacy of the source code ths case s gve by l K,,, R log log KP, P,, P. So, the average P ad K P, P,, P P. l

9 04 O. PARKASH, P. KAKKAR The absolute redudacy the case of Cambell s [3] mea codeword legth s the same as case of exoetated mea codeword legth, develoed by Parkash ad Kakkar [6] as gve (.). Thus, we see that smlar results as roved theorem (.) ad theorem (.3) hold for Cambell s case also. Note-II: Absolute redudacy whe we use Kaur s[4] mea codeword legth s gve by l J,,,. J R J,,, log,,, Theorem 5: The otmum robablty dstrbuto that mmzes the absolute redudacy of the source wth R ad mea codeword legth s gve by etroy l,,,,. (.9) l Proof: To mmze the redudacy, we eed to mmze subect to the costrat J,,, log J,,, (.30) To rove ths, we frst of all fd the extremum of log J,,, ad the usg the fact that J J,,, value of arameter. So, order to extremze J 0 s agrage s multler.,,, ettg 0,,,,, we get Substtutg (.33) (.3), we get. (.3) whch s equvalet to extremzg,,, s mmum or maxmum deedg uo the, we cosder the agraga gve by l l. (.3) l. (.33) Substtutg (.34) (.33), we get the result (.9). l Now,. l l l l. (.34)

10 O. PARKASH, P. KAKKAR 05 We see that Also, 0 0,. for 0 ad 0 P So, J,,, Therefore, log J,,, cosequetly observg the fucto J for. has maxmum value for 0 ad mmum value for. has maxmum value for 0 ad mmum value for ad,,,, we see that t has mmum value for 0,. The mmum value s gve by l J,,, log. m Theorem 6. The Kaur s [8] mea codeword legth satsfes the followg equalty f the source s ecoded usg Huffma rocedure. Proof: Proceedg as Theorem.3, we ca rove the Theorem 6. Ackowledgemets log (.35) The authors are thakful to Coucl of Scetfc ad Idustral Research, New elh, for rovdg the facal assstace for the rearato of the mauscrt. REFERENCES [] C. E. Shao, A Mathematcal Theory of Commucato, Bell System Techcal Joural, Vol. 7, 948, (Part I) (Part II). []. G. Kraft, A evce for Quatzg Groug ad Codg Amltude Modulated Pulses, M.S. Thess, MIT, Cambrdge, 949. [3].. Cambell, A Codg Theorem ad Rey s Etroy, Iformato ad Cotrol, Vol. 8, No. 4, 965, [4] J. N. Kaur, Etroy ad Codg, Mathematcal Sceces Trust Socety (MSTS), New elh, 998. [5] A. Rey, O Measures of Etroy ad Iformato, Proceedgs of the Fourth Berkeley Symosum o Mathematcal Statstcs ad Probablty, Vol., 96, [6] O. Parkash ad P. Kakkar, evelomet of Two New Mea Codeword egths, Iformato Sceces, Vol. 07, 0, [7]. Harte, Multfractals: Theory ad Alcatos, Chama ad Hall, odo, 00. [8] J. F. Bercher, Source Codg wth Escort strbutos ad Rey Etroy Bouds, Physcs etters A, Vol. 373, No. 36, 009, [9] J. F. Bercher, Tsalls strbuto as a Stadard Maxmum Etroy Soluto wth Tal Costrat, Physcs etters A, Vol. 37, No. 35, 008, [0] C. Beck ad F. Schloegl, Thermodyamcs of Chaotc Systems, Cambrdge Uversty Press, Cambrdge, 993. []. A. Huffma, A Method for the Costructo of Mmum Redudacy Codes, Proceedgs of the Isttute of Rado Egeers, Vol. 40, No. 0, 95,