Optimum Binary-Constrained Homophonic Coding
|
|
- Darcy Douglas
- 6 years ago
- Views:
Transcription
1 Optimum Binary-Constrained Homophonic Coding Valdemar C. da Rocha Jr. and Cecilio Pimentel Communications Research Group - CODEC Department of Electronics and Systems, P.O. Box 7800 Federal University of Pernambuco Recife PE, BRAZIL Abstract This paper introduces an optimum homophonic coding scheme to handle the situation in which the symbols in each homophonic codeword are independent and identically distributed binary random variables obeying an arbitrary probability distribution. An algorithm is given for decomposing the source probabilities which produces, at each step, the expanded alphabet of least entropy. A proof is given that this procedure is optimum in the sense of minimising the overall redundancy. Introduction Homophonic coding [, 2] is a cryptographic technique for reducing the redundancy of a message to be enciphered at the cost of some plaintext expansion. This technique consists of a replacement (one-to-many) of each letter of the original message by a substitute or homophone, in a larger alphabet, to form the plaintext message that is then encrypted. Each homophone is represented (one-to-one) by a codeword, usually in such a manner as to produce uniformly distributed and statistically independent code symbols, and consequently making the cipher more secure by increasing the cipher unicity distance [4]. For simplicity, we consider only the homophonic coding of the output sequence U, U 2, U 3,... of a K-ary discrete memoryless source (DMS), but the theory presented can be applied to sources with memory simply by replacing the probability distribution for U i with the conditional probability distribution for U i given the observed values of U, U 2,..., U i. For a K-ary DMS, the homophonic coding problem reduces to the problem of such coding for a single K-ary random variable U. Let U be a random variable taking values in the finite set {u, u 2,..., u K }. We assume with no loss of essential generality that all K values of U have non-zero probability and that K 2. Let V be a random variable representing the homophones for U, taking values in the set {v, v 2,...}, which may be finite or countably infinite, which is characterized by the fact The authors acknowledge partial support of this research by the Brazilian National Council for Scientific and Technological Development (CNPq) under Grants 30424/77-9 and /96-0.
2 that for each j there is exactly one i such that P (V = v j U = u i ) 0. For D-ary variable-length homophonic encoding we associate to each homophone a distinct codeword referred as a homophonic codeword. Let X, X 2,..., X W denote a homophonic codeword, where X i is a D-ary random variable and where the length W of the codeword is in general also a random variable. It is required that the homophonic codewords be assigned in such a manner that X, X 2,..., X W is a prefix-free encoding of V, i.e., such that the homophonic codewords x, x 2,..., x w are all distinct and none is the prefix of another. If the components X, X 2,..., X W of the homophonic codewords are independent and uniformly distributed D-ary random variables, the homophonic coding is said to be perfect [2]. Homophonic coding was defined in [2] to be optimum if it is perfect and minimises the average length of a homophone, E[W ], over perfect homophonic codings [3, 5]. In 200 we presented at the VI International Symposium on Communication Theory & Applications [6] an algorithm to perform homophonic coding in which the symbols in each homophonic codeword are independent and identically distributed binary random variables obeying an arbitrary probability distribution Π 2 = {p, p}, where p /2. The importance of this algorithm comes from the fact that it satisfies the upper bound for optimum binary-constrained prefix-free homophonic coding which states that the entropy of a homophone exceeds the source entropy by less than h(p)/p bits [6], where h(.) is the binary entropy function. In the sequel we shall refer to this binary-constrained algorithm as the BCA-200 algorithm. The BCA-200 algorithm provides a solution to the problem posed by Knuth and Yao [7, p.427] on the generation of probability distributions using a biased coin and was brought to our attention by Julia Abrahams [8]. Our motivation for the present paper was the observation that there are cases, as illustrated in Section 2, Example, where we can perform homophonic coding more efficiently than by using the BCA-200 algorithm. In Section 2 we also prove two lemmas that show how to expand the source alphabet with least increase in the expanded alphabet entropy. In Section 3 we introduce an algorithm to perform optimum binary-constrained homophonic coding and prove that it is optimum in the sense of minimising the overall redundancy. We also present an example where both the BCA-200 algorithm and the algorithm proposed here produce identical results, and show that the upper bound h(p)/p, referred earlier, is tight. Finally, in Section 4 we present some concluding remarks. 2 Motivation and basic results We present next an example illustrating that in some cases it is possible to perform homophonic coding more efficiently than by using the BCA-200 algorithm. This was our main motivation to try to improve the BCA-200 algorithm. Example Let U be the K = 2 DMS with P U (u ) = 5/9 and P U (u 2 ) = 4/9. We consider the perfect binary-constrained homophonic coding of U when Π 2 = {2/3, /3} is the code alphabet probability distribution. Applying the BCA-200 algorithm [6] we obtain P U (u ) = 4/9 + 8/3 4+2i 2 i=0
3 P U (u 2 ) = /3 + 2/27 + 8/3 5+2i, which lead to an average homophonic codeword length E(W ) = 9/9 and to a homophonic coding redundancy E(W ) H(U) = =.2 bits. On the other hand, by trial and error we obtain i=0 P U (u ) = 2/9 + 3/9 P U (u 2 ) = 4/9, which lead to an average homophonic codeword length E(W ) = 5/3 and to a homophonic coding redundancy E(W ) H(U) = = bits, i.e., a redundancy representing 60% of the redundancy obtained with the BCA- 200 algorithm. The following two lemmas provide the basis for constructing the algorithm presented in Section 3 for performing optimum binary-constrained homophonic coding. Lemma Let U be a K-ary DMS with probability distribution P U = {P U (u ), P U (u 2 ),..., P U (u K )} and entropy H(U). The entropy H(V ) of the probability distribution P V = {P U (u ), P U (u 2 ),..., βp U (u i ), ( β)p U (u i ),... P U (u K )}, containing K + terms and obtained by expanding the symbol u i from U into two new symbols with probabilities βp U (u i ) and ( β)p U (u i ), respectively, where 0 < β, is given by H(V ) = H(U) + P U (u i )h(β), where h(β) denotes the binary entropy function. Proof: We prove this lemma by expanding and simplifying the expression for H(V ) as follows H(V ) = βp U (u i ) log βp U (u + ( β)p i) U (u i ) log ( β)p U (u + i) K j=,j i P U (u j ) log P U (u j ) = βp U (u i ) log β + βp U (u i ) log P U (u + ( β)p i) U (u i ) log ( β) + ( β)p U (u i ) log P U (u + K i) j=,j i P U (u j ) log P U (u j) = K j= P U (u j ) log P U (u + P j) U (u i )[β log β + ( β) log ( β) ] = H(U) + P U (u i )h(β). Lemma 2 Let U be a K-ary DMS with probability distribution P U = {P U (u ), P U (u 2 ),..., P U (u K )} and entropy H(P U (u ), P U (u 2 ),..., P U (u i ),..., P U (u j ),..., P U (u K )). Let P V = {P U (u ), P U (u 2 ),..., α, P U (u i ) α,..., P U (u K )} be the probability distribution obtained by expanding the symbol u i from U into two new symbols with probabilities α and P U (u i ) α, respectively, and let P V2 = {P U (u ), P U (u 2 ),..., α, P U (u j ) α,..., P U (u K )} be the probability distribution obtained by expanding the symbol u j from U into two new symbols with probabilities α and P U (u j ) α, respectively, i.e., both P V and P V2 contain K + terms each and 0 < α min{p U (u i ), P U (u j )}. The entropy H(V ), associated with P V, is greater than the entropy H(V 2 ), associated with P V2, if and only if P U (u i ) > P U (u j ). 3
4 Proof: Let = P U (u i ) + P U (u j ) α > 0. If P U (u i ) > P U (u j ) it follows that either P U (u i ) > P U (u i ) α > P U (u j ) > P U (u j ) α or P U (u i ) > P U (u j ) > P U (u i ) α > P U (u j ) α. Or, equivalently, P U (u i )/ > (P U (u i ) α)/ > P U (u j )/ > (P U (u j ) α)/ or P U (u i )/ > P U (u j )/ > (P U (u i ) α)/ > (P U (u j ) α)/. In either case it follows that h(p U (u i )/ ) < h(p U (u j )/ ), i.e., h(p U (u j )/ ) h(p U (u i )/ ) > 0. By subtracting h(p U (u i )/ ) from h(p U (u j )/ ) it follows that h(p U (u j )/ ) h(p U (u i )/ ) = [ (P U (u i ) α) log (P U (u j ) α) log P U (u i ) α P U (u i ) log P U (u j ) α + P U (u j ) log However, by subtracting H(V 2 ) from H(V ) it follows that P U (u i ) ] > 0. P U (u j ) H(V ) H(V 2 ) = (P U (u i ) α) log P U (u i ) α P U (u i ) log P U (u i ) (P U (u j ) α) log P U (u j ) α + P U (u j ) log P U (u j ) = [h(p U (u j )/ ) h(p U (u i )/ )] > 0. () By making α = β i P U (u i ) = β j P U (u j ) in () it follows that H(V ) H(V 2 ) = P U (u i )h(β i ) P U (u j )h(β j ) 0. (2) Corollary (To Lemmas and 2) Let u r be a source symbol for which P U (u r ) α 0 is a minimum, r {, 2,..., K}. To expand the source alphabet by one symbol as in Lemma 2, with the least increase in the resulting expanded alphabet entropy, we must replace the symbol u r by two symbols whose probabilities are α and P U (u r ) α, respectively. 3 An optimum algorithm In standard D-ary homophonic coding the designer benefits from the fact that a given symbol probability P U (u i ), 0 < P U (u i ) <, has an essentially unique base D decomposition. This follows because P U (u i ) either has a unique decomposition as an infinite sum of negative powers of D, or it has both a decomposition as a finite sum of distinct negative powers of D and a decomposition as an infinite sum of distinct negative powers of D in which the smallest term in the finite decomposition is expanded as an infinite sum of successive negative powers of D. For example, for D = 3, P U (u i ) = 4/9 can be decomposed as either P U (u i ) = /3 + /9 or as P U (u i ) = /3 + (/27) i=0 (2/3)i. Constrained homophonic coding unfortunately does not inherit the essentially unique probability decomposition property described above. This means that in order to split the source symbols into homophones we need to work with the whole set of source symbol probabilities, instead of working with only one 4
5 symbol probability at a time as for the D-ary case. We will describe next a way to perform binary-constrained homophonic coding in the form of an algorithm which we will refer as the BCA-2003 algorithm. 3. The BCA-2003 Algorithm Let Π 2 = {p, p}, p /2, be the probability distribution for the homophonic codeword symbols. For a given source, the BCA-2003 algorithm simultaneously finds the decomposition of each source symbol probability as a sum (finite or infinite) of terms p λ ( p) l λ, and the corresponding prefix-free homophonic code, where λ is the number of s and l λ is the number of zeroes of a homophonic codeword of length l. The homophones are selected as terminal nodes in the binary rooted tree with probabilities T. From any non-terminal node in this tree two branches emanate with probabilities p and p = p, respectively. The label of a path in T is represented by the sequence of zeroes and ones associated with the branches constituting the path. The probability of a path of length l in T, containing λ s and l λ zeroes, is p λ ( p) l λ. In particular, for computing the probability of a terminal node we consider the path extending from the root node to that terminal node. Let v(i, j) denote the j th homophone assigned to the source symbol u i, i K, j =, 2,.... Let α(i, j) denote the probability of v(i, j). Definition We define the symbol running sum γ m (i), associated with the symbol u i, i K, at the m th iteration of the BCA-2003 algorithm as γ m (i) = P U (u i ) j α(i, k), with γ m (i) = P U (u i ) for j = 0, where j denotes the number of homophones allocated to u i up to the m th iteration. Definition 2 We define the running sum set Γ m at the m th iteration of the algorithm as Γ m = {γ m (i) γ m (i) > 0, i K}, with Γ 0 = {P U (u ), P U (u 2 ),..., P U (u K )}. Let γ max = max γ m (i) Γ m, i K. At m = 0 we grow T from the root, starting with only two leaves. We will expand each terminal node in T, whose probability exceeds γ max, by the least number of branches sufficient to make the resulting extended terminal node probability less than or equal to γ max. We call the resulting tree the processed binary rooted tree with probabilities, T p. At the m th iteration, m, a homophone is assigned to a terminal node of the corresponding T p, in a manner that the unused terminal node with largest probability P m is assigned as a homophone to the symbol u r with minimum nonnegative value for the difference between its homophone running sum γ m (r) and P m, i.e., such that min i {γ m (i) P m (γ m (i) P m ) 0} = γ m (r) P m 0, i K. The algorithm consists of the following steps.. Let m = 0. Let γ 0 (i) = P U (u i ), i K. Let Γ 0 = {P U (u ), P U (u 2 ),..., P U (u K )}. k= 5
6 2. Determine γmax and produce the tree T p for the m th iteration by expanding each terminal node in the tree from the (m ) th iteration, m, whose probability exceeds γ max, by the least number of branches sufficient to make the resulting extended terminal node probability less than or equal to γ max. 3. Find the unused path E l of length l in T p whose probability is largest among unused paths, and denote this largest probability by P m. 4. If, for i K, min i {γ m (i) P m (γ m (i) P m ) 0} = γ m (r) P m 0, then we associate to u r the homophone (terminal node) v(r, j) and the binary homophonic codeword of length l, whose digits constitute the labeling of E l in T p. This implies α(r, j) = P m. Compute the symbol running sum γ m(r) after this decomposition and let Γ m = Γ m {γ m (r)}. If γ m(r) = 0 then let Γ m+ = Γ m. The decomposition of P U (u r ) is now complete and contains j homophones, and if Γ m+ = φ then END. Otherwise, i.e., if γ m(r) > 0, then let Γ m+ = Γ m {γ m(r)}. 5. Let m m Go to step 2. Example 2 Let U be the K = 3 DMS with P U = {53/8, 6/8, 4/27}. We consider the perfect binary-constrained homophonic coding of U when Π 2 = {2/3, /3} is the code alphabet probability distribution. Applying the BCA-200 algorithm [6] we obtain P U (u ) = 53/8 = 4/9 + 4/27 + 4/8 + 2/ /3 7+2i i=0 P U (u 2 ) = 6/8 = 4/27 + 4/8 P U (u 3 ) = 4/27 = /9 + 2/8 + 8/3 6+2i, which lead to an average homophonic codeword length E(W ) = 24/8 and to a homophonic coding redundancy E(W ) H(U) = =.372 bits. On the other hand, by using the BCA-2003 algorithm we obtain i=0 P U (u ) = 53/8 = 4/9 + /9 + 2/27 + 2/8 P U (u 2 ) = 6/8 = 4/27 + 4/8 P U (u 3 ) = 4/27, which lead to an average homophonic codeword length E(W ) = 68/27 and to a homophonic coding redundancy E(W ) H(U) = =.25 bits, i.e., a redundancy representing 9% of the redundancy obtained with the BCA-200 algorithm. Proposition Let U be a K-ary DMS with probability distribution P U = {P U (u ), P U (u 2 ),..., P U (u K )} and entropy H(U). Let Π 2 = {p, p} be the homophonic code alphabet probability distribution. The perfect binary-constrained homophonic coding of U in the manner described in the BCA-2003 algorithm minimises the redundancy E(W ) H(U) and is therefore optimum. 6
7 Proof: It is well known [9] that the homophone entropy H(V ) in perfect homophonic coding is related to the average homophonic codeword length E(W ) by the expression H(V ) = h(p)e(w ). Therefore, by minimising H(V ) we also minimise E(W ). At each step of the BCA-2003 algorithm the alphabet increases by one symbol and, by Lemma 2, it follows that the associated alphabet entropy increases by the least amount. This procedure is repeated until the decomposition of U into homophones V is complete. It follows that H(V ) has the minimum possible value for a given P U and the proposition follows. The tightness of the upper-bound H(V U) h(p)/p [6] is demonstrated in the following example. Example 3 Let U be the K = 2 DMS with P U (u ) = ( p) n = p n and P U (u 2 ) = ( p) n = p n. We consider the perfect binary-constrained homophonic coding of U when Π 2 = {p, p} is the code alphabet probability distribution. Applying the BCA-2003 algorithm we obtain the following probabilities for the homophones representing u : α(, ) = p, α(, 2) = pp, α(, 3) = p 2 p,..., α(, j) = p (j ) p,..., α(, n) = p (n ) p, and for u 2 we obtain a single homophone v(2, ) whose probability is α(2, ) = p n. It follows that H(V U = u ) = H(U)/( p n ) + h(p)/p, H(V U = u 2 ) = 0 and thus H(V U) = P U (u )H(V U = u ) + P U (u 2 )H(V U = u 2 ) = ( p n )H(V U = u ) = H(U) + ( p n ) h(p) p. (3) However, since H(V ) = H(U) + H(V U) = ( p n )h(p)/p, it follows from (3) that h(p) lim H(V U) = n p, because lim n H(U) = 0. This example shows that the upper bound h(p)/p is tight. We remark that both BCA-200 and BCA-2003 algorithms produce identical results in this example because at each step of either algorithm there is only one possibility for performing the source expansion, i.e., γ m () P m > 0 and γ m (2) P m < 0, for m n. For m = n + we have γ n+ () = 0 and γ n+ (2) P n+ = 0. 4 Conclusions We introduced an algorithm for performing optimum binary-constrained homophonic coding, which has the interesting characteristic of simultaneously performing the selection of homophone probabilities and associated homophonic codewords. Furthermore we derived properties of this algorithm which allowed us to prove its optimality for minimising the redundancy of binary-constrained homophonic coding. We remark that the BCA-2003 algorithm produces the same results as those obtained by trial and error in Example. 7
8 References [] Ch. G. Günther, A universal algorithm for homophonic coding, pp in Advances in Cryptology-Eurocrypt 88, Lecture Notes in Computer Science, No.330. Heidelberg and New York: Springer, 988. [2] H. N. Jendal, Y. J. B. Kuhn and J. L. Massey, An informationtheoretic approach to homophonic substitution, pp in Advances in Cryptology-Eurocrypt 89 (Eds. J.-J. Quisquater and J. Vandewalle), Lecture Notes in Computer Science, No.434. Heidelberg and New York: Springer, 990. [3] V. C. da Rocha Jr. and J. L. Massey, On the entropy bound for optimum homophonic substitution, Proc. IEEE International Symposium on Information Theory, Ulm, Germany, 29 June - 4 July, 997, p.93. [4] C. E. Shannon, Communication theory of secrecy systems, Bell System Tech. J., vol. 28, pp , Oct., 949. [5] V. C. da Rocha Jr. and J. L. Massey, Better than optimum homophonic substitution, Proc. IEEE International Symposium on Information Theory. Sorrento, Italy, June, 2000, p. 24. [6] V.C. da Rocha Jr. and C. Pimentel, Binary-constrained homophonic coding, VI International Symposium on Communication Theory & Applications, 5-20 July 200, Ambleside, England, pp [7] D.W. Knuth and A.C. Yao, The complexity of random number generation, In J.F. Traub, editor,algorithms and Complexity: Recent Results and New Directions. Proceedings of the Symposium on New Directions and Recent Results in Algorithms and Complexity, Carnegie Mellon University, 976. Academic Press, New York, 976. [8] Julia Abrahams, Generation of Discrete Distributions form Biased Coins, IEEE Trans. Inform. Theory, vol. IT-42, pp , September 996. [9] J.L. Massey, Applied Digital Information Theory, Fach Nr G, 7. Semester, Class notes at the ETH Zurich, Chapter2, Wintersemester
Paper from European Trans. on Telecomm., Vol. 5, pp , July-August 1994.
Paper from European Trans. on Telecomm., Vol. 5, pp. 421-429, July-August 1994. Some Applications of Source Coding in Cryptography James L. Massey Signal and Information Processing Laboratory Swiss Federal
More informationA UNIVERSAL ALGORITHM FOR HOMOPHONIC CODING
A UNIVERSAL ALGORITHM FOR HOMOPHONIC CODING Christoph G. Gunther Asea Brown Boveri Corporate Research CH-5405 Baden, Switzerland ABSTRACT This contribution describes a coding technique which transforms
More informationand its Extension to Authenticity
EWSCS 06 almse, Estonia 5-10 March 2006 Lecture 1: Shannon s Theory of Secrecy and its Extension to Authenticity James L. Massey rof.-em. ETH Zürich, Adjunct rof., Lund Univ., Sweden, and Tech. Univ. of
More informationTight Upper Bounds on the Redundancy of Optimal Binary AIFV Codes
Tight Upper Bounds on the Redundancy of Optimal Binary AIFV Codes Weihua Hu Dept. of Mathematical Eng. Email: weihua96@gmail.com Hirosuke Yamamoto Dept. of Complexity Sci. and Eng. Email: Hirosuke@ieee.org
More informationChapter 3 Source Coding. 3.1 An Introduction to Source Coding 3.2 Optimal Source Codes 3.3 Shannon-Fano Code 3.4 Huffman Code
Chapter 3 Source Coding 3. An Introduction to Source Coding 3.2 Optimal Source Codes 3.3 Shannon-Fano Code 3.4 Huffman Code 3. An Introduction to Source Coding Entropy (in bits per symbol) implies in average
More informationInformation Theory with Applications, Math6397 Lecture Notes from September 30, 2014 taken by Ilknur Telkes
Information Theory with Applications, Math6397 Lecture Notes from September 3, 24 taken by Ilknur Telkes Last Time Kraft inequality (sep.or) prefix code Shannon Fano code Bound for average code-word length
More informationCorrecting Codes in Cryptography
EWSCS 06 Palmse, Estonia 5-10 March 2006 Lecture 2: Orthogonal Arrays and Error- Correcting Codes in Cryptography James L. Massey Prof.-em. ETH Zürich, Adjunct Prof., Lund Univ., Sweden, and Tech. Univ.
More informationAn Entropy Bound for Random Number Generation
244 An Entropy Bound for Random Number Generation Sung-il Pae, Hongik University, Seoul, Korea Summary Many computer applications use random numbers as an important computational resource, and they often
More information3F1: Signals and Systems INFORMATION THEORY Examples Paper Solutions
Engineering Tripos Part IIA THIRD YEAR 3F: Signals and Systems INFORMATION THEORY Examples Paper Solutions. Let the joint probability mass function of two binary random variables X and Y be given in the
More informationChapter 5: Data Compression
Chapter 5: Data Compression Definition. A source code C for a random variable X is a mapping from the range of X to the set of finite length strings of symbols from a D-ary alphabet. ˆX: source alphabet,
More informationEntropy as a measure of surprise
Entropy as a measure of surprise Lecture 5: Sam Roweis September 26, 25 What does information do? It removes uncertainty. Information Conveyed = Uncertainty Removed = Surprise Yielded. How should we quantify
More informationLecture 16. Error-free variable length schemes (contd.): Shannon-Fano-Elias code, Huffman code
Lecture 16 Agenda for the lecture Error-free variable length schemes (contd.): Shannon-Fano-Elias code, Huffman code Variable-length source codes with error 16.1 Error-free coding schemes 16.1.1 The Shannon-Fano-Elias
More informationCOMM901 Source Coding and Compression. Quiz 1
German University in Cairo - GUC Faculty of Information Engineering & Technology - IET Department of Communication Engineering Winter Semester 2013/2014 Students Name: Students ID: COMM901 Source Coding
More information3F1 Information Theory, Lecture 3
3F1 Information Theory, Lecture 3 Jossy Sayir Department of Engineering Michaelmas 2013, 29 November 2013 Memoryless Sources Arithmetic Coding Sources with Memory Markov Example 2 / 21 Encoding the output
More informationRun-length & Entropy Coding. Redundancy Removal. Sampling. Quantization. Perform inverse operations at the receiver EEE
General e Image Coder Structure Motion Video x(s 1,s 2,t) or x(s 1,s 2 ) Natural Image Sampling A form of data compression; usually lossless, but can be lossy Redundancy Removal Lossless compression: predictive
More informationCSCI 2570 Introduction to Nanocomputing
CSCI 2570 Introduction to Nanocomputing Information Theory John E Savage What is Information Theory Introduced by Claude Shannon. See Wikipedia Two foci: a) data compression and b) reliable communication
More informationLecture 3 : Algorithms for source coding. September 30, 2016
Lecture 3 : Algorithms for source coding September 30, 2016 Outline 1. Huffman code ; proof of optimality ; 2. Coding with intervals : Shannon-Fano-Elias code and Shannon code ; 3. Arithmetic coding. 1/39
More information1 Background on Information Theory
Review of the book Information Theory: Coding Theorems for Discrete Memoryless Systems by Imre Csiszár and János Körner Second Edition Cambridge University Press, 2011 ISBN:978-0-521-19681-9 Review by
More informationLecture 4 : Adaptive source coding algorithms
Lecture 4 : Adaptive source coding algorithms February 2, 28 Information Theory Outline 1. Motivation ; 2. adaptive Huffman encoding ; 3. Gallager and Knuth s method ; 4. Dictionary methods : Lempel-Ziv
More informationCoding of memoryless sources 1/35
Coding of memoryless sources 1/35 Outline 1. Morse coding ; 2. Definitions : encoding, encoding efficiency ; 3. fixed length codes, encoding integers ; 4. prefix condition ; 5. Kraft and Mac Millan theorems
More information3F1 Information Theory, Lecture 3
3F1 Information Theory, Lecture 3 Jossy Sayir Department of Engineering Michaelmas 2011, 28 November 2011 Memoryless Sources Arithmetic Coding Sources with Memory 2 / 19 Summary of last lecture Prefix-free
More information1 Introduction to information theory
1 Introduction to information theory 1.1 Introduction In this chapter we present some of the basic concepts of information theory. The situations we have in mind involve the exchange of information through
More informationUsing an innovative coding algorithm for data encryption
Using an innovative coding algorithm for data encryption Xiaoyu Ruan and Rajendra S. Katti Abstract This paper discusses the problem of using data compression for encryption. We first propose an algorithm
More informationSIGNAL COMPRESSION Lecture Shannon-Fano-Elias Codes and Arithmetic Coding
SIGNAL COMPRESSION Lecture 3 4.9.2007 Shannon-Fano-Elias Codes and Arithmetic Coding 1 Shannon-Fano-Elias Coding We discuss how to encode the symbols {a 1, a 2,..., a m }, knowing their probabilities,
More informationDecomposing Bent Functions
2004 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 49, NO. 8, AUGUST 2003 Decomposing Bent Functions Anne Canteaut and Pascale Charpin Abstract In a recent paper [1], it is shown that the restrictions
More informationThe simple ideal cipher system
The simple ideal cipher system Boris Ryabko February 19, 2001 1 Prof. and Head of Department of appl. math and cybernetics Siberian State University of Telecommunication and Computer Science Head of Laboratory
More informationLarge Deviations Performance of Knuth-Yao algorithm for Random Number Generation
Large Deviations Performance of Knuth-Yao algorithm for Random Number Generation Akisato KIMURA akisato@ss.titech.ac.jp Tomohiko UYEMATSU uematsu@ss.titech.ac.jp April 2, 999 No. AK-TR-999-02 Abstract
More informationIntro to Information Theory
Intro to Information Theory Math Circle February 11, 2018 1. Random variables Let us review discrete random variables and some notation. A random variable X takes value a A with probability P (a) 0. Here
More informationLecture 1: September 25, A quick reminder about random variables and convexity
Information and Coding Theory Autumn 207 Lecturer: Madhur Tulsiani Lecture : September 25, 207 Administrivia This course will cover some basic concepts in information and coding theory, and their applications
More informationAn Approximation Algorithm for Constructing Error Detecting Prefix Codes
An Approximation Algorithm for Constructing Error Detecting Prefix Codes Artur Alves Pessoa artur@producao.uff.br Production Engineering Department Universidade Federal Fluminense, Brazil September 2,
More information(Reprint of pp in Proc. 2nd Int. Workshop on Algebraic and Combinatorial coding Theory, Leningrad, Sept , 1990)
(Reprint of pp. 154-159 in Proc. 2nd Int. Workshop on Algebraic and Combinatorial coding Theory, Leningrad, Sept. 16-22, 1990) SYSTEMATICITY AND ROTATIONAL INVARIANCE OF CONVOLUTIONAL CODES OVER RINGS
More informationSIGNAL COMPRESSION Lecture 7. Variable to Fix Encoding
SIGNAL COMPRESSION Lecture 7 Variable to Fix Encoding 1. Tunstall codes 2. Petry codes 3. Generalized Tunstall codes for Markov sources (a presentation of the paper by I. Tabus, G. Korodi, J. Rissanen.
More informationInformation and Entropy
Information and Entropy Shannon s Separation Principle Source Coding Principles Entropy Variable Length Codes Huffman Codes Joint Sources Arithmetic Codes Adaptive Codes Thomas Wiegand: Digital Image Communication
More informationPERFECTLY secure key agreement has been studied recently
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 45, NO. 2, MARCH 1999 499 Unconditionally Secure Key Agreement the Intrinsic Conditional Information Ueli M. Maurer, Senior Member, IEEE, Stefan Wolf Abstract
More informationInformation Theory and Statistics Lecture 2: Source coding
Information Theory and Statistics Lecture 2: Source coding Łukasz Dębowski ldebowsk@ipipan.waw.pl Ph. D. Programme 2013/2014 Injections and codes Definition (injection) Function f is called an injection
More informationOn the redundancy of optimum fixed-to-variable length codes
On the redundancy of optimum fixed-to-variable length codes Peter R. Stubley' Bell-Northern Reserch Abstract There has been much interest in recent years in bounds on the redundancy of Huffman codes, given
More informationOn the exact bit error probability for Viterbi decoding of convolutional codes
On the exact bit error probability for Viterbi decoding of convolutional codes Irina E. Bocharova, Florian Hug, Rolf Johannesson, and Boris D. Kudryashov Dept. of Information Systems Dept. of Electrical
More informationChapter 9 Fundamental Limits in Information Theory
Chapter 9 Fundamental Limits in Information Theory Information Theory is the fundamental theory behind information manipulation, including data compression and data transmission. 9.1 Introduction o For
More information10-704: Information Processing and Learning Fall Lecture 10: Oct 3
0-704: Information Processing and Learning Fall 206 Lecturer: Aarti Singh Lecture 0: Oct 3 Note: These notes are based on scribed notes from Spring5 offering of this course. LaTeX template courtesy of
More informationUpper Bounds on the Capacity of Binary Intermittent Communication
Upper Bounds on the Capacity of Binary Intermittent Communication Mostafa Khoshnevisan and J. Nicholas Laneman Department of Electrical Engineering University of Notre Dame Notre Dame, Indiana 46556 Email:{mhoshne,
More informationTight Bounds on Minimum Maximum Pointwise Redundancy
Tight Bounds on Minimum Maximum Pointwise Redundancy Michael B. Baer vlnks Mountain View, CA 94041-2803, USA Email:.calbear@ 1eee.org Abstract This paper presents new lower and upper bounds for the optimal
More informationAn instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if. 2 l i. i=1
Kraft s inequality An instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if N 2 l i 1 Proof: Suppose that we have a tree code. Let l max = max{l 1,...,
More informationCryptography and Network Security Prof. D. Mukhopadhyay Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur
Cryptography and Network Security Prof. D. Mukhopadhyay Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Module No. # 01 Lecture No. # 08 Shannon s Theory (Contd.)
More informationA Weak Cipher that Generates the Symmetric Group
A Weak Cipher that Generates the Symmetric Group Sean Murphy Kenneth Paterson Peter Wild Information Security Group, Royal Holloway and Bedford New College, University of London, Egham, Surrey TW20 0EX,
More informationCryptography. Lecture 2: Perfect Secrecy and its Limitations. Gil Segev
Cryptography Lecture 2: Perfect Secrecy and its Limitations Gil Segev Last Week Symmetric-key encryption (KeyGen, Enc, Dec) Historical ciphers that are completely broken The basic principles of modern
More informationCommon Information. Abbas El Gamal. Stanford University. Viterbi Lecture, USC, April 2014
Common Information Abbas El Gamal Stanford University Viterbi Lecture, USC, April 2014 Andrew Viterbi s Fabulous Formula, IEEE Spectrum, 2010 El Gamal (Stanford University) Disclaimer Viterbi Lecture 2
More informationChapter 2 Date Compression: Source Coding. 2.1 An Introduction to Source Coding 2.2 Optimal Source Codes 2.3 Huffman Code
Chapter 2 Date Compression: Source Coding 2.1 An Introduction to Source Coding 2.2 Optimal Source Codes 2.3 Huffman Code 2.1 An Introduction to Source Coding Source coding can be seen as an efficient way
More informationChapter 2: Source coding
Chapter 2: meghdadi@ensil.unilim.fr University of Limoges Chapter 2: Entropy of Markov Source Chapter 2: Entropy of Markov Source Markov model for information sources Given the present, the future is independent
More informationFixed-to-Variable Length Distribution Matching
Fixed-to-Variable Length Distribution Matching Rana Ali Ajad and Georg Böcherer Institute for Counications Engineering Technische Universität München, Gerany Eail: raa2463@gail.co,georg.boecherer@tu.de
More informationUncertainity, Information, and Entropy
Uncertainity, Information, and Entropy Probabilistic experiment involves the observation of the output emitted by a discrete source during every unit of time. The source output is modeled as a discrete
More informationSmart Hill Climbing Finds Better Boolean Functions
Smart Hill Climbing Finds Better Boolean Functions William Millan, Andrew Clark and Ed Dawson Information Security Research Centre Queensland University of Technology GPO Box 2434, Brisbane, Queensland,
More informationSecurity in Locally Repairable Storage
1 Security in Locally Repairable Storage Abhishek Agarwal and Arya Mazumdar Abstract In this paper we extend the notion of locally repairable codes to secret sharing schemes. The main problem we consider
More informationData Compression. Limit of Information Compression. October, Examples of codes 1
Data Compression Limit of Information Compression Radu Trîmbiţaş October, 202 Outline Contents Eamples of codes 2 Kraft Inequality 4 2. Kraft Inequality............................ 4 2.2 Kraft inequality
More informationLecture 4 Channel Coding
Capacity and the Weak Converse Lecture 4 Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 15, 2014 1 / 16 I-Hsiang Wang NIT Lecture 4 Capacity
More informationOn the minimum neighborhood of independent sets in the n-cube
Matemática Contemporânea, Vol. 44, 1 10 c 2015, Sociedade Brasileira de Matemática On the minimum neighborhood of independent sets in the n-cube Moysés da S. Sampaio Júnior Fabiano de S. Oliveira Luérbio
More information(Preprint of paper to appear in Proc Intl. Symp. on Info. Th. and its Applications, Waikiki, Hawaii, Nov , 1990.)
(Preprint of paper to appear in Proc. 1990 Intl. Symp. on Info. Th. and its Applications, Waikiki, Hawaii, ov. 27-30, 1990.) CAUSALITY, FEEDBACK AD DIRECTED IFORMATIO James L. Massey Institute for Signal
More informationarxiv: v1 [cs.it] 5 Sep 2008
1 arxiv:0809.1043v1 [cs.it] 5 Sep 2008 On Unique Decodability Marco Dalai, Riccardo Leonardi Abstract In this paper we propose a revisitation of the topic of unique decodability and of some fundamental
More informationAsymptotic redundancy and prolixity
Asymptotic redundancy and prolixity Yuval Dagan, Yuval Filmus, and Shay Moran April 6, 2017 Abstract Gallager (1978) considered the worst-case redundancy of Huffman codes as the maximum probability tends
More informationNew Minimal Weight Representations for Left-to-Right Window Methods
New Minimal Weight Representations for Left-to-Right Window Methods James A. Muir 1 and Douglas R. Stinson 2 1 Department of Combinatorics and Optimization 2 School of Computer Science University of Waterloo
More informationIN this paper, we consider the capacity of sticky channels, a
72 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 54, NO. 1, JANUARY 2008 Capacity Bounds for Sticky Channels Michael Mitzenmacher, Member, IEEE Abstract The capacity of sticky channels, a subclass of insertion
More informationInformation Theory, Statistics, and Decision Trees
Information Theory, Statistics, and Decision Trees Léon Bottou COS 424 4/6/2010 Summary 1. Basic information theory. 2. Decision trees. 3. Information theory and statistics. Léon Bottou 2/31 COS 424 4/6/2010
More informationSuperposition Encoding and Partial Decoding Is Optimal for a Class of Z-interference Channels
Superposition Encoding and Partial Decoding Is Optimal for a Class of Z-interference Channels Nan Liu and Andrea Goldsmith Department of Electrical Engineering Stanford University, Stanford CA 94305 Email:
More informationHöst, Stefan; Johannesson, Rolf; Zigangirov, Kamil; Zyablov, Viktor V.
Active distances for convolutional codes Höst, Stefan; Johannesson, Rolf; Zigangirov, Kamil; Zyablov, Viktor V Published in: IEEE Transactions on Information Theory DOI: 101109/18749009 Published: 1999-01-01
More informationSource Coding. Master Universitario en Ingeniería de Telecomunicación. I. Santamaría Universidad de Cantabria
Source Coding Master Universitario en Ingeniería de Telecomunicación I. Santamaría Universidad de Cantabria Contents Introduction Asymptotic Equipartition Property Optimal Codes (Huffman Coding) Universal
More informationEntropy and Ergodic Theory Lecture 3: The meaning of entropy in information theory
Entropy and Ergodic Theory Lecture 3: The meaning of entropy in information theory 1 The intuitive meaning of entropy Modern information theory was born in Shannon s 1948 paper A Mathematical Theory of
More informationTurbo Compression. Andrej Rikovsky, Advisor: Pavol Hanus
Turbo Compression Andrej Rikovsky, Advisor: Pavol Hanus Abstract Turbo codes which performs very close to channel capacity in channel coding can be also used to obtain very efficient source coding schemes.
More informationPublic Key Cryptography
Public Key Cryptography Spotlight on Science J. Robert Buchanan Department of Mathematics 2011 What is Cryptography? cryptography: study of methods for sending messages in a form that only be understood
More informationLecture 4: Perfect Secrecy: Several Equivalent Formulations
Cryptology 18 th August 015 Lecture 4: Perfect Secrecy: Several Equivalent Formulations Instructor: Goutam Paul Scribe: Arka Rai Choudhuri 1 Notation We shall be using the following notation for this lecture,
More informationLecture 8 - Cryptography and Information Theory
Lecture 8 - Cryptography and Information Theory Jan Bouda FI MU April 22, 2010 Jan Bouda (FI MU) Lecture 8 - Cryptography and Information Theory April 22, 2010 1 / 25 Part I Cryptosystem Jan Bouda (FI
More informationBandwidth: Communicate large complex & highly detailed 3D models through lowbandwidth connection (e.g. VRML over the Internet)
Compression Motivation Bandwidth: Communicate large complex & highly detailed 3D models through lowbandwidth connection (e.g. VRML over the Internet) Storage: Store large & complex 3D models (e.g. 3D scanner
More informationSome results on the existence of t-all-or-nothing transforms over arbitrary alphabets
Some results on the existence of t-all-or-nothing transforms over arbitrary alphabets Navid Nasr Esfahani, Ian Goldberg and Douglas R. Stinson David R. Cheriton School of Computer Science University of
More informationarxiv: v2 [cs.ds] 28 Jan 2009
Minimax Trees in Linear Time Pawe l Gawrychowski 1 and Travis Gagie 2, arxiv:0812.2868v2 [cs.ds] 28 Jan 2009 1 Institute of Computer Science University of Wroclaw, Poland gawry1@gmail.com 2 Research Group
More informationLECTURE 15. Last time: Feedback channel: setting up the problem. Lecture outline. Joint source and channel coding theorem
LECTURE 15 Last time: Feedback channel: setting up the problem Perfect feedback Feedback capacity Data compression Lecture outline Joint source and channel coding theorem Converse Robustness Brain teaser
More informationA Simple Left-to-Right Algorithm for Minimal Weight Signed Radix-r Representations
A Simple Left-to-Right Algorithm for Minimal Weight Signed Radix-r Representations James A. Muir School of Computer Science Carleton University, Ottawa, Canada http://www.scs.carleton.ca/ jamuir 23 October
More informationMaximum Correlation Analysis of Nonlinear S-boxes in Stream Ciphers
Maximum Correlation Analysis of Nonlinear S-boxes in Stream Ciphers Muxiang Zhang 1 and Agnes Chan 2 1 GTE Laboratories Inc., 40 Sylvan Road LA0MS59, Waltham, MA 02451 mzhang@gte.com 2 College of Computer
More informationStatistical and Linear Independence of Binary Random Variables
Statistical and Linear Independence of Binary Random Variables Kaisa Nyberg Department of Computer Science, Aalto University School of Science, Finland kaisa.nyberg@aalto.fi October 10, 2017 Abstract.
More information4. Quantization and Data Compression. ECE 302 Spring 2012 Purdue University, School of ECE Prof. Ilya Pollak
4. Quantization and Data Compression ECE 32 Spring 22 Purdue University, School of ECE Prof. What is data compression? Reducing the file size without compromising the quality of the data stored in the
More informationRooted Trees with Probabilities Revisited
Rooted Trees with Probabilities Revisited Georg öcherer Institute for Communications Engineering Technische Universität München Email: georg.boecherer@tum.de arxiv:1302.0753v1 [cs.it] 4 Feb 2013 February
More informationEE229B - Final Project. Capacity-Approaching Low-Density Parity-Check Codes
EE229B - Final Project Capacity-Approaching Low-Density Parity-Check Codes Pierre Garrigues EECS department, UC Berkeley garrigue@eecs.berkeley.edu May 13, 2005 Abstract The class of low-density parity-check
More information1 Ex. 1 Verify that the function H(p 1,..., p n ) = k p k log 2 p k satisfies all 8 axioms on H.
Problem sheet Ex. Verify that the function H(p,..., p n ) = k p k log p k satisfies all 8 axioms on H. Ex. (Not to be handed in). looking at the notes). List as many of the 8 axioms as you can, (without
More informationShannon s Theory of Secrecy Systems
Shannon s Theory of Secrecy Systems See: C. E. Shannon, Communication Theory of Secrecy Systems, Bell Systems Technical Journal, Vol. 28, pp. 656 715, 1948. c Eli Biham - March 1, 2011 59 Shannon s Theory
More informationThe Method of Types and Its Application to Information Hiding
The Method of Types and Its Application to Information Hiding Pierre Moulin University of Illinois at Urbana-Champaign www.ifp.uiuc.edu/ moulin/talks/eusipco05-slides.pdf EUSIPCO Antalya, September 7,
More informationChannel combining and splitting for cutoff rate improvement
Channel combining and splitting for cutoff rate improvement Erdal Arıkan Electrical-Electronics Engineering Department Bilkent University, Ankara, 68, Turkey Email: arikan@eebilkentedutr arxiv:cs/5834v
More informationTransducers for bidirectional decoding of prefix codes
Transducers for bidirectional decoding of prefix codes Laura Giambruno a,1, Sabrina Mantaci a,1 a Dipartimento di Matematica ed Applicazioni - Università di Palermo - Italy Abstract We construct a transducer
More informationAn average case analysis of a dierential attack. on a class of SP-networks. Distributed Systems Technology Centre, and
An average case analysis of a dierential attack on a class of SP-networks Luke O'Connor Distributed Systems Technology Centre, and Information Security Research Center, QUT Brisbane, Australia Abstract
More informationOn the Cost of Worst-Case Coding Length Constraints
On the Cost of Worst-Case Coding Length Constraints Dror Baron and Andrew C. Singer Abstract We investigate the redundancy that arises from adding a worst-case length-constraint to uniquely decodable fixed
More informationUNIT I INFORMATION THEORY. I k log 2
UNIT I INFORMATION THEORY Claude Shannon 1916-2001 Creator of Information Theory, lays the foundation for implementing logic in digital circuits as part of his Masters Thesis! (1939) and published a paper
More informationTopics. Probability Theory. Perfect Secrecy. Information Theory
Topics Probability Theory Perfect Secrecy Information Theory Some Terms (P,C,K,E,D) Computational Security Computational effort required to break cryptosystem Provable Security Relative to another, difficult
More informationELEC546 Review of Information Theory
ELEC546 Review of Information Theory Vincent Lau 1/1/004 1 Review of Information Theory Entropy: Measure of uncertainty of a random variable X. The entropy of X, H(X), is given by: If X is a discrete random
More informationEECS 229A Spring 2007 * * (a) By stationarity and the chain rule for entropy, we have
EECS 229A Spring 2007 * * Solutions to Homework 3 1. Problem 4.11 on pg. 93 of the text. Stationary processes (a) By stationarity and the chain rule for entropy, we have H(X 0 ) + H(X n X 0 ) = H(X 0,
More informationConstant Weight Codes: An Approach Based on Knuth s Balancing Method
Constant Weight Codes: An Approach Based on Knuth s Balancing Method Vitaly Skachek 1, Coordinated Science Laboratory University of Illinois, Urbana-Champaign 1308 W. Main Street Urbana, IL 61801, USA
More informationAn Extended Fano s Inequality for the Finite Blocklength Coding
An Extended Fano s Inequality for the Finite Bloclength Coding Yunquan Dong, Pingyi Fan {dongyq8@mails,fpy@mail}.tsinghua.edu.cn Department of Electronic Engineering, Tsinghua University, Beijing, P.R.
More informationOn Scalable Coding in the Presence of Decoder Side Information
On Scalable Coding in the Presence of Decoder Side Information Emrah Akyol, Urbashi Mitra Dep. of Electrical Eng. USC, CA, US Email: {eakyol, ubli}@usc.edu Ertem Tuncel Dep. of Electrical Eng. UC Riverside,
More informationDigital communication system. Shannon s separation principle
Digital communication system Representation of the source signal by a stream of (binary) symbols Adaptation to the properties of the transmission channel information source source coder channel coder modulation
More informationFIBONACCI NUMBERS AND DECIMATION OF BINARY SEQUENCES
FIBONACCI NUMBERS AND DECIMATION OF BINARY SEQUENCES Jovan Dj. Golić Security Innovation, Telecom Italia Via Reiss Romoli 274, 10148 Turin, Italy (Submitted August 2004-Final Revision April 200) ABSTRACT
More informationA Simple Left-to-Right Algorithm for Minimal Weight Signed Radix-r Representations
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. XX, NO. X, MONTH 2007 1 A Simple Left-to-Right Algorithm for Minimal Weight Signed Radix-r Representations James A. Muir Abstract We present a simple algorithm
More informationLecture 2: Perfect Secrecy and its Limitations
CS 4501-6501 Topics in Cryptography 26 Jan 2018 Lecture 2: Perfect Secrecy and its Limitations Lecturer: Mohammad Mahmoody Scribe: Mohammad Mahmoody 1 Introduction Last time, we informally defined encryption
More informationThe Poisson Channel with Side Information
The Poisson Channel with Side Information Shraga Bross School of Enginerring Bar-Ilan University, Israel brosss@macs.biu.ac.il Amos Lapidoth Ligong Wang Signal and Information Processing Laboratory ETH
More informationInformation-theoretic Secrecy A Cryptographic Perspective
Information-theoretic Secrecy A Cryptographic Perspective Stefano Tessaro UC Santa Barbara WCS 2017 April 30, 2017 based on joint works with M. Bellare and A. Vardy Cryptography Computational assumptions
More informationIntroducing an Analysis in Finite Fields
Introducing an Analysis in Finite Fields Hélio M de Oliveira and Ricardo M Campello de Souza Universidade Federal de Pernambuco UFPE CP78-57-97 - Recife - PE, Brazil E-mail: {hmo,ricardo}@ufpebr Abstract
More information