Information Theory and Synthetic Steganography

Transcription

1 Information Theory and Synthetic Steganography CSM25 Secure Information Hiding Dr Hans Georg Schaathun University of Surrey Spring 2009 Week 8 Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 1 / 56

2 Learning Outcomes understand the relationship between steganography and established disciplines like communications, information theory, data compression, and coding theory. be familiar with at least one way of doing steganography by cover synthesis Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 2 / 56

3 Reading Core Reading Peter Wayner: Disappearing Cryptography Ch. 6-7 Core Reading Cox et al.: Appendix A Suggested Reading Lin & Costello: Error-Control Coding Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 3 / 56

4 Outline Communications essentials 1 Communications essentials Communications and Redundancy Anderson and Petitcolas 1999 Digital Communications Shannon Entropy Security Prediction 2 Compression 3 Miscellanea Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 4 / 56

5 Outline Communications essentials Communications and Redundancy 1 Communications essentials Communications and Redundancy Anderson and Petitcolas 1999 Digital Communications Shannon Entropy Security Prediction 2 Compression Huffmann Coding Huffmann Steganography 3 Miscellanea Synthesis by Grammar Redundancy in Images Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 5 / 56

6 Communications essentials Communications and Redundancy The communications problem Alice Bob m ˆm Bob s problem Estimate m, given (partly) random output ˆm from the channel How much (un)certainty does Bob have about m? Information theory and Shannon entropy. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 6 / 56

7 Communications essentials Communications and Redundancy The communications problem Alice Bob m Noisy channel ˆm Bob s problem Estimate m, given (partly) random output ˆm from the channel How much (un)certainty does Bob have about m? Information theory and Shannon entropy. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 6 / 56

8 Communications essentials Communications and Redundancy The communications problem Alice Bob m Noisy channel ˆm Bob s problem Estimate m, given (partly) random output ˆm from the channel How much (un)certainty does Bob have about m? Information theory and Shannon entropy. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 6 / 56

9 Communications essentials Communications and Redundancy The communications problem Alice Bob m Noisy channel ˆm Bob s problem Estimate m, given (partly) random output ˆm from the channel How much (un)certainty does Bob have about m? Information theory and Shannon entropy. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 6 / 56

10 Communications essentials The communications problem Communications and Redundancy Alice Bob m Noisy Encode c r Decode ˆm channel Bob s problem Estimate m, given (partly) random output ˆm from the channel How much (un)certainty does Bob have about m? Information theory and Shannon entropy. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 6 / 56

11 Communications essentials The communications problem Communications and Redundancy Alice Bob m Noisy Encode c r Decode ˆm channel Bob s problem Estimate m, given (partly) random output ˆm from the channel How much (un)certainty does Bob have about m? Information theory and Shannon entropy. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 6 / 56

12 Communications essentials Redundancy of English Communications and Redundancy Fact The English language is more than 50% redundant. Message destroyed on the channel Redundancy allows Bob to determine the original m. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 7 / 56

13 Communications essentials Redundancy of English Communications and Redundancy Fact The English language is more than 50% redundant. t** p*oce*s o**hid**g *ata**nsid* o*her**ata. For ex*****, a **xt f*le c**ld*** hid*** "in**de"****im*ge or***s**nd *ile* By look****at t*e im*g***or list***** to th**s**nd,*yo* w*u*d n*t *no**that***ere is *x*ra info******* *r*sent. from Message destroyed on the channel Redundancy allows Bob to determine the original m. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 7 / 56

14 Communications essentials Redundancy of English Communications and Redundancy Fact The English language is more than 50% redundant. t** p*oce*s o**hid**g *ata**nsid* o*her**ata. For ex*****, a **xt f*le c**ld*** hid*** "in**de"****im*ge or***s**nd *ile* By look****at t*e im*g***or list***** to th**s**nd,*yo* w*u*d n*t *no**that***ere is *x*ra info******* *r*sent. from Message destroyed on the channel Redundancy allows Bob to determine the original m. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 7 / 56

15 Communications essentials Redundancy of English Communications and Redundancy Fact The English language is more than 50% redundant. t*e p*oce*s o* hid**g *ata*insid* o*her*data. For ex*m***, a t*xt f*le c**ld*b* hidd** "ind*de" a**im*ge or*a*s*und *ile* By look**g*at t*e im*g*,*or list**in* to th* s**nd,*yo* w*uld n*t *no**that *here is *x*ra info*****on *r*sent. from Message destroyed on the channel Redundancy allows Bob to determine the original m. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 7 / 56

16 Communications essentials Redundancy of English Communications and Redundancy Fact The English language is more than 50% redundant. the process of hiding data inside other data. For example, a text file could be hidden "inside" an image or a sound file. By looking at the image, or listening to the sound, you would not know that there is extra information present. from Message destroyed on the channel Redundancy allows Bob to determine the original m. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 7 / 56

17 Communications essentials Communications and Redundancy Benefits of redundancy Cross-word puzzles Understand foreigners with imperfect pronounciation. How much would you understand of a lecture without redundancy? Hear in a noisy environment. Read bad hand writing How could I mark exam scripts without redundancy? Cryptanalysis? Steganalysis? Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 8 / 56

18 Communications essentials Communications and Redundancy Benefits of redundancy Cross-word puzzles Understand foreigners with imperfect pronounciation. How much would you understand of a lecture without redundancy? Hear in a noisy environment. Read bad hand writing How could I mark exam scripts without redundancy? Cryptanalysis? Steganalysis? Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 8 / 56

19 Communications essentials Communications and Redundancy Benefits of redundancy Cross-word puzzles Understand foreigners with imperfect pronounciation. How much would you understand of a lecture without redundancy? Hear in a noisy environment. Read bad hand writing How could I mark exam scripts without redundancy? Cryptanalysis? Steganalysis? Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 8 / 56

20 Communications essentials Communications and Redundancy Benefits of redundancy Cross-word puzzles Understand foreigners with imperfect pronounciation. How much would you understand of a lecture without redundancy? Hear in a noisy environment. Read bad hand writing How could I mark exam scripts without redundancy? Cryptanalysis? Steganalysis? Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 8 / 56

21 Communications essentials Communications and Redundancy Benefits of redundancy Cross-word puzzles Understand foreigners with imperfect pronounciation. How much would you understand of a lecture without redundancy? Hear in a noisy environment. Read bad hand writing How could I mark exam scripts without redundancy? Cryptanalysis? Steganalysis? Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 8 / 56

22 Communications essentials Communications and Redundancy Benefits of redundancy Cross-word puzzles Understand foreigners with imperfect pronounciation. How much would you understand of a lecture without redundancy? Hear in a noisy environment. Read bad hand writing How could I mark exam scripts without redundancy? Cryptanalysis? Steganalysis? Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 8 / 56

23 Communications essentials Communications and Redundancy Benefits of redundancy Cross-word puzzles Understand foreigners with imperfect pronounciation. How much would you understand of a lecture without redundancy? Hear in a noisy environment. Read bad hand writing How could I mark exam scripts without redundancy? Cryptanalysis? Steganalysis? Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 8 / 56

24 Communications essentials What if there were no redundancy? Communications and Redundancy No use for steganography! Any text would be meaningful, in particular, ciphertext would be meaningful Simple encryption would give a stegogramme indistinguishable from cover-text. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 9 / 56

25 Communications essentials What if there were no redundancy? Communications and Redundancy No use for steganography! Any text would be meaningful, in particular, ciphertext would be meaningful Simple encryption would give a stegogramme indistinguishable from cover-text. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 9 / 56

26 Communications essentials What if there were no redundancy? Communications and Redundancy No use for steganography! Any text would be meaningful, in particular, ciphertext would be meaningful Simple encryption would give a stegogramme indistinguishable from cover-text. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 9 / 56

27 Outline Communications essentials Anderson and Petitcolas Communications essentials Communications and Redundancy Anderson and Petitcolas 1999 Digital Communications Shannon Entropy Security Prediction 2 Compression Huffmann Coding Huffmann Steganography 3 Miscellanea Synthesis by Grammar Redundancy in Images Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 10 / 56

28 Perfect compression Communications essentials Anderson and Petitcolas 1999 Compression removes redundancy Minimises average string length (file size) Retaining information contents Decompression replaces the redundancy Recover original (loss-less compression) Perfect means no redundancy in compressed string Consequently all strings are used A(ny) random string can be decompressed... and yield sensible output Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 11 / 56

29 Perfect compression Communications essentials Anderson and Petitcolas 1999 Compression removes redundancy Minimises average string length (file size) Retaining information contents Decompression replaces the redundancy Recover original (loss-less compression) Perfect means no redundancy in compressed string Consequently all strings are used A(ny) random string can be decompressed... and yield sensible output Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 11 / 56

30 Communications essentials Anderson and Petitcolas 1999 Steganography by Perfect Compression Anderson and Petitcolas 1998 A perfect compression scheme. A secure cipher. Message Encryption adf!haj dgh a Decompress Key Once upon a time there was a red herring... Message Decrypt adf!haj dgh a Compress Steganography without data hiding. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 12 / 56

31 Communications essentials Anderson and Petitcolas 1999 Steganography by Perfect Compression Anderson and Petitcolas 1998 A perfect compression scheme. A secure cipher. Message Encryption adf!haj dgh a Decompress Key Once upon a time there was a red herring... Message Decrypt adf!haj dgh a Compress Steganography without data hiding. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 12 / 56

32 Outline Communications essentials Digital Communications 1 Communications essentials Communications and Redundancy Anderson and Petitcolas 1999 Digital Communications Shannon Entropy Security Prediction 2 Compression Huffmann Coding Huffmann Steganography 3 Miscellanea Synthesis by Grammar Redundancy in Images Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 13 / 56

33 Communications essentials Digital Communications Problems in natural language How efficient is the redundancy Natural languages are arbitrary Some words/sentences have a lot of redundancy Others have very little Unstructured: hard to automate correction Structured redundancy is necessary for digital comms Coding Theory Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 14 / 56

34 Communications essentials Digital Communications Problems in natural language How efficient is the redundancy Natural languages are arbitrary Some words/sentences have a lot of redundancy Others have very little Unstructured: hard to automate correction Structured redundancy is necessary for digital comms Coding Theory Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 14 / 56

35 Communications essentials Digital Communications Problems in natural language How efficient is the redundancy Natural languages are arbitrary Some words/sentences have a lot of redundancy Others have very little Unstructured: hard to automate correction Structured redundancy is necessary for digital comms Coding Theory Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 14 / 56

36 Communications essentials Digital Communications Problems in natural language How efficient is the redundancy Natural languages are arbitrary Some words/sentences have a lot of redundancy Others have very little Unstructured: hard to automate correction Structured redundancy is necessary for digital comms Coding Theory Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 14 / 56

37 Communications essentials Digital Communications Problems in natural language How efficient is the redundancy Natural languages are arbitrary Some words/sentences have a lot of redundancy Others have very little Unstructured: hard to automate correction Structured redundancy is necessary for digital comms Coding Theory Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 14 / 56

38 Communications essentials Digital Communications Problems in natural language How efficient is the redundancy Natural languages are arbitrary Some words/sentences have a lot of redundancy Others have very little Unstructured: hard to automate correction Structured redundancy is necessary for digital comms Coding Theory Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 14 / 56

39 Communications essentials Digital Communications Problems in natural language How efficient is the redundancy Natural languages are arbitrary Some words/sentences have a lot of redundancy Others have very little Unstructured: hard to automate correction Structured redundancy is necessary for digital comms Coding Theory Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 14 / 56

40 Communications essentials Digital Communications Coding Channel and source coding Source coding (aka. compression) Remove redundancy Make a compact representation Channel coding (aka. error-control coding) Add mathematically structured redundancy Computationally efficient error-correction Optimised (low error-rate, small space) Two aspect of Information Theory Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 15 / 56

41 Communications essentials Digital Communications Coding Channel and source coding Source coding (aka. compression) Remove redundancy Make a compact representation Channel coding (aka. error-control coding) Add mathematically structured redundancy Computationally efficient error-correction Optimised (low error-rate, small space) Two aspect of Information Theory Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 15 / 56

42 Communications essentials Digital Communications Coding Channel and source coding Source coding (aka. compression) Remove redundancy Make a compact representation Channel coding (aka. error-control coding) Add mathematically structured redundancy Computationally efficient error-correction Optimised (low error-rate, small space) Two aspect of Information Theory Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 15 / 56

43 Communications essentials Channel and Source Coding Digital Communications Message r Comp. Decom. Encode Channel Decode Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 16 / 56

44 Communications essentials Channel and Source Coding Digital Communications Message r Remove redundancy Comp. Decom. Encode Channel Decode Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 16 / 56

45 Communications essentials Channel and Source Coding Digital Communications Message r Remove redundancy Comp. Decom. Add redundancy Encode Channel Decode Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 16 / 56

46 Communications essentials Channel and Source Coding Digital Communications Message r Remove redundancy Comp. Decom. Scramble Encrypt. Decrypt. Add redundancy Encode Channel Decode Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 16 / 56

47 Outline Communications essentials Shannon Entropy 1 Communications essentials Communications and Redundancy Anderson and Petitcolas 1999 Digital Communications Shannon Entropy Security Prediction 2 Compression Huffmann Coding Huffmann Steganography 3 Miscellanea Synthesis by Grammar Redundancy in Images Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 17 / 56

48 Communications essentials Shannon Entropy Uncertainty Shannon Entropy m and r are stochastic variables (drawn at random from a distribution) How much uncertainty about the message m? Uncertainty measured by entropy H(m) before any message is received. H(m r) after receipt of the message Conditional entropy Mutual Information is derived from entropy I(m; r) = H(m) H(m r) I(m; r) is the amount of information contained in r about m. I(m; r) = I(r; m) Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 18 / 56

49 Communications essentials Shannon Entropy Uncertainty Shannon Entropy m and r are stochastic variables (drawn at random from a distribution) How much uncertainty about the message m? Uncertainty measured by entropy H(m) before any message is received. H(m r) after receipt of the message Conditional entropy Mutual Information is derived from entropy I(m; r) = H(m) H(m r) I(m; r) is the amount of information contained in r about m. I(m; r) = I(r; m) Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 18 / 56

50 Communications essentials Shannon Entropy Uncertainty Shannon Entropy m and r are stochastic variables (drawn at random from a distribution) How much uncertainty about the message m? Uncertainty measured by entropy H(m) before any message is received. H(m r) after receipt of the message Conditional entropy Mutual Information is derived from entropy I(m; r) = H(m) H(m r) I(m; r) is the amount of information contained in r about m. I(m; r) = I(r; m) Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 18 / 56

51 Shannon entropy Definition Communications essentials Shannon Entropy Random variable X X H q (X) = x X Pr(X = x) log q Pr(X = x) Usually q = 2, giving entropy in bits q = e (natural logarithm) gives entropy in nats If Pr(X = x i ) = p i for x 1, x 2,... X, we write H(X) = h(p 1, p 2,...) Example: One question Q; Yes/No is probability ( ) H(Q) = log 1 2 = 1 Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 19 / 56

52 Shannon entropy Definition Communications essentials Shannon Entropy Random variable X X H q (X) = x X Pr(X = x) log q Pr(X = x) Usually q = 2, giving entropy in bits q = e (natural logarithm) gives entropy in nats If Pr(X = x i ) = p i for x 1, x 2,... X, we write H(X) = h(p 1, p 2,...) Example: One question Q; Yes/No is probability ( ) H(Q) = log 1 2 = 1 Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 19 / 56

53 Shannon entropy Definition Communications essentials Shannon Entropy Random variable X X H q (X) = x X Pr(X = x) log q Pr(X = x) Usually q = 2, giving entropy in bits q = e (natural logarithm) gives entropy in nats If Pr(X = x i ) = p i for x 1, x 2,... X, we write H(X) = h(p 1, p 2,...) Example: One question Q; Yes/No is probability ( ) H(Q) = log 1 2 = 1 Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 19 / 56

54 Shannon entropy Definition Communications essentials Shannon Entropy Random variable X X H q (X) = x X Pr(X = x) log q Pr(X = x) Usually q = 2, giving entropy in bits q = e (natural logarithm) gives entropy in nats If Pr(X = x i ) = p i for x 1, x 2,... X, we write H(X) = h(p 1, p 2,...) Example: One question Q; Yes/No is probability ( ) H(Q) = log 1 2 = 1 Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 19 / 56

55 Communications essentials Shannon Entropy Example Alice has a 1-bit message m, with distribution The entropy (Bob s uncertainty) is H(m Binary Symmetric Channel with error rate of 25% i.e. 25% risk that Alice s message is flipped Alice s uncertainty about the received message is H(r 1) = H(r 0) = 0.25 log log H(r m) = 0.5H(r 0) + 0.5H(r 1) = The information received by Bob is I(m; r) = H(m) H(m r) = H(r) H(r m) = = What if the error rate is 50%? Or 10%? Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 20 / 56

56 Communications essentials Shannon Entropy Shannon entropy Properties 1 Additive, if X and Y are independent, then H(X, Y ) = H(X) + H(Y ). If you are uncertain about two completely different questions, the entropy is the sum of uncertainty for each question 2 If X is uniformly distributed, then H(X) increase when the size of X increases. The more possibilities, the more uncertainty 3 Continuity, h(p 1, p 2,...) is continuous in each p i. Shannon entropy is a measure in mathematical terms Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 21 / 56

57 What it tells us Shannon entropy Communications essentials Shannon Entropy Consider a message X of entropy k = H(X) (in bits) The average size of a file F describing X is at least k bits If the size of F is exactly k bits on average then we have found a perfect compression of F Each message bit contains one bit of information on average Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 22 / 56

58 What it tells us Shannon entropy Communications essentials Shannon Entropy Consider a message X of entropy k = H(X) (in bits) The average size of a file F describing X is at least k bits If the size of F is exactly k bits on average then we have found a perfect compression of F Each message bit contains one bit of information on average Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 22 / 56

59 Communications essentials Shannon Entropy Exemple banale A single bit may contain more than a 1 bit of information E.G. Image Compression 0: Mona Lisa 10: Lenna 110: Baboon 11100: Peppers 11110: F : Che Guevarra : other images However, on average, Maximum information in one bit is one bit (most of the time it is less) The example is based on Huffmann coding Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 23 / 56

60 Communications essentials Shannon Entropy Exemple banale A single bit may contain more than a 1 bit of information E.G. Image Compression 0: Mona Lisa 10: Lenna 110: Baboon 11100: Peppers 11110: F : Che Guevarra : other images However, on average, Maximum information in one bit is one bit (most of the time it is less) The example is based on Huffmann coding Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 23 / 56

61 Communications essentials Shannon Entropy Exemple banale A single bit may contain more than a 1 bit of information E.G. Image Compression 0: Mona Lisa 10: Lenna 110: Baboon 11100: Peppers 11110: F : Che Guevarra : other images However, on average, Maximum information in one bit is one bit (most of the time it is less) The example is based on Huffmann coding Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 23 / 56

62 Communications essentials Shannon Entropy Exemple banale A single bit may contain more than a 1 bit of information E.G. Image Compression 0: Mona Lisa 10: Lenna 110: Baboon 11100: Peppers 11110: F : Che Guevarra : other images However, on average, Maximum information in one bit is one bit (most of the time it is less) The example is based on Huffmann coding Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 23 / 56

63 Communications essentials Shannon Entropy Exemple banale A single bit may contain more than a 1 bit of information E.G. Image Compression 0: Mona Lisa 10: Lenna 110: Baboon 11100: Peppers 11110: F : Che Guevarra : other images However, on average, Maximum information in one bit is one bit (most of the time it is less) The example is based on Huffmann coding Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 23 / 56

64 Communications essentials Shannon Entropy Exemple banale A single bit may contain more than a 1 bit of information E.G. Image Compression 0: Mona Lisa 10: Lenna 110: Baboon 11100: Peppers 11110: F : Che Guevarra : other images However, on average, Maximum information in one bit is one bit (most of the time it is less) The example is based on Huffmann coding Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 23 / 56

65 Communications essentials Shannon Entropy Exemple banale A single bit may contain more than a 1 bit of information E.G. Image Compression 0: Mona Lisa 10: Lenna 110: Baboon 11100: Peppers 11110: F : Che Guevarra : other images However, on average, Maximum information in one bit is one bit (most of the time it is less) The example is based on Huffmann coding Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 23 / 56

66 Communications essentials Shannon Entropy Exemple banale A single bit may contain more than a 1 bit of information E.G. Image Compression 0: Mona Lisa 10: Lenna 110: Baboon 11100: Peppers 11110: F : Che Guevarra : other images However, on average, Maximum information in one bit is one bit (most of the time it is less) The example is based on Huffmann coding Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 23 / 56

67 Outline Communications essentials Security 1 Communications essentials Communications and Redundancy Anderson and Petitcolas 1999 Digital Communications Shannon Entropy Security Prediction 2 Compression Huffmann Coding Huffmann Steganography 3 Miscellanea Synthesis by Grammar Redundancy in Images Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 24 / 56

68 Cryptography Communications essentials Security Alice ciphertext Bob, m c m Eve Eve seeks information about m, observing c If I(m; c) > 0 then Eve succeeds in theory or if I(k; c) > 0 If H(m c) = H(m) then the system is absolutely secure. The above are strong statements Even if Eve has information I(m; c) > 0, she may be unable to make sense of it. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 25 / 56

69 Communications essentials Security Stegananalysis Question: Does Alice send secret information to Bob? Answer: X {yes, no} What is the uncertainty H(X)? Eve intercepts a message S, Is there any information I(X; S)? If H(X S) = H(X), then the system is absolutely secure. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 26 / 56

70 Communications essentials Security Stegananalysis Question: Does Alice send secret information to Bob? Answer: X {yes, no} What is the uncertainty H(X)? Eve intercepts a message S, Is there any information I(X; S)? If H(X S) = H(X), then the system is absolutely secure. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 26 / 56

71 Communications essentials Security Stegananalysis Question: Does Alice send secret information to Bob? Answer: X {yes, no} What is the uncertainty H(X)? Eve intercepts a message S, Is there any information I(X; S)? If H(X S) = H(X), then the system is absolutely secure. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 26 / 56

72 Communications essentials Security Stegananalysis Question: Does Alice send secret information to Bob? Answer: X {yes, no} What is the uncertainty H(X)? Eve intercepts a message S, Is there any information I(X; S)? If H(X S) = H(X), then the system is absolutely secure. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 26 / 56

73 Communications essentials Security Stegananalysis Question: Does Alice send secret information to Bob? Answer: X {yes, no} What is the uncertainty H(X)? Eve intercepts a message S, Is there any information I(X; S)? If H(X S) = H(X), then the system is absolutely secure. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 26 / 56

74 Outline Communications essentials Prediction 1 Communications essentials Communications and Redundancy Anderson and Petitcolas 1999 Digital Communications Shannon Entropy Security Prediction 2 Compression Huffmann Coding Huffmann Steganography 3 Miscellanea Synthesis by Grammar Redundancy in Images Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 27 / 56

75 Communications essentials Prediction Random sequences Text is a sequence of random samples (letters) (l 1, l 2, l 3,...); l i A = {A, B,..., Z} Each letter has a probability distribution P(l), l A. Statistical dependence (implies redundancy) P(l i l i 1 ) P(l i ) H(l i l i 1 ) < H(l i ): Letter i 1 contains information about l i Use this information to guess l i The more letters l i j,..., l i 1 we have seen the more reliable can we predict l i Wayner (Ch 6.1) gives example of first, second,..., fifth order prediction Using j = 0, 1, 2, 3, 4 Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 28 / 56

76 Communications essentials First-order prediction Example from Wayner Prediction Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 29 / 56

77 Communications essentials Second-order prediction Example from Wayner Prediction Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 30 / 56

78 Communications essentials Third-order prediction Example from Wayner Prediction Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 31 / 56

79 Communications essentials Fourth-order prediction Example from Wayner Prediction Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 32 / 56

80 Markov models Communications essentials Prediction Markov source is a sequence M 1, M 2,... of stochastic (random) variables An n-th order Markov source completely described by probability distributions P[M 1, M 2,..., M n] P[M i M i n,..., M i 1 ] (identical for all i) This is a finite-state machine (automaton) State of the source last n bits M i n,..., M i 1 determines probability distribution of next symbol The random texts from Wayner are generated using 1st, 2nd, 3rd, and 4th order Markov models Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 33 / 56

81 Communications essentials Prediction A related example A group of MIT students software generating random science papers random paper accepted for WMSCI 2005 You can generate your own paper on-line Source code available (SCIgen) If you are brave as a poster topic modify SCIgen for steganography Or maybe for your dissertation if you have a related topic you can tweek Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 34 / 56

82 Outline Compression 1 Communications essentials 2 Compression Huffmann Coding Huffmann Steganography 3 Miscellanea Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 35 / 56

83 Outline Compression Huffmann Coding 1 Communications essentials Communications and Redundancy Anderson and Petitcolas 1999 Digital Communications Shannon Entropy Security Prediction 2 Compression Huffmann Coding Huffmann Steganography 3 Miscellanea Synthesis by Grammar Redundancy in Images Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 36 / 56

84 Compression Huffmann Coding Compression F is set of binary strings of arbitrary length Definition A compression system is a function c : F F, such that E(length m) > E(length(c(m))) when m is drawn from F. The compressed string is expected to be shorter than the original. Definition A compression c is perfect E(length c(m)) = H(m). It follows from the definition that the compression is one-to-one Decompress any random string m, and c 1 (m) makes sense! Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 37 / 56

85 Huffmann Coding Compression Huffmann Coding Short codewords for frequent quantities Long codewords for unusual quantities Each symbol (bit) should be equally probable % % 25% Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 38 / 56

86 Example Compression Huffmann Coding % 25% 25% % % 7 1 % 4 4 Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 39 / 56

87 Compression Huffmann Coding Decoding Huffmann codes are prefix free No codeword is the prefix of another This simplifies the decoding This is expressed in the Huffmann tree, follow edges for each coded bit (only) leaf node resolves to a message symbol When a message symbol is recovered, start over for next symbol. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 40 / 56

88 Compression Huffmann Coding Ideal Huffmann code Each branch equally likely: P(b i b i 1, b i 2,...) = 1/2 Maximum entropy H(B i B i 1, B i 2,...) = 1 uniform distribution of compressed files implies perfect compression In practice, the probabilities are rarely powers of 1 2 hence the Huffmann code is imperfect Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 41 / 56

89 Outline Compression Huffmann Steganography 1 Communications essentials Communications and Redundancy Anderson and Petitcolas 1999 Digital Communications Shannon Entropy Security Prediction 2 Compression Huffmann Coding Huffmann Steganography 3 Miscellanea Synthesis by Grammar Redundancy in Images Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 42 / 56

90 Reverse Huffmann Compression Huffmann Steganography Core Reading Peter Wayner: Disappearing Cryptography Ch. 6-7 Use a Huffmann code for each state in the Markov model Stegano-encoder: Huffmann decompression Stegano-decoder: Huffmann compression Is this similar to Anderson & Petitcolas Steganography by Perfect Compression? Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 43 / 56

91 Compression Huffmann Steganography The Steganogram Steganogram looks like random text use probability distribution based on sample text higher-order statistics make it look natural Fifth-order statistics is reasonable Higher order will look more natural Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 44 / 56

92 Compression Huffmann Steganography The Steganogram Steganogram looks like random text use probability distribution based on sample text higher-order statistics make it look natural Fifth-order statistics is reasonable Higher order will look more natural Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 44 / 56

93 Compression Huffmann Steganography Example Fifth order For each 5-tupple of letters A 0, A 1, A 2, A 3, A 4, Let l i 4,..., l i be consecutive letters in natural text tabulate P(l i = A 0 l i j = A j, j = 1, 2, 3, 4) For each 4-tuple A 1, A 2, A 3, A 4 make an (approximate) Huffmann code for A 0. we may ommit some values of A 0, or have non-unique codewords We encode a message by Huffmann decompression using Huffmann code depending on the last four stegogramme symbols obtaining a fifth-order random text Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 45 / 56

94 Compression Huffmann Steganography Example Fifth order Consider four preceeding letters comp Next letter may be letter r e l a o probability 40% 12% 22% 18% 8% combined 52% 22% 26% rounded 50% 25% 25% Rounding to power of 1 2 Combining several letters reduces rounding error. The example is arbitrary and fictuous. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 46 / 56

95 Compression Huffmann Steganography Example The Huffmann code Huffmann code based on fifth-order conditional probabilities 0 1 r/e 0 1 l a/o When two letters are possible, choose at random (according to probability in natural text) decoding (compression) is still unique encoding (decompression) is not unique This evens out the statistics in the stegogramme Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 47 / 56

96 Outline Miscellanea 1 Communications essentials 2 Compression 3 Miscellanea Synthesis by Grammar Redundancy in Images Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 48 / 56

97 Outline Miscellanea Synthesis by Grammar 1 Communications essentials Communications and Redundancy Anderson and Petitcolas 1999 Digital Communications Shannon Entropy Security Prediction 2 Compression Huffmann Coding Huffmann Steganography 3 Miscellanea Synthesis by Grammar Redundancy in Images Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 49 / 56

98 Miscellanea Synthesis by Grammar Grammar A grammar describes the structure of a language Simple grammar sentence noun verb noun Mr. Brown Miss Scarlet verb eats drinks Each choice can map to a message symbol 0 : Mr. Brown, eats 1 : Miss Scarlet, drinks Two messages can be stego-encrypted No cover-text is input. Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 50 / 56

99 Miscellanea Synthesis by Grammar More complex grammar sentence noun verb addition noun Mr. Brown Miss Scarlet... Mrs. White verb eats drinks celebrates... cooks addition addition term term on Monday in March with Mr. Green... in Alaska at home general sentence question question Does noun verb addition? xgeneral general sentence, because sentence Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 51 / 56

100 Miscellanea Synthesis by Grammar More complex grammar sentence noun verb addition noun Mr. Brown Miss Scarlet... Mrs. White verb eats drinks celebrates... cooks addition addition term term on Monday in March with Mr. Green... in Alaska at home general sentence question question Does noun verb addition? xgeneral general sentence, because sentence Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 51 / 56

101 Miscellanea Synthesis by Grammar More complex grammar sentence noun verb addition noun Mr. Brown Miss Scarlet... Mrs. White verb eats drinks celebrates... cooks addition addition term term on Monday in March with Mr. Green... in Alaska at home general sentence question question Does noun verb addition? xgeneral general sentence, because sentence Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 51 / 56

102 Miscellanea Synthesis by Grammar Is this practical? Exercise Choose either the reverse Huffmann or the grammar-based steganography technique, and write a short critique (approx. 1 page) where you answer some of the following questions. How can you do steganalysis? Under what condition will it be secure? Is the system practical? Useful? Which implementation issues do you foresee? How could it be implemented? Could the technique extend to images? Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 52 / 56

103 Outline Miscellanea Redundancy in Images 1 Communications essentials Communications and Redundancy Anderson and Petitcolas 1999 Digital Communications Shannon Entropy Security Prediction 2 Compression Huffmann Coding Huffmann Steganography 3 Miscellanea Synthesis by Grammar Redundancy in Images Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 53 / 56

104 Miscellanea Redundancy in Images Returning to Images Communications deal with abstract and discrete data Arbitrary bit strings How does redundancy apply to images? Some say that the LSB is redundant... e.g. it does not change the semantics however, the LSB cannot be reconstructed from the other bits Characters removed from English text can be reconstructed I.e. the LSB-s contain little, but still some information any value in the LSB would be meaningful/valid Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 54 / 56

105 Miscellanea Redundancy in Images Returning to Images Communications deal with abstract and discrete data Arbitrary bit strings How does redundancy apply to images? Some say that the LSB is redundant... e.g. it does not change the semantics however, the LSB cannot be reconstructed from the other bits Characters removed from English text can be reconstructed I.e. the LSB-s contain little, but still some information any value in the LSB would be meaningful/valid Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 54 / 56

106 Miscellanea Redundancy in Images Returning to Images Communications deal with abstract and discrete data Arbitrary bit strings How does redundancy apply to images? Some say that the LSB is redundant... e.g. it does not change the semantics however, the LSB cannot be reconstructed from the other bits Characters removed from English text can be reconstructed I.e. the LSB-s contain little, but still some information any value in the LSB would be meaningful/valid Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 54 / 56

107 Miscellanea Redundancy in Images Returning to Images Communications deal with abstract and discrete data Arbitrary bit strings How does redundancy apply to images? Some say that the LSB is redundant... e.g. it does not change the semantics however, the LSB cannot be reconstructed from the other bits Characters removed from English text can be reconstructed I.e. the LSB-s contain little, but still some information any value in the LSB would be meaningful/valid Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 54 / 56

108 Miscellanea Redundancy in Images Returning to Images Communications deal with abstract and discrete data Arbitrary bit strings How does redundancy apply to images? Some say that the LSB is redundant... e.g. it does not change the semantics however, the LSB cannot be reconstructed from the other bits Characters removed from English text can be reconstructed I.e. the LSB-s contain little, but still some information any value in the LSB would be meaningful/valid Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 54 / 56

109 Miscellanea Redundancy in Images Returning to Images Communications deal with abstract and discrete data Arbitrary bit strings How does redundancy apply to images? Some say that the LSB is redundant... e.g. it does not change the semantics however, the LSB cannot be reconstructed from the other bits Characters removed from English text can be reconstructed I.e. the LSB-s contain little, but still some information any value in the LSB would be meaningful/valid Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 54 / 56

110 Miscellanea Redundancy in Images Lossy and loss-less compression The Huffmann code is loss-less decompression restores the original exactly How does image processing work? Lossy... i.e. information is unrevocably lost down-sampling (reduce resolution) reduce colour depth (e.g. discard LSB)... and similar approaches in the transform domain Loss-less image compression is still possible... but the loss/compression trade-off favours lossy compression Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 55 / 56

111 Miscellanea Redundancy in Images Lossy and loss-less compression The Huffmann code is loss-less decompression restores the original exactly How does image processing work? Lossy... i.e. information is unrevocably lost down-sampling (reduce resolution) reduce colour depth (e.g. discard LSB)... and similar approaches in the transform domain Loss-less image compression is still possible... but the loss/compression trade-off favours lossy compression Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 55 / 56

112 Miscellanea Redundancy in Images Lossy and loss-less compression The Huffmann code is loss-less decompression restores the original exactly How does image processing work? Lossy... i.e. information is unrevocably lost down-sampling (reduce resolution) reduce colour depth (e.g. discard LSB)... and similar approaches in the transform domain Loss-less image compression is still possible... but the loss/compression trade-off favours lossy compression Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 55 / 56

113 Miscellanea Redundancy in Images LSB in English text Thought experiment Hide in redundant characters in English prose analogously to LSB Would it work? Why (not)? Correct character can be predicted spelling mistakes would be suspicious Would it work using spelling mistakes in an MSc dissertation by an overseas student? Dr Hans Georg Schaathun Information Theory and Synthetic Steganography Spring 2009 Week 8 56 / 56