Advanced Topics in Information Theory

Transcription

1 Advanced Topics in Information Theory Lecture Notes Stefan M. Moser c Copyright Stefan M. Moser Signal and Information Processing Lab ETH Zürich Zurich, Switzerland Institute of Communications Engineering National Chiao Tung University (NCTU) Hsinchu, Taiwan You are welcome to use these lecture notes for yourself, for teaching, or for any other noncommercial purpose. If you use extracts from these lecture notes, please make sure to show their origin. The author assumes no liability or responsibility for any errors or omissions. 3 rd Edition Version 3.3. Compiled on 12 December For the latest version see

2 E 2 E 1 B 2 B 3 B 1 B 5 B6 B 4 B 7 E 3

3 v 1 v 5 v 6 v 2 v v 4 v 3

4 p a p p b

5 p

6 y 1 1 x y 1 x

7 y y x x

8 Q F Q P(X) Q

9 q q q

10 q > 90 degrees q q

11 Q > 90 degrees Q Q F

12 Q > 90 degrees Q Q Q F

13 Q Q Q F

14 I II Q 2 Q 1 III M M c

15 F A Q P(X) Q

16 Q Q A F

17 Q Q A F

18 V U W

19 Error exponents R 0 E G (R) E B (R) 0 R crit R 0 C R

20 E 0 (1,Q) E 0 ( 1 2,Q) E 0 ( 1 3,Q) E 0 ( 1 4,Q) max {E 0(ρ,Q) ρr} 0 ρ 1 E 0 ( 1 8,Q) R

21 Error exponents E SP (R) R 0 E G (R) E (R) 0 R crit C R

22 Error exponents Error exponents R 0 E SP (R) R 0 E SP (R) E G (R) E G (R) E (R) E (R) 0 R R crit C R 0 R crit R C R

23 E G (R) R 0 E G (R) R 0 R crit R 0 C R C R

24 C I (E s ) line above C I (E s ) here a discontinuity is theoretically possible E s

25 Destination ˆM Dec. ψ n FB θ k Y k DMC Q Y X X k Enc. φ k M Uniform Source F k+1 Delay F k

26 1 δ 0 0 X δ δ? Y 1 1 δ 1

27 independent description 49 points dependent description 45 points

28

29 ˆx 1 ˆx 2 θ

30

31 1 D 0 0 ˆX D D D X

32 R I (D) here a discontinuity is theoretically possible line below R I (D) D

33 rate information rate distortion function (with discontinuity) R I (D) R I (D+ǫ) D D+ǫ distortion

34 rate λ E [ d(x, ˆX) ] point ( E [ d(x, ˆX) ],I(X; ˆX) ) line of slope λ I(X; ˆX) E [ d(x, ˆX) ] distortion

35 rate R 0 (q,λ) R( ) λ E q [ d(x, ˆX) ] = λd I q (X; ˆX) = R(D) D distortion

36 rate R( ) R 0 (q,λ) achievable by q D distortion

37 rate R 0 (q,λ) R 0 (q,λ) R( ) achievable by q achievable by q distortion

38 R(D) R 0 (q,λ 1 ) R 0 (q,λ 2 ) slope discontinuities two different tangents with slopes λ 1 and λ 2 D

39 R(D) D

40 Û 1,...,ÛK Dest. Decoder Y 1,...,Y n X 1,...,X n U 1,...,U K DMC Encoder DMS

41 Encoder V 1...,V K Lossy Compressor U 1...,U K Binary DMS Destination ˆV 1,..., ˆV K Decoder Y 1,...,Y n DMC X 1,...,X n

42 DMC X 1,...,X n Channel Encoder W RD Encoder U 1...,U K Binary DMS Destination V 1,...,V K RD Decoder Ŵ Channel Decoder Y 1,...,Y n

43

44 σ 2 i σ 2 4 σ 2 1 σ 2 5 λ σ 2 7 σ 2 2 D 1 σ 2 3 D 4 D 5 D 7 D 2 D 3 σ 2 6 D 6 X 1 X 2 X 3 X 4 X 5 X 6 X 7

45 ˆx x x ˆx

46 the sources Q that do not work because for the given R and D: R( Q,D) > R inf D( Q Q) = D P(X) Q

47 W (1) Enc. φ (1) n Destination ˆX 1,..., ˆX n Dec. ψ (i) n X 1,...,X n Q W (2) Enc. φ (2) n

48 ˆX (2) Q ˆX(1), ˆX (2) X (, 0) 0 1 Q ˆX(1) X ( 0) ˆX (1) Q ˆX(2) X ( 0) ˆX (2) Q ˆX(1), ˆX (2) X (, 1) 0 1 Q ˆX(1) X ( 1) ˆX (1) Q ˆX(2) X ( 1) 0 1

49 ˆX (2) Q ˆX(1), ˆX (2) (, ) 0 1 Q ˆX(1)( ) ˆX (1) 0 1 Q ˆX(2)( )

50 Destination ˆX 1,..., ˆX n Dec. ψ n W Enc. φ n X 1,...,X n Q X,Y Y 1,...,Y n

51 codeword 1 codeword e nr bin 1 bin 2 bin 3 bin ( e nr 1 ) bin e nr

52 Destination ˆX 1,..., ˆX n Dec. ψ n W Enc. φ n X 1,...,X n BSS Y 1,...,Y n p p 1 p 1 p

53 W (1) Enc. φ (1) n X 1,...,X n Destination ˆX 1,..., ˆX n Ŷ 1,...,Ŷn Dec. ψ n W (2) Enc. φ (2) n Y 1,...,Y n Q X,Y

54 R (2) H(X,Y) separate compression and decompression H(Y) H(Y X) joint encoding H(X Y) H(X) H(X, Y) R (1)

55 Hsinchu X Taichung Y Q X,Y (, ) rain sun Hsinchu total rain sun Taichung total

56 Destination ˆM (1) ˆM (2) Dec. ψ n Y Channel Q Y X (1),X (2) X (1) X (2) Enc. φ (1) n Enc. φ (2) n M (1) M (2) Uniform Source 1 Uniform Source 2

57 0 X (1) 1 1 ǫ 1 ǫ 1 ǫ ǫ 1 1 Y 1 ǫ 2 2 X (2) 0 ǫ 2 ǫ 2 1 ǫ 2 3 1

58 R (2) C (2) C (1) R (1)

59 R (2) 1 1 R (1)

60 1 2 0 X (2) Y 1 2 2

61 R (2) R (1)

62 R (2) I 3 I 2 D C I 1 0 E 0 B A R (1)

63 R (2) R (1)

64 R (2) I ( X (2) ;Y X (1)) D C I ( X (2) ;Y ) E I ( X (1) ;Y ) B A I ( X (1) ;Y X (2) ) R (1)

65 R (2) convex hull of C a C b C b C a R (1)

66 R (2) C [ϑ] R (1)

67 R (2) R (2) 20 inactive constraint C a 10 C b R (1) R (1)

68 R (2) inactive constraint C [1 2] R (1)

69 R (2) ( [ϑ]) 0,I 2 D C ( [ϑ] ) I 3 I[ϑ] 2,I[ϑ] 2 E (0,0) B A ( [ϑ] I 1,0) ( [ϑ] ) I 1,I[ϑ] 3 I[ϑ] 1 R (1)

70 R (2) C D B E A R (1)

71 R (2) ( ) C E (2) σ 2 C ( ) R (1) +R (2) = C E (1) +E (2) σ 2 ( ) C E (2) E (1) +σ 2 B ( ) C E (1) E (2) +σ 2 ( ) C E (1) σ 2 R (1)

72 R (2) E E R (1)

73 R (2) α = 0 TDMA for α from 0 to 1 α = E(1) E (1) +E (2) α = 1 R (1)

74 Destination Û n 1 ˆV n 1 Dec. ψ Y n 1 MAC Q Y X (1),X (2) { (1)} n X k 1 { (2)} n X k 1 Enc. φ (1) Enc. φ (2) U n 1 V n 1 Q U,V

75 Encoder X (1) MAC Enc. 1 W (1) SW Enc. 1 U MAC Q Y X (1),X (2) Q U,V X (2) MAC Enc. 2 W (2) SW Enc. 2 V Destination Û ˆV SW Dec. Ŵ (1) Ŵ (2) Decoder MAC Dec. Y

76 U V

77 Q S S S Destination ˆM Dec. ψ Y Q Y X,S X Enc. φ M Uniform Source

78 1 q 0 0 X 1 q q Y 1 q 1

79 q 0 0 X q 1 q Y q

80 1 p 0 0 X p p Y p

81 λ α α 1 2 S 1 2λ 2 U 1 α λ α 1 1 2

82 Dest. 1 Dest. 2 ˆM (1) ˆM (0) ˆM (0) ˆM (2) Dec. ψ (1) Dec. ψ (2) Y (1) Y (2) BC Q Y (1),Y (2) X X Enc. φ M (1) M (0) M (2) Uniform Source 1 Uniform Source 0 Uniform Source 2

83 Q Y (2) Y (1) Y (1) Q Y (1) X X Y (2) Y (1)

84 V Z (1) + Y (1) + X Y (2) Y (1)

85 codeword X cloud center U

86 R (1) I ( X;Y (2)) +I ( X;Y (1) U ) R (1) = R (0) +I ( U;Y (2)) +I ( X;Y (1) U ) I ( X;Y (1) U ) I ( U;Y (2)) R (0)

87 R (2) = R (1) +I ( U (1) ;Y (1)) +I ( U (2) ;Y (2)) I ( U (1) ;U (2)) R (2) R (1) = I ( U (1) ;Y (1)) I ( U (2) ;Y (2)) A R (2) = I ( U (2) ;Y (2)) I ( U (2) ;Y (2)) I ( U (1) ;U (2)) B I ( U (1) ;Y (1)) I ( U (1) ;U (2)) I ( U (1) ;Y (1)) R (1)

88 X (1) f GP U (1) GP Enc. M (1) X f U (2) U (2) U (2) U (2) f: U (1) U (2) X Enc. 2 M (2)

89 R (0) +R (2) ( ) 1 2 log 1+ E σ(2) 2 α = 0 time-sharing α = 1 ( ) 1 2 log 1+ E σ(1) 2 R (1)

90 ˆM (1) X (1) Enc. φ (1) M (1) Uniform Source 1 Destination ˆM (0) ˆM (2) Dec. ψ Y MAC Q Y X (1),X (2) M (0) Uniform Source 0 X (2) Enc. φ (2) M (2) Uniform Source 2

91 R (0) R (2) R (1)

92 M (1) Terminal 1 M (2) ˆM (3) (2) X (1) ˆM (1) (5) ˆM(2) (5) Terminal 2 X (2) Y (5) Terminal 5 Y (2) DMN Channel ˆM (3) (5) ˆM(4) (5) X (3) Y (3) Y (4) X (4) Terminal 3 M (3) M (4) Terminal 4 ˆM (2) (4)

93 ˆM (1) ˆM (0) (1) ˆM (0) (2) ˆM (2) Terminal 1 Terminal 2 Y (1) Y (2) Broadcast Channel Q Y (1),Y (2) X (3) S 1 S 2 X (3) S 3 Terminal 3 M (1) M (0) M (2)

94 ˆM (1) ˆM (2) Terminal 3 Y (3) S 3 MAC Q Y (3) X (1),X (2) S 1 X (1) X (2) S 2 Terminal 1 Terminal 2 M (1) M (2)

95 S 2 Terminal 2 X (2) Y (2) S 1 ˆM Terminal 3 Y (3) Q Y (2),Y (3) X (1),X (2) X (1) Terminal 1 M

96 S 2 Terminal 2 S 4 X (2) Y (2) S 1 ˆM Terminal 4 Y (4) Q Y (2),Y (3),Y (4) X (1),X (2),X (3) X (1) Terminal 1 M X (3) Y (3) S 3 Terminal 3

97 Dest. 1 ˆM (1) Dec. ψ (1) Y (1) IC X (1) Enc. φ (1) M (1) Uniform Source 1 Dest. 2 ˆM (2) Dec. ψ (2) Y (2) Q Y (1),Y (2) X (1),X (2) X (2) Enc. φ (2) M (2) Uniform Source 2

98 1 ǫ X (1) Y (1) ǫ 1 ǫ ǫ 1 1 ǫ X (2) Y (2) ǫ 2 ǫ ǫ 2

99 R (2) C (2) C (1) R (1)

100 R (2) C (2) C (1) R (1)

101 ˆM (1) Terminal 3 Y (1) X (1) Terminal 1 M (1) IC Q Y (1),Y (2) X (1),X (2) ˆM (2) Terminal 4 Y (2) X (2) Terminal 2 M (2)

102 1.6 a 12 = 0.15, a 21 = a 12 = 0.35, a 21 = R (2) [bits] R (2) [bits] R (1) [bits] R (1) [bits] a 12 = 0.55, a 21 = a 12 = 0.85, a 21 = R (2) [bits] R (2) [bits] R (1) [bits] R (1) [bits] a 12 = 1.15, a 21 = a 12 = 2.15, a 21 = R (2) [bits] R (2) [bits] R (1) [bits] R (1) [bits]

103 R (2) [bits] R (1) +2R (2) 7.09 bits R (2) 3.26 bits R (1) +R (2) 4.19 bits 2R (1) +R (2) 7.09 bits R (1) 3.26 bits R (1) [bits] 3 3.5

104 d sym weak medium strong very strong a