Advanced Topics in Information Theory Lecture Notes Stefan M. Moser c Copyright Stefan M. Moser Signal and Information Processing Lab ETH Zürich Zurich, Switzerland Institute of Communications Engineering National Chiao Tung University (NCTU) Hsinchu, Taiwan You are welcome to use these lecture notes for yourself, for teaching, or for any other noncommercial purpose. If you use extracts from these lecture notes, please make sure to show their origin. The author assumes no liability or responsibility for any errors or omissions. 3 rd Edition 2018. Version 3.3. Compiled on 12 December 2018. For the latest version see http://moser-isi.ethz.ch/scripts.html
E 2 E 1 B 2 B 3 B 1 B 5 B6 B 4 B 7 E 3
v 1 v 5 v 6 v 2 v v 4 v 3
p a p p b
p
y 1 1 x y 1 x
y y x x
Q F Q P(X) Q
q q q
q > 90 degrees q q
Q > 90 degrees Q Q F
Q > 90 degrees Q Q Q F
Q Q Q F
I II Q 2 Q 1 III M M c
F A Q P(X) Q
Q Q A F
Q Q A F
V U W
Error exponents R 0 E G (R) E B (R) 0 R crit R 0 C R
E 0 (1,Q) E 0 ( 1 2,Q) E 0 ( 1 3,Q) E 0 ( 1 4,Q) max {E 0(ρ,Q) ρr} 0 ρ 1 E 0 ( 1 8,Q) R
Error exponents E SP (R) R 0 E G (R) E (R) 0 R crit C R
Error exponents Error exponents R 0 E SP (R) R 0 E SP (R) E G (R) E G (R) E (R) E (R) 0 R R crit C R 0 R crit R C R
E G (R) R 0 E G (R) R 0 R crit R 0 C R C R
C I (E s ) line above C I (E s ) here a discontinuity is theoretically possible E s
Destination ˆM Dec. ψ n FB θ k Y k DMC Q Y X X k Enc. φ k M Uniform Source F k+1 Delay F k
1 δ 0 0 X δ δ? Y 1 1 δ 1
independent description 49 points dependent description 45 points
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 4 3 2 1 0 1 2 3 4
ˆx 1 ˆx 2 θ 0.5 1 0.75 0.3578 1.3288 0.8433 0.3973 1.4011 0.8992 0.4202 1.4450 0.9326 0.4336 1.4714 0.9525 0.4414 1.4872 0.9643 0.4461 1.4966 0.9714 0.4488 1.5022 0.9755. 0.4528 1.5104 0.9816
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 4 3 2 1 0 1 2 3 4
1 D 0 0 ˆX D D 1 1 1 D X
R I (D) here a discontinuity is theoretically possible line below R I (D) D
rate information rate distortion function (with discontinuity) R I (D) R I (D+ǫ) D D+ǫ distortion
rate λ E [ d(x, ˆX) ] point ( E [ d(x, ˆX) ],I(X; ˆX) ) line of slope λ I(X; ˆX) E [ d(x, ˆX) ] distortion
rate R 0 (q,λ) R( ) λ E q [ d(x, ˆX) ] = λd I q (X; ˆX) = R(D) D distortion
rate R( ) R 0 (q,λ) achievable by q D distortion
rate R 0 (q,λ) R 0 (q,λ) R( ) achievable by q achievable by q distortion
R(D) R 0 (q,λ 1 ) R 0 (q,λ 2 ) slope discontinuities two different tangents with slopes λ 1 and λ 2 D
R(D) D
Û 1,...,ÛK Dest. Decoder Y 1,...,Y n X 1,...,X n U 1,...,U K DMC Encoder DMS
Encoder V 1...,V K Lossy Compressor U 1...,U K Binary DMS Destination ˆV 1,..., ˆV K Decoder Y 1,...,Y n DMC X 1,...,X n
DMC X 1,...,X n Channel Encoder W RD Encoder U 1...,U K Binary DMS Destination V 1,...,V K RD Decoder Ŵ Channel Decoder Y 1,...,Y n
σ 2 i σ 2 4 σ 2 1 σ 2 5 λ σ 2 7 σ 2 2 D 1 σ 2 3 D 4 D 5 D 7 D 2 D 3 σ 2 6 D 6 X 1 X 2 X 3 X 4 X 5 X 6 X 7
ˆx x x ˆx
the sources Q that do not work because for the given R and D: R( Q,D) > R inf D( Q Q) = D P(X) Q
W (1) Enc. φ (1) n Destination ˆX 1,..., ˆX n Dec. ψ (i) n X 1,...,X n Q W (2) Enc. φ (2) n
ˆX (2) Q ˆX(1), ˆX (2) X (, 0) 0 1 Q ˆX(1) X ( 0) ˆX (1) 0 3 2 2 2 1 2 2 1 2 1 0 2 1 Q ˆX(2) X ( 0) 2 2 2 1 ˆX (2) Q ˆX(1), ˆX (2) X (, 1) 0 1 Q ˆX(1) X ( 1) ˆX (1) 0 0 0 0 1 0 1 1 Q ˆX(2) X ( 1) 0 1
ˆX (2) Q ˆX(1), ˆX (2) (, ) 0 1 Q ˆX(1)( ) ˆX (1) 0 1 Q ˆX(2)( ) 3 2 2 2 2 1 2 2 1 2 2 2 1 2 1 2 2 2 1 2 2 2 2
Destination ˆX 1,..., ˆX n Dec. ψ n W Enc. φ n X 1,...,X n Q X,Y Y 1,...,Y n
codeword 1 codeword e nr bin 1 bin 2 bin 3 bin ( e nr 1 ) bin e nr
Destination ˆX 1,..., ˆX n Dec. ψ n W Enc. φ n X 1,...,X n BSS Y 1,...,Y n p p 1 p 1 p
W (1) Enc. φ (1) n X 1,...,X n Destination ˆX 1,..., ˆX n Ŷ 1,...,Ŷn Dec. ψ n W (2) Enc. φ (2) n Y 1,...,Y n Q X,Y
R (2) H(X,Y) separate compression and decompression H(Y) H(Y X) joint encoding H(X Y) H(X) H(X, Y) R (1)
Hsinchu X Taichung Y Q X,Y (, ) rain sun Hsinchu total rain 0.445 0.055 0.5 sun 0.055 0.445 0.5 Taichung total 0.5 0.5
Destination ˆM (1) ˆM (2) Dec. ψ n Y Channel Q Y X (1),X (2) X (1) X (2) Enc. φ (1) n Enc. φ (2) n M (1) M (2) Uniform Source 1 Uniform Source 2
0 X (1) 1 1 ǫ 1 ǫ 1 ǫ 1 0 1 ǫ 1 1 Y 1 ǫ 2 2 X (2) 0 ǫ 2 ǫ 2 1 ǫ 2 3 1
R (2) C (2) C (1) R (1)
R (2) 1 1 R (1)
1 2 0 X (2) 0 1 1 2 1 2 1 Y 1 2 2
R (2) 1 1 2 1 2 1 R (1)
R (2) I 3 I 2 D C I 1 0 E 0 B A R (1)
R (2) R (1)
R (2) I ( X (2) ;Y X (1)) D C I ( X (2) ;Y ) E I ( X (1) ;Y ) B A I ( X (1) ;Y X (2) ) R (1)
R (2) convex hull of C a C b C b C a R (1)
R (2) C [ϑ] R (1)
R (2) R (2) 20 inactive constraint 20 10 C a 10 C b 10 20 R (1) 10 20 R (1)
R (2) inactive constraint 20 10 C [1 2] 10 20 R (1)
R (2) ( [ϑ]) 0,I 2 D C ( [ϑ] ) I 3 I[ϑ] 2,I[ϑ] 2 E (0,0) B A ( [ϑ] I 1,0) ( [ϑ] ) I 1,I[ϑ] 3 I[ϑ] 1 R (1)
R (2) C D B E A R (1)
R (2) ( ) C E (2) σ 2 C ( ) R (1) +R (2) = C E (1) +E (2) σ 2 ( ) C E (2) E (1) +σ 2 B ( ) C E (1) E (2) +σ 2 ( ) C E (1) σ 2 R (1)
R (2) E E R (1)
R (2) α = 0 TDMA for α from 0 to 1 α = E(1) E (1) +E (2) α = 1 R (1)
Destination Û n 1 ˆV n 1 Dec. ψ Y n 1 MAC Q Y X (1),X (2) { (1)} n X k 1 { (2)} n X k 1 Enc. φ (1) Enc. φ (2) U n 1 V n 1 Q U,V
Encoder X (1) MAC Enc. 1 W (1) SW Enc. 1 U MAC Q Y X (1),X (2) Q U,V X (2) MAC Enc. 2 W (2) SW Enc. 2 V Destination Û ˆV SW Dec. Ŵ (1) Ŵ (2) Decoder MAC Dec. Y
U V 1 2 3 4 5 6 7 1 2 0 3 4 5 6 0 7
Q S S S Destination ˆM Dec. ψ Y Q Y X,S X Enc. φ M Uniform Source
1 q 0 0 X 1 q q Y 1 q 1
q 0 0 X q 1 q Y 1 1 1 q
1 p 0 0 X p p Y 1 1 1 p
λ α 0 0 1 2 1 α 1 2 S 1 2λ 2 U 1 α λ 1 2 1 α 1 1 2
Dest. 1 Dest. 2 ˆM (1) ˆM (0) ˆM (0) ˆM (2) Dec. ψ (1) Dec. ψ (2) Y (1) Y (2) BC Q Y (1),Y (2) X X Enc. φ M (1) M (0) M (2) Uniform Source 1 Uniform Source 0 Uniform Source 2
Q Y (2) Y (1) Y (1) Q Y (1) X X Y (2) Y (1)
V Z (1) + Y (1) + X Y (2) Y (1)
codeword X cloud center U
R (1) I ( X;Y (2)) +I ( X;Y (1) U ) R (1) = R (0) +I ( U;Y (2)) +I ( X;Y (1) U ) I ( X;Y (1) U ) I ( U;Y (2)) R (0)
R (2) = R (1) +I ( U (1) ;Y (1)) +I ( U (2) ;Y (2)) I ( U (1) ;U (2)) R (2) R (1) = I ( U (1) ;Y (1)) I ( U (2) ;Y (2)) A R (2) = I ( U (2) ;Y (2)) I ( U (2) ;Y (2)) I ( U (1) ;U (2)) B I ( U (1) ;Y (1)) I ( U (1) ;U (2)) I ( U (1) ;Y (1)) R (1)
X (1) f GP U (1) GP Enc. M (1) X f U (2) U (2) U (2) U (2) f: U (1) U (2) X Enc. 2 M (2)
R (0) +R (2) ( ) 1 2 log 1+ E σ(2) 2 α = 0 time-sharing α = 1 ( ) 1 2 log 1+ E σ(1) 2 R (1)
ˆM (1) X (1) Enc. φ (1) M (1) Uniform Source 1 Destination ˆM (0) ˆM (2) Dec. ψ Y MAC Q Y X (1),X (2) M (0) Uniform Source 0 X (2) Enc. φ (2) M (2) Uniform Source 2
R (0) R (2) R (1)
M (1) Terminal 1 M (2) ˆM (3) (2) X (1) ˆM (1) (5) ˆM(2) (5) Terminal 2 X (2) Y (5) Terminal 5 Y (2) DMN Channel ˆM (3) (5) ˆM(4) (5) X (3) Y (3) Y (4) X (4) Terminal 3 M (3) M (4) Terminal 4 ˆM (2) (4)
ˆM (1) ˆM (0) (1) ˆM (0) (2) ˆM (2) Terminal 1 Terminal 2 Y (1) Y (2) Broadcast Channel Q Y (1),Y (2) X (3) S 1 S 2 X (3) S 3 Terminal 3 M (1) M (0) M (2)
ˆM (1) ˆM (2) Terminal 3 Y (3) S 3 MAC Q Y (3) X (1),X (2) S 1 X (1) X (2) S 2 Terminal 1 Terminal 2 M (1) M (2)
S 2 Terminal 2 X (2) Y (2) S 1 ˆM Terminal 3 Y (3) Q Y (2),Y (3) X (1),X (2) X (1) Terminal 1 M
S 2 Terminal 2 S 4 X (2) Y (2) S 1 ˆM Terminal 4 Y (4) Q Y (2),Y (3),Y (4) X (1),X (2),X (3) X (1) Terminal 1 M X (3) Y (3) S 3 Terminal 3
Dest. 1 ˆM (1) Dec. ψ (1) Y (1) IC X (1) Enc. φ (1) M (1) Uniform Source 1 Dest. 2 ˆM (2) Dec. ψ (2) Y (2) Q Y (1),Y (2) X (1),X (2) X (2) Enc. φ (2) M (2) Uniform Source 2
1 ǫ 1 0 0 X (1) Y (1) ǫ 1 ǫ 1 1 1 1 ǫ 1 1 ǫ 2 0 0 X (2) Y (2) ǫ 2 ǫ 2 1 1 1 ǫ 2
R (2) C (2) C (1) R (1)
R (2) C (2) C (1) R (1)
ˆM (1) Terminal 3 Y (1) X (1) Terminal 1 M (1) IC Q Y (1),Y (2) X (1),X (2) ˆM (2) Terminal 4 Y (2) X (2) Terminal 2 M (2)
1.6 a 12 = 0.15, a 21 = 0.05 1.6 a 12 = 0.35, a 21 = 0.25 1.4 1.4 1.2 1.2 R (2) [bits] 1.0 0.8 0.6 R (2) [bits] 1.0 0.8 0.6 0.4 0.4 0.2 0.2 0 0 0.2 0.4 0.6 0.8 1.0 R (1) [bits] 1.2 1.4 1.6 0 0 0.2 0.4 0.6 0.8 1.0 R (1) [bits] 1.2 1.4 1.6 1.6 a 12 = 0.55, a 21 = 0.45 1.6 a 12 = 0.85, a 21 = 0.75 1.4 1.4 1.2 1.2 R (2) [bits] 1.0 0.8 0.6 R (2) [bits] 1.0 0.8 0.6 0.4 0.4 0.2 0.2 0 0 0.2 0.4 0.6 0.8 1.0 R (1) [bits] 1.2 1.4 1.6 0 0 0.2 0.4 0.6 0.8 1.0 R (1) [bits] 1.2 1.4 1.6 1.6 a 12 = 1.15, a 21 = 1.05 1.6 a 12 = 2.15, a 21 = 2.05 1.4 1.4 1.2 1.2 R (2) [bits] 1.0 0.8 0.6 R (2) [bits] 1.0 0.8 0.6 0.4 0.4 0.2 0.2 0 0 0.2 0.4 0.6 0.8 1.0 R (1) [bits] 1.2 1.4 1.6 0 0 0.2 0.4 0.6 0.8 1.0 R (1) [bits] 1.2 1.4 1.6
R (2) [bits] 3.5 3 2.5 2 1.5 1 R (1) +2R (2) 7.09 bits R (2) 3.26 bits R (1) +R (2) 4.19 bits 2R (1) +R (2) 7.09 bits 0.5 0 0 0.5 1 R (1) 3.26 bits 1.5 2 2.5 R (1) [bits] 3 3.5
d sym weak medium strong very strong 1 2 3 1 2 0 0 1 2 2 3 1 3 2 2 5 2 a