Entropy, Inference, and Channel Coding

Transcription

1 Entropy, Inference, and Channel Coding Sean Meyn Department of Electrical and Computer Engineering University of Illinois and the Coordinated Science Laboratory NSF support: ECS , ITR and CCF

2 Overview Hypothesis testing and channel coding Structure of optimal codes Error exponents Algorithms E r ( R) Optimal code QAM R

3 References Large deviations Dembo and Zeitouni, Large Deviations Techniques And Applications, 1998 Kontoyiannis, Lastras-Montano and Meyn, Relative Entropy and Exponential Deviation Bounds for General Markov Chains, ISIT, 2005 Pandit and Meyn, Extremal Distributions and Worst-Case Large-Deviation Bounds, 2004 Hypothesis testing D&Z 1998 Zeitouni and Gutman. On universal hypothesis testing via large deviations, IT-37, 1991 Pandit, Meyn and Veeravalli, Asymptotic Robust Neyman-Pearson Testing Based on Moment Classes, ISIT, 2004.

4 References Channel coding Csiszar and Korner. Information theory: Coding Theorems for Discrete Memoryless Systems. Academic Press New York, 1997 MacKay, Information Theory, Inference, and Learning Algorithms, CUP, Blahut, Hypothesis testing and information theory, IT-20, 1974

5 Outline (today) Introduction Relative entropy & Large deviations Hypothesis testing Channel capacity Conclusions

6 Memoryless Channel Model Memoryless channel with input sequence X, output sequence Y Channel kernel P (dy x) =P{Y t dx X t = x} If X is i.i.d. with marginal distribution µ Then, Y is i.i.d. with marginal distribution π π( )= P ( x)µ(dx)

7 Random codebook Channel kernel P (dy x) =P{Y t dx X t = x} N-dimensional code words X i, i =1, 2,...,e NR N-dimensional output Y received: i.i.d., with marginal distribution π

8 IEEE Std a SUPPLEMENT TO IEEE STANDARD FOR INFORMATION TECHNOLOGY BPSK Q I QPSK b 0 Q b 0 b I 64-QAM Q b 0 b 1 b 2 b 3 b 4 b QAM Q b 0 b 1 b 2 b I I

9 Questions & Objectives 1. What is the structure of optimal µ? 2. Construct algorithms based on this structure 3. Worst-case modeling to simplify code construction 4. Decoding algorithms and evaluation

10 Questions & Objectives 1. What is the structure of optimal µ? 2. Construct algorithms based on this structure 3. Worst-case modeling to simplify code construction 4. Decoding algorithms and evaluation Methodology & Viewpoint: Hypothesis testing Large deviations Convex & linear optimization theory

11 Example: Rayleigh Channel Y = AX + N A and N are i.i.d. and mutually independent: σ 2 A =1,σ2 N =1,andσ2 P =26.4 (SNR=14.2 db)

12 Example: Rayleigh Channel Y = AX + N A and N are i.i.d. and mutually independent: σ 2 A =1,σ2 N =1,andσ2 P =26.4 (SNR=14.2 db) Standard: Rate: 16-point QAM I =0.2 nats/symbol. 16-point QAM

13 Example: Rayleigh Channel Y = AX + N A and N are i.i.d. and mutually independent: σ 2 A =1,σ2 N =1,andσ2 P =26.4 (SNR=14.2 db) point QAM Three-point constellation

14 Example: Rayleigh Channel Y = AX + N A and N are i.i.d. and mutually independent: σ 2 A =1,σ2 N =1,andσ2 P =26.4 (SNR=14.2 db) Er ( R ) point distribution: three-fold improvement over 16-point QAM R

15 Outline Introduction Relative entropy & Large deviations Hypothesis testing Channel capacity Conclusions

16 Large Deviations X = {X 1,X 2,...} a nice Markov chain on X, marginal distribution µ Simulate a function g : X R ĉ n = n 1 n t=1 g(x t )

17 Large Deviations X = {X 1,X 2,...} a nice Markov chain on X, marginal distribution µ Simulate a function g : X R ĉ n = n 1 n g(x t ) c 0 = µ(g) t=1 Probability of over-estimate c>c 0 n 1 log P {n n 1 t=1 } g(x t ) c Λ (c)

18 Large Deviations X = {X 1,X 2,...} a nice Markov chain on X, marginal distribution µ Simulate a function g : X R ĉ n = n 1 n g(x t ) c 0 = µ(g) t=1 Probability of over-estimate c>c 0 = µ(g), n 1 log P {n n 1 t=1 } g(x t ) c Λ (c) Rate function & log-moment generating function Λ (c) =sup[θc Λ(θ)] θ>0 [ ( Λ(θ) = lim n n 1 log E exp θ n t=1 )] g(x t )

19 Hoeffding's Bound X = {X 1,X 2,...} is i.i.d. on X =[0, 1] g(x) =x Marginal distribution µ unknown ĉ n = n 1 n t=1 X t c 0 = µ(g) Worst-case rate function & log-moment generating function inf{λ µ(c) :µ(g) =c 0 } sup{λ µ (θ) :µ(g) =c 0 }

20 Hoeffding's Bound X = {X 1,X 2,...} is i.i.d. on X =[0, 1] g(x) =x Marginal distribution µ unknown ĉ n = n 1 n t=1 X t c 0 = µ(g) Worst-case rate function & log-moment generating function inf{λ µ(c) :µ(g) =c 0 } sup{λ µ (θ) :µ(g) =c 0 } Solution: µ is binary on {0, 1}

21 Bennett's Lemma X = {X 1,X 2,...} is i.i.d. on X =[0, 1] Mean and variance given Marginal distribution µ unknown g(x) =x ĉ n = n 1 n t=1 X t Worst-case rate function & log-moment generating function inf{λ µ(c) :µ(g i )=c i, i =1, 2} sup{λ µ (θ) :µ(g i )=c i, i =1, 2}

22 Bennett's Lemma X = {X 1,X 2,...} is i.i.d. on X =[0, 1] Mean and variance given Marginal distribution µ unknown g(x) =x ĉ n = n 1 n t=1 X t Worst-case rate function & log-moment generating function inf{λ µ(c) :µ(g i )=c i, i =1, 2} sup{λ µ (θ) :µ(g i )=c i, i =1, 2} Solution: µ is binary on x 0 {, 1}

23 Generalized Bennett's Lemma X = {X 1,X 2,...} is i.i.d. on X =[0, 1] n moments given g i Marginal distribution µ unknown ĉ n = n 1 n t=1 g(x t ) Worst-case moment generating function: λ(θ) =E[e θg(x t) ]= µ, e θg

24 Generalized Bennett's Lemma X = {X 1,X 2,...} is i.i.d. on X =[0, 1] n moments given Marginal distribution µ unknown g i ĉ n = n 1 n g(x t ) t=1 Worst-case moment generating function: λ(θ) =E[e θg(x t) ]= µ, e θg Linear program over M: max µ, e θg s. t. µ, g i = c i, i =1,...,n. µ is discrete

25 Sanov's Theorem State space: Notation: X Probability measures: M µ, g = µ(g):= Empirical measures: g(y) µ(dy) µ ameasure g afunction on X L n := 1 n n 1 t=0 δ Xt L n M for n 1 L n,g = 1 n n 1 t=0 g(x t )

26 Sanov's Theorem State space: Notation: X Probability measures: M µ, g = µ(g):= Empirical measures: g(y) µ(dy) µ ameasure g afunction on X L n := 1 n n 1 t=0 δ Xt L n M for n 1 Relative entropy: D(ν µ) = ν, log ( dν ) = dµ ( dν ) log ν(dx) dµ

27 Sanov's Theorem Law of large numbers: L n := 1 n n 1 δ Xt L n µ, n t=0 L n µ

28 Sanov's Theorem Convex set of probability measures K M µ K n 1 log P{L n K}? L n µ? K

29 Sanov's Theorem Convex set of probability measures K M µ K n 1 log P{L n K} η = inf J(ν) ν K L n Q η µ K Q η = {ν : J (ν) <η}

30 Sanov's Theorem i.i.d. source: J(ν) =D(ν µ) Markov: J(ν) =inf D(ν ˇP ν P ) : ˇP tr. kernel with ν invariant L n Q η µ K Q η = {ν : J (ν) <η}

31 Sanov's Theorem Example: K = {ν : ν, g c} n 1 log P{L n K} η = inf J(ν) = ν, g c Λ (c)

32 Sanov's Theorem Example: K = {ν : ν, g c} n 1 log P{L n K} η = inf J(ν) = ν, g c Λ (c) ν, g = c L n Q η µ K Q η = {ν : J (ν) <η}

33 Outline Introduction Relative entropy & Large deviations Hypothesis testing Channel capacity Conclusions

34 Neyman Pearson Hypothesis Testing Observations X = {X t : t =1, 2,...N} X i.i.d. with marginal π j under H j, j =0, 1 Hypothesis test: Error Probabilities φ(x) = 1 ifh1 is declared true, based on N observations P e,0 =P 0 {φ(x) =1}, P e,1 =P 1 {φ(x) =0} N-P Criterion: inf φ P e,1 subject to P e,0 e Nη

35 Neyman Pearson Hypothesis Testing Observations X = {X t : t =1, 2,...N} X i.i.d. with marginal π j under H j, j =0, 1 ErrorProbabilities P e,0 =P 0 {φ(x) =1}, P e,1 =P 1 {φ(x) =0} π 1 Solution: φ(x) =0 if L n Q η (π 0 ) Q η (π 0 ) π 0 N-P Criterion: inf φ P e,1 subject to P e,0 e Nη

36 Neyman Pearson Hypothesis Testing Solution: φ(x) =0 if L n Q η (π 0 ) lim N N 1 log P 0 {φ N =1} = η lim N 1 log P 1 {φ N =0} = β N π 1 Q η (π 0 ) π 0

37 Neyman Pearson Hypothesis Testing Solution: φ(x) =0 if L n Q η (π 0 ) lim N N 1 log P 0 {φ N =1} = η Q (π 1 ) β lim N 1 log P 1 {φ N =0} = β N ν, l = c π 1 β =inf{j 1 (ν) :J 0 (ν) η} Q η (π 0 ) π 0 =inf{β >0:Q β (π 1 ) Q η (π 0 ) }

38 Robust Neyman Pearson Hypothesis Testing Uncertainty classes defined by moment constraints π 0 P 0 π 1 P 1 P 1 P 0

39 Robust Neyman Pearson Hypothesis Testing Uncertainty classes defined by moment constraints π 0 P 0 π 1 P 1 P 1 P 0 Q (P 0 ) η

40 Robust Neyman Pearson Hypothesis Testing Uncertainty classes defined by moment constraints There exist π 0 P 0,π 1 P 1,andµ solving, β = inf π 1 P 1 inf D(µ π 1 ) µ Q η (P 0 ) π 1 µ P 1 π 0 P 0 Q (P 0 ) η

41 Robust Neyman Pearson Hypothesis Testing Uncertainty classes defined by moment constraints There exist π 0 P 0,π 1 P 1,andµ solving, β = inf π 1 P 1 inf D(µ π 1 ) µ Q η (P 0 ) Q β (P 1 ) Q (P 0 ) η π 1 µ π 0 P 1 P 0 µ, log(l) = µ, log(l ) Optimizers again discrete

42 Outline Introduction Relative entropy & Large deviations Hypothesis testing Channel capacity Conclusions

43 Channel Coding and Sanov's Theorem Channel kernel P (dy x) =P{Y t dy X t = x} N-dimensional code words X i, i =1, 2,...,e NR N-dimensional output Y received X is i.i.d. with marginal distribution µ Y is i.i.d. with marginal distribution π π( )= P ( x)µ(dx)

44 Channel Coding and Sanov's Theorem Channel kernel P (dy x) =P{Y t dy X t = x} N-dimensional code words X i, i =1, 2,...,e NR N-dimensional output Y received If i is the true codeword then ( i, ) has marginal distribution X Y Otherwise, independence: µ P (dx, dy) =µ(dx)p (dy x) µ π (dx, dy) =µ(dx)π(dy)

45 Channel Coding and Sanov's Theorem Two hypotheses based on observations: H : 0 µ π (dx, dy) =µ(dx) π(dy) 1 µ P (dx, dy) =µ(dx)p (dy x) H : µ P µ π

46 Channel Coding and Sanov's Theorem Two hypotheses based on observations: H : 0 H : 1 µ π (dx, dy) =µ(dx) π(dy) µ P (dx, dy) =µ(dx)p (dy x) µ P Solution: Reject codeword i ( φ =0) if L n Q η (π 0 ) Empirical distributions for joint observations i (, ) X Y Q η (π 0 ) µ π

47 Channel Coding and Sanov's Theorem Solution: φ =0 if L n Q η (π 0 ) lim N 1 log P 0 {φ N =1} = η N The error probability e Nη must be multiplied by e NR µ P For vanishing error, e NR e Nη < 1 That is, R < η Q η (π 0 ) µ π

48 Channel Coding and Sanov's Theorem Solution: φ =0 if L n Q η (π 0 ) lim N N 1 log P 0 {φ N =1} = η The error probability e Nη must be multiplied by e NR µ P R<η max = D(µ P µ π) = mutual information Q η (π 0 ) max µ π

49 { Error Exponent E(R,µ ) = lim N 1 log P { error } N Formula expressed as solution to a robust hypothesis testing problem: For a given input distribution µ, denote product measures on X Y with first marginal µ, P 0 = { µ ν : ν is a probability measure on Y

50 { Error Exponent E(R,µ ) = lim N 1 log P { error } N Formula expressed as solution to a robust hypothesis testing problem: For a given input distribution µ, denote product measures on X Y with first marginal µ, P 0 = { µ ν : ν is a probability measure on Y H 0 Hypothesis : Code word i not sent; (Xj i ) (Y j ) independent Test: Empirical distributions within entropy ball around P 0

51 { { Error Exponent H 0 : {(X i j, Y j):j =1,...,N } has marginal distribution π 0 P 0 H 1 : {(X i j, Y j):j =1,...,N } has marginal distribution π 1 := µ p Entropy neighborhood of P 0 Q + R (P 0)={ γ :min ν D(γ µ ν) R Entropy neighborhood of π 1 Q + ( ) = { γ : β D(γ µ p ) β π 1

52 Error Exponent E(R,µ ) = lim N 1 log P { error } N β = infimum over β such that these entropy neighborhoods meet: µ p + Q β (µ p) + Q R (P 0 ) µ pˆ µ ˆp µ P 0

53 Error Exponent E(R,µ ) = = lim N 1 log P { error } N inf β { } β : Q + β (µ p) Q+ R (P 0) E(R ) = random coding exponent = supremum over µ µ p + Q β (µ p) + Q R (P 0 ) µ pˆ µ ˆp µ P 0

54 Outline Introduction Relative entropy & Large deviations Hypothesis testing Channel capacity Conclusions

55 Summary Large Deviations is the grand unifying principle of Information Theory

56 Summary Standard coding based on AWGN models May be unrealistic in wireless models with fading Discrete distributions arise in coding, and other applications involving optimization over M Extremal distributions arise in worst-case models

57 What's Next? II Channel models Convex optimization and channel coding Cutting plane algorithm III Worst-case models Extremal distributions