Delay, feedback, and the price of ignorance

Transcription

1 Delay, feedback, and the price of ignorance Anant Sahai based in part on joint work with students: Tunc Simsek Cheng Chang Wireless Foundations Department of Electrical Engineering and Computer Sciences University of California at Berkeley Major Support from NSF ITR EPFL Summer Research Institute: July 8th, 26 Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 / 55

2 Shannon tells us Architectural implication: separate source and channel coding Delay is the most basic price of reliability Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 2 / 55

3 Shannon tells us Architectural implication: separate source and channel coding Delay is the most basic price of reliability [The duality between source and channel coding] can be pursued further and is related to a duality between past and future and the notions of control and knowledge. Thus we may have knowledge of the past and cannot control it; we may control the future but have no knowledge of it. Claude Shannon 959 Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 2 / 55

4 Shannon tells us Architectural implication: separate source and channel coding Delay is the most basic price of reliability [The duality between source and channel coding] can be pursued further and is related to a duality between past and future and the notions of control and knowledge. Thus we may have knowledge of the past and cannot control it; we may control the future but have no knowledge of it. Claude Shannon 959 What did he mean? Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 2 / 55

5 Review of block coding Long block codes are the traditional info theory approach Source: X n B Rn X n Channel: B Rn Y n Zn B Rn Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 3 / 55

6 Review of block coding Long block codes are the traditional info theory approach Source: X n B Rn X n Channel: B Rn Y n Zn B Rn No real sense of time, except trivial interpretation Source-coding: randomness is before encoding Channel-coding: randomness is after encoding Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 3 / 55

7 Review of block coding Long block codes are the traditional info theory approach Source: X n B Rn X n Channel: B Rn Y n Zn B Rn No real sense of time, except trivial interpretation Source-coding: randomness is before encoding Channel-coding: randomness is after encoding Block error exponents: P e exp( ne(r)) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 3 / 55

8 Review of block coding Long block codes are the traditional info theory approach Source: X n B Rn X n Channel: B Rn Y n Zn B Rn No real sense of time, except trivial interpretation Source-coding: randomness is before encoding Channel-coding: randomness is after encoding Block error exponents: P e exp( ne(r)) Source coding: E b (R) = min Q:H(Q) R D(Q P) = sup ρ E (ρ) = ln[ x ρr E (ρ) P(x) +ρ ] (+ρ) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 3 / 55

9 Review of block coding Long block codes are the traditional info theory approach Source: X n B Rn X n Channel: B Rn Y n Zn B Rn No real sense of time, except trivial interpretation Source-coding: randomness is before encoding Channel-coding: randomness is after encoding Block error exponents: P e exp( ne(r)) Channel sphere-packing bound: E sp (R) = max q = sup ρ min D (G P q) G:I( q,g) R E (ρ) ρr E (ρ) = max ln q z [ q y p y ] (+ρ) +ρ y,z Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 3 / 55

10 Outline Motivation and introduction 2 Fixed-delay channel coding Without feedback The BEC example The focusing bound Approaching the focusing bound with feedback 3 The source-coding analog Without side-information With side-information 4 Conclusions Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 4 / 55

11 What about fixed delay? B B 2 B 3 B 4 B 5 B 6 B 7 B 8 B 9 B B B 2 B 3 Y Y 2 Y 3 Y 4 Y 5 Y 6 Y 7 Y 8 Y 9 Y Y Y 2 Y 3 Y 4 Y 5 Y 6 Y 7 Y 8 Y 9 Y 2 Y 2 Y 22 Y 23 Y 24 Y 25 Y 26 Z Z 2 Z 3 Z 4 Z 5 Z 6 Z 7 Z 8 Z 9 Z Z Z 2 Z 3 Z 4 Z 5 Z 6 Z 7 Z 8 Z 9 Z 2 Z 2 Z 22 Z 23 Z 24 Z 25 Z 26 B B 2 B 3 B 4 B 5 B 6 B 7 B 8 B 9 fixed delay d = 7 Consider hard deadlines today. ( Soft deadlines allow erasures ) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 6 / 55

12 What about fixed delay? B B 2 B 3 B 4 B 5 B 6 B 7 B 8 B 9 B B B 2 B 3 Y Y 2 Y 3 Y 4 Y 5 Y 6 Y 7 Y 8 Y 9 Y Y Y 2 Y 3 Y 4 Y 5 Y 6 Y 7 Y 8 Y 9 Y 2 Y 2 Y 22 Y 23 Y 24 Y 25 Y 26 Z Z 2 Z 3 Z 4 Z 5 Z 6 Z 7 Z 8 Z 9 Z Z Z 2 Z 3 Z 4 Z 5 Z 6 Z 7 Z 8 Z 9 Z 2 Z 2 Z 22 Z 23 Z 24 Z 25 Z 26 B B 2 B 3 B 4 B 5 B 6 B 7 B 8 B 9 fixed delay d = 7 Consider hard deadlines today. ( Soft deadlines allow erasures ) Can achieve E r (R) with delay using convolutional codes. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 6 / 55

13 What about fixed delay? B B 2 B 3 B 4 B 5 B 6 B 7 B 8 B 9 B B B 2 B 3 Y Y 2 Y 3 Y 4 Y 5 Y 6 Y 7 Y 8 Y 9 Y Y Y 2 Y 3 Y 4 Y 5 Y 6 Y 7 Y 8 Y 9 Y 2 Y 2 Y 22 Y 23 Y 24 Y 25 Y 26 Z Z 2 Z 3 Z 4 Z 5 Z 6 Z 7 Z 8 Z 9 Z Z Z 2 Z 3 Z 4 Z 5 Z 6 Z 7 Z 8 Z 9 Z 2 Z 2 Z 22 Z 23 Z 24 Z 25 Z 26 B B 2 B 3 B 4 B 5 B 6 B 7 B 8 B 9 fixed delay d = 7 Consider hard deadlines today. ( Soft deadlines allow erasures ) Can achieve E r (R) with delay using convolutional codes. Pinsker (967: PPI ) claimed that the block-exponents continued to govern the non-block case with and without feedback. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 6 / 55

14 Nonblock codes without feedback Time Infinite binary tree, with iid random labels: Choose a path through the tree based on data bits Transmit the path labels through the channel Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 8 / 55

15 Nonblock codes without feedback Time. ML decoding Disjoint paths are pairwise independent of the true path. Er (R) analysis applies: future events dominate. Infinite binary tree, with iid random labels: Choose a path through the tree based on data bits Transmit the path labels through the channel Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 8 / 55

16 Nonblock codes without feedback Time. Infinite binary tree, with iid random labels: Choose a path through the tree based on data bits Transmit the path labels through the channel ML decoding Disjoint paths are pairwise independent of the true path. Er (R) analysis applies: future events dominate. Can implement with time-varying random convolutional code. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 8 / 55

17 Nonblock codes without feedback Time. Infinite binary tree, with iid random labels: Choose a path through the tree based on data bits Transmit the path labels through the channel ML decoding Disjoint paths are pairwise independent of the true path. Er (R) analysis applies: future events dominate. Can implement with time-varying random convolutional code. Achieves P e (d) K exp( E r (R)d) for every d for all R < C Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 8 / 55

18 Pinsker s bounding construction explained Without feedback: E sp (R) continues to be a bound. Consider a code with target delay d Use it to construct a block-code with blocksize n >> d Genie-aided decoder: has the truth of all bits before i B feedforward delay Fixed B Z delay decoder Z Fixed delay XOR XOR decoder... t B (t d)r B (t d)r B Causal Y encoder DMC B (t d)r B Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 / 55

19 Pinsker s bounding construction explained Without feedback: E sp (R) continues to be a bound. Consider a code with target delay d Use it to construct a block-code with blocksize n >> d Genie-aided decoder: has the truth of all bits before i Error events for genie-aided system depend only on last d B feedforward delay Fixed B Z delay decoder Z Fixed delay XOR XOR decoder... t B (t d)r B (t d)r B Causal Y encoder DMC B (t d)r B Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 / 55

20 Pinsker s bounding construction explained Without feedback: E sp (R) continues to be a bound. Consider a code with target delay d Use it to construct a block-code with blocksize n >> d Genie-aided decoder: has the truth of all bits before i Error events for genie-aided system depend only on last d Apply a change of measure argument B feedforward delay Fixed B Z delay decoder Z Fixed delay XOR XOR decoder... t B (t d)r B (t d)r B Causal Y encoder DMC B (t d)r B Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 / 55

21 My favorite example: The BEC δ δ e δ δ Simple capacity δ bits per channel use With perfect feedback, simple to achieve: retransmit until it gets through Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 2 / 55

22 My favorite example: The BEC δ δ e δ δ Error Exponent (base 2) Simple capacity δ bits per channel use With perfect feedback, simple to achieve: retransmit until it gets through Rate (in bits) Classical bounds Sphere-packing bound D( R δ) Random coding bound max ρ [,] E (ρ) ρr Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 2 / 55

23 My favorite example: The BEC δ δ e δ δ Error Exponent (base 2) Simple capacity δ bits per channel use With perfect feedback, simple to achieve: retransmit until it gets through Rate (in bits) Classical bounds Sphere-packing bound D( R δ) Random coding bound max ρ [,] E (ρ) ρr What happens with feedback? Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 2 / 55

24 BEC with feedback and fixed blocks At rate R <, have Rn bits to transmit in n channel uses. Typically ( δ)n code bits will be received. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 3 / 55

25 BEC with feedback and fixed blocks At rate R <, have Rn bits to transmit in n channel uses. Typically ( δ)n code bits will be received. Block errors caused by atypical channel behavior. Doomed if fewer than Rn bits arrive intact. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 3 / 55

26 BEC with feedback and fixed blocks At rate R <, have Rn bits to transmit in n channel uses. Typically ( δ)n code bits will be received. Block errors caused by atypical channel behavior. Doomed if fewer than Rn bits arrive intact. Feedback can not save us. D( R δ) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 3 / 55

27 BEC with feedback and fixed blocks At rate R <, have Rn bits to transmit in n channel uses. Typically ( δ)n code bits will be received. Block errors caused by atypical channel behavior. Doomed if fewer than Rn bits arrive intact. Feedback can not save us. D( R δ) Dobrushin showed that this type of behavior is common. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 3 / 55

28 BEC with feedback and fixed delay R = 2 example: δ 2 δ 2 δ 2 δ 2 ( δ) 2 ( δ) 2 ( δ) 2 ( δ) Birth-death chain: positive recurrent if δ < 2 Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 5 / 55

29 BEC with feedback and fixed delay R = 2 example: δ 2 δ 2 δ 2 δ 2 ( δ) 2 ( δ) 2 ( δ) 2 ( δ) Birth-death chain: positive recurrent if δ < 2 Delay exponent easy to see: P(D d) = P(L > d 2 ) = K( δ δ )d Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 5 / 55

30 BEC with feedback and fixed delay R = 2 example: δ 2 δ 2 δ 2 δ 2 ( δ) 2 ( δ) 2 ( δ) 2 ( δ) Birth-death chain: positive recurrent if δ < 2 Delay exponent easy to see: P(D d) = P(L > d 2 ) = K( δ δ )d.584 vs.294 for block-coding with δ =.4 Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 5 / 55

31 BEC with feedback and fixed delay R = 2 example: δ 2 δ 2 δ 2 δ 2 ( δ) 2 ( δ) 2 ( δ) 2 ( δ) Birth-death chain: positive recurrent if δ < 2 Delay exponent easy to see: P(D d) = P(L > d 2 ) = K( δ δ )d.584 vs.294 for block-coding with δ =.4 Pinsker was wrong! Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 5 / 55

32 Where is this boost coming from? 3 Progres (in bits) time (in BEC uses) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 6 / 55

33 Outline Motivation and introduction 2 Fixed-delay channel coding Without feedback The BEC example The focusing bound Approaching the focusing bound 3 The source-coding analog Without side-information With side-information 4 Conclusions Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 7 / 55

34 Using E sp to bound α in general Past behavior λn λ λ d Future ( λ)n d λr n bits within deadline bits to ignore The block error probability is like e α( λ)n which cannot exceed the sphere-packing bound e Esp(λR)n Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 9 / 55

35 Using E sp to bound α in general Past behavior λn λ λ d Future ( λ)n d λr n bits within deadline bits to ignore The block error probability is like e α( λ)n which cannot exceed the sphere-packing bound e Esp(λR)n α (R) E sp(λr) λ Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 9 / 55

36 Using E sp to bound α in general Past behavior λn λ λ d Future ( λ)n d λr n bits within deadline bits to ignore The block error probability is like e α( λ)n which cannot exceed the sphere-packing bound e Esp(λR)n α (R) E sp(λr) λ The error events involve both the past and the future. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 9 / 55

37 Uncertainty focusing bound for symmetric DMCs Minimize over λ for symmetric DMCs to sweep out frontier by varying ρ > : R(ρ) = E (ρ) ρ E + a (ρ) = E (ρ) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 2 / 55

38 Uncertainty focusing bound for symmetric DMCs Minimize over λ for symmetric DMCs to sweep out frontier by varying ρ > : R(ρ) = E (ρ) ρ E + a (ρ) = E (ρ) Same form as Viterbi s convolutional coding bound for constraint-lengths, but a lot more fundamental! Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 2 / 55

39 Upper bound tight for the BEC with feedback 7 Error Exponent (base 2) Rate (in bits) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 2 / 55

40 Outline Motivation and introduction 2 Fixed-delay channel coding Without feedback The BEC example The focusing bound Approaching the focusing bound 3 The source-coding analog Without side-information With side-information 4 Conclusions Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

41 A spoonful of sugar helps the bits get across. Original forward DMC channel uses 5 -Fortification noiseless forward side channel uses S S 2 S 3 S 4 S 5 S 6 2 Error Exponent (base e) Rate (in nats) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

42 Harnessing the power of flow control Forward DMC channel uses... Noiseless forward side channel uses one chunk... deny deny confirm Previous block disambiguation disambiguation Previous block confirmation Group bits into miniblocks of size nr. (n << d) 2 Transmit using an -length random codebook. 3 Use the sugar to tell decoder when it s done. No decoding errors, just queuing plus transmission delays. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

43 Approaching the focusing bound Error Exponent (base e) (5,8,6) (,3,2) (2,4,3) Rate (in nats) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

44 The dominant error events: past vs future Ratio (in db) of future to past in dominant error event Future behavior dominates Rate (in nats) Past behavior dominates Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

45 Why this works: operational interpretation of E (ρ) Variable block transmission time T can be bounded by a constant plus a geometric random variable. Error Exponent (base e) Rate (in nats) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

46 Why this works: operational interpretation of E (ρ) Variable block transmission time T can be bounded by a constant plus a geometric random variable. Error Exponent (base e) Rate (in nats) Need to do list-decoding at low rates. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

47 Reduces to the low-rate erasure case Pick R < R < C and aim for E + (R ) = E (ρ ) exponent. Error Exponent (base e) Rate (in nats) If n large, effective point-message rate (n( R R )) is small. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

48 But low-rate erasure exponent log δ Error Exponent (base 2) Rate (in bits) log δ = E (ρ ) in our context. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 3 / 55

49 Channels with positive zero-error capacity ( θ)c θc ( θ)c θc ( θ)c θc first chunk second chunk third chunk random increment deny random increment deny random increment.... (l + )-bit zero-error feedback block codes confirm + disambiguation Throw away a fraction θ of channel uses for flow-control overhead Asymptotically achieves the focusing bound as θ. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

50 Can do well even without sugar Time-share flow-control and data and optimize fraction θ for flow-control. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

51 Channel coding final comments Computation per channel use does not depend on probability of error = infinite computational exponent Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

52 Channel coding final comments Computation per channel use does not depend on probability of error = infinite computational exponent The code is anytime in that it is delay universal application can pick what latency is desired. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

53 Channel coding final comments Computation per channel use does not depend on probability of error = infinite computational exponent The code is anytime in that it is delay universal application can pick what latency is desired. Queuing delay dominates at all rates. Transmission delay exponents are bounded away from zero at all rates up to capacity. (partially explains Horstein s weird positive error exponents at capacity) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

54 Outline Motivation and introduction 2 Fixed-delay channel coding Without feedback The BEC example The focusing bound Approaching the focusing bound 3 The source-coding analog Without side-information With side-information 4 Conclusions Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

55 The source coding problem {ˆX t} {X t} Source Encoder Encoded Bitstream Fixed Rate R Source Decoder Assume {X t } iid Application-level interface Symbol error probability: Pe = P(X t ˆX t ) End-to-end latency: d (measured in source timescale) Channel-code interface: fixed rate R (assumed noiseless) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

56 The source coding problem {ˆX t} {X t} Source Encoder Encoded Bitstream Fixed Rate R Source Decoder Assume {X t } iid Application-level interface Symbol error probability: Pe = P(X t ˆX t ) End-to-end latency: d (measured in source timescale) Channel-code interface: fixed rate R (assumed noiseless) What are the fundamental tradeoffs? Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

57 Using E b to bound E s in general Past behavior Future β d d β dr dr n symbols within deadline symbols to ignore The error probability is bounded by K exp( de s (R)) which cannot exceed the block-coding bound exp( ne b (βr)) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

58 Using E b to bound E s in general Past behavior Future β d d β dr dr n symbols within deadline symbols to ignore The error probability is bounded by K exp( de s (R)) which cannot exceed the block-coding bound exp( ne b (βr)) E s (R) E b(βr) β Only the past matters! Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

59 Achieving the focusing bound R(ρ) = E (ρ) ρ E s (ρ) = E (ρ) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 4 / 55

60 Achieving the focusing bound R(ρ) = E (ρ) ρ E s (ρ) = E (ρ) fixed-rate source variable-length buffer code FIFO Queue fixed-rate bitstream Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 4 / 55

61 Achieving the focusing bound R(ρ) = E (ρ) ρ E s (ρ) = E (ρ) fixed-rate source variable-length buffer code FIFO Queue fixed-rate bitstream Pick miniblock n large enough but small relative to d Variable-length codes turn into variable delay at the receiver. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 4 / 55

62 Achieving the focusing bound R(ρ) = E (ρ) ρ E s (ρ) = E (ρ) fixed-rate source variable-length buffer code FIFO Queue fixed-rate bitstream Pick miniblock n large enough but small relative to d Variable-length codes turn into variable delay at the receiver. Queuing delay dominates Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 4 / 55

63 A simple example Unfair coin tosses P(H) = Reliability Rate Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

64 Side-information at the decoder X, X 2,... (X i, Y i ) p XY Y, Y 2,... Rate R E D X, X 2,.... Encoder may or may not be ignorant of the side-information If P X,Y symmetric with uniform marginals, can do no better than E b (R) with delay. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

65 Side-information at the decoder X, X 2,... (X i, Y i ) p XY Y, Y 2,... Rate R E D X, X 2,.... Encoder may or may not be ignorant of the side-information If P X,Y symmetric with uniform marginals, can do no better than E b (R) with delay Reliability Rate Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

66 Side-information at the decoder X, X 2,... (X i, Y i ) p XY Y, Y 2,... Rate R E D X, X 2,.... Encoder may or may not be ignorant of the side-information If P X,Y symmetric with uniform marginals, can do no better than E b (R) with delay Reliability.3.2..e5.5e4.e4.5e3 Ratio.e3.5e Rate.e Rate Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

67 Long vs. Large deviations Error Height Rate Slope Past Future Typical Slope Entropy Slope Shorter deviation periods must be larger. Smaller deviations must be over longer periods. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

68 Conclusions [The duality between source and channel coding] can be pursued further and is related to a duality between past and future and the notions of control and knowledge. Thus we may have knowledge of the past and cannot control it; we may control the future but have no knowledge of it. Claude Shannon 959 Error events are dominated by the: Future: Channel coding without feedback. [Pinsker] Past: Point-to-point lossless source coding [ITW6] Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

69 Conclusions [The duality between source and channel coding] can be pursued further and is related to a duality between past and future and the notions of control and knowledge. Thus we may have knowledge of the past and cannot control it; we may control the future but have no knowledge of it. Claude Shannon 959 Error events are dominated by the: Future: Channel coding without feedback. [Pinsker] Past: Point-to-point lossless source coding [ITW6] Future: Symmetric source coding with receiver side-information [ISIT6] Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

70 Conclusions [The duality between source and channel coding] can be pursued further and is related to a duality between past and future and the notions of control and knowledge. Thus we may have knowledge of the past and cannot control it; we may control the future but have no knowledge of it. Claude Shannon 959 Error events are dominated by the: Future: Channel coding without feedback. [Pinsker] Past: Point-to-point lossless source coding [ITW6] Future: Symmetric source coding with receiver side-information [ISIT6] Combination: Channel coding with feedback Combination: Non-symmetric source-coding with receiver side-information. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

71 Conclusions [The duality between source and channel coding] can be pursued further and is related to a duality between past and future and the notions of control and knowledge. Thus we may have knowledge of the past and cannot control it; we may control the future but have no knowledge of it. Claude Shannon 959 Error events are dominated by the: Future: Channel coding without feedback. [Pinsker] Past: Point-to-point lossless source coding [ITW6] Future: Symmetric source coding with receiver side-information [ISIT6] Combination: Channel coding with feedback Combination: Non-symmetric source-coding with receiver side-information. Architectural guidance: Make your messages as short as possible while avoiding integer effects. Use variable-length coding where you can Use feedback for hybrid ARQ, not retransmissions Use queues to smooth out your data rates Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

72 n = 6 total chunks in a block Potential slack chunks True minimum number of chunks n R C required based on Shannon capacity Minimum number of chunks t(ρ) n R C(ρ) required based on target reliability E (ρ) n( E (ρ) ρr E (ρ) ) slack worth at least E (ρ) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, 26 5 / 55

73 . i 2 i i 2 nr bit message block arrival times i i i + Target delay d chunks after arrival Essential delay t(ρ) Extra delay d t(ρ) i i T i 2 T i T i i 3 i 2 i i possible message block decoding times leading to an error Assumed renewal time new block enters an empty queue point messages vs slack. Extra delay d t(ρ) i 3 i 2 i i i + i + 2 i 3 i T i 2 (ρ) T i (ρ) T i (ρ) i 2 i Assumed renewal time Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

74 Optimize fraction θ for flow-control ( θ)c θc ( θ)c θc ( θ)c θc.... first chunk second chunk third chunk.. random increment deny random increment deny random increment.... confirm + disambiguation Flow-control encoded with -length convolutional code. Low-rate feedback anytime code Flow-control exponent θe () Data exponent ( θ)e (ρ) Data rate ( θ) E (ρ) ρ θ = E (ρ) E () + E (ρ) E (ρ) = E (ρ)e () E (ρ) + E () R (ρ) = E (ρ) ρ E (ρ) = E (ρ) Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

75 Low rate feedback convolutional codes At R < E (), sequential decoding expands only a finite number of nodes on average. Each expansion costs the (growing) constraint length. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

76 Low rate feedback convolutional codes At R < E (), sequential decoding expands only a finite number of nodes on average. Each expansion costs the (growing) constraint length. Idea: run a copy of the decoder at the encoder Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

77 Low rate feedback convolutional codes At R < E (), sequential decoding expands only a finite number of nodes on average. Each expansion costs the (growing) constraint length. Idea: run a copy of the decoder at the encoder Convolve against: (B + B (n)), (B 2 + B 2 (n)),..., (B n + B n (n)), B n Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

78 Low rate feedback convolutional codes At R < E (), sequential decoding expands only a finite number of nodes on average. Each expansion costs the (growing) constraint length. Idea: run a copy of the decoder at the encoder Convolve against: (B + B (n)), (B 2 + B 2 (n)),..., (B n + B n (n)), B n Identical distance properties Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

79 Low rate feedback convolutional codes At R < E (), sequential decoding expands only a finite number of nodes on average. Each expansion costs the (growing) constraint length. Idea: run a copy of the decoder at the encoder Convolve against: (B + B (n)), (B 2 + B 2 (n)),..., (B n + B n (n)), B n Identical distance properties But only a finite number of expected nonzero terms Infinite-constraint length performance at a finite price! Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

80 Low rate feedback convolutional codes At R < E (), sequential decoding expands only a finite number of nodes on average. Each expansion costs the (growing) constraint length. Idea: run a copy of the decoder at the encoder Convolve against: (B + B (n)), (B 2 + B 2 (n)),..., (B n + B n (n)), B n Identical distance properties But only a finite number of expected nonzero terms Infinite-constraint length performance at a finite price! Achieves E r (R) exponent with delay. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55

81 Low rate feedback convolutional codes At R < E (), sequential decoding expands only a finite number of nodes on average. Each expansion costs the (growing) constraint length. Idea: run a copy of the decoder at the encoder Convolve against: (B + B (n)), (B 2 + B 2 (n)),..., (B n + B n (n)), B n Identical distance properties But only a finite number of expected nonzero terms Infinite-constraint length performance at a finite price! Achieves E r (R) exponent with delay. Another trick due to Pinsker can extend computational advantage to higher rates at the cost of lower delay exponents. Anant Sahai (UC Berkeley) Delay and Feedback Jul 8, / 55