Information Sources. Professor A. Manikas. Imperial College London. EE303 - Communication Systems An Overview of Fundamentals

Size: px

Start display at page:

Download "Information Sources. Professor A. Manikas. Imperial College London. EE303 - Communication Systems An Overview of Fundamentals"

Wilfred Stone
5 years ago
Views:

1 Information Sources Professor A. Manikas Imperial College London EE303 - Communication Systems An Overview of Fundamentals Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

2 Table of Contents 1 Introduction 2 Classification of Information Sources 3 Discrete Memoryless Sources (DMS) 4 Measure of Information Generated by a DMS Source Source Entropy Source Information Rate Examples 5 A Joint DMS Source 6 Markov Discrete Information Sources 7 Continuous Sources/Signals 8 Measure of Information Generated by a Continuous Source Continuous Source: Di erential Entropy Important Relationships 9 A Note on Information Sinks Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

3 Introduction Introduction Wireline and fiber communications as well as wireless communications are fully digital. The general block structure of a Digital Communication System is shown in the following page. it is common practice its quality to be expressed in terms of the accuracy with which the binary digits delivered at the output of the detector (point bb2) represent the binary digits that were fed into the digital modulator (point B2). It is generally taken that it is the fraction of the binary digits that are delivered back in error that is a measure of the quality of the communication system. This fraction, or rate, is referred to as the bit error probability,or,bit-error-rate BER (point bb2). Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

4 Introduction Block Structure of a Digital Comm System Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

5 Classification of Information Sources N.B. - terminology: "continuous" = "analogue" Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23 Classification of Information Sources Information sources (or communication sources), or simply sources can be classified as I Discrete F Discrete Memoryless Sources (MDS), F with Memory (e.g. Markov Sources) I Continuous F non-gaussian F Gaussian Examples with reference to previous page s figure: I continuous: up to points A, A1, ort I discrete: F up to points A2 -levelsofquantiser,or F up to points B, B1,or B2 -binarydigitsorbinarycodewords. The "inverse" of an information/communication source is an information/communication sink (discrete or continuous)

6 Discrete Memoryless Sources (DMS) Discrete Memoryless Sources A source is called a Discrete Source if produces a sequence {X [n]} of symbols one after another, with each symbol being drawn from a finite alphabet X, {x 1, x 2,...,x M } (1) with a rate r X symbols/, and in which each symbol x m 2 X is produced at the output of the source with some associated probability Pr(x m ) - abbreviated p m,i.e. Pr(x m ), p m (2) If successive outputs from a discrete source are statistically independent, or in other words, if at each instant of time the source chooses for transmission one symbol from the set X = {x 1, x 2,...,x M } and its choices are independent from one time instant to the next, then the source is called a Discrete Memoryless Source (DMS). Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

7 Discrete Memoryless Sources (DMS) If p represents the vector with elements the probabilities associated with the symbols of a source i.e. then the set p, [p 1, p 2,...,p M ] T (3) " X, p #, {(x1, p 1 ), (x 2, p 2 ),...,(x M, p M )} (4) with is defined as the Source Ensemble M p m = 1 (5) m=1 Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

8 Discrete Memoryless Sources (DMS) A DMS source can be fully described by its ensemble " X, p # and its symbol rate (symbols per ond) r X The point A2 may be considered as the input of a Digital Communication System where messages consist of sequences of "symbols" selected from an alphabet e.g. levels of a quantizer or telegraph letters, numbers and punctuations. Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

9 Measure of Information Generated by a DMS Source Source Entropy Measure of Information Generated by a DMS Source Source Entropy Consider a discrete memoryless information source 9 >= " X, p # = 8> < >: (x 1, p 1 ) (x 2, p 2 )... (x M, p M ) >; with M p m = 1 (6) m=1 Then, the average information per symbol generated by the source is given by the so-called entropy of the source i.e. H X, H X (p) = " M m=1 p m log 2 (p m ) bits/symbol (7) {z },"I(x m ) = "p T log 2 (p) bits/symbol (8) Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

10 Measure of Information Generated by a DMS Source Source Information Rate Source Information Rate H X is a measure of the a priori uncertainty associated with the source output or, equivalently, a measure of the information obtained when the source output is observed. The notion of entropy is not restricted to the case where the ensemble is finite i.e. M may be. It can also be shown that H X is the minimum number of binary digits bits needed to encode the source output. Based on entropy, the average information bit-rate at the output of the source is defined as follows: r inf = r X# symbols. H X # bits symbol bits/ (9) Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

11 Measure of Information Generated by a DMS Source Source Information Rate Thus, we have two types of bits and, therefore, two types of rates. That is, I data rate : r b in data bits or simply bits I info rate : r inf in information bits or simply bits NB: I I In general: In ideal systems: r inf $ r b (10) r inf = r b (11) Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

12 Measure of Information Generated by a DMS Source Examples Example 1 If X = {0, 1} with probablities / 0 Pr(0) p = = Pr(1) / (12) then i.e. data rate : r b = r X = 10 bits entropy : H X = 1 bits = 1 info bit data bit info rate : r inf = r X.H X = 10 bits r b = r inf = 10 bits (13) Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

13 Measure of Information Generated by a DMS Source Examples Example 2 If X = {0, 1} with probablities / 0 Pr(0) p = = Pr(1) / (14) then i.e. data rate : r b = r X = 10 bits entropy : H X = bits info rate : r inf = r X.H X = bits = info bit data bit r b > r inf (15) Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

14 A Joint DMS Source A Joint DMSSource I " X, p # " Y, q # = = 8 >< >: 8 >< >: (x 1, p 1 ) (x 2, p 2 )... (x M, p M ) (y 1, q 1 ) (y 2, q 2 )... (y K, q K ) 9 >= >; 9 >= >; Consider two information sources with the ensembles " X, p # and " Y, q #, respectively, defined as follows with with M p m = 1 (16) m=1 K p k = 1 (17) k =1 where p 1 p 2 z } { z } { z } { p = [ Pr(x 1 ), Pr(x 2 ),..., Pr(x M )] T (18) q 1 q 2 z } { z } { z } { q = [ Pr(y 1 ), Pr(y 2 ),..., Pr(y K )] T (19) p M q K Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

15 A Joint DMS Source Let us form a joined source where its symbols are taken to be pairs of symbols drawn from the two original sources X and Y. The new source with alphabet {(x 1, y 1 ), (x 1, y 2 ),...,(x 2, y 1 ),...,(x m, y k ),...,(x M, y K )} is known as joint source and its ensemble as joint ensemble defined as follows: = 8 >< (X % Y, J) = (x 1, y 1 ), Pr(x 1, y 1 ) (x 1, y 2 ), Pr(x 1, y 2 )... (x m, y k ), Pr(x m, y k )... (x M, y K ), Pr(x M, y K ) 1 >: >; 80 9 >< >= B m, y k ), Pr(x m, y k ) A, 8mk : 1 $ m $ M, 1 $ k $ K >: {z } >; =J km 9 >= (20) Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

16 A Joint DMS Source Note that the joint probabilistic relationship between the symbols of X and Y, described by the so-called joint-probability matrix, 2 3T,J 11,J 12 z } { z } 1,J 1K 1 { z } { Pr(x 1, y 1 ), Pr(x 1, y 2 ),..., Pr(x 1, y K ),J 21 z } 2,J 22,J 2K { z } { z } 2 { J = Pr(x 2, y 1 ), Pr(x 2, y 2 ),..., Pr(x 2, y K )...,...,..., ,J M 1,J M 2 z } M,J MK { z } M { z } M { 5 Pr(x M, y 1 ), Pr(x M, y 2 ),..., Pr(x M, y K ) Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

17 A Joint DMS Source H X, " H Y, " H X %Y, " M m=1 K k=1 M m=1 p m log 2 (p m ) {z },"I(x m ) q k log 2 (q k ) {z },"I(x m ) K k=1 0 J km log 2 (J km ) {z } = "p T log 2 (p) = "q T log 2 (q),"i(x m,y k ) 1 = "1 T B C log 2 (J) A 1 {z } M K %M matrix bits symbol bits symbol bits symbol (21) (22) (23) (24) where 1 M = a column vector of M ones Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

18 Markov Discrete Information Sources Markov Discrete Information Sources Entropy A DMS can be modelled as with a single-state transition diagram Markov sources are modelled by Markov state transition diagrams of N states (N > 1), e.g. Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

19 Markov Discrete Information Sources The entropy of a Markov Source is given as follows: where H = N Pr(s i ).H i i=1 N = number of states bits symbol (25) Pr(s i ) = Probability to be on the i th state s i H i = entropy of the i th state (considered as a DMS) Example: Consider a two-state Markov source which gives the symbols A,B,C according to the following model: Then (for you) I H 1 =? I H 2 =? I H =? bits/symbol Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

20 Continuous Sources/Signals Continuous Sources/Signals A source whose output is a continuous signal in both amplitude and time is called a Continuous Source. However, this definition may be relaxed to include also signals that are continuous in amplitude but discrete in time, e.g. point A2. That is the condition is relaxed to only "continuous in amplitude". A continuous source which relates directly to analogue signal waveforms is described by its ensemble " g(t), pdf g (g) # where pdf g (g) denotes the amplitude probability density function of g(t). However, in order to describe fully a continuous source g(t), in addition to the source ensemble, the following parameters of g(t) should be specified/identified I the power spectral density PSD g (f ), I the bandwidth. Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

21 Measure of Information Generated by a Continuous Source Continuous Source: Di erential Entropy Measure of Information Generated by a Continuous Source Continuous Source: Di erential Entropy Entropy of an analoque source = Di erential entropy: Z H g, " pdf g (g). log 2 (pdf g (g)).dg (26) " The term "entropy" in this course (for analogue signals) will be used to refer to "di erential entropy" Entropy of a Gaussian Source H g = log 2 q2πeσ 2 g = max (27) Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

22 Measure of Information Generated by a Continuous Source Important Relationships Important Relationships At the output of an information source g(t) the following are very important: Entropy : H g (28) max(entropy) : Gaussian Entropy = log 2 q2πeσ 2 g (29) Entropy Power : N g, 1 2πe 22H g (30) Average Power : P g, E? g(t) 2@ (31) In general : P g = N g (32) if pdf g = Gaussian ) P g = N g (33) if pdf g 6= Gaussian ) P g > N g (34) Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

23 A Note on Information Sinks ANoteonInformationSinks A communication sink is the destination of the symbols produced by a source and transmitted from the input to the output of a channel. It has one input and no output and can be seen as the inverse of a source. That is, matching its associated source, a sink is either I continuous, or I discrete Examples (with reference to the block structure of a comm system): I I continuous: from ponts ba, ba1, orbt discrete: form points bb, bb1 or bb2, Asinkisdescribedby 1 a performance (fidelity) criterion ρ(x, Y ) F F e.g. criterion for a discrete sink: BER e.g. criterion for a continuous sink: SNR 2 a maximum allowed distortion D max (e.g. D max =10 "3 i.e. BER<10 "3 ) Prof. A. Manikas (Imperial College) EE303: Information Sources 24 Oct / 23

Digital Communications: An Overview of Fundamentals

Digital Communications: An Overview of Fundamentals IMPERIAL COLLEGE of SCIENCE, TECHNOLOGY and MEDICINE, DEPARTMENT of ELECTRICAL and ELECTRONIC ENGINEERING. COMPACT LECTURE NOTES on COMMUNICATION THEORY. Prof Athanassios Manikas, Spring 2001 Digital Communications: