A Mathematcal Theory of Communcaton Claude hannon s aer resented by Kate Jenkns 2/19/00
Publshed n two arts, July 1948 and October 1948 n the Bell ystem Techncal Journal Foundng aer of Informaton Theory Frst erson to use a robablstc model of communcaton Develoed around same tme as Codng Theory uge Imact: now the mathematcal theory of communcaton follow on aers dea that all nformaton s essentally dgtal telecommuncatons, CD layers, comuter networks alcatons to bology, artfcal ntellgence..
Questons: ow much nformaton s roduced by a source? (nfo/symbol or nfo/sec) ow quckly can nformaton be transmtted through a channel? (nfo/sec) What s best achevable transmsson rate (source symbols/sec)? If channel has nose, under what condtons can the sent message be reconstructed from the receved message?
What s nformaton? Acqurng nformaton Reducng uncertanty Amount of nformaton Level of surrse { s :1 robablty that s Informaton( s 1/16, 0, n} set of all ossble events ) Examle: {0,1} 0 s 0 0 N 1 log occurs 1/ 2 "bts" {0,1} 1/ 2 Info(0) Info(1) 1bt 1 s 1 N 1/ 2 Info( s) N bts 15/16 Info(0) 4 bts 1 Info(1) 0 bts N N
Channel caacty measured n bts/sec N ( T ) number C of lm T log allowed N ( T ) / T sgnals of duraton T Examle: Dgtal channel All{0,1}sequences allowed, roduce N( T ) 2 rt C r bts/sec r symbols/sec. Allows more comlcated channel structures: varyng tme er symbol restrctons on allowed sequences of symbols
Defne nformaton generated by source (measured n bts/symbol) to be exected amount of nformaton generated er symbol. Recall, o, Info( s ) log1/, s E(Info) s Call ths quantty the Entroy of the source. Use the symbol. (x) Where x s a random varable reresentng our sgnal. s log1/ log
Nce roertes of Entroy: If (x, y) (x) + (y) only f x, y ndeendent. x (y) uose x, y two events, () 0 0 only f 1for some n, s maxmzed when 1/n then (x, y) - (y) Then (x, y) (x) +,j,j (,j) log (,j) (x) + (y) Defne Condtonal Entroy (uncertanty of y gven value of x): - - (,j) log x,j () (j) log (j) (y), and (y) (j) x (y)
Now consder messagesof uosesource roduceseach symbolndeendently at random. Then wth hgh robablty, for a message m #of for N large,have 2 occurencesof log so m N 2 log N Ths result also holds for more comlcated source models. robable,equally lkely messages, For ergodcmarkov rocesses,use entroy m m s N n length N m N N, N N large. P states
A channel wth nose: Consder two dstnct sgnals x sgnal nut nto the channel y sgnal receved at the other end Channel caacty C Equvocaton Rate of actual transmsson R(x) max nfo sources R(x) y (x) (x) - max nfo sources y (x) bts/sec ((x) y (x))
The Fundamental Theorem for a Dscrete Channel wth Nose: Let a dscretechannelhavecaacty C, entroy bts/second. that theoutut of arbtrarly smallerrors. Proof: RecallC uose encodng has nut entroy 2 2 (x)t (y)t max encodngs If and a dscretesourcehave < C, thereexsts a codngsystemsuch thesourcecan be transmtted over thechannelwth R(x) attans ths (x), outut entroy robablenut messages of robable receved messages of maxmum (or arbtrarly close). duraton T, (y). duraton T, and o there are 2 x (y)t robablenuts for a gven outut.
Construct a bartte grah, where each node s a robable nut or outut message of duraton T for source. Connect nodes A and B by an edge f message A s an nut lkely to roduce outut B. Let R be the source we re nterested n, wth entroy < C. Encode R by randomly assgnng messages of duraton T to nodes n the left column of the grah. Gven an outut message, the robablty that t s connected to more than one R-nut message s (2 R T / 2 T )2 s x T 2 ( R ( x ) T 2 ( R C) T 0 as T
Extensons to hannon s work: Contnuous source/channel (n 2nd art of aer) Consder mult-termnal case Consder mult-way channels (lke telehone lnes!) Consder more comlcated source structures (non-ergodc!) and dfferent memory models for transmtters. Kolmogorov aled hannon s deas to solve long-standng roblems n ergodc theory. Alcatons to bology: Entroy of DNA to dentfy bndng stes Intra-organsm communcaton
Dscusson Tocs: Any questons? Any hannon anecdotes? Requred readng at NA Wrote good artcle on the mathematcs of jugglng Made a maze-learnng mouse out of hone-relays Marred a numercal analyst from Bell Labs hannon says (.413) that no exlct descrton s known of aroxmatons to the deal codng for a nosy channel. I understand ths s stll the case. Comments on what s done n ractce? Other alcatons/mact of nformaton theory? Any deas about entroy of Englsh and crossword uzzles? (.399) ow to go about rovng such a result?