Lecture 27: Entropy and Information Prof. WAN, Xin

Transcription

1 General Pysis I Leture 27: Entropy and Information Prof. WAN, Xin xinwan@zju.edu.n ttp://zimp.zju.edu.n/~xinwan/

2 Outline Introduing entropy e meaning of entropy Reversibility Disorder Information Seleted topis on information

3 1st & 2nd Laws of ermodynamis e 1st law speifies tat we annot get more energy out of a yli proess by work tan te amount of energy we put in. U Q W e 2nd law states tat we annot break even beause we must put more energy in, at te iger temperature, tan te net amount of energy we get out by work. W Q 1 arnot 1 Q Q

4 Carnot s Engine

5 Effiieny of a Carnot Engine All Carnot engines operating between te same two temperatures ave te same effiieny.

6 An Equality Now putting in te proper signs, positive Q Q Carnot Cyle 0 dq negative 0

7 A Sum of Carnot Cyles P adiabats,i Any reversible proess an be approximated by a sum of Carnot yles, ene i Q, i, i Q, i, i 0 C dq 0,i

8 Clausius Definition of Entropy Entropy is a state funtion, te ange in entropy during a proess depends only on te end points and is independent of te atual pat followed. C 2 2 C ds ds 12 dq C 1, 2 reversible ds,21 C ds 0 1 C 1 S 2 S 1 ds ds C1,1 2 C2,21 C 2 ds,1 2

9 Return to Inexat Differential Assume (2,1) (1,1) dg (2,2) (2,1) dx x y dx dy x y dy 1 2ln 2 (1,2) (1,1) (2,2) (1,2) dx x y dy ln 2 1 Note: df dg x dx x dy y is an exat differential. Integrating fator f ( x, y) ln x ln y f 0

10 Bak to te First Law Heat is pat dependent. dq du Pd erefore, 1/ is really te integrating fator for te differential form of eat. Now we an reast te 1st law of termodynamis as du ds Pd Entropy is also a state funtion, as is te internal energy or volume.

11 Entropy of an Ideal Gas (1 mole) p(, ) R U mol ( ) C fr 2 ds 1 du pd C mol d Rd Integrating from ( 0, 0 ) to (, ) S(, ) mol S0 C ln R ln 0 0

12 Carnot s eorem No real eat engine operating between two energy reservoirs an be more effiient tan Carnot s engine operating between te same two reservoirs. positive negative e' 1 Q' Q' 1 Q' Q' Wat does tis mean? Still, for any engine in a yle (S is a state funtion!) ds 0 0

13 Counting te Heat Bats in S Q' Q' > 0 S gas ds 0 after a yle S Q' Q' < 0 S S S gas S Q' 0 Q' 0

14 Counting te Heat Bats in S Q' Q' > 0 S gas ds 0 after a yle S Q' Q' < 0 e total entropy of an isolated system tat undergoes a ange an never derease.

15 Example 1: Clausius Statement S Q S Q S S S Q Q 0 Irreversible!

16 Example 2: Kelvin Statement S Q 0 Irreversible!

17 Speifi Heat Note: Last time we defined molar speifi eat. In pysis, we also use speifi eat per partile.

18 Example 3: Mixing Water Example 3: Mixing Water A A Q B B A B Q A < B B A B B A A B A B A m m m m m m A A A m Q : m Q B B B :

19 Example 3: Mixing Water A B S A < B Q A B A B Q A B m : ln Ad S A ma 0 A A m : ln Bd SB mb 0 B B For simpliity, assume S A S B m A m m ln B 2 A m B 0 / 2, A B Irreversible!

20 e Seond Law in terms of Entropy e total entropy of an isolated system tat undergoes a ange an never derease. If te proess is irreversible, ten te total entropy of an isolated system always inreases. In a reversible proess, te total entropy of an isolated system remains onstant. e ange in entropy of te Universe must be greater tan zero for an irreversible proess and equal to zero for a reversible proess. ΔSS Universe 0

21 Order versus Disorder Isolated systems tend toward disorder and tat entropy is a measure of tis disorder. Ordered: all moleules on te left side Disordered: moleules on te left and rigt

22 Example 4: Free Expansion U Q W? 0 S 0 We an only alulate S wit a reversible proess! In tis ase, we replae te free expansion by te isotermal proess wit te same initial and final states. S i f dq i f Pd i f nrd f nr ln 0 i Irreversible!

23 Entropy: A Measure of Disorder Entropy: A Measure of Disorder ln 2 ln B i f B Nk Nk S W k S B ln N m f f W N m i i W N i f i f W W We assume tat ea moleule oupies some mirosopi volume m. suggesting (Boltzmann)

24 Information, An Example Consider te following tree statements. 1. Newton s birtday falls on a partiular day of te year. (P = 1) 2. Newton s birtday falls in te seond alf of te year. (P = 1/2) 3. Newton s birtday falls on te 25t of a mont. (P = 12/365) As P inreases, te information ontent inreases. Combining 2 and 3, P =1/2 * 12/365 = 6/365. It is natural to assume information ontent is additive. e information ontent Q of a statement (wit probability P) an ten be defined by Q = log P

25 Information and Entropy (1927) Bell Labs, Ralp Hartley Measure for information in a message Logaritm: 8 bit = 2 8 = 256 different numbers (1948) Bell Labs, Claude Sannon A matematial teory of ommuniation Probability of a partiular message Average information ~ entropy? S infomation i P i log P i Sannon entropy

26 Sannon Entropy It is already in use under tat name. and besides, it will give you great edge in debates beause nobody really knows wat entropy is anyway. Jon von Neumann

27 Entropy of A Bernoulli rial Consider te simplest ase: Bernoulli trial a two-outome random variable wit probability p and 1-p of te two outomes. S P S = i P i log P i = p log p (1 p)log(1 p)

28 e Entropy of Mixing Consider two different ideal gases (all tem 1 and 2) wi are in separate vessels wit volume x and (1-x) respetively at te same pressure p and temperature. If te pipe tat onnets te two vessels is opened, te gas will spontaneously mix, resulting an inrease in entropy. Δ S = xnk B x 1 + (1 x) Nk B (1 x) 2 d 1 d 2 = Nk B [x ln x + (1 x)ln(1 x)] Distinguisability is an important piee of information!

29 Entropy, from Information iewpoint Entropy is a measure of our unertainty of a system, based on our limited knowledge of its properties and ignorane about wi of its mirostates it is in. In making inferenes on te basis of partial information, we an assign probabilities on te basis tat we maximize entropy subjet to te onstraints provided by wat is known about te system. Exerise: Maximize entropy S = - k B S i P i ln P i subjet to 1 = S i P i and U = S i E i P i. Wat do you expet?

30 Solution Using te metod of Lagrange multipliers, in wi we maximize S /k B α ( i P i 1) β ( i P i E i U ) were a and b are Lagrange multipliers. We vary tis expression wit respet to one of te probability P j and get so tat P j ( i P i ln P i α P i β P i E i ) = 0 ln P j 1 α β E j = 0 is an be rearranged to give P j = e β E j e 1+α Boltzmann probability

31 Seleted opis Maxwell s demon Data ompression Relative entropy and ross entropy Quantum information

32 I. Maxwell s Demon o determine weter to let a moleule troug, te demon must aquire information about te state of te moleule. However well prepared, te demon will eventually run out of information storage spae and must begin to erase te information it as previously gatered. Erasing information is a termodynamially irreversible proess tat inreases te entropy of a system.

33 Information is Pysial Consider a devie wi as stored 1 bit of information and is onneted to a termal reservoir of temperature. e bit an be eiter 1 or 0. We an erase it by setting te bit to zero. Before erasure, p 1 = 1/2, p 2 = 1/2. S = log 2. After erasure, p 1 = 1, p 2 = 0. S = 0. Erasure is irreversible. Entropy goes down by k B ln 2. For te total entropy of te universe not to derease, te entropy of te surroundings must go up by k B ln 2 and we must dissipate eat in te surroundings.

34 Landauer s Priniple & erifiation Computation needs to involve eat dissipation only wen you do someting irreversible wit te information. Lutz group (2012) k Q B ln

35 For ose Wo Are Interested Reading (downloadable from my website): Carles Bennett and Rolf Landauer, e fundamental pysial limits of omputation. Antoine Bérut et al., Experimental verifiation of Landauer s priniple linking information and termodynamis, Nature (2012). Set Lloyd, Ultimate pysial limits to omputation, Nature (2000). Dare to adventure were you ave not been!

36 II. Data Compression Information must be stored/transmitted. Compressible? Example: Classial information is stored in te form of binary digits 0 (wit probability p) and 1 (wit probability 1-p). Suppose p = 0.9, te data ontain less information tan in te ase of p = 0.5. wo-bit enoding: 00 0 p 2 = p(1-p) = p(1-p) = (1-p) 2 = 0.01 Code effiiently te typial sequene! Average lengt: * * *0.01 = 1.3

37 Generalization to n-bit Enoding e probability of finding a sequene x 1,, x n is P(x 1, x 2,..., x n ) = P(x 1 )P(x 2 )... P(x n ) p np n(1 p) (1 p) e information ontent is log P(x 1, x 2,..., x n ) = np log p n(1 p)log(1 p) = ns were S is te entropy for a Bernoulli trial wit probability p. Hene P(x 1, x 2,..., x n ) = 1 2 ns is sows tat tere are at most 2 ns typial sequenes and ene it only requires ns bits to ode tem.

38 Noiseless Cannel Coding eorem A ompression algoritm will take a typial sequene of n terms x 1,, x n and turn tem into a string of lengt nr. Hene te smaller R is, te greater te ompression. If we ave a soure of information wit entropy S, and if R > S, ten tere exists a reliable ompression seme of ompression fator R. Conversely, if R < S, ten any ompression seme will not be reliable. e entropy S sets te ultimate ompression limit on a set of data.

39 III. Relative Entropy e Kullbak Leibler divergene (also alled relative entropy) is a measure of ow one probability distribution diverges from a seond, expeted probability distribution. D KL (P Q) = i P i log Q i P i = i P i log Q i ( i P i log P i ) ross entropy (Sannon) entropy e KL divergene is always non-negative (Gibbs inequality), wit DKL = 0 if and only if P = Q almost everywere. KL divergene and ross entropy an be used to define te loss funtion in maine learning and optimization. e true probability P is te true label, and te given distribution Q is te predited value of te urrent model.

40 I. Quantum Information Quantum information is omposed of quantum bits (known as qubits), wi are two-level quantum systems tat an be represented by linear ombinations of te states 0 and 1. Quantum entanglement: no lassial ounterpart Bell state ( )/ 2 Information transmission faster tan te speed of ligt? No! Entanglement entropy Quantum no-loning teorem: It is impossible to make a opy a non-ortogonal quantum meanial states.