Alan Wlson Plenary Sesson Entroy, Comlexty, & Informaton n Satal Analyss Mchael Batty m.batty@ucl.ac.uk @jmchaelbatty htt://www.comlexcty.nfo/ htt://www.satalcomlexty.nfo/ for Advanced Satal Analyss CentreCentre for Advanced Satal Analyss, Unversty College London
Outlne of the Talk Two Vews of Entroy A Methods for Dervng Models, followng Wlson B Substantve Insghts: Entroy as Varety, Sread and Comlexty Defnng Entroy: Probablty (Poulaton Denstes Interretng Entroy Entroy Maxmsng: Dervng the Densty Model Informaton and Entroy Satal Entroy Sze and Shae and Dstrbuton Satal Entroy as Comlexty Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Defnng Entroy: Probablty (Poulaton Denstes Alan Wlson artculated hs entroy-maxmsng model for a two-dmensonal satal system because hs focus was on nteracton/transortaton but n fact most treatments of entroy deal wth one dmenson: we wll follow ths route here to begn wth We frst defne the robablty as the roorton of the oulaton n but we could take any attrbute we use oulaton because t s an easy to understand attrbute of a geograhcal system. We thus defne the robablty as = P P Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
The oulaton P sums to P as P = P And ths means that the robabltes wll sum to = Now let us defne raw nformaton n terms of But f an event occurs and then another event occurs whch s ndeendent of the frst one, then the jont nfo should be j = Note that when the robablty s small the nformaton s large and vce versa. I.E. hgh nfo occurs when the event s unlkely and we get a lot of nfo f and when t occurs. j Now nformaton ganed should n fact be addtve, we should be able to add the frst nfo + the second nfo to get ths but Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Centre for Advanced Satal Analyss, Unversty College London Centre for Advanced Satal Analyss The only functon whch wll allow ths s the log of And we thus wrte the nformaton as follows Now f we take the average or exected value of all these robabltes n the set, we multly the nfo by the robablty of each and sum j j + ¹ log ï þ ï ý ü - = - - + = log( log( log( ( ( ( 2 2 2 2 F F F
to get for n events H = - n = log Ths s the entroy. The mnmum value of ths functon s clearly 0 whch occurs when =, and the rest are j = 0, " j ¹ And we can easly fnd out that the entroy s at a maxmum when the robabltes are all equal and H=log n when = n Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Back to the entroy of two events wth entroy to the log base 2 ths s the classc dagram note P=-P2 Max H=log 2 (/2+log 2 (/2= H(P P=-P2 Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
I wrote about all ths a very long tme ago 972, well not 50 years but 45! And there are occasonal aers snce then Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Interretng Entroy Entroy has two asects that are relevant to comlexty. These are based on the dstrbuton and on the number of the events.e. the sze of the system n terms of the number of objects or oulatons n Ths mmedately means there s a tradeoff between the shae of the dstrbuton and the number of events. In general, as the number of events goes u.e. n gets bgger then the entroy H gets larger. But the shae of the dstrbuton also makes a dfference. Let us magne that we are lookng at the oulaton densty rofle from the centre for the edge of a cty. Ths s a onedmensonal dstrbuton. Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
We can grah ths as follows: = = K ex( -ld = n We can calculate H as mnmum 0 where =, =0, > H maxmum = log n for the unform dstrbuton For the negatve exonental = log K + l Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London H d
Bascally what all ths mles s that when we have an extreme dstrbuton, the entroy or nformaton s zero. Ths means that f the robablty s, and the event occurs, then the nformaton we get s zero. In the case where the robablty s the same everywhere, f an event occurs then the nformaton s at a maxmum. Now also as the number of events goes u, we get more nformaton. There s thus a tradeoff. We can have any system wth entroy from zero to log n. But as n goes u, then we can have a system wth very few n and greater entroy than a system wth many n but wth an extreme dstrbuton. Ths entroy measures dstrbuton as well as number; dstrbuton s a lttle lke shae as the revous grahs show. Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Entroy Maxmsng: Dervng the Densty Model Let us lnk some of ths to Wlson. In E-M, we choose a robablty dstrbuton so that we let there be as much uncertanty as ossble subject to what nformaton we know whch s certan Ths s not the easest ont to gras why would we want to maxmse ths knd of uncertanty well because f we ddn t we would be assumng more than we knew f we know there s some more nfo, then we ut t n as constrants. If we know =, we say so n the constrants. Let us revew the rocess, Maxmse Subject to H = - = n = log and c = C Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
We can thnk of ths as a one dmensonal robablty densty model where ths mght be oulaton densty And we then get the classc negatve exonental densty functon whch can be wrtten as ex( -lc = - = K ex( lc, = ex( -lc Now we don t know that ths s a negatve functon, t mght be ostve t deends on how we set u the roblem but n workng out robabltes wrt to costs, t mles the hgher the cost, the lower the robablty of locaton. We can now show how we get a ower law smly by usng a log constrant on travel cost nstead of the lnear constrant. Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
We thus maxmse entroy subject to a normalsaton constrant on robabltes and now a logarthmc cost constrant of the form Max H = - n = log Subject to = and logc = C Note the meanng of the log cost constrant. Ths s vewed as the fact that travellers erceve costs logarthmcally accordng the Weber-Fechner law and n some crcumstances, ths s as t should be. Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
If we do all ths we get the followng model where we could smly ut = logc ex( -l logc ex( -l logc nto the negatve exonental gettng Þ A ower law. But ths s not the rank sze relaton as n the sort of scalng that I have dealt wth elsewhere. We wll see f we can get such a relaton below but frst let me gve one reference at ths ont to my GA 200 aer = c -l -l c Sace, Scale, and Scalng n Entroy Maxmzng, Geograhcal Analyss 42 (200 395 42 whch s at htt://www.comlexcty.nfo/fles/20/06/batty-ga- 200.df Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Before I look at the rank sze dervaton, let me show you a smle model of how we can generate an entroy maxmsng dstrbuton whch s negatve exonental. We assume that we start wth a random dstrbuton of robabltes whch n fact we can assume are resources.e. money Now assume each zone has c (t unts and two of these chosen at random engage n swang a small unt of ther resource say one unt of money n each tme erod. In short at each tme erod, two zones and j are chosen randomly and then one of them gves one unt of resource to another, agan determned randomly; then c (t+=c (t- and c j (t+=c j (t+. In ths way, the total resources are conserved.e. c ( t = C Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Now ths s lke a rocess of random collsons. In much the same way that we can show how networks generate large hubs through referental attachment, and the way ctes get bgger or smaller through random growth through roortonate effect, then oulaton unts gan or lose n the same knd of comettve fashon Ths leads to a negatve exonental dstrbuton It s knd of obvous but we need to demo t and the followng rogram shows how ths occurs: The Random Collsons Model Note ths model starts wth somethng dfferent from a unform dstrbuton an extreme dstrbuton wth H=0 and then entroy ncreases as the collsons move the money around Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
If we start wth an extreme dstrbuton wth H=log n, then the entroy reduces to that of the negatve exonental Prvate Sub Command_Clck( Dm Peole(00 As Sngle Money = 00 SwaMoney = n = 40 For = To n Peole( = Money 'Prnt, Peole( Next t = For = t To 000000 = Int((Rnd( * n + jj = Int((Rnd( * n + If = jj Then GoTo 777 If Peole( = 0 Then Peole( = : GoTo 777 If Peole(jj = 0 Then Peole(jj = : GoTo 777 d = Rnd( If d > 0.5 Then fd = SwaMoney fjd = -fd End If Peole( = Peole( + fd Peole(jj = Peole(jj + fjd Total = 0 For z = To n Total = Total + Peole(z Next z 'Prnt, jj, fd, fjd, Peole(, Peole(jj, Total 777 Next For = To n Prnt, Peole( Next NewFle = "Money.txt" Oen NewFle For Outut As #2 For = To n Prnt #2,, Peole( Next Close #2 End Sub 0 9 8 7 6 5 4 3 2 0 y = -3.340Ln(x + 2.934 R 2 = 0.8379 0 200 400 600 800 Ths s my own rogram whch gradually converges on an exonental as the grah shows after mllon runs Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Our last foray nto generatng ower laws usng EM nvolves showng how we can get a rank sze dstrbuton. There s a key dfference between entroy-maxmsng locaton models whch tend to look at locaton robabltes as functons of cost and beneft of the locatons, and scalng models of cty sze or frm sze or ncome sze whch tend to look at robabltes of szes whch have nothng to do wth costs Thus the roblems of generatng a locaton model or a sze model are qute dfferent. Thus we must maxmse entroy wth resect to average cty sze not average locatonal cost and then we get the robabltes of locatng n small ctes much hgher than n large ctes as cty sze s lke cost. Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
It s entrely ossble of course for robabltes of locatng n bg ctes to be hgher than n small ctes but as there are so many more small ctes than bg ctes, small ones domnate. So we to look at the cty sze roblem, we must substtute cost wth sze and we thus set u the roblem as max H = ex( -l log P ex( -l log P = - = log st and log P = And then we take the frequency as and then the sze as P, form the counter cumulatve whch s the rank and then twst the equaton round to get the rank sze rule and hey resto we can connect u wth many models that generate rank-sze Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London Þ = P -l -l P P
Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Informaton and Entroy There s another measure of nformaton whch s mortant n satal analyss and that s the nformaton dfference. Imagne we have ror and osteror robablty dstrbutons q where q = where = We could form the entroy for each and make comarsons but there s an ntegrated formula based on the entroy of each wth resect to the osteror robabltes only, that s H( : q log q H( - log I( : q = - = H( : q - H( = = Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London log q
Ths s the Kullback nformaton dfference formula and t s always ostve from the way we have formed t In fact what we mght do s not maxmse ths nformaton dfference but mnmse t and we can set u the roblem as one where we mn I = q = log st and c = C Ths then leads to a model n whch the ror robablty aears n the model as one whch s moderated by the addtonal nformaton on cost, that s = q ex( -lc q ex( -lc Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
In fact f we then set q =/n, that s, the unform dstrbuton, then ths ror robablty has no effect and the model smlfes to the usual EM model As a artng shot on ths, consder what haens when the ror robablty s equal to the sace avalable for oulaton, that s q ~ D Then our model becomes x Dx ex( -lc = and thus r = Dx ex( -lc Note that ths densty can n fact be derved rather dfferently by develong a satal verson of entroy S and ths we wll now do. It s n fact equvalent formally to I Dx Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Satal Entroy Sze and Shae and Dstrbuton Imagne that we now want to fnd the entroy of the robablty densty whch s r = Dx We can smly take the exected value of the log of the nverse of ths, that s the exected value of log = -log r r So the satal entroy formula becomes S = - log r = - If we follow through the logc of EM then we get the same model as the one we have just shown but ths tme by maxmsng S, not mnmsng I Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London log Dx
Now what we are dong here s usng a rather dfferent equaton satal entroy s really entroy wth an addtonal comonent Let us exand t as S = - = - log log r = - + log log Dx Dx Ths s the area sze effect Ths s the dstrbuton and the number sze effect n terms of n n entroy In fact ths satal entroy s really only the dstrbuton effect for the number sze effect s cancelled out.e. the second term cancels the number effect but n a convoluted way Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
In fact the satal entroy s really just the dscrete aroxmaton to the contnuous entroy whch deals only wth the dstrbuton/densty not smly the sze effect We can wrte satal entroy thus or entroy as + D S = H log x H = S - log Dx Now here we have an excellent defnton of satal comlexty because we have n entroy both a sze and dstrbuton effect. Note that the contnuous equvalent of S s S = - ò r( xlog r( x x By ntroducng satal entroy, we get at both dstrbuton and number-area sze effects and are able to dsaggregate ths. Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Satal Entroy as Comlexty What we can now do s examne how entroy as comlexty changes under dfferent assumtons of the dstrbuton and the sze. Frst let us note what haens when the robablty s unform, that s = n S = log n + log D n x Then f we also have a unform dstrbuton of land D X x = x / n n = D Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Then we get S as S = log n + log X + n = log X n log n We could of course maxmse S and then we can easly see ths. We thus have dfferent ways of comutng the comonents of sze and dstrbuton and makng comarsons of the shae of the dstrbuton what entroy comes from ths and the sze of the dstrbuton what entroy comes from that Moreover we can also emloy extensve satal dsaggregaton of these log lnear measures. And I refer you back to the entroy aer n GA n 200 Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
As a concluson, let us return to Alan s satal nteracton model and look at ts entroy ths s now H = - j j log j And there are varous versons of satal entroy S S = - j log Dx = - j j j j j log Dx Dx j These can be exanded but they are qute dfferent: the frst assumes that the sace s an j term whereas the second assumes sace s at and j searately that nteracton s not a sace but that sace s a locaton. Ths s not just a lay on words the j sace could be the sace of the network Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
To conclude I wll smly refer you to our recent aer n Geograhcal Systems but the essental concluson s that we need to nterret what entroy means as well as maxmsng ths We need also to sort out dmensonal consderatons n terms of dstrbutons and denstes really throughout ths knd of modellng we should be workng wth denstes wth satal entroy or wth models that roduce denstes Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London
Questons, maybe There are a few alcatons of these deas contaned n my two recent books MIT Press, 2005 and 203 And on my blogs www.comlexty.nfo www.satalcomlexty.nfo Centre Centre for Advanced for Advanced Satal Satal Analyss, Unversty College London