Maximum-Entropy Models in Science and Engineering (Revised Edition) J. N. Kapur JOHN WILEY & SONS New York Chichester Brisbane Toronto Singapore
p Contents Preface iü 1. Maximum-Entropy Probability Distributions: Principles, Formalism and Techniques 1 1.1 Measures of entropy 2 1.2 Maximum-entropy principle 9 1.3 Maximum-entropy formalism 15 1.4 Proofs of some needed mathematical results 20 1.5 Bibliographical and historical remarks 28 2. Maximum-Entropy Discrete Univariate Probability Distributions 30 2.1 When the discrete variate takes only a finite set of values 30 2.2 When the discrete variate takes a countable infinite set of values 35 2.3 When the prior probability distribution is given and mean is prescribed 37 2.4 Lagrangian distributions 40 2 5 Bibliographical and historical remarks 42 3. Maximum-Entropy Continuous Univariate Probability Distributions 44 3.1 When the ränge of the variate is ( oo, co) 44 3.2 When the variate varies over the semi-infinite interval [0, oo) 55 3 3 When the variate ranges over a finite interval [0, R] 71 3.4 Bibliographical and historical Remarks 83 4. Maximum-Entropy Discrete Multivariate Probability Distributions 88 4.1 The Bose-Einstein, Fermi-Dirac and multinomial distributions 88 4.2 A general class of discrete multivariate distributions 96 4.3 Multivariate generalised geometrical, negative binomial, generalised negative binomial and Poisson distributions 100 4.4 Multivariate Lagrangian distributions 103 4.5 General discussion 110
viii Contents 4.6 Discrete multivariate distributions for queueing networks 115 4.7 Bibliographical and historical remarks 116 5 Maximum-Entropy Continuous Multivariate Probability Distributions 118 5.1 Maximum-Entropy distributions obtained by specifying covariances 119 5.2 Maximum-Entropy distributions when sumof variates is a constant 123 5.3 Maximum-Entropy distributions obtained by also specifying expected values of functions of all variates 132 5 4 Maximum-Entropy distribution obtained by finding distributions of function of given independent variates 139 5.5 Some other multivariate distributions 142 5.6 Bibliographical and historical remarks 145 6. Maximum-Entropy Distributions in Statistical Mechanics 146 6.1 Maxwell-Boltzmann distribution 146 6.2 Bose-Einstein distribution 156 6.3 Fermi-Dirac distribution 164 6.4 Intermediate statistics distributions 167 6.5 Bibliographical and historical remarks 171 7. Minimum Discrepancy Measures 173 7.1 Some general classes of measures of directed divergence or discrepancy 173 7.2 Some special cases 175 7.3 Minimum discriminationprobability distributions 182 7.4 Properties of minimum discrimination functions 185 7.5 Efficiency of minimum discrepancy estimators 191 7.6 Generalised measures of generalised directed divergence 193 7.7 Bibliographical and historical remarks 195 8. Concavity (Convexity) of Maximum-Entropy (Minimum Information) Functions 196 8.1 Concavity and convexity of functions 196 8.2 Concavity and convexity of functions useful in Information theory 203 8.3 Consequences of convexity or concavity of functions 211 8 4 Concavity of S mix for Renyi's entropy of second order 219 8.5 Extension of results to continuous variate distributions and density matrices 223 8.6 Bibliographical and historical remarks 226
Contents ix 9. Eqaivalence of Maximum-Entropy Principle and Gauss's Principle of Density Estimation 228 9.1 Minimum discrimination information principle 228 9.2 Maximum likelihood estimators for the parameters of the minimum discrimination information distribution 234 9.3 An alternative proof of equivalence and naturalness of measures 237 9.4 Relations among six basic principles of Statistical inference 244 9.5 Bibliographical and historical remarks 250 10. Maximum-Entropy Principle and Contingency Tables 252 10.1 Multidimensional contingency tables 252 10.2 Contingency tables with elements as random variables 256 10.3 The maximum-entropy principle and chi-square test 262 10.4 Bibliographical and historical remarks 265 11 Maximum-Entropy Principle and Statistics 266 11.1 Statistical inference 266 11.2 Jayne's entropy concentration theorem 271 11.3 The role of MEP in statistics 277 11.4 Non-parametric density estimation 285 11.5 Bibliographical and historical remarks 291 12. Entropy Maximization and Statistical Thermodynamics 292 12.1 Thermodynamics of closed Systems 292 12 2 Thermodynamics of open or diffusive systems 298 12.3 Thermodynamic functions 301 12.4 Bose-Einstein and Fermi-Dirac distributions 305 12.5 Bibliographical and historical remarks 309 13. Maximum-Entropy Models in Regional and Urban Planning 310 13.1 Discrete population distribution modeis 310 13.2 Transportation modeis 323 13.3 Population distribution and transportation Systems (continuous modeis) 340 13.4 Fermi-Dirac, Bose-Einstein and intermediate distributions for residential location and trip distribution 349 13.5 Bibliographical and historical remarks 355 14. Maximum-Entropy Models in Marketing and Elections 358 14.1 Herniter's model for brand switching 358 14.2 Generalisations of Herniter's model 369 14.3 Maximum-Entropy brand switching and vote-switching modeis with different structures 378
x Contents 14.4 Vote-Switching among political parties 386 14.5 Derivation of some purchase incidence stochastic modeis from maximum-entropy principle 395 14.6 Bibliographical and historical remarks 406 15. Maximum-Entropy Models in Economics, Finance, Insurance and Accountancy 409 15.1 The effect of taxation on reduction of inequalities of income 409 15.2 International trade modeis 425 15.3 The maximum-entropy distribution of the future market price of a stock 428 15.4 Maximum-entropy distributions in accounting and insurance 436 15.5 Bibliographical and historical remarks 442 16. Maximum-Entropy Spectral Analysis 444 16.1 Maximum-Entropy probability distribution 444 16.2 The spectral density function 451 16.3 Relation of MESA with other methods of spectral analysis 454 16.4 Multidimensional maximum-entropy spectral analysis 465 16.5 Bibliographical and historical remarks 467 17 Maximum-Entropy Image Reconstruction 469 17.1 Some physical problems requiring image reconstruction 469 17.2 The mathematical problems and discretisation 474 17.3 Solutions of the problem of estimation 478 17.4 Measures of entropy used in image reconstruction 483 17.5 Entropiegray-level picture thresholding 488 17.6 Bibliographical and historical remarks 495 18. Maximum and Minimum-Entropy Models in Pattern Recognition 497 18.1 Optimizing properties of Karhunen-Loeve expansion 497 18.2 Use of measures of divergence in feature extraction 506 18.3 Pattern recognition as a quest for minimumentropy 513 18.4 Bibliographical and historical remarks 524 19. Maximum-Entropy Principle in Operations Research 526 19.1 Maximum-Entropy principle in search theory 526 19.2 Maximum-Entropy principle in reliability theory 531' 19.3 Maximum-Entropy principle in queuing theory 534 19.4 Maximum-Entropy principle in theory of games 539 19.5 Optimal portfolio selection 544
Contents xi 19.6 Maximum-Entropy principle in risk sharing 549 19.7 Maximum-Entropy principle and mathematical Programming 553 19.8 Bibliographical and historical remarks 564 20. Maximum-Entropy Models in Biology, Medicine and Agriculture 567 20.1 Maximum-Entropy principle in pharmacokinetics (or in compartment analysis) 567 20.2 Maximum-Entropy principle in stochastic prey-predator and epidemic modeis 574 20.3 Maximum-Entropy principle in ecological modelling 579 20.4 Maximum-Entropy principle in multispecies ecological modeis 582 20.5 Maximum-Entropy estimation of missing data in design of experiments 584 20.6 MEP derivation of logistic law of population growth 586 20.7 Bibliographical and historical remarks 588 References Author Index Subject Index 591 627 633