BAYESIAN PROBABILITY THEORY

Transcription

1 BAYESIAN PROBABILITY THEORY From the basics to the forefront of modern research, this book presents all aspects of probability theory, statistics and data analysis from a Bayesian perspective for physicists and engineers. The book presents the roots, applications and numerical implementation of probability theory, and covers advanced topics such as maximum entropy distributions, stochastic processes, parameter estimation, model selection, hypothesis testing and experimental design. In addition, it explores state-of-the-art numerical techniques required to solve demanding real-world problems. The book is ideal for students and researchers in physical sciences and engineering. Wolfgang von der Linden is Professor of Theoretical and Computational Physics at the Graz University of Technology. His research area is statistical physics, with a focus on strongly correlated quantum many-body physics based on computational techniques. Volker Dose is a former Director of the Surface Physics Division of the Max Planck Institute for Plasma Physics. He has contributed to Bayesian methods in physics, astronomy and climate research. Udo von Toussaint is a Senior Scientist in the Material Research Division of the Max Planck Institute for Plasma Physics, where he works on Bayesian experimental design, data fusion, molecular dynamics and inverse problems in the field of plasma wall interactions.

2

3 BAYESIAN PROBABILITY THEORY Applications in the Physical Sciences WOLFGANG VON DER LINDEN Graz University of Technology, Institute for Theoretical and Computational Physics, Graz, Austria VOLKER DOSE Max Planck Institute for Plasma Physics, Garching, Germany UDO VON TOUSSAINT Max Planck Institute for Plasma Physics, Garching, Germany

4 University Printing House, Cambridge CB2 8BS, United Kingdom Cambridge University Press is part of the University of Cambridge. It furthers the University s mission by disseminating knowledge in the pursuit of education, learning and research at the highest international levels of excellence. Information on this title: / W. von der Linden, V. Dose and U. von Toussaint 2014 This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 2014 Printed in the United Kingdom by Clays, St Ives plc A catalogue record for this publication is available from the British Library Library of Congress Cataloguing in Publication data Linden, Wolfgang von der, author. Bayesian probability theory / Prof. Dr. Wolfgang von der Linden, Graz University of Technology (Austria), Institute for Theoretical and Computational Physics, [and] Prof. Dr. Dr. h.c. Volker Dose Max-Planck-Institute for Plasma Physics (Garching, Germany), [and] Dr. Udo von Toussaint, Max-Planck-Institute for Plasma Physics (Garching, Germany). pages cm Includes bibliographical references and index. ISBN (Hardback) 1. Probabilities. 2. Bayesian statistical decision theory. I. Dose, Volker, author. II. Toussaint, Udo von, author. III. Title. QC P76L dc ISBN Hardback Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate.

5 Contents Preface page xi PART I INTRODUCTION 1 1 The meaning of probability Classical definition of probability Statistical definition of probability Bayesian understanding of probability 10 2 Basic definitions for frequentist statistics and Bayesian inference Definition of mean, moments and marginal distribution Worked example: The three-urn problem Frequentist statistics versus Bayesian inference 28 3 Bayesian inference Propositions Selected examples Ockham s razor 43 4 Combinatorics Preliminaries Partitions, binomial and multinomial distributions Occupation number problems Geometric and hypergeometric distributions The negative binomial distribution 66 5 Random walks First return First lead Random walk with absorbing wall 80 6 Limit theorems Stirling s formula de Moivre Laplace theorem/local limit theorem Bernoulli s law of large numbers Poisson s law 87

6 vi Contents 7 Continuous distributions Continuous propositions Distribution function and probability density functions Application in statistical physics Definitions for continuous distributions Common probability distributions Order statistic Transformation of random variables Characteristic function Error propagation Helmert transformation The central limit theorem The theorem Stable distributions Proof of the central limit theorem Markov chain Monte Carlo (MCMC) The multivariate case Poisson processes and waiting times Stochastic processes Three ways to generate Poisson points Waiting time paradox Order statistic of Poisson processes Various examples 157 PART II ASSIGNING PROBABILITIES Prior probabilities by transformation invariance Bertrand s paradox revisited Prior for scale variables The prior for a location variable Hyperplane priors The invariant Riemann measure (Jeffreys prior) Testable information and maximum entropy Discrete case Properties of the Shannon entropy Maximum entropy for continuous distributions Quantified maximum entropy The entropic prior Derivation of the entropic prior Saddle-point approximation for the normalization Posterior probability density 204

7 Contents vii 12.5 Regularization and good data A technical trick Application to ill-posed inversion problems Global smoothness A primer on cubic splines Second derivative prior First derivative prior Fisher information prior 221 PART III PARAMETER ESTIMATION Bayesian parameter estimation The estimation problem Loss and risk function Confidence intervals Examples Frequentist parameter estimation Unbiased estimators The maximum likelihood estimator Examples Stopping criteria for experiments Is unbiasedness desirable at all? Least-squares fitting The Cramer Rao inequality Lower bound on the variance Examples Admissibility of the Cramer Rao limit 251 PART IV TESTING HYPOTHESES The Bayesian way Some illustrative examples Independent measurements with Gaussian noise The frequentist approach Introduction Neyman Pearson lemma Sampling distributions Mean and median of i.i.d. random variables Mean and variance of Gaussian samples z-statistic Student s t-statistic Fisher Snedecor F -statistic 302

8 viii Contents 19.6 Chi-squared in case of missing parameters Common hypothesis tests Comparison of Bayesian and frequentist hypothesis tests Prior knowledge is prior data Dependence on the stopping criterion 325 PART V REAL-WORLD APPLICATIONS Regression Linear regression Models with nonlinear parameter dependence Errors in all variables Consistent inference on inconsistent data Erroneously measured uncertainties Combining incompatible measurements Unrecognized signal contributions The nuclear fission cross-section 239 Pu (n, f ) Electron temperature in a tokamak edge plasma Signal background separation Change point problems The Bayesian change point problem Change points in a binary image Neural network modelling Thin film growth detected by Auger analysis Function estimation Deriving trends from observations Density estimation Integral equations Abel s integral equation The Laplace transform The Kramers Kronig relations Noisy kernels Deconvolution Model selection Inelastic electron scattering Signal background separation Spectral line broadening Adaptive choice of pivots Mass spectrometry 484

9 Contents ix 28 Bayesian experimental design Overview of the Bayesian approach Optimality criteria and utility functions Examples N-step-ahead designs Experimental design: Perspective 504 PART VI PROBABILISTIC NUMERICAL TECHNIQUES Numerical integration The deterministic approach Monte Carlo integration Beyond the Gaussian approximation Monte Carlo methods Simple sampling Variance reduction Markov chain Monte Carlo Expectation value of the sample mean Equilibration Variance of the sample mean Taming rugged PDFs by tempering Evidence integral and partition function Nested sampling Motivation The theory behind nested sampling Application to the classical ideal gas Statistical uncertainty Concluding remarks 594 Appendix A Mathematical compendium 595 Appendix B Selected proofs and derivations 611 Appendix C Symbols and notation 619 References 620 Index 631

10

11 Preface The present book is comprehensive and application-oriented, written by physicists with the emphasis on physics-related topics. However, the general concepts, ideas and numerical techniques presented here are not restricted to physics but are equally applicable to all natural sciences, as well as to engineering. Physics is a fairly expansive discipline in the natural sciences, both financially and intellectually. Considerable efforts and financial means go into the planning, design and operation of modern physics experiments. Disappointingly less attention is usually paid to the analysis of the collected data, which hardly ever goes beyond the 200-year-old method of least squares. A possible reason for this imbalance of efforts lies in the problems which physicists encounter with traditional frequentist statistics. The great statistician G. E. Box hit this point already in 1962: I believe, for instance that it would be very difficult to persuade an intelligent physicist that current statistical practise was sensible, but there would be much less difficulty with an approach via likelihood and Bayes theorem. This citation describes fairly precisely the adventure we have experienced with growing enthusiasm during the last 20 years. Bayesian reasoning is nothing but common physicists logic, however, expressed in a rigorous and consistent mathematical form. Data analysis without a proper background in probability theory and statistics is like performing an experiment without knowing what the electronic devices are good for and how they are used properly. As we will see, guided by numerous examples, the consequent use of probability theory reveals that there is incredibly more information in the data than is usually expected. More than that: probability theory is at the heart of any science, it represents in the words of E. T. Jaynes the logic of science [104]. Besides E. T. Jaynes, M. Tribus, the former director of the Centre for Advanced Engineering Study at MIT, recognized the strength of the Bayesian approach to engineering problems, which he summarized in Rational Descriptions, Decisions and Design [206]. Strangely enough, most scientists never had a thorough education in probability theory. True enough, probability theory not only plays an albeit very important second fiddle, but in theories such as quantum mechanics (QM) and statistical physics it is at the very heart. The importance of Bayesian probability theory, particularly as far as the fundamental interpretation of QM is concerned, has been clearly outlined by L. E. Ballentine in Quantum Mechanics: A Modern Development [9]. For historical reasons, the traditional approach to statistical physics is based

12 xii Preface primarily on frequentist statistics. However, a more powerful and systematic derivation, based on Bayesian probability theory, has been presented by W. T. Grandy in Foundations of Statistical Mechanics I, II [89]. The goal of the present book is to give a comprehensive overview of probability theory and those aspects of frequentist statistics that physicists, both experimentalists and theoreticians, need to know. The book sets a homogeneous framework for all problems occurring in physics and most other sciences that are directly or indirectly amenable to probability theory. Concepts and applications are gathered which are usually fragmented over diverse books and lectures. The first half of this book is based on a course presented to physics students at the Graz University of Technology and represents a comprehensive introduction to probability theory, statistics and data analysis for physicists from a Bayesian perspective. Probability theory is increasingly important in computational physics or engineering as it forms the basis of various powerful numerical techniques. We will present some of these state-of-the-art techniques which are both interesting from the probabilistic point of view and required to solve challenging data analysis problems. It is worth mentioning that the computational effort involved in the solution of specific data analysis problems may be so high that it would have been prohibitive until a few years ago. Progress in the performance of modern computers renders this problem progressively less important. The basic concepts and ideas of Bayesian probability theory have already been presented in numerous books for a broad readership. However, in order to stimulate the interest of physicists in the Bayesian choice, it is in our opinion necessary to discuss a wide range of realistic physics applications along with a detailed discussion of the in some cases elaborate solution. The second half of this volume is therefore devoted to a wide variety of problems arising in physical data analysis and to Bayesian experimental design. This combination, we hope, will make this volume attractive for advanced students as well as for active researchers. The table on the following page contains a classification of the content of this book according to the target audience. It is intended to help the reader decide which sections are most appropriate. Finally, one of us (V. D.) wants to thank Mrs I. Zeising for preparing some parts of the manuscript and S. Gori for his never-ending patience in the preparation of figures. All three of us have benefited from the collaboration with R. Fischer and R. Preuss. The development of Bayesian activity at our institute has also profited very much from the short stays of R. Silver, A. Garrett, T. Loredo, D. Keren and the continuous information exchange with the participants of the workshop series on Bayesian Inference and Maximum Entropy Methods in Science and Engineering and ISBA. We are particularly grateful to J. Skilling for providing valuable insight into nested sampling.

13 Preface xiii Classification of the content according to the target audience Target audience Chapters Undergraduate students 1 18, without 16 Graduate students 3, 10 25, Readers interested in stochastic integration techniques Readers interested in concepts 10 13, Experimentalists interested in simple data analysis problems 3, 7, 10 12, 14, 21 problems beyond the least-squares level focus on regression 3, 7, 10 16, focus on model comparison 3, 7, 17 20, 27, focus on experimental design 3, 7, 10 14, 28, Readers interested in the probabilistic foundation and instruments of statistical physics 1 12, 29 31