Information, Physics, and Computation

Transcription

1 Information, Physics, and Computation Marc Mezard Laboratoire de Physique Thdorique et Moales Statistiques, CNRS, and Universit y Paris Sud Andrea Montanari Department of Electrical Engineering and Department of Statistics, Stanford University, and Laboratoire de Physique Thdorique de l'ens, Paris OXFORD UNIVERSITY PRESS

2 Contents PART 1 BACKGROUND 1 Introduction to information theory Random variables Entropy Sequences of random variables and their entropy rate Correlated variables and mutual information Data compression Data transmission 16 Notes 21 2 Statistical physics and probability theory The Boltzmann distribution Thermodynamic potentials The fluctuation-dissipation relations The thermodynamic limit Ferromagnets and Ising models The Ising spin glass 44 Notes 46 3 Introduction to combinatorial optimization A first example: The minimum spanning tree General definitions More examples Elements of the theory of computational complexity Optimization and statistical physics Optimization and coding 61 Notes 62 4 A probabilistic toolbox Many random variables: A qualitative preview Large deviations for independent variables Correlated variables The Gibbs free energy The Monte Carlo method Simulated annealing Appendix: A physicist's approach to Sanov's theorem 87 Notes 89

3 x Contents PART II INDEPENDENCE 5 The random energy model Definition of the model Thermodynamics of the REM The condensation phenomenon A comment on quenched and annealed averages The random subcube model 103 Notes The random code ensemble Code ensembles The geometry of the random code ensemble Communicating over a binary symmetric channel Error-free communication with random codes Geometry again: Sphere packing Other random codes A remark on coding theory and disordered systems Appendix: Proof of Lemma Notes Number partitioning A fair distribution into two groups? Algorithmic issues Partition of a random list: Experiments The random tost model Partition of a random list: Rigorous results 140 Notes Introduction to replica theory Replica solution of the random energy model The fully connected p-spin glass model Extreme value statistics and the REM Appendix: Stability of the RS saddle point 166 Notes 169 PART III MODELS ON GRAPHS 9 Factor graphs and graph ensembles Factor graphs Ensembles of factor graphs: Definit ions Random factor graphs: Basic properties Random factor graphs: The giant component The locally tree-like structure of random graphs 191 Notes Satisfiability The satisfiability problem 197

4 Contents xi 10.2 Algorithms Random K-satisfiability ensembles Random 2-SAT The phase transition in random K(> 3)-SAT 209 Notes Low-density parity-check codes Definitions The geometry of the codebook LDPC codes for the binary symmetric channel A simple decoder: Bit flipping 236 Notes Spin glasses Spin glasses and factor graphs Spin glasses: Constraints and frustration What is a glass phase? An example: The phase diagram of the SK model 262 Notes Bridges: Inference and the Monte Carlo method Statistical inference The Monte Carlo method: Inference via sampling Free-energy barriers 281 Notes 287 PART IV SHORT-RANGE CORRELATIONS 14 Belief propagation Two examples Belief propagation an tree graphs Optimization: Max-product and min-sum Loopy BP General message-passing algorithms Probabilistic analysis 317 Notes Decoding with belief propagation BP decoding: The algorithm Analysis: Density evolution BP decoding for an erasure channel The Bethe free energy and MAP decoding 347 Notes The assignment problem The assignment problem and random assignment ensembles Message passing and its probabilistic analysis A polynomial message-passing algorithm 366

5 xii Contents 16.4 Combinatorial results An exercise: Multi-index assignment 376 Notes Ising models an random graphs The BP equations for Ising spins RS cavity analysis Ferromagnetic model Spin glass models 391 Notes 399 PART V LONG-RANGE CORRELATIONS 18 Linear equations with Boolean variables Definitions and general remarks Belief propagation Core percolation and BP The SAT UNSAT threshold in random XORSAT The Hard-SAT phase: Clusters of solutions An alternative approach: The cavity method 422 Notes The 1RSB cavity method Beyond BP: Many states The 1RSB cavity equations A first application: XORSAT The special value x = Survey propagation The nature of 1RSB phases Appendix: The SP(y) equations for XORSAT 463 Notes Random K-satisfiability 20.1 Belief propagation and the replica-symmetric analysis 20.2 Survey propagation and the 1RSB phase 20.3 Some ideas about the full phase diagram 20.4 An exercise: Colouring random graphs Notes 21 Glassy states in coding theory 21.1 Local search algorithms and metastable states 21.2 The binary erasure channel 21.3 General binary memoryless Symmetrie channels 21.4 Metastable states and near-codewords Notes 22 An ongoing story 22.1 Gibbs measures and Jong-range correlations

6 Contents xiii 22.2 Higher levels of replica symmetry breaking Phase structure and the behaviour of algorithms 535 Notes 538 Appendix A Symbols and notation 541 A.1 Equivalence relations 541 A.2 Orders of growth 542 A.3 Combinatorics and probability 543 A.4 Summary of mathematical notation 544 A.5 Information theory 545 A.6 Factor graphs 545 A.7 Cavity and message-passing methods 545 References 547 Index 565