An Introduction to Stochastic Modeling

F An Introduction to Stochastic Modeling Fourth Edition Mark A. Pinsky Department of Mathematics Northwestern University Evanston, Illinois Samuel Karlin Department of Mathematics Stanford University Stanford, California PT ~P\TTPV AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY TOKYO

F Contents 1.:: Preface to the Fourth Edition Preface to the Third Edition Preface to the First Edition To the Instructor Acknowledgments xi xiii XV xvii xix 1 Introduction 1 1.1 Stochastic Modeling 1 1.1.1 Stochastic Processes 4 1.2 Probability Review 4 1.2.1 Events and Probabilities 4 1.2.2 Random Variables 5 1.2.3 Moments and Expected Values 7 1.2.4 Joint Distribution Functions 8 1.2.5 Sums and Convolutions 10 1.2.6 Change of Variable 10 1.2.7 Conditional Probability 11 1.2.8 Review of Axiomatic Probability Theory 12 1.3 The Major Discrete Distributions 19 1.3.1 Bernoulli Distribution 20 1.3.2 Binomial Distribution 20 1.3.3 Geometric and Negative Binominal Distributions 21 1.3.4 The Poisson Distribution 22 1.3.5 The Multinomial Distribution 24 1.4 Important Continuous Distributions 27 1.4.1 The Normal Distribution 27 1.4.2 The Exponential Distribution 28 1.4.3 The Uniform Distribution 30 1.4.4 The Gamma Distribution 30 1.4.5 The Beta Distribution 31 1.4.6 The Joint Normal Distribution 31 1.5 Some Elementary Exercises 34 1.5.1 Tail Probabilities 34 1.5.2 The Exponential Distribution 37 1.6 Useful Functions, Integrals, and Sums 42

vi Content 2 Conditional Probability and Conditional Expectation 4~ 2.1 The Discrete Case 4 2.2 The Dice Game Craps s 2.3 Random Sums 5" 2.3.1 Conditional Distributions: The Mixed Case 5 2.3.2 The Moments of a Random Sum 5 2.3.3 The Distribution of a Random Sum 6 2.4 Conditioning on a Continuous Random Variable 6 2.5 Martingales 7 2.5.1 The Definition 7 2.5.2 The Markov Inequality 7 2.5.3 The Maximal Inequality for Nonnegative Martingales 7 3 Markov Chains: Introduction 7 3.1 Definitions 7 3.2 Transition Probability Matrices of a Markov Chain 8 3.3 Some Markov Chain Models 8 3.3.1 An Inventory Model 8 3.3.2 The Ehrenfest Urn Model 8 3.3.3 Markov Chains in Genetics g 3.3.4 A Discrete Queueing Markov Chain s 3.4 First Step Analysis s 3.4.1 Simple First Step Analyses s 3.4.2 The General Absorbing Markov Chain 1C 3.5 Some Special Markov Chains 1 J 3.5.1 The Two-State Markov Chain 1 J 3.5.2 Markov Chains Defined by Independent Random Variables 3.5.3 One-Dimensional Random Walks l! 3.5.4 Success Runs 1: 3.6 Functionals of Random Walks and Success Runs J: 3.6.1 The General Random Walk 1: 3.6.2 Cash Management J: 3.6.3 The Success Runs Markov Chain J: 3.7 Another Look at First Step Analysis J: 3.8 Branching Processes J. 3.8.1 Examples of Branching Processes I 3.8.2 The Mean and Variance of a Branching Process I 3.8.3 Extinction Probabilities '], 3.9 Branching Processes and Generating Functions 3.9.1 Generating Functions and Extinction Probabilities 3.9.2 Probability Generating Functions and Sums of Tndeoendent Random Variables I I

F Contents vii 4 The Long Run Behavior of Markov Chains 165 4.1 Regular Transition Probability Matrices 165 4.1.1 Doubly Stochastic Matrices 170 4.1.2 Interpretation of the Limiting Distribution 171 4.2 Examples 178 4.2.1 Including History in the State Description 178 4.2.2 Reliability and Redundancy 179 4.2.3 A Continuous Sampling Plan 181 4.2.4 Age Replacement Policies 183 4.2.5 Optimal Replacement Rules 185 4.3 The Classification of States 194 4.3.1 Irreducible Markov Chains 195 4.3.2 Periodicity of a Markov Chain 196 4.3.3 Recurrent and Transient States 198 4.4 The Basic Limit Theorem of Markov Chains 203 4.5 Reducible Markov Chains 215 5 Poisson Processes 223 5.1 The Poisson Distribution and the Poisson Process 223 5.1.1 The Poisson Distribution 223 5.1.2 The Poisson Process 225 5.1.3 Nonhomogeneous Processes 226 5.1.4 Cox Processes 227 5.2 The Law of Rare Events 232 5.2.1 The Law of Rare Events and the Poisson Process 234 5.2.2 Proof of Theorem 5.3 237 5.3 Distributions Associated with the Poisson Process 241 5.4 The Uniform Distribution and Poisson Processes 247 5.4.1 Shot Noise 253 5.4.2 Sum Quota Sampling 255 5.5 Spatial Poisson Processes 259 5.6 Compound and Marked Poisson Processes 264 5.6.1 Compound Poisson Processes 264 5.6.2 Marked Poisson Processes 267 6 Continuous Time Markov Chains 277 6.1 Pure Birth Processes 277 6.1.1 Postulates for the Poisson Process 277 6.1.2 Pure Birth Process 278 6.1.3 The Yule Process 282 6.2 Pure Death Processes 286 6.2.1 The Linear Death Process 287

8 Brownian Motion and Related Processes 3 8.1 Brownian Motion and Gaussian Processes 3S 8.1.1 A Little History 3~ 8.1.2 The Brownian Motion Stochastic Process 8.1.3 The Central Limit Theorem and the Invariance Principle 3! 8.1.4 Gaussian Processes 3! 8.2 The Maximum Variable and the Reflection Principle 41 8.2.1 The Reflection Principle 41 8.2.2 The Time to First Reach a Level 41 8.2.3 The Zeros of Brownian Motion 41 8.3 Variations and Extensions 4 8.3.1 Reflected Brownian Motion 4 8.3.2 Absorbed Brownian Motion 4 Rii The Brownian Bridge 4 viii Content 6.3 Birth and Death Processes 295 6.3.1 Postulates 295 6.3.2 Sojourn Times 296 6.3.3 Differential Equations of Birth and Death Processes 299 6.4 The Limiting Behavior of Birth and Death Processes 304 6.5 Birth and Death Processes with Absorbing States 316 6.5.1 Probability of Absorption into State 0 31E 6.5.2 Mean Time Until Absorption 31~ 6.6 Finite-State Continuous Time Markov Chains 321 6.7 A Poisson Process with a Markov Intensity 33~ 7 Renewal Phenomena 34'i 7.1 Definition of a Renewal Process and Related Concepts 34~ 7.2 Some Examples of Renewal Processes 35: 7.2.1 Brief Sketches of Renewal Situations 35: 7.2.2 Block Replacement 35< 7.3 The Poisson Process Viewed as a Renewal Process 35: 7.4 The Asymptotic Behavior of Renewal Processes 36: 7.4.1 The Elementary Renewal Theorem 36. 7.4.2 The Renewal Theorem for Continuous Lifetimes 36 7.4.3 The Asymptotic Distribution of N(t) 36 7.4.4 The Limiting Distribution of Age and Excess Life 36 7.5 Generalizations and Variations on Renewal Processes 37 7.5.1 Delayed Renewal Processes 37 7.5.2 Stationary Renewal Processes 37 7.5.3 Cumulative and Related Processes 37 7.6 Discrete Renewal Theory 37 7.6.1 The Discrete Renewal Theorem 3~ 7.6.2 Deterministic Population Growth with Age Distribution 3~

f Contents ix 8.4 Brownian Motion with Drift 419 8.4.1 The Gambler's Ruin Problem 420 8.4.2 Geometric Brownian Motion 424 8.5 The Omstein-Uhlenbeck Process 432 8.5.1 A Second Approach to Physical Brownian Motion 434 8.5.2 The Position Process 437 8.5.3 The Long Run Behavior 439 8.5.4 Brownian Measure and Integration 441 9 Queueing Systems 9.1 Queueing Processes 9.1.1 The Queueing Formula L = A. W 9.1.2 A Sampling of Queueing Models 9.2 Poisson Arrivals, Exponential Service Times 9.2.1 The MIMil System 9.2.2 TheMIMioo System 9.2.3 The MIMI s System 9.3 General Service Time Distributions 9.3.1 TheMIGI1 System 9.3.2 The MIGioo System 9.4 Variations and Extensions 9.4.1 Systems with Balking 9.4.2 Variable Service Rates 9.4.3 A System with Feedback 9.4.4 A Two-Server Overflow Queue 9.4.5 Preemptive Priority Queues 9.5 Open Acyclic Queueing Networks 9.5.1 The Basic Theorem 9.5.2 Two Queues in Tandem 9.5.3 Open Acyclic Networks 9.5.4 Appendix: Time Reversibility 9.5.5 Proof of Theorem 9.1 9.6 General Open Networks 9.6.1 The General Open Network 447 447 448 449 451 452 456 457 460 460 465 468 468 469 470 470 472 480 480 481 482 485 487 488 492 10 Random Evolutions 495 10.1 Two-State Velocity Model 495 10.1.1 Two-State Random Evolution 498 10.1.2 The Telegraph Equation 500 10.1.3 Distribution Functions and Densities in the Two-State Model 501 10.1.4 Passage Time Distributions 505 10.2 N-State Random Evolution 507 10.2.1 Finite Markov Chains and Random Velocity Models 507

X Content 10.2.3 Random Evolution Processes 10.2.4 Existence-Uniqueness of the First-Order System (10.26) 10.2.5 Single Hyperbolic Equation 10.2.6 Spectral Properties of the Transition Matrix 10.2.7 Recunence Properties of Random Evolution 10.3 Weak Law and Central Limit Theorem 10.4 Isotropic Transport in Higher Dimensions 10.4.1 The Rayleigh Problem of Random Flights 10.4.2 Three-Dimensional Rayleigh Model 508 509 51C 512 51: 51( 521 521 52: 11 Characteristic Functions and Their Applications 52! 11.1 Definition of the Characteristic Function 52~ 11.1.1 Two Basic Properties of the Characteristic Function 52< 11.2 Inversion Formulas for Characteristic Functions 52 11.2.1 Fourier Reciprocity/Local Non-Uniqueness 53( 11.2.2 Fourier Inversion and Parseval's Identity 53 11.3 Inversion Formula for General Random Variables 53: 11.4 The Continuity Theorem 53. 11.4.1 Proof of the Continuity Theorem 53-11.5 Proof of the Central Limit Theorem 53 11.6 Stirling's Formula and Applications 53 11.6.1 Poisson Representation of n! 53 11.6.2 Proof of Stirling's Formula 53 11.7 Local demoivre-laplace Theorem 53 Further Reading Answers to Exercises Index 54 54 55