Index. Eigenvalues and eigenvectors of [Pl, Elementary renewal theorem, 96

Size: px

Start display at page:

Download "Index. Eigenvalues and eigenvectors of [Pl, Elementary renewal theorem, 96"

Annice Lindsey
5 years ago
Views:

Bibliography [BeI57], Bellman, R., Dynamic Programming, Princeton University Press, Princeton, N.J., 1957. [Ber87] Bertsekas, D. P., Dynamic Programming-Deterministic and Stochastic Models, Prentice Hall, N.

1 Bibliography [BeI57], Bellman, R., Dynamic Programming, Princeton University Press, Princeton, N.J., [Ber87] Bertsekas, D. P., Dynamic Programming-Deterministic and Stochastic Models, Prentice Hall, N.J., [BhW90] Bhattacharya, RN. & E.C. Waymire, Stochastic Processes with Applications, Wiley, [DeZ93] Dembo, A. & O. Zeitouni, Large Deviation Techniques and Applications, Jones & Bartlett Publishers, [Do053] Doob, 1. L., Stochastic Processes, Wiley, New York [Dra67] Drake, A., Fundamentals of Applied Probability Theory, McGraw Hill, [FeI68] Feller, w., An Introduction to Probability Theory and its Applications, vol 1, Third Edition, Wiley, N.Y., [FeI66] Feller, w., An Introduction to Probability Theory and its Applications, vol 2, Wiley, N.Y., [Gan59] Gantmacher, Applications of the Theory of Matrices, (English Translation), Interscience, N.Y., [Har63] Harris, T. E., The Theory of Branching Processes, Springer Verlag, Berlin, and Prentice Hall, Englewood Cliffs, N.J., [How60] Howard, R, Dynamic Programming and Markov Processes, M.I.T. Press, [Ke179] Kelly, F. P., Reversibility and Stochastic Networks, Wiley & Sons, [KoI50] Kolmogorov, A. N., Foundations of the Theory of Probability, Chelsea [Odo69] Odoni, A.R, "On finding the maximal gain for Markov Decision processes," Operations Res. 17 (1969) pp [Ros83] Ross, S., Stochastic Processes, Wiley & Sons, [Ros94] Ross, S., A First Course in Probability, 4th Ed., Mcmillan & Co [ScF67] Schweitzer, P. 1. &A. Federgruen, "The Asymptotic Behavior of Un discounted Value Iteration in Markov Decision Problems," Math. of Op. Res., 2, Nov. 1967, pp [Str88] Strang, G., Linear Algebra and its Applications, Third Edition, Harcourt, Brace, Jovanovich, Fort Worth, [WoI89] Wolff, R w., Stochastic Modeling and the Theory of Queues, Prentice Hall, [Yat90] Yates, R D., High Speed Round Robin Queueing Networks, LIDS-TH-1983, Laboratory of Information and Decision Systems, M.LT., Cambridge, MA.,

2 Index Accessible states, los, 150 Age, 96; ensemble average, 77-88; time average, Aggregate reward, 121 ; per inter-renewal interval, 73-74, 83 Aperiodic states, 106 Arithmetic distributions, 68 Arithmetic renewal process, 69 Arrival epochs, 32,57 Arrival processes, 1-3,31-33,82 Asymptotic relative gain vector, 125 Baby Bernoulli process, 37-38, 42, 50 Backward transition rates, 200 Bernoulli random variable, 25 Birth death Markov chains, 149, , 163, ; reversibility, 202 Blackwell's theorem, 68-69,153 Branching processes, ; limiting behavior, 256; scaled as Martingale, 244; variance, 182 Bulk arrivals, 92 Burke's theorem, 202; sampled time, 168, 184 Cauchy random variable, 21 Central limit theorem, 15-18; for renewal processes, 61 Chapman Kolmogorov equations: for Markov chain, 110; for Markov process, 192 Characteristic function, 12 Chebyshev inequality, 13 Chernoff bound, 14 Class; of states, 105 Classification of states, , Communicating states, 105, 150 Complementary distribution function, 8 Conditional probability,s Continuous time processes, 103 Convergence: in mean square sense, 14; in probability, 15; with probability one, 23 Convex functions, Convolution equation, 10 Countable, 30 Counting process, 32, 57,103; random rate, 51 Decision maker, 119 Decision theory, Delayed renewal processes, 87-92; Blackwell's theorem, 89; elementary renewal theorem, 89; strong law, 88; Transient behavior, Delayed renewal reward processes, Detection, Directed graph, 104 Directed path, los, 147 Directed walk, 105, 147 Directly Riemann integrable, 80 Distribution function, 6; arithmetic, 68; complementary, 8; joint, 7; marginal,7 Doubly stochastic matrix, 137 Duration, 96; ensemble average, 77-78; time average, 73 Dynamic programming, 119, ; algorithm, ; monotonicity property, 133 Eigenvalues and eigenvectors of [Pl, Elementary renewal theorem,

3 268 Index Embedded Markov chain, 174, 187 Endogenous inputs, 205 Epochs, 31; anivru, 32 Equilibrium process, Ergodic chain, 107 Ergodic class, 107 Ergodic Markov chains, Ergodic plus transient, 118 Ergodic theory, 60 Erlang random variables, 25, 35, 49, 93 Events, 1-3 Exogenous inputs, 205 Expectation, 8-11; number of renewrus, Expected gain: steady state, 123 Exponential bound, 14 Exponential random variables, 25, 32 FCFS. See First come first serve FIFO. See First in first out First come first serve, 82 First in first out, 102 First passage times, ll9-120, ISO Flags, 100ll Frobenius theorem, lis G/GII queue, 87, , 234; waiting time distribution, 226 G/G/m queue, 57 Gamma density, 35 Gaussian distribution, 16 Gaussian random variables, 25 Geometric random variables, 25 Graph: of Markov chain, 104; of Markov process, 188 Homogenous Markov chains, 104 Howard's policy improvement rugorithm, Hypothesis testing, , 262 lid. See Independent identically distributed Independence: conditional, 5; of events, 5; of random variables, 7; pairwise, 27 Independent identically distributed, 32 Independent increment property, 35 Indicator functions, 18; for stopping rules, 65 Inequalities, 12; Chebyshev, 13; Chernoff, 14; Exponentiru, 14; Markov, Inf,30 Infimum, 30 Integer time processes, 103 Inter-anivru intervals, 31, 57 Inter-renewru intervals for a given state, 152 Irreduci1;lle Markov chains, 155 Irregular Markov processes, Jackson networks, ; closed, 2ll- 213; queue dependent service rate, 211; with multiple servers per node, 221 Jensen's inequruity, 247 Joint distribution function, 7 Joint PMF, 7 Joint probability density, 7 Jordan form, 1I2 Key renewru theorem, Kingman bound, 234 Kolmogorov differentiru equations, ; eigenvector solution, 194; Laplace transform solution, 195; matrix solution, Kolmogorov inequruities, Laplace transform, 12; of Kolmogorov differentiru equation, 195; for expected number of renewals, Large deviation theory, 229 Largest real eigenvalue, 115 Last come first serve MlG/I queues, Laws of large numbers, 12-25; strong law, 21-25; weak law, LCFS. See Last come first serve Left moving chain, 166 Little's theorem, 81-86, 173 Log likelihood ratios, 228 M/GIl queue, 85-87, 169, 174, ; LCFS, MlG/oo queue, 43-44, 52 MlMll queue,

4 Index 269 MlMII sampled time Markov chain, , 173 MlMIm queue, 198 MAP rule, 228 Markov chains, ; birth death, 149, ,163; classification of states, ; countable state spaces, ; definition, 103; embedded, 174; finite state, ; graph representation, ; homogenous, 104; irreducible, 155; MlMII sampled time, , 173; matrix representation, 104, ; non-homogenous, 104; reversibility, ; state, 104; with rewards, Markov decision theory, 119, Markov inequality, I :b1.1 MW'kr;v!TIQdijjij!i.)4 random WtllK3, 241; walas Im:nnty. ;nz; wtul Markov processes, i87-222; embedded Markov chain, 187; epochs of transitions, 243; graph, 188; irreducible, 190; irregular, , Kolmogorov differential equations, ; pathological cases, ; reversibility, ; sampled time approximation, 189; steady state equations, 190; transition rate from a state, 187; transition rate from i to j, 188; uniformitization, Martingale convergence theorem, 256 Martingales, 223, ; binary product, 240, 250; convex functions of, ; Kolmogorov inequality, 253; product form, ; relative to joint processes, 243; sums of dependent variables, 240 Maximum aposteriori probability, 228 Maximum likelihood detection, 228 Maximum of random variables, 26 Memoryless property of Poisson process Minimum of random variables, 26 Moment generating function, Murphy's law, 181 Neyman Pear.on test, 228, 259 Non-homogeneous Markov chains, 104 Normal distribution, 16 Null recurrent states, (.),36 Odoni bound, 145 Order statistics, Overshoots, "':?.5, 235 PASTA property, 102 Periodic states, 106 Perron theorem, Peron-Frobenius theory, PMF. See probability mass function P9licy maker. 119 Poisson p r O e e, 31-"6; A! c ~ c6m\)inaliolll, j(l-j9; d6flmi(1!!, JJ, 36, '.n,!!!!! ~ f ~1! ~ U i! iu 1U. J 8 m~ E~ 3i U ~, memoryli! Pf(jP[jI'fY. :n-33; non-homogenous, 41-43; stationary increments, 35; subdividing (splitting), 39-41, 44; two dimensional, 54 Poisson random variables, 25, 27, 36 Policy, 127; Howard's algorithm, ; optimal dynamic, ; optimal stationary, ; stationary, 127 Pollaczek-Khinchin formula, 87 Positive recurrent states, Possibly defective stopping rules, 248 Probability: axioms, 4; conditional, 5; density, 7; distribution function, 6; mass function, 7; model, 2; one, 5; zero, 4; experiment, 1 Processor sharing, Queueing: G/GIl, 87, , 234; G/G/ m, 57; Mld/mlm, 221; MIGIl, 85-87,169,174, ;MI G/oo,43-44,52;MlMIl, ; MlMII sampled time, ; M/M/m, ; standard notation, 43; tandem, 203; with feedback, 204

270 Index Random incidence, 76 Random processes, 1 Random variables, 5-12; Bernoulli, 25; Cauchy, 21; complex, 6; defective, 6; distribution function, 6; Erlang, 25,35,49, 93;exponential,25,32;

5 270 Index Random incidence, 76 Random processes, 1 Random variables, 5-12; Bernoulli, 25; Cauchy, 21; complex, 6; defective, 6; distribution function, 6; Erlang, 25,35,49, 93;exponential,25,32; Gaussian, 16,25; geometric, 25; maximum and minimum, 26; Poisson, 25, 27, 36; tilted, 260; uncorrelated, 27; uniform, 25, 28, 46-47; vector, 6 Random walks, ; integer valued, 224; Kolmogorov inequality, 254; overshoots, 225; relation to renewal process, 224; simple, ; with thresholds, 250; with two thresholds, Rate of a Poisson process, Recurrent class of states, 105 Recurrent plus transient, 116 Recurrent states, 105, Relative frequency, Relative gain vector, 123, 125, 134 Renewal equation, 63, 93 Renewal processes, 31-32, , 151; applied to semi-markov processes, ; arithmetic, 81, 156; generalized with P(X>O»O, 92, 101; nonarithmetic, 81 Renewal reward processes, 69-87; delayed, 90; ensemble averages, 77-87, 100; time averages, Renewal theorems: Blackwell, 68-69; delayed processes, 88-89; elementary, 67-68, 96; Key, 80-81; strong law, Residual life, 67,96; distribution, 75-76; of customer in service, 85-86; time average, Reversibility: for birth death chains, 163; for birth death processes, 202; for MOO1 queues, 202; for M/ MIl sampled time chains, ; for Markov chains, ; for Markov processes, Reward: aggregate, ; steady state, 119; transient, 119 Reward functions: for renewal processes, 69-70,74-75 Right moving chain, 166 Round robin service, Sample functions, 18 Sample median, 52 Sample points, 1-3 Sample space, 1-3 Sampled time approximation to Markov process, 189 Semi-invariant moment generating function, 230 Semi-Markov processes, ; epochs of transitions, 243; fraction of time in each state, 178; fraction of time in each transition, 178; pathological cases, 192 Sequential analysis, 65, 229 Slow truck effect, 87,169,174 Span of arithmetic distribution, 68 Stages, 122 Standard deviation, 9 State of Markov chain, 104 S tate rewards, 119 Stationary distribution, 110 Stationary increment property, 35 Statistics, 2 Steady state distribution, 110, Steady state equations for Markov processes, 190 Steady state probabilities, 149,207 Steady state probability vector, 110 Stieitjes integral, 8 Stochastic matrix, 109 Stochastic processes, 1; arrival, 1; continuous time, 103; discrete, 2; independent processes, 55; integer time, 103 Stopped processes, Stopping rules, 65-66, 93, 94, ; for martingales relative to a process, 251; possibly defective,248 Strong law oflarge numbers, 21-25; for renewal processes, 58-61; proof,

6 Index 271 Submartingales, ; Kolmogorov inequality, 253 Sup, 30 Supermartingales, Supremum, 30 Tandem queues, Threshold tests, 228 Thresholds, 250; Markov modulated random walks, ; crossing probabilities, ; crossing time, Tilted random variables, 260 Transforms, Transient classes, 105 Transient states, 105, Transient probabilities, 104; eigenvalues and eigenvectors, ; n'th step, 109 Transition rates: from state to state, 188; out of a state, 187 Transition rewards, 119 Trapping states, , 158 Truncation, 19-21,60 Truncation ofreversible chains, 182 Uncorrelated random variables, 27 Unfinished work, Uniform random variables, 25, 28, Uniformization, Variance,9 Wald's equality, 64-67, 94, 233 Wald's identity, ; Markov modulated random walks, 252; proof, Weak law ofjarge numbers, 12-21,28; finite variance, 14-15; infinite variance, Z transform, 12

Stochastic Processes. Theory for Applications. Robert G. Gallager CAMBRIDGE UNIVERSITY PRESS

Stochastic Processes. Theory for Applications. Robert G. Gallager CAMBRIDGE UNIVERSITY PRESS Stochastic Processes Theory for Applications Robert G. Gallager CAMBRIDGE UNIVERSITY PRESS Contents Preface page xv Swgg&sfzoMj ybr zmjfr%cforj owf fmdy xix Acknowledgements xxi 1 Introduction and review