A First Course in Stochastic Models. Henk C. Tijms Vrije Universiteit, Amsterdam, The Netherlands

Size: px

Start display at page:

Download "A First Course in Stochastic Models. Henk C. Tijms Vrije Universiteit, Amsterdam, The Netherlands"

Barry Green
5 years ago
Views:

1 A First Course in Stochastic Models Henk C. Tijms Vrije Universiteit, Amsterdam, The Netherlands

3 A First Course in Stochastic Models

5 A First Course in Stochastic Models Henk C. Tijms Vrije Universiteit, Amsterdam, The Netherlands

6 Copyright c 2003 John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) (for orders and customer service enquiries): cs-books@wiley.co.uk Visit our Home Page on or All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or ed to permreq@wiley.co.uk, or faxed to (+44) This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA , USA Wiley-VCH Verlag GmbH, Boschstr. 12, D Weinheim, Germany John Wiley & Sons Australia Ltd, 33 Park Road, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Library of Congress Cataloging-in-Publication Data Tijms, H. C. A first course in stochastic models / Henk C. Tijms. p. cm. Includes bibliographical references and index. ISBN (acid-free paper) ISBN (pbk. : acid-free paper) 1. Stochastic processes. I. Title. QA274.T dc British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN (Cloth) ISBN (Paper) Typeset in 10/12pt Times from LATEX files supplied by the author, by Laserwords Private Limited, Chennai, India Printed and bound in Great Britain by T J International Ltd, Padstow, Cornwall This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production.

7 Contents Preface ix 1 The Poisson Process and Related Processes Introduction The Poisson Process The Memoryless Property Merging and Splitting of Poisson Processes The M/G/ Queue The Poisson Process and the Uniform Distribution Compound Poisson Processes Non-Stationary Poisson Processes Markov Modulated Batch Poisson Processes 24 Exercises 28 Bibliographic Notes 32 References 32 2 Renewal-Reward Processes Introduction Renewal Theory The Renewal Function The Excess Variable Renewal-Reward Processes The Formula of Little Poisson Arrivals See Time Averages The Pollaczek Khintchine Formula A Controlled Queue with Removable Server An Up- And Downcrossing Technique 69 Exercises 71 Bibliographic Notes 78 References 78 3 Discrete-Time Markov Chains Introduction The Model 82

8 vi CONTENTS 3.2 Transient Analysis Absorbing States Mean First-Passage Times Transient and Recurrent States The Equilibrium Probabilities Preliminaries The Equilibrium Equations The Long-run Average Reward per Time Unit Computation of the Equilibrium Probabilities Methods for a Finite-State Markov Chain Geometric Tail Approach for an Infinite State Space Metropolis Hastings Algorithm Theoretical Considerations State Classification Ergodic Theorems 126 Exercises 134 Bibliographic Notes 139 References Continuous-Time Markov Chains Introduction The Model The Flow Rate Equation Method Ergodic Theorems Markov Processes on a Semi-Infinite Strip Transient State Probabilities The Method of Linear Differential Equations The Uniformization Method First Passage Time Probabilities Transient Distribution of Cumulative Rewards Transient Distribution of Cumulative Sojourn Times Transient Reward Distribution for the General Case 176 Exercises 179 Bibliographic Notes 185 References Markov Chains and Queues Introduction The Erlang Delay Model The M/M/1 Queue The M/M/c Queue The Output Process and Time Reversibility Loss Models The Erlang Loss Model The Engset Model Service-System Design Insensitivity A Closed Two-node Network with Blocking The M/G/1 Queue with Processor Sharing A Phase Method 209

9 CONTENTS vii 5.6 Queueing Networks Open Network Model Closed Network Model 219 Exercises 224 Bibliographic Notes 230 References Discrete-Time Markov Decision Processes Introduction The Model The Policy-Improvement Idea The Relative Value Function Policy-Iteration Algorithm Linear Programming Approach Value-Iteration Algorithm Convergence Proofs 267 Exercises 272 Bibliographic Notes 275 References Semi-Markov Decision Processes Introduction The Semi-Markov Decision Model Algorithms for an Optimal Policy Value Iteration and Fictitious Decisions Optimization of Queues One-Step Policy Improvement 295 Exercises 300 Bibliographic Notes 304 References Advanced Renewal Theory Introduction The Renewal Function The Renewal Equation Computation of the Renewal Function Asymptotic Expansions Alternating Renewal Processes Ruin Probabilities 326 Exercises 334 Bibliographic Notes 337 References Algorithmic Analysis of Queueing Models Introduction Basic Concepts 341

10 viii CONTENTS 9.2 The M/G/1 Queue The State Probabilities The Waiting-Time Probabilities Busy Period Analysis Work in System The M X /G/1 Queue The State Probabilities The Waiting-Time Probabilities M/G/1 Queues with Bounded Waiting Times The Finite-Buffer M/G/1 Queue An M/G/1 Queue with Impatient Customers The GI /G/1 Queue Generalized Erlangian Services Coxian-2 Services The GI /P h/1 Queue The Ph/G/1 Queue Two-moment Approximations Multi-Server Queues with Poisson Input The M/D/c Queue The M/G/c Queue The MX/G/c Queue The GI /G/c Queue The GI /M/c Queue The GI /D/c Queue Finite-Capacity Queues The M/G/c/c + N Queue A Basic Relation for the Rejection Probability The M X /G/c/c + N Queue with Batch Arrivals Discrete-Time Queueing Systems 417 Exercises 420 Bibliographic Notes 428 References 428 Appendices 431 Appendix A. Useful Tools in Applied Probability 431 Appendix B. Useful Probability Distributions 440 Appendix C. Generating Functions 449 Appendix D. The Discrete Fast Fourier Transform 455 Appendix E. Laplace Transform Theory 458 Appendix F. Numerical Laplace Inversion 462 Appendix G. The Root-Finding Problem 470 References 474 Index 475

11 Preface The teaching of applied probability needs a fresh approach. The field of applied probability has changed profoundly in the past twenty years and yet the textbooks in use today do not fully reflect the changes. The development of computational methods has greatly contributed to a better understanding of the theory. It is my conviction that theory is better understood when the algorithms that solve the problems the theory addresses are presented at the same time. This textbook tries to recognize what the computer can do without letting the theory be dominated by the computational tools. In some ways, the book is a successor of my earlier book Stochastic Modeling and Analysis. However, the set-up of the present text is completely different. The theory has a more central place and provides a framework in which the applications fit. Without a solid basis in theory, no applications can be solved. The book is intended as a first introduction to stochastic models for senior undergraduate students in computer science, engineering, statistics and operations research, among others. Readers of this book are assumed to be familiar with the elementary theory of probability. I am grateful to my academic colleagues Richard Boucherie, Avi Mandelbaum, Rein Nobel and Rien van Veldhuizen for their helpful comments, and to my students Gaya Branderhorst, Ton Dieker, Borus Jungbacker and Sanne Zwart for their detailed checking of substantial sections of the manuscript. Julian Rampelmann and Gloria Wirz-Wagenaar were helpful in transcribing my handwritten notes into a nice Latex manuscript. Finally, users of the book can find supporting educational software for Markov chains and queues on my website

13 CHAPTER 1 The Poisson Process and Related Processes 1.0 INTRODUCTION The Poisson process is a counting process that counts the number of occurrences of some specific event through time. Examples include the arrivals of customers at a counter, the occurrences of earthquakes in a certain region, the occurrences of breakdowns in an electricity generator, etc. The Poisson process is a natural modelling tool in numerous applied probability problems. It not only models many real-world phenomena, but the process allows for tractable mathematical analysis as well. The Poisson process is discussed in detail in Section 1.1. Basic properties are derived including the characteristic memoryless property. Illustrative examples are given to show the usefulness of the model. The compound Poisson process is dealt with in Section 1.2. In a Poisson arrival process customers arrive singly, while in a compound Poisson arrival process customers arrive in batches. Another generalization of the Poisson process is the non-stationary Poisson process that is discussed in Section 1.3. The Poisson process assumes that the intensity at which events occur is time-independent. This assumption is dropped in the non-stationary Poisson process. The final Section 1.4 discusses the Markov modulated arrival process in which the intensity at which Poisson arrivals occur is subject to a random environment. 1.1 THE POISSON PROCESS There are several equivalent definitions of the Poisson process. Our starting point is a sequence X 1,X 2,... of positive, independent random variables with a common probability distribution. Think of X n as the time elapsed between the (n 1)th and nth occurrence of some specific event in a probabilistic situation. Let n S 0 = 0 and S n = X k, n = 1, 2,.... A First Course in Stochastic Models H.C. Tijms c 2003 John Wiley & Sons, Ltd. ISBNs: (HB); (PB) k=1

14 2 THE POISSON PROCESS AND RELATED PROCESSES Then S n is the epoch at which the nth event occurs. For each t 0, define the random variable N(t) by N(t) = the largest integer n 0 for which S n t. The random variable N(t) represents the number of events up to time t. Definition The counting process {N(t), t 0} is called a Poisson process with rate λ if the interoccurrence times X 1,X 2,... have a common exponential distribution function P {X n x} =1 e λx, x 0. The assumption of exponentially distributed interoccurrence times seems to be restrictive, but it appears that the Poisson process is an excellent model for many real-world phenomena. The explanation lies in the following deep result that is only roughly stated; see Khintchine (1969) for the precise rationale for the Poisson assumption in a variety of circumstances (the Palm Khintchine theorem). Suppose that at microlevel there are a very large number of independent stochastic processes, where each separate microprocess generates only rarely an event. Then at macrolevel the superposition of all these microprocesses behaves approximately as a Poisson process. This insightful result is analogous to the well-known result that the number of successes in a very large number of independent Bernoulli trials with a very small success probability is approximately Poisson distributed. The superposition result provides an explanation of the occurrence of Poisson processes in a wide variety of circumstances. For example, the number of calls received at a large telephone exchange is the superposition of the individual calls of many subscribers each calling infrequently. Thus the process describing the overall number of calls can be expected to be close to a Poisson process. Similarly, a Poisson demand process for a given product can be expected if the demands are the superposition of the individual requests of many customers each asking infrequently for that product. Below it will be seen that the reason of the mathematical tractability of the Poisson process is its memoryless property. Information about the time elapsed since the last event is not relevant in predicting the time until the next event The Memoryless Property In the remainder of this section we use for the Poisson process the terminology of arrivals instead of events. We first characterize the distribution of the counting variable N(t). To do so, we use the well-known fact that the sum of k independent random variables with a common exponential distribution has an Erlang distribution. That is,

15 THE POISSON PROCESS 3 k 1 λt (λt)j P {S k t} =1 e, t 0. (1.1.1) j! j=0 The Erlang (k, λ) distribution has the probability density λ k t k 1 e λt /(k 1)!. Theorem For any t>0, λt (λt)k P {N(t) = k} =e, k = 0, 1,.... (1.1.2) k! That is, N(t) is Poisson distributed with mean λt. Proof The proof is based on the simple but useful observation that the number of arrivals up to time t is k or more if and only if the kth arrival occurs before or at time t. Hence P {N(t) k} =P {S k t} k 1 λt (λt)j = 1 e. j! The result next follows from P {N(t) = k} =P {N(t) k} P {N(t) k + 1}. The following remark is made. To memorize the expression (1.1.1) for the distribution function of the Erlang (k, λ) distribution it is easiest to reason in reverse order: since the number of arrivals in (0,t) is Poisson distributed with mean λt and the kth arrival time S k is at or before t only if k or more arrivals occur in (0,t), it follows that P {S k t} = j=k e λt (λt) j /j!. j=0 The memoryless property of the Poisson process Next we discuss the memoryless property that is characteristic for the Poisson process. For any t 0, define the random variable γ t as γ t = the waiting time from epoch t until the next arrival. The following theorem is of utmost importance. Theorem For any t 0, the random variable γ t has the same exponential distribution with mean 1/λ. That is, independently of t. P {γ t x} =1 e λx, x 0, (1.1.3)

A First Course in Stochastic Models. Henk C. Tijms Vrije Universiteit, Amsterdam, The Netherlands

A First Course in Stochastic Models Henk C. Tijms Vrije Universiteit, Amsterdam, The Netherlands A First Course in Stochastic Models A First Course in Stochastic Models Henk C. Tijms Vrije Universiteit,