Probability Theory, Random Processes and Mathematical Statistics

Probability Theory, Random Processes and Mathematical Statistics

Mathematics and Its Applications Managing Editor: M.HAZEWINKEL Centre for Mathematics and Computer Science, Amsterdam, The Netherlands Volume 344

Probability Theory, Random Processes and Mathematical Statistics by Yu. A. Rozanov Steklov Mathematicallnstitute, Moscow, Russia 1iII... " SPRINGER SCIENCE+BUSINESS, MEDIA, B.V.

A C.I.P. Catalogue record for this book is available from the Library of Congress. ISBN 978-94-010-4201-7 ISBN 978-94-011-0449-4 (ebook) DOI 10.1007/978-94-011-0449-4 This is a completely revised and updated translation of the original Russian work of the same title, Moscow, Nauka, 1980 (second edition). Translated by the author. Printed on acid-free paper Ali Rights Reserved 1995 Springer Science+Business Media Dordrecht Originally published by Kluwer Academic Publishers in 1995 Softcover reprint of the hardcover 2nd edition 1995 No part of the material protected by this copyright notice may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording or by any information storage and retrieval system, without written permis sion from the copyright owner.

CONTENTS Preface Annotation IX XI Chapter 1. Introductory Probability Theory 1 1. The Notion of Probability 1 1.1. Equiprobable outcomes 1 1.2. Examples 2 1.3. Conditional probability 4 1.4. Independent events 7 1.5. Probability and frequency 9 2. Some Probability Models 10 2.1. Trials with countable outcomes 10 2.2. Bernoulli trials 11 2.3. Limit Poisson distribution 12 2.4. Finite number of events 15 2.5. The general model of probability theory 17 2.6. Some examples 26 3. Random Variables 32 3.1. Probability distributions 32 3.2. Joint probability distribution 35 3.3. Independent random variables 38 3.4. Conditional distributions 41 3.5. Functions of random variables 42 3.6. Random variables in the general model of probability theory 44 4. Mathematical Expectation 45 4.1. Mean value of discrete variable 45 4.2. Limit mean values 51 4.3. Some limit properties 54 4.4. Conditional expectation 60 5. Correlation 62 5.1. Variance and correlation 62 5.2. Normal correlations 66 5.3. Properties of the variance and the law of large numbers 69 6. Characteristic Functions 73 6.1. Some examples 73 6.2. Elementary analysis of characteristic functions 78

vi 6.3. The inverse formula of probability distributions 80 6.4. Weak convergence of distributions 82 7. The Central Limit Theorem 83 7.1. Some limit properties of probabilities 83 7.2. The central limit theorem 87 Chapter 2. Random Processes 91 1. Random Processes with Discrete State Space 91 1.1. The Poisson process and related processes 91 1.2. The Kolmogorov equations 96 1.3. Example (Branching processes) 100 1.4. The (limit) stationary probability distribution 107 2. Random Processes with Continuous States 113 2.1. The Brownian motion 113 2.2. Trajectories of the Brownian motion 115 2.3. Maxima and hitting times 122 2.4. Diffusion processes 126 Chapter 3. An Introduction to Mathematical Statistics 131 1. Some Examples of Statistical Problems and Methods 131 1.1. Estimation of the success probability in Bernoulli trials 131 1.2. Estimation of parameters in a normal sample 133 1.3. Chi-square criterion for probability testing 137 1.4. Sequential analysis of alternative hypotheses 141 1.5. Bayesian approach to hypotheses testing and parameters estimation 144 1.6. Maximum likelihood method 147 1.7. Sample distribution function and the method of moments 149 1.8. The method of least squares 151 2. Optimality of Statistical Decisions 154 2.1. The most powerful criterion 154 2.2. Sufficient statistics 156 2.3. Lower bound for the mean square error 163 2.4. Asymptotic normality and efficiency of the maximum likelihood estimate 166 Chapter 4. Basic Elements of Probability Theory 171 1. General Probability Distributions 171 1.1. Mappings and a-algebras 171 1.2. Approximation of events 175 1.3. 0-1 law 178 1.4. Mathematical expectation as the Lebesgue integral 179 1.5...cp-spaces 181 2. Conditional Probabilities and Expectations 187

CONTENTS vii 2.1. Preliminary remarks 187 2.2. Conditional expectation and its properties 189 2.3. Conditional probability 191 3. Conditional Expectations and Martingales 194 3.1. General properties 194 Chapter 5. Elements of Stochastic Analysis and Stochastic Differential Equations 201 1. Stochastic Series 201 1.1. Series of independent random variables 201 1.2. Three series' criterion 203 2. Stochastic Integrals 207 2.1. Random functions (Preliminary remarks) 207 2.2. Integration in l-space 209 2.3. Stochastic integrals in 2-space 212 2.4. Stochastic Ito integral in 2-space 218 3. Stochastic Integral Representations 222 3.1. Canonical representations 222 3.2. Spectral representation of a stationary process and its applications 227 3.3. Stochastic integral representation of a process with independent increments 231 4. Stochastic Differential Equations 237 4.1. Stochastic differentials 237 4.2. Linear stochastic differential equations 238 4.3. Linear differential equations with constant coefficients 242 4.4. The Kalman-Bucy filter 246 Subject Index 253

Preface Probability Theory, Theory of Random Processes and Mathematical Statistics are important areas of modern mathematics and its applications. They develop rigorous models for a proper treatment for various 'random' phenomena which we encounter in the real world. They provide us with numerous tools for an analysis, prediction and, ultimately, control of random phenomena. Statistics itself helps with choice of a proper mathematical model (e.g., by estimation of unknown parameters) on the basis of statistical data collected by observations. This volume is intended to be a concise textbook for a graduate level course, with carefully selected topics representing the most important areas of modern Probability, Random Processes and Statistics. The first part (Ch. 1-3) can serve as a self-contained, elementary introduction to Probability, Random Processes and Statistics. It contains a number of relatively simple and typical examples of random phenomena which allow a natural introduction of general structures and methods. Only knowledge of elements of real/complex analysis, linear algebra and ordinary differential equations is required here. The second part (Ch. 4-6) provides a foundation of Stochastic Analysis, gives information on basic models of random processes and tools to study them. Here a familiarity with elements of functional analysis is necessary. Our intention to make this course fast-moving made it necessary to present important material in a form of examples. Yu. Rozanov

Annotation The book consists of two parts which differ one from another in their contents and the style of exposition. The first one discusses many relatively simple problems which lead to different models of probability and random processes, as well as basic methods of mathematical statistics, including typical applications. The second part presents elements of general analysis of random processes.