Probability for Statistics and Machine Learning

Transcription

1 ~Springer Anirban DasGupta Probability for Statistics and Machine Learning Fundamentals and Advanced Topics

2 Contents Suggested Courses with Diffe~ent Themes xix 1 Review of Univariate Probability I 1.1 Experiments and Sample Spaces l 1.2 Conditional Probability and Independence Integer-Valued and Discrete Random Variables CDF and Independence Expectation and Moments Inequalities Generating and Moment-Generating Functions * Applications of Generating Functions to a Pattern Problem Standard Discrete Distributions Poisson Approximation to Binomial Continuous Random Variables Functions of a Continuous Random Variable Expectation and Moments Moments and the Tail of a CDF Moment-Generating Function and Fundamental Inequalities I * Inversion of an MGF and Post's Formula Some Special Continuous Distributions Normal Distribution and Confidence Interval for a Mean I 1.14 Stein's Lemma *Chernoff's Variance Inequality * Various Characterizations of Normal Distributions Normal Approximations and Central Limit Theorem Binomial Confidence Interval ErroroftheCLT Normal Approximation to Poisson and Ga,mma Confidence Intervals * Convergence of Densities and Edgeworth Expansions References

3 I Ill xii Contents 2 Multivariate Discrete Distributions Bivariate Joint Distributions and Expectations of Functions Conditional Distributions and Conditional Expectations Examples on Conditional Distributions and Expectations OJ 2.3 Using Conditioning to Evaluate Mean and Variance I Covariance and Correlation I Multivariate Case JointMGF Il Multinomial Distribution * The Poissonization Technique Multidimensiopal Densities Joint Density Function and Its Role Expectation of Functions Bivariate Normal Conditional Densities and Expectations Examples on Conditional Densities and Expectations Posterior Densities, Likelihood Functions, and Bayes Estimates Maximum Likelihood Estimates Bivariate Normal Conditional Distributions * Useful Formulas and Characterizations for Bivariate Normal Computing Bivariate Normal Probabilities *Conditional Expectation Given a Set and Borel's Paradox References Advanced Distribution Theory J onvolution nd Exam pi Products and Quoti.ent and d1e /- and F -Distribution Transformati no Appli ation of Jacobian Formula Polar Coordinates in Two Dimen i n * n-dimen. ional Polar and Helmerr' Transformation.... o Efficien t Spherical alculatious with Polar Coordinates Independence of Mean and Variance in Normal Ca oe... 0 o o The I Confidence Interval The Dirichlet Di tribution * Picking a Point from the Surface fa Sphere *Poincare' Lemma..' * Ten Important Higb-Dimen ional Formula for Easy Reference References

4 Contents 5 Multivariate Normal and Related Distributions Definition and Some Basic Properties Conditional Distributions Exchangeable Normal Variables Sampling Distributions Useful in Statistics *Wishart Expectation Identities * Hotelling's T 2 and Distribution of Quadratic Forms *Distribution of Correlation Coefficient Noncentral Distributions Some Important Inequalities for Easy Reference References Finite Sample Theory of Order Statistics and Extremes Basic Distribution Theory More Advanced Distribution Theory Quantile Transformation and Existence of Moments Spacings Exponential Spacings and Reyni's Representation Uniform Spacings Conditional Distributions and Markov Property Some Applications *Records The Empirical CDF *Distribution of the Multinomial Maximum References Essential Asymptotics and Applications Some Basic Notation and Convergence Concepts Laws of Large Numbers Convergence Preservation Convergence in Distribution Preservation of Convergence and Statistical Applications Slutsky's Theorem Delta Theorem Variance Stabilizing Transformations Convergence of Moments Uniform Integrability l;:he Moment Problem and Convergence in Distribution Approximation of Moments Convergence of Densities and Scheffe's Theorem References xiii

5 xiv Content 8 Characteristic Functions and Applications Characteristic Functions of Standard Distributions Inversion and Uniqueness Taylor Expansions, Differentiability, and Moments Continuity Theorems Proof of the CLT and the WLLN *Producing Characteristic Functions Error of the Central Limit Theorem Lindeberg- Feller Theorem for General Independent Case *Infinite Divisibility and Stable Laws *Some Useful Inequalities References Asymptotics of Extremes and Order Statistics Central-Order Statistics Single-Order tali tic Two tali ti cal Applications Several Order Statistics Extremes Easily Applicable Limit Theorems The Convergence of Types Theorem * Fisher-Tippett Family and Putting it Together References Markov Chains and Applications Notation and Basic Definitions Examples and Various Applications as a Model Chapman-Kolmogorov Equation Communicating Classes Gambler's Ruin First Passage, Recurrence, and Transience Long Run Evolution and Stationary Distributions References Random Walks..' Random Walk on the Cubic Lattice Some Distribution Theory Recurrence and Tran ieuc * P6lya's Formula for the Return Probability First Passage Time and AJ Si ne Law TheLoca1Time Practically Useful Generalizations Wald's Identity Fate of a Random Walk

6 p Contents XV 11.7 Chung-Fuchs Theorem Six Important Inequalities References Brownian Motion and Gaussian Processes Preview of Connections to the Random Walk Basic Definitions Condition for a Gaussian Process to be Markov *Explicit Construction of Brownian Motion Basic Distributional Properties Reflection Principle and Extremes Path Properties and Behavior Near Zero and Infinity *FractalNatureofLevelSets The Dirichlet Problem and Boundary Crossing Probabilities Recurrence and Transience The Local Time of Brownian Motion Invariance Principle and Statistical Applications Strong Invariance Principle and the KMT Theorem Brownian Motion with Drift and Ornstein-Uhlenbeck Process Negative Drift and Density of Maximum *Transition Density and the Heat Equation * The Ornstein-Uhlenbeck Process References Poisson Processes and Applications Notation Defining a Homogeneous Poisson Process Important Properties and Uses as a Statistical Model *Linear Poisson Process and Brownian Motion: A Connection Higher-Dimensional Poisson Point Processes The Mapping Theorem One-Dimensional Nonhomogeneous Processes *Campbell's Theorem and Shot Noise Poisson Process and Stable Laws References Discrete Time Martingales and Concentration Inequalities Illustrative Examples and Applications in Statistics Stopping Times and Optional Stopping Stopping Times Optional Stopping Sufficient Conditions for Optional Stopping Theorem Applications of Optional Stopping

7 xvi Contents 14.3 Martingale and Concentration Inequalities Maximal Inequality * Inequalities of Burkholder, Davis, and Gundy Inequalities of Hoeffding and Azuma *Inequalities of McDiarmid and Devroye The Upcrossing Inequality Convergence of Martingales The Basic Convergence Theorem Convergence in L 1 and L * Reverse Martingales and Proof of SLLN Martingale Central Limit Theorem References Probability Metrics Standard Probability Metrics Useful in Statistics Basic Properties of the Metrics Metric Inequalities Differential Metrics for Parametric Families *Fisher Information and Differential Metrics * Rao's Geodesic Distances on Distributions _ References _ Empirical Processes and VC Theory Basic Notation and Definitions Classic Asymptotic Properties of the Empirical Process In variance Principle and Statistical Applications *Weighted Empirical Process The Quantile Process Strong Approximations of the Empirical Process. _. _ Vapnik-Chervonenkis Theory _. _ Basic Theory Concrete Examples CLTs for Empirical Measures and Applications Notation and Formulation Entropy Bounds and Specific CLTs Concrete Examples Maximal Inequalities and Symmetrization _ *Connection to the Poisson Process References Large Deviations Large Deviations for Sample Means The Cramer -Chernoff Theorem in R Properties of the Rate Function Cramer's Theorem for General Sets

8 Contents xvii 17.2 The Gartner-Ellis Theorem and Markov Chain Large Deviations The t-statistic Lipschitz Functions and Talagrand's Inequality Large Deviations in Continuous Time *Continuity of a Gaussian Process *Metric Entropy oft and Tail of the Supremum References The Exponential Family and Statistical Applications One-Parameter Exponential Family Definition and First Examples The Canonical Form and Basic Properties Convexity Properties Moments and Moment Generating Function Closure Properties Multi parameter Exponential Family Sufficiency and Completeness * Neyman-Fisher Factorization and Basu 's Theorem *Applications of Basu's Theorem to Probability Curved Exponential Family References Simulation and Markov Chain Monte Carlo The Ordinary Monte Carlo Basic Theory and Examples Monte Carlo P-Values Rao-Blackwellization Textbook Simulation Techniques Quantile Transformation and Accept-Reject Importance Sampling and Its Asymptotic Properties Optimal Importance Sampling Distribution Algorithms for Simulating from Common Distributions Markov Chain Monte Carlo Reversible Markov Chains Metropolis Algorithms The Gibbs Sampler Convergence ofmcmc and Bounds on Errors Spectral Bounds * Dobrushin's Inequality and Diaconis-Fill- Stroock Bound *Drift and Minorization Methods

9 xviii Content! References MCMC on General Spaces General Theory and Metropolis Schemes Convergence Convergence of the Gibbs Sampler Practical Convergence Diagnostics Useful Tools for Statistics and Machine Learning The Bootstrap Consistency of the Bootstrap Further Examples * Higher-Order Accuracy of the Bootstrap Bootstrap for Dependent Data I 20.2 The EM Algorithm The Algorithm and Examples Monotone Ascent and Convergence of EM * Modifications of EM Kernels and Classification Smoothing by Kernels l Some Common Kernels in Use Kernel Density Estimation l Kernels for Statistical Classification Mercer's Theorem and Feature Maps References A Symbols, Useful Formulas, and Normal Table A.1 Glossary of Symbols A.2 Moments and MGFs of Common Distributions A.3 Normal Table Author Index Subject Index