Linear Algebra and Probability

Linear Algebra and Probability for Computer Science Applications Ernest Davis CRC Press Taylor!* Francis Group Boca Raton London New York CRC Press is an imprint of the Taylor Sc Francis Croup, an informa business AN A K PETERS BOOK

Contents Preface xiii 1 MATLAB 1 1.1 Desk Calculator Operations 1 1.2 Booleans 2 1.3 Nonstandard Numbers 3 1.4 Loops and Conditionals 4 1.5 Script File 6 1.6 Functions 7 1.7 Variable Scope and Parameter Passing 9 1 Linear Algebra 15 2 Vectors 17 2.1 Definition ofvectors 17 2.2 Applications of Vectors 17 2.2.1 General Comments about Applications 20 2.3 Basic Operations on Vectors 21 2.3.1 Algebraic Properties of the Operations 21 2.3.2 Applications of Basic Operations 22 2.4 Dot Product 23 2.4.1 Algebraic Properties of the Dot Product 23 2.4.2 Application of the Dot Product: Weighted Sum 23 2.4.3 Geometric Properties of the Dot Product 24 2.4.4 Metacomment: How to Read Formula Manipulations 27 2.4.5 Application of the Dot Product: Similarity of Two Vectors 28 2.4.6 Dot Product and Linear Transformations 30 vii

viii Contents 2.5 Vectors in MATLAB: Basic Operations 31 2.5.1 Creating a Vector and Indexing 31 2.5.2 Creating a Vector with Elements in Arithmetic Sequence... 31 2.5.3 Basic Operations 33 2.5.4 Element-by-Element Operations 33 2.5.5 Useful Vector Functions 34 2.5.6 Random Vectors 35 2.5.7 Strings: Arrays of Characters 36 2.5.8 Sparse Vectors 36 2.6 Plotting Vectors in MATLAB 37 2.7 Vectors in Other Programming Languages 41 3 Matrices 47 3.1 Definition of Matrices 47 3.2 Applications of Matrices 47 3.3 Simple Operations on Matrices 49 3.4 Multiplying a Matrix Times a Vector 49 3.4.1 Applications of Multiplying a Matrix Times a Vector 51 3.5 Linear Transformation 55 3.6 Systems of Linear Equations 56 3.6.1 Applications of Systems of Linear Equations 57 3.7 Matrix Multiplication 62 3.8 Vectors as Matrices 65 3.9 Algebraic Properties of Matrix Multiplication 66 3.9.1 Matrix Exponentiation 67 3.10 Matrices in MATLAB 68 3.10.1 Inputting Matrices 68 3.10.2 Extracting Submatrices 70 3.10.3 Operations on Matrices 71 Matrices 73 3.10.4 Sparse 3.10.5 Cell Arrays 75 4 Vector Spaces 81 4.1 Fundamentals of Vector Spaces 81 4.1.1 Subspaces 82 4.1.2 Coordinates, Bases, Linear Independence 84 4.1.3 Orthogonaland Orthonormal Basis 87 4.1.4 Operations on Vector Spaces 88 4.1.5 Null Space, Image Space, and Rank 89 4.1.6 Systems oflinear Equations 91 4.1.7 Inverses 93 4.1.8 Null Space and Rank in MATLAB 93 4.2 Proofs and Other Abstract Mathematics (Optional) 94 4.2.1 Vector Spaces 94 4.2.2 Linear Independence and Bases 95 4.2.3 Sum of Vector Spaces 98

Contents ix 4.2.4 Orthogonality 99 4.2.5 Functions 102 4.2.6 Linear Transformations 104 4.2.7 Inverses 106 4.2.8 Systems of Linear Equations 107 4.3 Vector Spaces in General (Very Optional) 108 4.3.1 The General Definition of Vector Spaces 108 5 Algorithms 115 5.1 Gaussian Elimination: Examples 115 5.2 Gaussian Elimination: Discussion 117 5.2.1 Gaussian Elimination on Matrices 120 5.2.2 Maximum ElementRow Interchange 121 5.2.3 Testing on Zero 122 5.3 Computing a Matrix Inverse 123 5.4 Inverse and Systems of Equations inmatlab 127 5.5 Ill-Conditioned Matrices 131 5.6 Computational Complexity 135 5.6.1 Viewpoints on Numerical Computation 135 5.6.2 Running Times 136 6 Geometry 143 6.1 Arrows 143 6.2 Coordinate Systems 145 6.3 Simple Geometric Calculations 146 6.3.1 Distance and Angle 146 6.3.2 Direction 147 6.3.3 Lines in Two-Dimensional Space 148 6.3.4 Lines and Planes in Three-Dimensional Space 150 6.3.5 Identity, Incidence, Parallelism, and Intersection 152 6.3.6 Projections 154 6.4 Geometric Transformations 155 6.4.1 Translations 156 6.4.2 Rotation around the Origin 157 6.4.3 Rigid Motions and the Homogeneous Representation 161 6.4.4 Similarity Transformations 165 6.4.5 Affine Transformations 167 6.4.6 Image of a Distant Object 169 6.4.7 Determinants 172 6.4.8 Coordinate Transformation on Image Arrays 174 7 Change of Basis, DFT, and SVD 181 7.1 Change of Coordinate System 181 7.1.1 Affine Coordinate Systems 182 7.1.2 Duality of Transformation and Coordinate Change; Handedness 184 7.1.3 Application: Robotic Arm 185

x Contents 7.2 The Formula for Basis Change 189 7.3 Confusion and How to Avoid It 191 7.4 Nongeometric Change of Basis 192 7.5 Color Graphics 192 7.6 Discrete Fourier Transform (Optional) 193 7.6.1 Other Applications of the Fourier Transform 198 7.6.2 The Complex Fourier Transform 199 7.7 Singular Value Decomposition 200 7.7.1 Matrix Decomposition 202 7.7.2 Proof of Theorem 7.4 (Optional) 203 7.8 Further Properties of the SVD 205 7.8.1 Eigenvalues of a Symmetric Matrix 208 7.9 Applications of the SVD 209 7.9.1 Condition Number 209 7.9.2 ComputingRank in the Presence of Roundoff 210 7.9.3 Lossy Compression 211 7.10 MATLAB 213 7.10.1 The SVD in MATLAB 213 7.10.2 The DFT in MATLAB 213 II Probability 221 8 Probability 223 8.1 The Interpretations of Probability Theory 223 8.2 Finite Sample Spaces 224 8.3 Basic Combinatorial Formulas 226 8.3.1 Exponential 226 8.3.2 Permutations of n Items 226 8.3.3 Permutations of fc Items out of n 227 8.3.4 Combinations ofk Items out ofn 227 8.3.5 Partition into Sets 227 8.3.6 Approximation of Central Binomial 228 8.3.7 Examples of Combinatorics 228 8.4 The Axioms of ProbabilityTheory 230 8.5 Conditional Probability 231 8.6 The Likelihood Interpretation 232 8.7 Relation between Likelihood and Sample Space Probability 235 8.8 Bayes' Law 236 8.9 Independence 238 8.9.1 Independent Evidence 239 8.9.2 Application: Secret Sharing in Cryptography 242 8.10 Random Variables 243 8.11 Application: Naive Bayes Classification 245

Contents xi 9 Numerical Random Variables 257 9.1 Marginal Distribution 259 9.2 Expected Value 260 9.3 DecisionTheory 262 9.3.1 Sequence of Actions: Decision Trees 263 9.3.2 Decision Theory and the Value of Information 266 9.4 Variance and Standard Deviation 269 9.5 Random Variables over Infinite Sets of Integers 271 9.6 Three Important Discrete Distributions 273 9.6.1 The Bernoulli Distribution 273 9.6.2 The Binomial Distribution 273 9.6.3 The Zipf Distribution 274 9.7 Continuous Random Variables 279 9.8 Two Important Continuous Distributions 287 9.8.1 The Continuous Uniform Distribution 287 9.8.2 The Gaussian Distribution 287 9.9 MATLAB 291 10 Markov Models 299 10.1 Stationary ProbabilityDistribution 301 10.1.1 Computing the Stationary Distribution 303 10.2 PageRank and Link Analysis 304 10.2.1 The Markov Model 306 10.2.2 Pages with No Outlinks 308 10.2.3 Nonuniform Variants 309 10.3 Hidden Markov Models and the K-Gram Model 309 10.3.1 The Probabilistic Model 310 10.3.2 Hidden Markov Models 312 10.3.3 The Viterbi Algorithm 314 10.3.4 Part of Speech Tagging 315 10.3.5 The Sparse Data Problem and Smoothing 318 11 Confidence Intervals 323 11.1 The Basic Formulafor Confidence Intervals 323 11.2 Application: Evaluating a Classifier 325 11.3 Bayesian Statistical Inference (Optional) 328 11.4 Confidence Intervals in the FrequentistViewpoint (Optional) 330 11.5 Hypothesis Testing and Statistical Significance 333 11.6 Statistical Inference and ESP 336 12 Monte Carlo Methods 339 12.1 Finding Area 339 12.2 Generating Distributions 341 12.3 Counting 342 12.4 Counting Solutions to a DNF Formula (Optional) 343 12.5 Sums, Expected Values, and Integrals 345

xii Contents 12.6 Probabilistic Problems 347 12.7 Resampling 348 12.8 Pseudorandom Numbers 350 12.9 Other Probabilistic Algorithms 351 12.10 MATLAB 351 13 Information and Entropy 359 13.1 Information 359 13.2 Entropy 361 13.3 Conditional Entropy and Mutual Information 362 13.4 Coding 364 13.4.1 Huffman Coding 374 13.5 Entropy of Numeric and Continuous Random Variables 375 13.6 The Principle of Maximum Entropy 376 13.6.1 The Principle ofmaximum Entropy 377 13.6.2 Consequences of the Maximum Entropy Principle 377 13.7 Statistical Inference 379 14 Maximum Likelihood Estimation 383 14.1 Sampling 384 14.2 Uniform Distribution 385 14.3 Gaussian Distribution: Known Variance 385 14.4 Gaussian Distribution: Unknown Variance 386 14.5 Least Squares Estimates 387 14.5.1 Least Squares in MATLAB 390 14.6 Principal Component Analysis 391 14.7 Applications of Principal Component Analysis 393 14.7.1 Visualization 393 14.7.2 Data Analysis 393 14.7.3 Bounding Box 394 14.7.4 Surface Normals 395 14.7.5 Latent Semantic Analysis 396 References 401 Notation 405