Linear Algebra and Probability

Transcription

1 Linear Algebra and Probability for Computer Science Applications Ernest Davis CRC Press Taylor!* Francis Group Boca Raton London New York CRC Press is an imprint of the Taylor Sc Francis Croup, an informa business AN A K PETERS BOOK

2 Contents Preface xiii 1 MATLAB Desk Calculator Operations Booleans Nonstandard Numbers Loops and Conditionals Script File Functions Variable Scope and Parameter Passing 9 1 Linear Algebra 15 2 Vectors Definition ofvectors Applications of Vectors General Comments about Applications Basic Operations on Vectors Algebraic Properties of the Operations Applications of Basic Operations Dot Product Algebraic Properties of the Dot Product Application of the Dot Product: Weighted Sum Geometric Properties of the Dot Product Metacomment: How to Read Formula Manipulations Application of the Dot Product: Similarity of Two Vectors Dot Product and Linear Transformations 30 vii

3 viii Contents 2.5 Vectors in MATLAB: Basic Operations Creating a Vector and Indexing Creating a Vector with Elements in Arithmetic Sequence Basic Operations Element-by-Element Operations Useful Vector Functions Random Vectors Strings: Arrays of Characters Sparse Vectors Plotting Vectors in MATLAB Vectors in Other Programming Languages 41 3 Matrices Definition of Matrices Applications of Matrices Simple Operations on Matrices Multiplying a Matrix Times a Vector Applications of Multiplying a Matrix Times a Vector Linear Transformation Systems of Linear Equations Applications of Systems of Linear Equations Matrix Multiplication Vectors as Matrices Algebraic Properties of Matrix Multiplication Matrix Exponentiation Matrices in MATLAB Inputting Matrices Extracting Submatrices Operations on Matrices 71 Matrices Sparse Cell Arrays 75 4 Vector Spaces Fundamentals of Vector Spaces Subspaces Coordinates, Bases, Linear Independence Orthogonaland Orthonormal Basis Operations on Vector Spaces Null Space, Image Space, and Rank Systems oflinear Equations Inverses Null Space and Rank in MATLAB Proofs and Other Abstract Mathematics (Optional) Vector Spaces Linear Independence and Bases Sum of Vector Spaces 98

4 Contents ix Orthogonality Functions Linear Transformations Inverses Systems of Linear Equations Vector Spaces in General (Very Optional) The General Definition of Vector Spaces Algorithms Gaussian Elimination: Examples Gaussian Elimination: Discussion Gaussian Elimination on Matrices Maximum ElementRow Interchange Testing on Zero Computing a Matrix Inverse Inverse and Systems of Equations inmatlab Ill-Conditioned Matrices Computational Complexity Viewpoints on Numerical Computation Running Times Geometry Arrows Coordinate Systems Simple Geometric Calculations Distance and Angle Direction Lines in Two-Dimensional Space Lines and Planes in Three-Dimensional Space Identity, Incidence, Parallelism, and Intersection Projections Geometric Transformations Translations Rotation around the Origin Rigid Motions and the Homogeneous Representation Similarity Transformations Affine Transformations Image of a Distant Object Determinants Coordinate Transformation on Image Arrays Change of Basis, DFT, and SVD Change of Coordinate System Affine Coordinate Systems Duality of Transformation and Coordinate Change; Handedness Application: Robotic Arm 185

5 x Contents 7.2 The Formula for Basis Change Confusion and How to Avoid It Nongeometric Change of Basis Color Graphics Discrete Fourier Transform (Optional) Other Applications of the Fourier Transform The Complex Fourier Transform Singular Value Decomposition Matrix Decomposition Proof of Theorem 7.4 (Optional) Further Properties of the SVD Eigenvalues of a Symmetric Matrix Applications of the SVD Condition Number ComputingRank in the Presence of Roundoff Lossy Compression MATLAB The SVD in MATLAB The DFT in MATLAB 213 II Probability Probability The Interpretations of Probability Theory Finite Sample Spaces Basic Combinatorial Formulas Exponential Permutations of n Items Permutations of fc Items out of n Combinations ofk Items out ofn Partition into Sets Approximation of Central Binomial Examples of Combinatorics The Axioms of ProbabilityTheory Conditional Probability The Likelihood Interpretation Relation between Likelihood and Sample Space Probability Bayes' Law Independence Independent Evidence Application: Secret Sharing in Cryptography Random Variables Application: Naive Bayes Classification 245

6 Contents xi 9 Numerical Random Variables Marginal Distribution Expected Value DecisionTheory Sequence of Actions: Decision Trees Decision Theory and the Value of Information Variance and Standard Deviation Random Variables over Infinite Sets of Integers Three Important Discrete Distributions The Bernoulli Distribution The Binomial Distribution The Zipf Distribution Continuous Random Variables Two Important Continuous Distributions The Continuous Uniform Distribution The Gaussian Distribution MATLAB Markov Models Stationary ProbabilityDistribution Computing the Stationary Distribution PageRank and Link Analysis The Markov Model Pages with No Outlinks Nonuniform Variants Hidden Markov Models and the K-Gram Model The Probabilistic Model Hidden Markov Models The Viterbi Algorithm Part of Speech Tagging The Sparse Data Problem and Smoothing Confidence Intervals The Basic Formulafor Confidence Intervals Application: Evaluating a Classifier Bayesian Statistical Inference (Optional) Confidence Intervals in the FrequentistViewpoint (Optional) Hypothesis Testing and Statistical Significance Statistical Inference and ESP Monte Carlo Methods Finding Area Generating Distributions Counting Counting Solutions to a DNF Formula (Optional) Sums, Expected Values, and Integrals 345

7 xii Contents 12.6 Probabilistic Problems Resampling Pseudorandom Numbers Other Probabilistic Algorithms MATLAB Information and Entropy Information Entropy Conditional Entropy and Mutual Information Coding Huffman Coding Entropy of Numeric and Continuous Random Variables The Principle of Maximum Entropy The Principle ofmaximum Entropy Consequences of the Maximum Entropy Principle Statistical Inference Maximum Likelihood Estimation Sampling Uniform Distribution Gaussian Distribution: Known Variance Gaussian Distribution: Unknown Variance Least Squares Estimates Least Squares in MATLAB Principal Component Analysis Applications of Principal Component Analysis Visualization Data Analysis Bounding Box Surface Normals Latent Semantic Analysis 396 References 401 Notation 405