Math for ML: review. Milos Hauskrecht 5329 Sennott Square, x people.cs.pitt.edu/~milos/courses/cs1675/

Transcription

1 Math for ML: review Milos Hauskrecht 5 Sennott Square, -5 people.cs.pitt.edu/~milos/courses/cs75/

2 Administrivia Recitations Held on Wednesdays at :00am and :00pm This week: Matlab tutorial Office hours: Milos: Wednesday at :0am-:00am Thursday at :00pm-5:00pm Jeongmin: Tuesday at :00-5:00pm Wednesday: :00-5:00pm Thursday: TBA

3 ML and knowledge of other fields ML solutions and algorithms rely on knowledge of many other disciplines: Algebra Calculus Probability Statistics Control theory Decision theory Net: a review of the basics of algebra and calculus one typically needs for ML

4 Notation Notation: Scalar: a Eample: a= Vector: v or Eample: v v Note: a vector is by default a column vector Matri: M Eample: M 0

5 Terminology Matri: M U 0 0 matri Elements of the matri: M, M,,, M,, matri (square matri) 0 v Vector is a special case of a matri

6 Why vectors and matrices Data: Data instances are often represented using vectors, and datasets using matrices. Eample: Weather information Temperature Pressure Humidity Cloud-cover (clear) (partly cloudy) (partly cloudy). Data can be naturally represented as a matri: 0 D Data instances in rows Attributes in columns

7 Basic Operations Matri Transpose The transpose of a matri is found by flipping the matri over its main diagonal. The main diagonal is the diagonal that begins at the element located at the first row and first column of the matri. Vector transpose:, T v v 0, 0 T A A

8 Basic Operations Scalar/matri operations: Addition Add the scalar to every element in the matri. The sum of a matri and a scalar is a matri. 0, M a a M M a

9 Basic Operations Scalar/matri operations: Multiplication Multiply every element of the matri by the scalar. The product of a matri and a scalar is a matri. a, M 0 am Ma * 0 0 7

10 Basic Operations Matri/vector operations: Matri Vector Addition Add the elements of the vector to each element in the corresponding row of the matri. The sum of a vector and a matri is a matri. 7 5, 5 M v v v M M

11 Basic Operations Matri operations: Matri-Matri Multiplication The inner dimensions of the two matrices must be the same. The product matri will have the same number of rows as the first matri and the same number of columns as the second matri. Inner dimensions must agree Outer dimensions define the result A 0, B 5 C AB 5 7

12 Basic Operations Matri operations: Matri-Matri Multiplication When performing matri multiplication, take the sum of the products of the elements in the row of the first matri and the column of the second matri. AB 0 5 ( ) 0 5 ( ) ( 0) 0 5 ( ) 5 ( ) 0 ( ) 5 7

13 Basic Operations Matri operations: Matri -Matri Multiplication When performing matri multiplication, take the sum of the products of the elements in the row of the first matri and the column of the second matri. AB 0 5 ( ) 0 5 ( ) ( 0) 0 5 ( ) 5 ( ) 0 ( ) 5 7

14 Basic Operations Matri operations: Matri-Matri Multiplication The product of A and B is not equal to the product of B and A. AB 0 5 ( ) 0 5 ( ) ( 0) 0 5 ( ) 5 ( ) 0 ( ) 5 7 BA ( ) 50 ( ) ( ) ( ) 5 7 0

15 Matri/Vector operations: Matri-Vector Multiplication Basic Operations Multiplication of a matri and a vector is similar to matrimatri multiplication. The inner dimensions of the matri and the vector must match. A 0, v, u va 0 7 Au 0

16 Basic Operations Matri/vector operations: Vector-Vector Multiplication The product of two vectors of the same length is either a scalar or a matri, depending on how the vectors are multiplied. Inner (dot) product Outer product, 7 u v 7 ) ( 7 v T u T vu

17 Basic Operations Matri/Vector operations: Matri Inverse The product of a matri and its inverse is the identity matri AA A A I A ( ) A 0. ( 0.) ( 0.) 0 0 Note: The inverse of a matri can be found by hand by augmenting the matri with an identity matri and using elementary row operations (additions or subtraction, multiplication by a constant, or swapping rows

18 Eigenvectors and Eigenvalues of the matri Decomposition allows us to see functional properties of a matri The eigenvector of a square matri M is a nonzero vector v such that, when multiplying the matri M by the eigenvector, only the scale of the eigenvector changes. Mv v Eample: Eigenvalue Eigenvector A v,

19 Matri determinant The determinant of a matri maps matrices to real scalars The determinant is a measure of how much the space (vector) epands or contracts when multiplied by the matri. Eamples: A det( A) * * 0 M 0 det( M ) (* *0) (* *) (*0 *)

20 Norms A norm measures the size of a vector. It is a map from a vector to a non-negative scalar. Properties of a norm:. f ( ) 0 0. f ( y) f ( ) f ( y) triangle inequality. a R f ( a) a f ( ) Eamples of norms: Euclidean (l norm) d i i

21 Eamples of norms: Norms Squared Euclidean (squared l norm) l norm i Ma norm (l infinity norm) d d i i i ma i i

22 Functions Functions of one variable: Function of many variables ) ( f f log ) ( ), ( f ),, ( f

23 Function derivatives Function derivatives are useful to analyze functions and their behaviors: First derivatives: increasing, decreasing trends and etremes f ( ) d f ( )' at = (increasing) at =- - (decreasing) at = 0 0 (an etreme - minimum) Solving for f ( )' 0 helps us to find the function etremes

24 Function derivatives The same applies for multivariate functions First derivatives: increasing, decreasing trends and etremes Gradient a vector of partial derivatives Solving for helps us to find the function etremes ) ( f T ], [ ) ( ) ( ) ( f f f 0 ) ( f