Some practical remarks (recap) Statistical Natural Language Processing. Today s lecture. Linear algebra. Why study linear algebra?

Transcription

1 Some practical remarks (recap) Statistical Natural Language Processing Mathematical background: a refresher Çağrı Çöltekin Universit of Tübingen Seminar für Sprachwissenschaft Summer Semester 017 Course web page: Please join the Moodle page Reminder: there are Easter eggs (in the version presented in the class) Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Toda s lecture Some concepts from linear algebra A (ver) short refresher on Derivatives: we are interested in maximizing/minimizing (objective) functions (mainl in machine learning) Integrals: mainl for probabilit theor This is onl a high-level, informal introduction/refresher Linear algebra Linear algebra is the field of mathematics that studies vectors and matrices A vector is an ordered sequence of numbers v (6, 17) A matrix is a rectangular arrangement of numbers A A well-known application of linear algebra is solving a set of linear equations [ ] [ ] [ ] x 1 + x 6 1 x1 6 x 1 + 4x x Ç Çöltekin, SfS / Universit of Tübingen Summer Semester 017 / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Wh stud linear algebra? Wh stud linear algebra? Consider an application counting words in multiple documents the and of to in document document document You should alread be seeing vectors and matrices here Insights from linear algebra is helpful in understanding man NLP methods In machine learning, we tpicall represent input, output, parameters as vectors or matrices It makes notation concise and manageable In programming, man machine learning libraries make use of vectors and matrices explicitl Vectorized operations ma run much faster on GPUs, and on modern CPUs Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Vectors Geometric interpretation of vectors A vector is an ordered list of numbers v (v 1, v, v n ), The vector of n real numbers is said to be in vector space R n (v R n ) In this course we will onl work with vectors in R n Tpical notation for vectors: v v (v 1, v, v 3 ) v 1, v, v 3 v v 3 Vectors are (geometric) objects with a magnitude and a direction v 1 direction magnitude Vectors are represented b arrows from the origin The endpoint of the vector v (v 1, v ) correspond to the Cartesian coordinates defined b v 1, v The intuitions often (!) generalize to higher dimensional spaces ( 1, 3) (1, 3) (1, 1) Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38

2 Vector norms L norm The norm of a vector is an indication of its size (magnitude) The norm of a vector is the distance from its tail to its tip Norms are related to distance measures Vector norms are particularl important for understanding some machine learning techniques Euclidean norm, or L (or L ) norm is the most commonl used norm For v (v 1, v ), v v 1 + v (3, 3) (3, 3) L norm is often written without a subscript: v x Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 L1 norm L P norm Another norm we will often encounter is the L1 norm v 1 v 1 + v (3, 3) In general, L P norm, is defined as ( n ) 1 p v p v i p i1 (3, 3) L1 norm is related to Manhattan distance x We will onl work with than L1 and L norms, but L 0 and L are also common Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Multipling a vector with a scalar Vector addition and subtraction For a vector v (v 1, v ) and a scalar a, av (av 1, av ) v v (1, ) For vectors v (v 1, v ) and w (w 1, w ) v+w (v 1 + w 1, v + w ) (1, ) + (, 1) (3, 3) v w v v + w w multipling with a scalar scales the vector 05v v w v + ( w) (1, ) (, 1) ( 1, 1) w Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Dot product For vectors w (w 1, w ) and v (v 1, v ), Cosine similarit The cosine of the angle between two vectors or, wv w 1 v 1 + w v wv w v cos α The dot product of two orthogonal vectors is 0 ww w Dot product ma be used as a similarit measure between two vectors α v v cos α w cos α vw v w is often used as another similarit metric, called cosine similarit The cosine similarit is related to the dot product, but ignores the magnitudes of the vectors For unit vectors (vectors of length 1) cosine similarit is equal to the dot product The cosine similarit is bounded in range [ 1, +1] Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38

3 Matrices Transpose of a matrix a 1,1 a 1, a 1,3 a 1,m a,1 a, a,3 a,m A a n,1 a n, a n,3 a n,m We can think of matrices as collection of row or column vectors A matrix with n rows and m columns is in R nm Transpose of a n m matrix is a m n matrix whose rows are the columns of the original matrix Transpose of a matrix A is denoted with A T a If A c e b d, A T f [ ] a c e b d f Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Multipling a matrix with a scalar Matrix addition and subtraction Similar to vectors, each element is multiplied b the scalar [ ] [ ] 4 8 Each element is added to (or subtracted from) the corresponding element Note: + [ ] 0 1 [ ] 4 Matrix addition and subtraction is defined on matrices of the same dimension Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Matrix multiplication if A is a n k matrix, and B is a k m matrix, their product C is a n m matrix Elements of C, c i,j, are defined as c ij k a il b lj l0 Note: c i,j is the dot product of the i th row of A and the j th column of B Matrix multiplication (demonstration) a 11 a 1 a 1k a 1 a a k a n1 a n a nk b11 b1 b1m b 1 b b m b k1 b k b km c ij a i1 b 1j + a i b j + a ik b kj c 11 c 1 c 1m c 1 c c m c n1 c n c nm Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Dot product as matrix multiplication Outer product In machine learning literature, the dot product of two vectors is often written as w T v For example, w (, ) and v (, ), [ ] [ ] Although, this notation is somewhat slopp, since the result of matrix multiplication is in fact not a scalar The outer product of two column vectors is defined as Note: The result is a matrix vw T [ 1 3 ] [ 1 ] The vectors do not have to be the same length Ç Çöltekin, SfS / Universit of Tübingen Summer Semester 017 / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38

4 Identit matrix Matrix multiplication as transformation A square matrix in which all the elements of the principal diagonal are ones and all other elements are zeros, is called identit matrix and often denoted I Multipling a matrix with the identit matrix does not change the original matrix Multipling a vector with a matrix transforms the vector Result is another vector (possibl in a different vector space) Man operations on vectors can be expressed with multipling with a matrix (linear transformations) IA A Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 identit stretch along the x axis Identit transformation maps a vector to itself In two dimensions: [ ] 0 1 [ ] x [ ] x [ ] [ ] 3 (1, ) (3, ) Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 rotation Matrix-vector representation of a set of linear equations Our earlier example set of linear equations [ cos θ ] sin θ sin θ cos θ [ ] 0 1 [ ] 1 (, 1) (1, ) x 1 + x 6 x 1 + 4x 17 can be written as: [ ] [ ] [ ] 1 x1 6 x 17 }{{}}{{}}{{} W x b One can solve the above equation using Gaussian elimination (we will not cover it toda) Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Inverse of a matrix Determinant of a matrix Inverse of a square matrix W is defined denoted W 1, and defined as WW 1 W 1 W I The inverse can be used to solve equation in our previous example: Wx b W 1 Wx W 1 b Ix W 1 b x W 1 b a c b d ad bc The above formula generalizes to higher dimensional matrices through a recursive definition, but ou are unlikel to calculate it b hand Some properties: A matrix is invertible if it has a non-zero determinant A sstem of linear equations has a unique solution if the coefficient matrix has a non-zero determinant Geometric interpretation of determinant is the (signed) changed in the volume of a unit (hper)cube caused b the transformation defined b the matrix Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38

5 Eigenvalues and eigenvectors of a matrix Derivatives An eigenvector, v and corresponding eigenvalue, λ, of a matrix A is defined as Av λv Eigenvalues an eigenvectors have man applications from communication theor to quantum mechanics A better known example (and close to home) is Google s PageRank algorithm We will return to them while discussing PCA and SVD (and mabe more topics/concepts) Derivative of a function f(x) is another function f (x) indicating the rate of change in f(x) Alternativel: df df(x) dx (x), dx Example from phsics: velocit is the derivative of the position Our main interest: the points where the derivative is 0 are the stationar points (maxima / minima / saddle points) the derivative evaluated at other points indicate the direction and steepness of the curve Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Finding minima and maxima of a function Partial derivatives and gradient Man machine learning problems are set up as optimization problems: Define an error function Learning involves finding the minimum error We search for f (x) 0 The value of f (x) on other points tell us which direction to go (and how fast) f (3) 4 f(x) x x f ( 05) 3 f (1) 0 In ML, we are often interested in (error) functions of man variables A partial derivative is derivative of a multi-variate function with respect to a single variable, noted f x A ver useful quantit, called gradient, is the vector of partial derivatives with respect to each variable ( ) f f f(x 1,, x n ),, x 1 x n Gradient points to the direction of the steepest change Example: if f(x, ) x 3 + x f(x, ) ( 3x +, x ) Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Integrals Numeric integrals & infinite sums Integral is the reverse of the derivative (anti-derivative) The indefinite integral of f(x) is noted F(x) f(x)dx We are often interested in definite integrals b a f(x)dx F(b) F(a) Integral gives the area under the curve When integration is not possible with analtic methods, we resort to numeric integration This also shows that integration is infinite summation Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Summar & next week Some understanding of linear algebra and calculus is important for understanding man methods in NLP (and ML) See bibliograph at the end of the slides if ou want a more complete refresher/introduction Wed We will do a similar excursion to probabilit theor Fri There will be a short tutorial on Pthon nump Further reading A classic reference book in the field is Strang (009) Shifrin and Adams (011) and Farin and Hansford (014) are textbooks with a more practical/graphical orientation Cherne, Denton, and Waldron (013) and Beezer (014) are two textbooks that are freel available Beezer, Robert A (014) A First Course in Linear Algebra version 340 Congruent Press isbn: url: Cherne, David, Tom Denton, and Andrew Waldron (013) Linear algebra mathucdavisedu url: Farin, Gerald E and Dianne Hansford (014) Practical linear algebra: a geometr toolbox Third edition CRC Press isbn: Shifrin, Theodore and Malcolm R Adams (011) Linear Algebra A Geometric Approach nd W H Freeman isbn: Strang, Gilbert (009) Introduction to Linear Algebra, Fourth Edition 4th ed Wellesle Cambridge Press isbn: Ç Çöltekin, SfS / Universit of Tübingen Summer Semester / 38 Ç Çöltekin, SfS / Universit of Tübingen Summer Semester 017 A1