Recitation. Shuang Li, Amir Afsharinejad, Kaushik Patnaik. Machine Learning CS 7641,CSE/ISYE 6740, Fall 2014
|
|
- Virgil Hubbard
- 5 years ago
- Views:
Transcription
1 Recitation Shuang Li, Amir Afsharinejad, Kaushik Patnaik Machine Learning CS 7641,CSE/ISYE 6740, Fall 2014
2 Probability and Statistics 2
3 Basic Probability Concepts A sample space S is the set of all possible outcomes of a conceptual or physical, repeatable experiment. (S can be finite or infinite.) E.g., S may be the set of all possible outcomes of a dice roll: S E.g., S may be the set of all possible nucleotides of a DNA site: S A C G T E.g., S may be the set of all possible time-space positions of an aircraft on a radar screen. An Event A is any subset of S Seeing "1" or "6" in a dice roll; observing a "G" at a site; UA007 in space-time interval 3
4 Probability An event space E is the possible worlds the outcomes can happen. All dice-rolls, reading a genome, monitoring the radar signal A probability P(A) is a function that maps an event A onto the interval [0,1]. P(A) is also called the probability measure or probability mass of A. Sample space of all possible worlds. Its area is 1 Worlds in which A is true Worlds in which A is false P(a) is the area of the oval 4
5 Kolmogorov Axioms For any event A, B S: 1 P(A) 0 P S = 1 P A B = P A + P B P(A B) 5
6 Random Variable A random variable is a function that associates a unique numerical value (a token) with every outcome of an experiment. (The value of the r.v. will vary the experiment is repeated) RV Distributions: Continuous RV: The outcome of observing the measured location of an aircraft The outcome of recording the true location of an aircraft Discrete RV: The outcome of a dice-roll The outcome of a coin toss 6
7 Discrete Prob. Distribution A probability distribution P defined on a discrete sample space S is an assignment of a non-negative real number P(s) to each sample s S : Probability Mass Function (PMF):p x x i = P[X = x i ] Cumulative Mass Function (CMF): F x x i = P X x i = j:x j <x i p x x j Properties: p x x i 0 and i p X x i = 1 Examples: Bernoulli Distribution: 1 p for x = 0 p for x = 1 Binomial Distribution: P(X = k)= n k pk (1 p) n k 7
8 Continuous Prob. Distribution A continuous random variable X is defined on a continuous sample space: an interval on the real line, a region in a high dimensional space, etc. It is meaningless to talk about the probability of the random variable assuming a particular value --- P(x) = 0 Instead, we talk about the probability of the random variable assuming a value within a given interval, or half interval, or arbitrary Boolean combination of basic propositions. Cumulative Distribution Function (CDF): F x x = P[X x] x Probability Density Function (PDF):F x x = fx (x) dx or f x x = d F x (x) dx Properties: f x (x) 0 and fx (x) dx = 1 Interpretation: f x x = P[X x,x+δ Δ ] 8
9 Continuous Prob. Distribution Examples: Uniform Density Function: 1 f x x = for a x b b a 0 otherwise Exponential Density Function: f x x = 1 μ e x μ for x 0 x F x x = 1 e μ for x 0 Gaussian(Normal) Density Function f x x = 1 2πσ e (x μ) 2 2σ 2 9
10 Continuous Prob. Distribution Gaussian Distribution: If Z~N(0,1) F x x = Φ x = x f x z dz = 1 2π x e z2 2 dz This has no closed form expression, but is built in to most software packages (eg. normcdf in matlab stats toolbox). 10
11 Characterizations Expectation: The mean value, center of mass, first moment: E X g(x) = N-th moment: g x = x n g x p X x dx = μ N-th central moment: g x = (x μ) n Mean: E X X = xpx x dx E αx = αe[x] E α + X = α + E[X] Variance(Second central moment): Var x = E X (X E X X ) 2 = E X X 2 E X [X] 2 Var αx = α 2 Var X Var α + X = Var X 11
12 Central Limit Theorem If (X 1,X 2, X n ) are i.i.d. continuous random variables, then the joint distribution is f X CLT proves that f X is Gaussian with mean E[X i ] and Var[X i ] X = f X 1, X 2,, X n = 1 n i=1 n X i as n Somewhat of a justification for assuming Gaussian noise 12
13 Joint RVs and Marginal Densities Joint densities: F X,Y x, y = P X x Y y = x y fx,y α, β dαdβ Marginal densities: f X x = fx,y x, β dβ p X x i = j p X,Y (x i, y j ) Expectation and Covariance: E X + Y = E X + E[Y] cov X, Y = E[ X E X X Y E Y Y = E XY E X E[Y] Var X + Y = Var X + 2cov X, Y + Var(Y) 13
14 Conditional Probability P(X Y)= Fraction of the worlds in which X is true given that Y is also true. For example: H= Having a headache F= Coming down with flu P(Headche Flu) = fraction of flu-inflicted worlds in which you have a headache. How to calculate? Definition: Corollary: P X Y = P(X Y) P(Y) = P Y X P X P Y P X Y = P Y X P(X) This is called Bayes Rule 14
15 The Bayes Rule P Headache Flu = = P Headache,Flu P Flu P Flu Headache P Headache P Flu Other cases: P X Y P(Y) P Y X = P X Y P Y +P X Y P( Y) P X Y P(Y) P Y = y i X = i S P X Y = y i P Y=y i P Y X, Z = P X Y, Z P(Y,Z) P(X,Z) = P X Y, Z P(Y,Z) P X Y, Z P Y,Z +P X Y, Z P( Y,Z) 15
16 Rules of Independence Recall that for events E and H, the probability of E given H, written as P(E H), is P E H P E H = P H E and H are (statistically) independent if P E H = P E P(H) Or equivalently P E = P(E H) That means, the probability of E is true doesn't depend on whether H is true or not E and F are conditionally independent given H if P E H, F = P(E H) Or equivalently P E, F H = P E H P(F H) 16
17 Rules of Independence Examples: P(Virus Drink Beer) = P(Virus) iff Virus is independent of Drink Beer P(Flu Virus;DrinkBeer) = P(Flu Virus) iff Flu is independent of Drink Beer, given Virus P(Headache Flu;Virus;DrinkBeer) = P(Headache Flu;DrinkBeer) iff Headache is independent of Virus, given Flu and Drink Beer Assume the above independence, we obtain: P(Headache;Flu;Virus;DrinkBeer) =P(Headache Flu;Virus;DrinkBeer) P(Flu Virus;DrinkBeer) P(Virus Drink Beer) P(DrinkBeer) =P(Headache Flu;DrinkBeer) P(Flu Virus) P(Virus) P(DrinkBeer) 17
18 Multivariate Gaussian p x μ, Σ = 1 2π n/2 Σ 1/2 exp{ 1 2 x μ Σ 1 x μ } Moment Parameterization μ = E(X) Σ = Cov X = E[ X μ X μ ] Mahalanobis Distance 2 = x = μ Σ 1 x μ Canonical Parameterization p x η, Λ = exp{a + η x 1 2 x Λx} whereλ = Σ 1, η = Σ 1 μ, a = 1 (n log2π log Λ + 2 η Λη ) Tons of applications (MoG, FA, PPCA, Kalman filter, ) 18
19 Multivariate Gaussian P (X1, X2) Joint Gaussian P X 1, X 2 Marginal Gaussian μ = μ 1 μ 2 = μ 2 m = μ 2 Σ 2 m = Σ 2 Conditional Gaussian P X 1 X 2 = x 2 μ 1 2 = μ 1 + Σ 12 Σ 1 22 x 2 μ 2 Σ 1 2 = Σ 11 Σ 12 Σ 1 22 Σ 21 19
20 Operations on Gaussian R.V. The linear transform of a Gaussian r.v. is a Gaussian. Remember that no matter how x is distributed E AX + b = AE X + b Cov AX + b = ACov X A this means that for Gaussian distributed quantities: X N μ, Σ AX + b N(Aμ + b, AΣA ) The sum of two independent Gaussian r.v. is a Gaussian Y = X 1 + X 2, X 1 X 2 μ y = μ 1 + μ 2, Σ y = Σ 1 + Σ 2 The multiplication of two Gaussian functions is another Gaussian function (although no longer normalized) N a, A N b, B N c, C, where C = A 1 + B 1 1, c = CA 1 a + CB 1 b 20
21 MLE Example: toss a coin Objective function: l θ; Head = log P Head θ = log θ n 1 θ N n = n log θ + N n log( 1 θ) We need to maximize this w.r.t. θ Take derivatives w.r.t. θ dl dθ = n θ N n 1 θ = 0 θ MLE = n N 21
22 Linear Algebra 22
23 Outline Motivating Example Eigenfaces Basics Dot and Vector Products Identity, Diagonal and Orthogonal Matrices Trace Norms Rank and linear independence Range and Null Space Column and Row Space Determinant and Inverse of a matrix Eigenvalues and Eigenvectors Singular Value Decomposition Matrix Calculus 23
24 Motivating Example - EigenFaces Consider the task of recognizing faces from images. Given images of size 512x512, each image contains 262,144 dimensions or features. Not all dimensions are equally important in classifying faces. Solution To use ideas from linear algebra, especially eigenvectors, to form a new set of reduced features. 24
25 Linear Algebra Basics - I Linear algebra provides a way of compactly representing and operating on sets of linear equations 4x 1 5x 2 = 13 2x 1 + 3x 2 = 9 can be written in the form of Ax = b A = b = 13 9 A R m n denotes a matrix with m rows and n columns, where elements belong to real numbers. x R n denotes a vector with n real entries. By convention an n dimensional vector is often thought as a matrix with n rows and 1 column. 25
26 Linear Algebra Basics - II Transpose of a matrix results from flipping the rows and columns. Given A R m n, transpose is A R n m For each element of the matrix, the transpose can be written as A ij = Aji The following properties of the transposes are easily verified A = A (AB) = B A (A + B) = A + B A square matrix A R n n is symmetric if A = A and it is anti-symmetric if A = A. Thus each matrix can be written as a sum of symmetric and anti-symmetric matrices. 26
27 Vector and Matrix Multiplication - I The product of two matrices A R m n and B R n p is given by C R m p n, where C ij = A ik B kj k=1 Given two vectors x, y R n, the term x y (also x y) is called the inner product or dot product of the vectors, and is a real number given by n x i y i. For example, i=1 x y = x 1 x 2 x 3 y 3 2 = i=1 x i y i y 3 y 1 Given two vectors x R n, y R m, the term xy is called the outer product of the vectors, and is a matrix given by (x i y j ) = x i y j.. For example, 27
28 Vector and Matrix Multiplication - II xy = x 1 x 2 y 1 y 2 y 3 = x 3 x 1 y 1 x 1 y 2 x 1 y 3 x 2 y 1 x 2 y 2 x 2 y 3 x 3 y 1 x 3 y 2 x 3 y 3 The dot product also has a geometrical interpretation, for vectors in x, y R 2 with angle θ between them x x y = x y cosθ θ y which leads to use of dot product for testing orthogonality, getting the Euclidean norm of a vector, and scalar projections. 28
29 Trace of a Matrix The trace of a matrix A R n n, denoted as tr(a), is the sum of the diagonal elements in the matrix tr(a) = i=1 n A ii The trace has the following properties For A R n n, tr(a) = tra For A, B R n n, tr(a + B) = tr(a) + tr(b) For A R n n, t R, tr ta = t tr(a) For A, B, C such that ABC is a square matrix tr(abc) = tr(bca) = tr(cab) The trace of a matrix helps us easily compute norms and eigenvalues of matrices as we will see later 29
30 Linear Independence and Rank A set of vectors x 1, x 2,..., x n R m are said to be (linearly) independent if no vector can be represented as a linear combination of the remaining vectors. That is if x n = n 1 i=1 α i x i for some scalar values α 1, α 2, R then we say that the vectors are linearly dependent; otherwise the vectors are linearly independent The column rank of a matrix A R m n is the size of the largest subset of columns of A that constitute a linearly independent set. Row rank of a matrix is defined similarly for rows of a matrix. For any matrix row rank = column rank 30
31 Range and Null Space The span of a set of vectors x 1, x 2,..., x n is the set of all vectors that can be expressed as a linear combination of the set v: v = n i=1 α i x i, α i R If x 1, x 2,..., x n R n is a set of linearly independent set of vectors, then span( x 1, x 2,..., x n ) = R n The range of a matrix A R m n, denoted as R(A), is the span of the columns of A The nullspace of a matrix A R m n, denoted N A, is the set of all vectors that equal 0 when multiplied by A N A = x R n Ax = 0 31
32 Column and Row Space The row space and column space are the linear subspaces generated by row and column vectors of a matrix Linear subspace, is a vector space that is a subset of some other higher dimension vector space For a matrix A R m n Col space(a) = span(columns of A) Rank(A) = dim(row space(a)) = dim(col space(a)) 32
33 Norms I Norm of a vector x of a vector is informally a measure of the length More formally, a norm is any function f : R n R that satisfies For all x R n, f(x) 0 (non-negativity) f(x) = 0 is and only if x = 0 (definiteness) For x R n, t R, f tx = t f x (homogeneity) For all x, y R n, f(x + y) f(x) + f(y) (triangle inequality) Common norms used in machine learning are l 2 norm x 2 = n i=1 x 2 i 33
34 Norm - II l 1 norm n x 1 = i=1 x i l norm x = max i x i All norms presented so far are examples of the family of l p norms, which are parameterized by a real number p 1 n x p = i=1 x i p 1 p Norms can be defined for matrices, such as the Frobenius norm. m n A F = i=1 j=1 A ij2 = tr(a A) 34
35 Identity, Diagonal and Orthogonal Matrices The identity matrix, denoted by I R n n is a square matrix with ones on the diagonal and zeros everywhere else A diagonal matrix is matrix where all non-diagonal matrices are 0. This is typically denoted as D = diag(d 1, d 2, d 3,..., d n ) Two vectors x, y R n are orthogonal if x. y = 0. A square matrix U R n n is orthogonal if all its columns are orthogonal to each other and are normalized It follows from orthogonality and normality that U T U = I = UU T Ux 2 = x 2 35
36 Determinant and Inverse of a Matrix The determinant of a square matrix A R n n is a function f: R n n R, denoted by A or det A, and is calculated as A = n i=1 ( 1) i+j a ij A \i, \j (for any j 1,2,, n) The inverse of a square matrix A R n n is denoted A 1 is the unique matrix such that A 1 A = I = AA 1 and For some square matrices A 1 may not exist, and we say that A is singular or non-invertible. In order for A to have an inverse, A must be full rank. For non-square matrices the inverse, denoted by A +,is given by A + = A A 1 A T called the pseudo inverse 36
37 Eigenvalues and Eigenvectors - I Given a square matrix A R n n we say that λ C is an eigenvalue of A and x C n is an eigenvector if Ax = λx, x 0 Intuitively this means that upon multiplying the matrix A with a vector x, we get the same vector, but scaled by a parameter λ Geometrically, we are transforming the matrix A from its original orthonormal basis/co-ordinates to a new set of orthonormal basis x with magnitude as λ 37
38 Eigenvalues and Eigenvectors - II Source - Wikipedia 38
39 Eigenvalues and Eigenvectors - III All the eigenvectors can be written together as AX = XΛ where the diagonals of X are the eigenvectors of A, and Λ is a diagonal matrix whose elements are eigenvalues of A If the eigenvectors of A are invertible, then A = XΛX 1 There are several properties of eigenvalues and eigenvectors n Tr(A) = i=1 λ i A = i=1 n λ i Rank of A is the number of non-zero eigenvalues of A If A is non-singular then 1/λ i are the eigenvalues of A 1 The eigenvalues of a diagonal matrix are the diagonal elements of the matrix itself! 39
40 Eigenvalues and Eigenvectors - IV For a symmetric matrix A, it can be shown that eigenvalues are real and the eigenvectors are orthonormal. Thus it can be represented as UΛU Considering quadratic form of A, x Ax = x UΛU x = y Λy = n λ i y i 2 (where y = U x) Since y i 2 is always positive the sign of the expression always depends on λ i. If λ i >0 then the matrix A is positive definite, if λ i 0 then the matrix A is positive semidefinite i=1 For a multi variate Gaussian, the variances of x and y do not fully describe the distribution. The eigenvectors of this covariance matrix capture the directions of highest variance and eigenvalues the variance 40
41 Eigenvalues and Eigenvectors - V We can rewrite the original equation in the following manner Ax = λx, x 0 λi A x = 0, x 0 This is only possible if λi A is singular, that is λi A = 0. Thus, eigenvalues and eigenvectors can be computed. Compute the determinant of A λi. This results in a polynomial of degree n. Find the roots of the polynomial by equating it to zero. The n roots are the n eigenvalues of A. They make A λi singular. For each eigenvalue λ, solve A λi x to find an eigenvector x 41
42 Singular Value Decomposition Singular value decomposition, known as SVD, is a factorization of a real matrix with applications in calculating pseudo-inverse, rank, solving linear equations, and many others. For a matrix M R m n assume n m M = UΣV where U R m m, V R n n, Σ R m n The m columns of U, and the n columns of V are called the left and right singular vectors of M. The diagonal elements of Σ, Σ ii are known as the singular values of M. Let v be the i th column of V, and u be the i th column of U, and σ be the i th diagonal element of Σ Mv = σu and M u = σv 42
43 Singular Value Decomposition - II Σ 11 Σ 1n M = u 1 u 2 u n v 1 v 2 v n Σ m1 Σ mn principal directions Singular value decomposition is related to eigenvalue decomposition Suppose X = x 1 u x 2 u x m u R m n Then covariance matrix is C = 1 m XX Starting from singular vector pair M u = σv MM u = σmv MM u = σ 2 u Cu = λu Scaling factor Projection in principal directions T 43
44 Singular Value Decomposition - III Source - Wikipedia 44
45 Matrix Calculus For a vector x, b R n, let f x = b x, then x b x is equal to b f(x) xk = x k i=1 n b i x i = bk Now for a quadratic function, f x = x Ax, with A S n, f(x) xk f(x) = 2Ax xk = x k n i=1 n i=1 A ij x i x j = i k A ik x i + j k A kj x j + 2Akkxk = 2 n i=1 A ki x i Let f X = X 1, then X 1 = X 1 ( X)X 1 45
46 References for self study Best Resources for review of material - Linear Algebra Review and Reference by Zico Kotler Matrix Cookbook by KB Peterson 46
Review (Probability & Linear Algebra)
Review (Probability & Linear Algebra) CE-725 : Statistical Pattern Recognition Sharif University of Technology Spring 2013 M. Soleymani Outline Axioms of probability theory Conditional probability, Joint
More informationKnowledge Discovery and Data Mining 1 (VO) ( )
Knowledge Discovery and Data Mining 1 (VO) (707.003) Review of Linear Algebra Denis Helic KTI, TU Graz Oct 9, 2014 Denis Helic (KTI, TU Graz) KDDM1 Oct 9, 2014 1 / 74 Big picture: KDDM Probability Theory
More informationReview (probability, linear algebra) CE-717 : Machine Learning Sharif University of Technology
Review (probability, linear algebra) CE-717 : Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Some slides have been adopted from Prof. H.R. Rabiee s and also Prof. R. Gutierrez-Osuna
More informationAarti Singh. Lecture 2, January 13, Reading: Bishop: Chap 1,2. Slides courtesy: Eric Xing, Andrew Moore, Tom Mitchell
Machine Learning 0-70/5 70/5-78, 78, Spring 00 Probability 0 Aarti Singh Lecture, January 3, 00 f(x) µ x Reading: Bishop: Chap, Slides courtesy: Eric Xing, Andrew Moore, Tom Mitchell Announcements Homework
More informationVectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1. x 2. x =
Linear Algebra Review Vectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1 x x = 2. x n Vectors of up to three dimensions are easy to diagram.
More informationLinear Algebra Review. Vectors
Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors
More information10-701/ Recitation : Linear Algebra Review (based on notes written by Jing Xiang)
10-701/15-781 Recitation : Linear Algebra Review (based on notes written by Jing Xiang) Manojit Nandi February 1, 2014 Outline Linear Algebra General Properties Matrix Operations Inner Products and Orthogonal
More informationFundamentals. CS 281A: Statistical Learning Theory. Yangqing Jia. August, Based on tutorial slides by Lester Mackey and Ariel Kleiner
Fundamentals CS 281A: Statistical Learning Theory Yangqing Jia Based on tutorial slides by Lester Mackey and Ariel Kleiner August, 2011 Outline 1 Probability 2 Statistics 3 Linear Algebra 4 Optimization
More information2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian
FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian
More informationCS 246 Review of Linear Algebra 01/17/19
1 Linear algebra In this section we will discuss vectors and matrices. We denote the (i, j)th entry of a matrix A as A ij, and the ith entry of a vector as v i. 1.1 Vectors and vector operations A vector
More information[POLS 8500] Review of Linear Algebra, Probability and Information Theory
[POLS 8500] Review of Linear Algebra, Probability and Information Theory Professor Jason Anastasopoulos ljanastas@uga.edu January 12, 2017 For today... Basic linear algebra. Basic probability. Programming
More informationCS 143 Linear Algebra Review
CS 143 Linear Algebra Review Stefan Roth September 29, 2003 Introductory Remarks This review does not aim at mathematical rigor very much, but instead at ease of understanding and conciseness. Please see
More informationStatistics for scientists and engineers
Statistics for scientists and engineers February 0, 006 Contents Introduction. Motivation - why study statistics?................................... Examples..................................................3
More informationLinear Algebra. Session 12
Linear Algebra. Session 12 Dr. Marco A Roque Sol 08/01/2017 Example 12.1 Find the constant function that is the least squares fit to the following data x 0 1 2 3 f(x) 1 0 1 2 Solution c = 1 c = 0 f (x)
More informationStat 206: Linear algebra
Stat 206: Linear algebra James Johndrow (adapted from Iain Johnstone s notes) 2016-11-02 Vectors We have already been working with vectors, but let s review a few more concepts. The inner product of two
More informationStatistical Pattern Recognition
Statistical Pattern Recognition A Brief Mathematical Review Hamid R. Rabiee Jafar Muhammadi, Ali Jalali, Alireza Ghasemi Spring 2012 http://ce.sharif.edu/courses/90-91/2/ce725-1/ Agenda Probability theory
More informationMatrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =
30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can
More informationLecture 8: Linear Algebra Background
CSE 521: Design and Analysis of Algorithms I Winter 2017 Lecture 8: Linear Algebra Background Lecturer: Shayan Oveis Gharan 2/1/2017 Scribe: Swati Padmanabhan Disclaimer: These notes have not been subjected
More informationLinear Algebra Review and Reference
Linear Algebra Review and Reference Zico Kolter (updated by Chuong Do) October 7, 2008 Contents 1 Basic Concepts and Notation 2 11 BasicNotation 2 2 Matrix Multiplication 3 21 Vector-VectorProducts 4 22
More informationReview of Some Concepts from Linear Algebra: Part 2
Review of Some Concepts from Linear Algebra: Part 2 Department of Mathematics Boise State University January 16, 2019 Math 566 Linear Algebra Review: Part 2 January 16, 2019 1 / 22 Vector spaces A set
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationBackground Mathematics (2/2) 1. David Barber
Background Mathematics (2/2) 1 David Barber University College London Modified by Samson Cheung (sccheung@ieee.org) 1 These slides accompany the book Bayesian Reasoning and Machine Learning. The book and
More informationPerhaps the simplest way of modeling two (discrete) random variables is by means of a joint PMF, defined as follows.
Chapter 5 Two Random Variables In a practical engineering problem, there is almost always causal relationship between different events. Some relationships are determined by physical laws, e.g., voltage
More informationDuke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014
Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014 Linear Algebra A Brief Reminder Purpose. The purpose of this document
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More informationMath Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88
Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant
More informationReview of Linear Algebra
Review of Linear Algebra Definitions An m n (read "m by n") matrix, is a rectangular array of entries, where m is the number of rows and n the number of columns. 2 Definitions (Con t) A is square if m=
More informationA Quick Tour of Linear Algebra and Optimization for Machine Learning
A Quick Tour of Linear Algebra and Optimization for Machine Learning Masoud Farivar January 8, 2015 1 / 28 Outline of Part I: Review of Basic Linear Algebra Matrices and Vectors Matrix Multiplication Operators
More information1. General Vector Spaces
1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule
More informationLecture 2: Linear Algebra Review
EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 2-6 of the web textbook 1 2.1 Vectors 2.1.1
More information1 Proof techniques. CS 224W Linear Algebra, Probability, and Proof Techniques
1 Proof techniques Here we will learn to prove universal mathematical statements, like the square of any odd number is odd. It s easy enough to show that this is true in specific cases for example, 3 2
More informationMathematical foundations - linear algebra
Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More informationRecitation 2: Probability
Recitation 2: Probability Colin White, Kenny Marino January 23, 2018 Outline Facts about sets Definitions and facts about probability Random Variables and Joint Distributions Characteristics of distributions
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More information(a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? Solution: dim N(A) 1, since rank(a) 3. Ax =
. (5 points) (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? dim N(A), since rank(a) 3. (b) If we also know that Ax = has no solution, what do we know about the rank of A? C(A)
More informationMathematical foundations - linear algebra
Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More informationLecture 1: Review of linear algebra
Lecture 1: Review of linear algebra Linear functions and linearization Inverse matrix, least-squares and least-norm solutions Subspaces, basis, and dimension Change of basis and similarity transformations
More informationMachine Learning for Large-Scale Data Analysis and Decision Making A. Week #1
Machine Learning for Large-Scale Data Analysis and Decision Making 80-629-17A Week #1 Today Introduction to machine learning The course (syllabus) Math review (probability + linear algebra) The future
More informationLinear Algebra Formulas. Ben Lee
Linear Algebra Formulas Ben Lee January 27, 2016 Definitions and Terms Diagonal: Diagonal of matrix A is a collection of entries A ij where i = j. Diagonal Matrix: A matrix (usually square), where entries
More information2. Review of Linear Algebra
2. Review of Linear Algebra ECE 83, Spring 217 In this course we will represent signals as vectors and operators (e.g., filters, transforms, etc) as matrices. This lecture reviews basic concepts from linear
More informationReview of some mathematical tools
MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical
More informationVectors and Matrices Statistics with Vectors and Matrices
Vectors and Matrices Statistics with Vectors and Matrices Lecture 3 September 7, 005 Analysis Lecture #3-9/7/005 Slide 1 of 55 Today s Lecture Vectors and Matrices (Supplement A - augmented with SAS proc
More informationBasic Elements of Linear Algebra
A Basic Review of Linear Algebra Nick West nickwest@stanfordedu September 16, 2010 Part I Basic Elements of Linear Algebra Although the subject of linear algebra is much broader than just vectors and matrices,
More informationMachine Learning. Theory of Classification and Nonparametric Classifier. Lecture 2, January 16, What is theoretically the best classifier
Machine Learning 10-701/15 701/15-781, 781, Spring 2008 Theory of Classification and Nonparametric Classifier Eric Xing Lecture 2, January 16, 2006 Reading: Chap. 2,5 CB and handouts Outline What is theoretically
More informationDimensionality Reduction: PCA. Nicholas Ruozzi University of Texas at Dallas
Dimensionality Reduction: PCA Nicholas Ruozzi University of Texas at Dallas Eigenvalues λ is an eigenvalue of a matrix A R n n if the linear system Ax = λx has at least one non-zero solution If Ax = λx
More informationReview problems for MA 54, Fall 2004.
Review problems for MA 54, Fall 2004. Below are the review problems for the final. They are mostly homework problems, or very similar. If you are comfortable doing these problems, you should be fine on
More information10-725/36-725: Convex Optimization Prerequisite Topics
10-725/36-725: Convex Optimization Prerequisite Topics February 3, 2015 This is meant to be a brief, informal refresher of some topics that will form building blocks in this course. The content of the
More informationAPPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the n-dimensional column vector with components x 1 x 2.
APPENDIX A Background Mathematics A. Linear Algebra A.. Vector algebra Let x denote the n-dimensional column vector with components 0 x x 2 B C @. A x n Definition 6 (scalar product). The scalar product
More informationExercise Sheet 1.
Exercise Sheet 1 You can download my lecture and exercise sheets at the address http://sami.hust.edu.vn/giang-vien/?name=huynt 1) Let A, B be sets. What does the statement "A is not a subset of B " mean?
More informationLinear Algebra for Machine Learning. Sargur N. Srihari
Linear Algebra for Machine Learning Sargur N. srihari@cedar.buffalo.edu 1 Overview Linear Algebra is based on continuous math rather than discrete math Computer scientists have little experience with it
More information3. Probability and Statistics
FE661 - Statistical Methods for Financial Engineering 3. Probability and Statistics Jitkomut Songsiri definitions, probability measures conditional expectations correlation and covariance some important
More informationThe Singular Value Decomposition
The Singular Value Decomposition Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) SVD Fall 2015 1 / 13 Review of Key Concepts We review some key definitions and results about matrices that will
More informationGlossary of Linear Algebra Terms. Prepared by Vince Zaccone For Campus Learning Assistance Services at UCSB
Glossary of Linear Algebra Terms Basis (for a subspace) A linearly independent set of vectors that spans the space Basic Variable A variable in a linear system that corresponds to a pivot column in the
More informationLinear Algebra Review. Fei-Fei Li
Linear Algebra Review Fei-Fei Li 1 / 37 Vectors Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, pixel brightnesses, etc. A vector
More informationLinear Algebra in Computer Vision. Lecture2: Basic Linear Algebra & Probability. Vector. Vector Operations
Linear Algebra in Computer Vision CSED441:Introduction to Computer Vision (2017F Lecture2: Basic Linear Algebra & Probability Bohyung Han CSE, POSTECH bhhan@postech.ac.kr Mathematics in vector space Linear
More information1. Linear systems of equations. Chapters 7-8: Linear Algebra. Solution(s) of a linear system of equations (continued)
1 A linear system of equations of the form Sections 75, 78 & 81 a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2 a m1 x 1 + a m2 x 2 + + a mn x n = b m can be written in matrix
More informationLinear Algebra & Geometry why is linear algebra useful in computer vision?
Linear Algebra & Geometry why is linear algebra useful in computer vision? References: -Any book on linear algebra! -[HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia
More informationLinear Algebra Primer
Linear Algebra Primer David Doria daviddoria@gmail.com Wednesday 3 rd December, 2008 Contents Why is it called Linear Algebra? 4 2 What is a Matrix? 4 2. Input and Output.....................................
More informationBasic Concepts in Matrix Algebra
Basic Concepts in Matrix Algebra An column array of p elements is called a vector of dimension p and is written as x p 1 = x 1 x 2. x p. The transpose of the column vector x p 1 is row vector x = [x 1
More informationDS-GA 1002 Lecture notes 10 November 23, Linear models
DS-GA 2 Lecture notes November 23, 2 Linear functions Linear models A linear model encodes the assumption that two quantities are linearly related. Mathematically, this is characterized using linear functions.
More informationGaussian Models (9/9/13)
STA561: Probabilistic machine learning Gaussian Models (9/9/13) Lecturer: Barbara Engelhardt Scribes: Xi He, Jiangwei Pan, Ali Razeen, Animesh Srivastava 1 Multivariate Normal Distribution The multivariate
More informationConceptual Questions for Review
Conceptual Questions for Review Chapter 1 1.1 Which vectors are linear combinations of v = (3, 1) and w = (4, 3)? 1.2 Compare the dot product of v = (3, 1) and w = (4, 3) to the product of their lengths.
More informationSTA141C: Big Data & High Performance Statistical Computing
STA141C: Big Data & High Performance Statistical Computing Numerical Linear Algebra Background Cho-Jui Hsieh UC Davis May 15, 2018 Linear Algebra Background Vectors A vector has a direction and a magnitude
More informationMA 575 Linear Models: Cedric E. Ginestet, Boston University Revision: Probability and Linear Algebra Week 1, Lecture 2
MA 575 Linear Models: Cedric E Ginestet, Boston University Revision: Probability and Linear Algebra Week 1, Lecture 2 1 Revision: Probability Theory 11 Random Variables A real-valued random variable is
More informationSingular Value Decomposition (SVD)
School of Computing National University of Singapore CS CS524 Theoretical Foundations of Multimedia More Linear Algebra Singular Value Decomposition (SVD) The highpoint of linear algebra Gilbert Strang
More informationLinear Algebra: Matrix Eigenvalue Problems
CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given
More informationLecture Notes 1: Vector spaces
Optimization-based data analysis Fall 2017 Lecture Notes 1: Vector spaces In this chapter we review certain basic concepts of linear algebra, highlighting their application to signal processing. 1 Vector
More informationMATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP)
MATH 20F: LINEAR ALGEBRA LECTURE B00 (T KEMP) Definition 01 If T (x) = Ax is a linear transformation from R n to R m then Nul (T ) = {x R n : T (x) = 0} = Nul (A) Ran (T ) = {Ax R m : x R n } = {b R m
More informationLecture Notes 1 Probability and Random Variables. Conditional Probability and Independence. Functions of a Random Variable
Lecture Notes 1 Probability and Random Variables Probability Spaces Conditional Probability and Independence Random Variables Functions of a Random Variable Generation of a Random Variable Jointly Distributed
More information8. Diagonalization.
8. Diagonalization 8.1. Matrix Representations of Linear Transformations Matrix of A Linear Operator with Respect to A Basis We know that every linear transformation T: R n R m has an associated standard
More informationECE 650 Lecture 4. Intro to Estimation Theory Random Vectors. ECE 650 D. Van Alphen 1
EE 650 Lecture 4 Intro to Estimation Theory Random Vectors EE 650 D. Van Alphen 1 Lecture Overview: Random Variables & Estimation Theory Functions of RV s (5.9) Introduction to Estimation Theory MMSE Estimation
More informationProbability Theory for Machine Learning. Chris Cremer September 2015
Probability Theory for Machine Learning Chris Cremer September 2015 Outline Motivation Probability Definitions and Rules Probability Distributions MLE for Gaussian Parameter Estimation MLE and Least Squares
More informationIV. Matrix Approximation using Least-Squares
IV. Matrix Approximation using Least-Squares The SVD and Matrix Approximation We begin with the following fundamental question. Let A be an M N matrix with rank R. What is the closest matrix to A that
More informationLinear Algebra March 16, 2019
Linear Algebra March 16, 2019 2 Contents 0.1 Notation................................ 4 1 Systems of linear equations, and matrices 5 1.1 Systems of linear equations..................... 5 1.2 Augmented
More informationVector spaces. DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis.
Vector spaces DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_fall17/index.html Carlos Fernandez-Granda Vector space Consists of: A set V A scalar
More informationSTAT 100C: Linear models
STAT 100C: Linear models Arash A. Amini April 27, 2018 1 / 1 Table of Contents 2 / 1 Linear Algebra Review Read 3.1 and 3.2 from text. 1. Fundamental subspace (rank-nullity, etc.) Im(X ) = ker(x T ) R
More information1. What is the determinant of the following matrix? a 1 a 2 4a 3 2a 2 b 1 b 2 4b 3 2b c 1. = 4, then det
What is the determinant of the following matrix? 3 4 3 4 3 4 4 3 A 0 B 8 C 55 D 0 E 60 If det a a a 3 b b b 3 c c c 3 = 4, then det a a 4a 3 a b b 4b 3 b c c c 3 c = A 8 B 6 C 4 D E 3 Let A be an n n matrix
More informationChap 3. Linear Algebra
Chap 3. Linear Algebra Outlines 1. Introduction 2. Basis, Representation, and Orthonormalization 3. Linear Algebraic Equations 4. Similarity Transformation 5. Diagonal Form and Jordan Form 6. Functions
More informationBasics on Probability. Jingrui He 09/11/2007
Basics on Probability Jingrui He 09/11/2007 Coin Flips You flip a coin Head with probability 0.5 You flip 100 coins How many heads would you expect Coin Flips cont. You flip a coin Head with probability
More informationStat 159/259: Linear Algebra Notes
Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the
More information2. Matrix Algebra and Random Vectors
2. Matrix Algebra and Random Vectors 2.1 Introduction Multivariate data can be conveniently display as array of numbers. In general, a rectangular array of numbers with, for instance, n rows and p columns
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2
EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 2 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 5, 2012 Andre Tkacenko
More informationME 597: AUTONOMOUS MOBILE ROBOTICS SECTION 2 PROBABILITY. Prof. Steven Waslander
ME 597: AUTONOMOUS MOBILE ROBOTICS SECTION 2 Prof. Steven Waslander p(a): Probability that A is true 0 pa ( ) 1 p( True) 1, p( False) 0 p( A B) p( A) p( B) p( A B) A A B B 2 Discrete Random Variable X
More informationMath 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination
Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column
More informationUNIT 6: The singular value decomposition.
UNIT 6: The singular value decomposition. María Barbero Liñán Universidad Carlos III de Madrid Bachelor in Statistics and Business Mathematical methods II 2011-2012 A square matrix is symmetric if A T
More informationAssignment 1 Math 5341 Linear Algebra Review. Give complete answers to each of the following questions. Show all of your work.
Assignment 1 Math 5341 Linear Algebra Review Give complete answers to each of the following questions Show all of your work Note: You might struggle with some of these questions, either because it has
More informationLinear Regression and Its Applications
Linear Regression and Its Applications Predrag Radivojac October 13, 2014 Given a data set D = {(x i, y i )} n the objective is to learn the relationship between features and the target. We usually start
More informationMTH 464: Computational Linear Algebra
MTH 464: Computational Linear Algebra Lecture Outlines Exam 2 Material Prof. M. Beauregard Department of Mathematics & Statistics Stephen F. Austin State University March 2, 2018 Linear Algebra (MTH 464)
More informationMAT Linear Algebra Collection of sample exams
MAT 342 - Linear Algebra Collection of sample exams A-x. (0 pts Give the precise definition of the row echelon form. 2. ( 0 pts After performing row reductions on the augmented matrix for a certain system
More informationStudy Guide for Linear Algebra Exam 2
Study Guide for Linear Algebra Exam 2 Term Vector Space Definition A Vector Space is a nonempty set V of objects, on which are defined two operations, called addition and multiplication by scalars (real
More informationThe Singular Value Decomposition (SVD) and Principal Component Analysis (PCA)
Chapter 5 The Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) 5.1 Basics of SVD 5.1.1 Review of Key Concepts We review some key definitions and results about matrices that will
More informationLinear Algebra Review. Fei-Fei Li
Linear Algebra Review Fei-Fei Li 1 / 51 Vectors Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, pixel brightnesses, etc. A vector
More informationPROBABILITY AND INFORMATION THEORY. Dr. Gjergji Kasneci Introduction to Information Retrieval WS
PROBABILITY AND INFORMATION THEORY Dr. Gjergji Kasneci Introduction to Information Retrieval WS 2012-13 1 Outline Intro Basics of probability and information theory Probability space Rules of probability
More information1 9/5 Matrices, vectors, and their applications
1 9/5 Matrices, vectors, and their applications Algebra: study of objects and operations on them. Linear algebra: object: matrices and vectors. operations: addition, multiplication etc. Algorithms/Geometric
More informationLecture II: Linear Algebra Revisited
Lecture II: Linear Algebra Revisited Overview Vector spaces, Hilbert & Banach Spaces, etrics & Norms atrices, Eigenvalues, Orthogonal Transformations, Singular Values Operators, Operator Norms, Function
More informationLinear Algebra. Matrices Operations. Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0.
Matrices Operations Linear Algebra Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0 The rectangular array 1 2 1 4 3 4 2 6 1 3 2 1 in which the
More informationMATH 304 Linear Algebra Lecture 19: Least squares problems (continued). Norms and inner products.
MATH 304 Linear Algebra Lecture 19: Least squares problems (continued). Norms and inner products. Orthogonal projection Theorem 1 Let V be a subspace of R n. Then any vector x R n is uniquely represented
More informationIntroduction to Matrix Algebra
Introduction to Matrix Algebra August 18, 2010 1 Vectors 1.1 Notations A p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the line. When p
More informationMATH 315 Linear Algebra Homework #1 Assigned: August 20, 2018
Homework #1 Assigned: August 20, 2018 Review the following subjects involving systems of equations and matrices from Calculus II. Linear systems of equations Converting systems to matrix form Pivot entry
More informationLarge Scale Data Analysis Using Deep Learning
Large Scale Data Analysis Using Deep Learning Linear Algebra U Kang Seoul National University U Kang 1 In This Lecture Overview of linear algebra (but, not a comprehensive survey) Focused on the subset
More information