1 MATHEMATICS FOR COMPUTER VISION WEEK 2 LINEAR SYSTEMS Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year 2013-14
OUTLINE OF WEEK 2 Linear Systems and solutions Systems of linear equations Solution sets Properties, under- and over-constrained systems Solving a linear system Variable elimination Row reduction Cramer s rule Matrix solution Systems with special structure Using LU, Cholenski and QR decomposition 2
3 SYSTEMS OF LINEAR EQUATIONS Week 2 Linear Systems
SYSTEM OF LINEAR EQUATIONS collection of linear equations on the same set of variables linear equation: each term is constant or product of a constant and a single variable example: solution: assignment of numbers to the variables there exist systems of nonlinear equations as well can be approximated by linear systems 4
GENERAL FORM AND MATRIX FORM system of m linear equations in n unknowns each unknown is a coefficient of a linear combination: so that the system can be expressed in matrix form: 5
SOLUTION SET solution set: set of all assignments to the variables such that the system is satisfied three possible cases: the system has no solution the system has one solution the system has infinitely many solutions why no 2 or 5 solutions?? a geometric representation can help show why 6
GEOMETRIC INTERPRETATION consider a system in two variables: x and y each equation determines a line in the plane of all possible values of (x,y) solution = intersection of the lines example: two equations what are the possible configurations? they can be parallel: no intersection (solution) they can intersect in one point (one solution) they can coincide: infinitely many solutions 7
GEOMETRIC INTERPRETATION - CONTINUED for three variables, each equation determines a plane in a three-dimensional space example: two equations in three variables, solution is generally a line general case: each linear equation determines a hyperplane in n-dimensional space 8
UNDER/OVERDETERMINED SYSTEMS fewer equations than unknown: infinitely many solutions system is said underdetermined same number of equations and unknowns: single unique solution system with more unknowns than equations: overdetermined this just most common behaviour: actual behaviour is determined by whether the equations are linearly dependent or inconsistent 9
INDEPENDENCE independence -> no equation can be derived from the others by algebraic means that means by multiplying them for a scalar and or adding them very related to linear independence of vectors (Week 1) example: are these independent? if not, how do you prove it? 10
CONSISTENCY (EXISTENCE OF SOLUTION) a linear system is consistent if there exists a solution inconsistent otherwise case of two equations in two variables: more interesting case: three equations can be inconsistent, even when each pair of them is consistent! example: 11
12 SOLVING A LINEAR SYSTEM Week 2 Linear Systems
SOLVING A LINEAR SYSTEM many different algorithms for solving a linear system we are going to describe the most common ones here this section: general purpose algorithm variable elimination row reduction Cramer s rule matrix solution next section: solution of systems with special structure by LU decomposition by Choleski decomposition by QR decomposition 13
VARIABLE ELIMINATION recursive algorithm: solve for one of the variables in the first equation (in terms of the other) plug resuling expression into other equations do the same for the result system with n - 1 unknowns and m - 1 equations example solving for x = 5 + 2z 3y in the first equation of the first system Solving for y = 2 + 3z in the first eq of the second system 14
ROW REDUCTION (GAUSSIAN ELIMINATION) the linear system is represented by an augmented matrix example: same system as before matrix is modified by elementary row operations 1. swap two rows 2. multiply row by a nonzero scalr 3. add to one row a scalar multiple of another at each step the new system is equivalent to original (same solution set) 15
CRAMER S RULE 16
MATRIX SOLUTION AND PSEUDOINVERSE 17
18 SYSTEMS WITH SPECIAL STRUCTURE Week 2 Linear Systems
SYSTEMS WITH SPECIAL STRUCTURES if A has some special structure, this can be exploited get faster or more accurate algorithms we consider a number of cases here: 4. if A is symmetric and definite positive Cholesky decomposition 5. if A is a Toeplitz matrix Levinson recursion if the matrix is sparse (very few non-zero entries) or..if the matrix if very large iterative methods, e.g. Jacobi 19
BY LU FACTORISATION applies to square systems only LUP factorisation: PA = LU where L and U are lower and upper triangular, respectively (Week 1) used to solve linear systems by decomposing the problem into two simpler ones Ly = Pb and Ux = y we get two systems with triangular matrices can be solved by forward substitution 20
BY CHOLENSKI DECOMPOSITION applies to systems with symmetric, definite positive matrix decomposes such a matrix A into A = L T L L is lower triangular, with positive diagonal entries (we only consider real numbers here) Problem is decomposed into two steps: Solving Ly = b by forward substitution Solving L T x = y by back substitution for linear systems that can be put in symmetric form, Cholesky factorisation has best performance in terms of efficiency and numerical stability 21
ITERATIVE METHODS: JACOBI 22
JACOBI SOLUTION: EXAMPLE 23
24 SUMMARY Week 2 Linear Systems
SUMMARY OF WEEK 2 LINEAR SYSTEMS systems of linear equations and solutions matrix form number of solutions geometric interpretation independence and consistency solution methods variable elimination Gaussian elimination Cramer s rule matrix solution and pseudoinverse systems with special structure Choleski and LU factorisation iterative methods (Jacobi) for large and/or sparse matrices 25