Statistical Geometry Processing Winter Semester 2011/2012


 Jody Montgomery
 1 years ago
 Views:
Transcription
1 Statistical Geometry Processing Winter Semester 2011/2012 Linear Algebra, Function Spaces & Inverse Problems
2 Vector and Function Spaces
3 3 Vectors vectors are arrows in space classically: 2 or 3 dim. Euclidian space
4 4 Vector Operations w v v + w Adding Vectors: Concatenation
5 5 Vector Operations 2.0 v 1.5 v v 1.0 v Scalar Multiplication: Scaling vectors (incl. mirroring)
6 6 You can combine it... v 2w + v w Linear Combinations: This is basically all you can do. n r v i1 i i
7 Vector Spaces Vector space: Set of vectors V Based on field F (we use only F = ) Two operations: Adding vectors u = v + w (u, v, w V) Scaling vectors w = v (u V, F) Vector space aioms: 7
8 8 Additional Tools More concepts: Subspaces, linear spans, bases Scalar product Angle, length, orthogonality GramSchmidt orthogonalization Cross product (R 3 ) Linear maps Matrices Eigenvalues & eigenvectors Quadratic forms (Check your old math books)
9 Eample Spaces Function spaces: Space of all functions f: Space of all smooth C k functions f: Space of all functions f: [0..1] 5 8 etc... + =
10 Function Spaces Intuition: Start with a finite dimensional vector Increase sampling density towards infinity Real numbers: uncountable amount of dimensions [f 1,f 2,...,f 9 ] T [f 1,f 2,...,f 18 ] T f() d = 9 d = 18 d = 10
11 Dot Product on Function Spaces Scalar products For squareintegrable functions f, g: n, the standard scalar product is defined as: It measures an abstract norm and angle between function (not in a geometric sense) Orthogonal functions: f g : f ( ) g( ) d Do not influence each other in linear combinations. Adding one to the other does not change the value in the other ones direction. 11
12 Approimation of Function Spaces Finite dimensional subspaces: Function spaces with infinite dimension are hard to represented on a computer For numerical purpose, finitedimensional subspaces are used to approimate the larger space Two basic approaches 12
13 Approimation of Function Spaces Task: Given: Infinitedimensional function space V. Task: Find f V with a certain property. Recipe: Finite Differences Sample function f on discrete grid Approimate property discretely Derivatives => finite differences Integrals => Finite sums Optimization: Find best discrete function 13
14 14 Approimation of Function Spaces actual solution function space basis approimate solution Recipe: Finite Elements Choose basis functions b 1,..., b d V Find f = d i=1 λ i b i that matches the property best f is described by ( 1,..., d )
15 15 Eamples 2 2 1,8 1,6 1,5 1,4 1 1,2 1 0,8 0,5 0 0,6 0,4 0,20, ,5 1 1,5 21,5 0,5 1 1, Monomial basis Fourier basis Bspline basis, Gaussian basis
16 16 Best Match Linear combination matches best Solution 1: Least squares minimization n f λ i b i d min R i=1 2 Solution 2: Galerkin method Both are equivalent i = 1.. n: f λ i b i n i=1, b i = 0
17 Optimality Criterion Given: Subspace W V An element v V Then we get: V W v w w W minimizes the quadratic error w v 2 (i.e. the Euclidean distance) if and only if: the residual w v is orthogonal to W Least squares = minimal Euclidean distance 17
18 Formal Derivation Leastsquares n 2 E f = f λ i b i d R i=1 n n n = f 2 2 λ i f b i + λ i λ j b i b j R i=1 i=1 i=1 Setting derivatives to zero: d E f = 2 λ 1 f, b 1 λ n f, b n + λ 1,, λ n b i, b j Result: j = 1.. n: f λ i b i n i=1, b j = 0 18
19 Linear Maps
20 Linear Maps A Function f: V W between vector spaces V, W is linear if and only if: v 1,v 2 V: f (v 1 +v 2 ) = f (v 1 ) + f (v 2 ) vv, F: f (v) = f (v) Constructing linear mappings: A linear map is uniquely determined if we specify a mapping value for each basis vector of V. 20
21 21 Matri Representation Finite dimensional spaces Linear maps can be represented as matrices For each basis vector v i of V, we specify the mapped vector w i. Then, the map f is given by: n n n v v v v f f w w v )... ( 1 1 1
22 22 Matri Representation This can be written as matrivector product: The columns are the images of the basis vectors (for which the coordinates of v are given) n n v v v f 1 1 ) ( w w
23 Linear Systems of Equations Problem: Invert an affine map Given: M = b We know M, b Looking for Solution Set of solutions: always an affine subspace of n, or the empty set. Point, line, plane, hyperplane... Innumerous algorithms for solving linear systems 23
24 Solvers for Linear Systems Algorithms for solving linear systems of equations: Gaussian elimination: O(n 3 ) operations for n n matrices We can do better, in particular for special cases: Band matrices: constant bandwidth Sparse matrices: constant number of nonzero entries per row Store only nonzero entries Instead of (3.5, 0, 0, 0, 7, 0, 0), store [(1:3.5), (5:7)] 24
25 Solvers for Linear Systems Algorithms for solving linear systems of n equations: Band matrices, O(1) bandwidth: Modified O(n) elimination algorithm. Iterative GaussSeidel solver converges for diagonally dominant matrices Typically: O(n) iterations, each costs O(n) for a sparse matri. Conjugate Gradient solver Only symmetric, positive definite matrices Guaranteed: O(n) iterations Typically good solution after O( n) iterations. More details on iterative solvers: J. R. Shewchuk: An Introduction to the Conjugate Gradient Method Without the Agonizing Pain,
26 Eigenvectors & Eigenvalues Definition: Linear map M, nonzero vector with M = an is eigenvalue of M is the corresponding eigenvector. 26
27 Eample Intuition: In the direction of an eigenvector, the linear map acts like a scaling Eample: two eigenvalues (0.5 and 2) Two eigenvectors Standard basis contains no eigenvectors 27
28 Eigenvectors & Eigenvalues Diagonalization: In case an n n matri M has n linear independent eigenvectors, we can diagonalize M by transforming to this coordinate system: M = TDT
29 Spectral Theorem Spectral Theorem: If M is a symmetric n n matri of real numbers (i.e. M = M T ), there eists an orthogonal set of n eigenvectors. This means, every (real) symmetric matri can be diagonalized: M = TDT T with an orthogonal matri T. 29
30 Computation Simple algorithm Power iteration for symmetric matrices Computes largest eigenvalue even for large matrices Algorithm: Start with a random vector (maybe multiple tries) Repeatedly multiply with matri Normalize vector after each step Repeat until ration before / after normalization converges (this is the eigenvalue) Intuition: Largest eigenvalue = dominant component/direction 30
31 31 Powers of Matrices What happens: A symmetric matri can be written as: M TDT T 1 T n T Taking it to the kth power yields: M k TDT T TDT T k 1 T T k T T... TDT TD T T T Bottom line: Eigenvalue analysis key to understanding powers of matrices. k n
32 Improvements Improvements to the power method: Find smallest? use inverse matri. Find all (for a symmetric matri)? run repeatedly, orthogonalize current estimate to already known eigenvectors in each iteration (Gram Schmidt) How long does it take? ratio to net smaller eigenvalue, gap increases eponentially. There are more sophisticated algorithms based on this idea. 32
33 Generalization: SVD Singular value decomposition: Let M be an arbitrary real matri (may be rectangular) Then M can be written as: M = U D V T The matrices U, V are orthogonal D is a diagonal matri (might contain zeros) The diagonal entries are called singular values. U and V are different in general. For diagonalizable matrices, they are the same, and the singular values are the eigenvalues. 33
34 34 Singular Value Decomposition Singular value decomposition M U D = orthogonal V T orthogonal
35 Singular Value Decomposition Singular value decomposition Can be used to solve linear systems of equations For full rank, square M: M = U D V T M 1 = (U D V T ) 1 = (V T ) 1 D 1 (U 1 ) = V D 1 U T Good numerical properties (numerically stable) More epensive than iterative solvers The OpenCV library provides a very good implementation of the SVD 35
36 Linear Inverse Problems
37 Inverse Problems Settings A (physical) process f takes place It transforms the original input into an output b Task: recover from b Eamples: 3D structure from photographs Tomography: values from line integrals 3D geometry from a noisy 3D scan 37
38 Linear Inverse Problems Assumption: f is linear and finite dimensional f () = b M f = b Inversion of f is said to be an illposed problem, if one of the following three conditions hold: There is no solution There is more than one solution There is eactly one solution, but the SVD contains very small singular values. 38
39 39 Ill posed Problems Ratio: Small singular values amplify errors Assume ineact input Measurement noise Numerical noise Reminder: M 1 = V D 1 U T does not hurt does not hurt (orthogonal) (orthogonal) this one is decisive Orthogonal transforms preserve norm of, so V and U do not cause problems
40 40 Ill posed Problems Ratio: Small singular values amplify errors Reminder: = M 1 b = (V D 1 U T )b Say D looks like that: D : Any input noise in b in the direction of the fourth right singular vector will be amplified by If our measurement precision is less than that, the result will be unusable. Does not depend on how we invert the matri. Condition number: ma / min
41 41 Ill Posed Problems Two problems: (1) Mapping destroys information goes below noise level cannot be recovered by any means (2) Inverse mapping amplifies noise D : yields garbage solution even remaining information not recovered etremely large random solutions are obtained We can do something about problem #2
42 Regularization Regularization Avoiding destructive noise caused by inversion Various techniques Goal: ignore the misleading information Subspace inversion: Ignore subspace with small singular values needs an SVD, risk of ringing Additional assumptions: smoothness (or something similar) make compound problem (f 1 + assumptions) well posed We will look at this in detail later 42
43 43 Illustration of the Problem f g forward f g problem original function smoothed function
44 44 Illustration of the Problem f inverse f g problem reconstructed function smoothed function
45 45 Illustration of the Problem f inverse f g problem regularized reconstructed function smoothed function
46 46 Illustration of the Problem f g original function G G 1 regularization Frequency Domain Frequency Domain
47 47 Illustration of the Problem f g forward f g problem original function smoothed function * =
48 Illustration of the Problem f g forward f g problem original function * = smoothed function solution (from 2 digits) correct 48
49 Analysis Matri 1,4 1,2 1 0,8 0,6 0,4 0, Spectrum 0,6 0,6 0,4 0,4 0,2 00,2 4 3,2 3,2 0,2 00,2 0,2 0,20,40,40, Dominant Eigenvectors 0, Smallest Eigenvectors 49
50 50 Digging Deeper... Let s look at this eample again: Convolution operation: f () = R Shiftinvariant linear operator f g continuous f(t) g t dt discrete
51 Convolution Theorem Fourier Transform Function space: f: 0.. 2π R (suff. smooth) Fourier basis: 1, cos k, sin k k = 1,2, Properties The Fourier basis is an orthogonal basis (standard scalar prod.) The Fourier basis diagonalizes all shiftinvariant linear operators 51
52 Convolution Theorem This means: Let F: N R be the Fourier transform (FT) of f: [0.. 2π] R Then: f g = FT 1 (F G) Consequences Fourier spectrum of convolution operator ( filter ) determines wellposedness of inverse operation (deconvolution) Certain frequencies might be lost In our eample: high frequencies 52
53 Quadratic Forms
54 Multivariate Polynomials A multivariate polynomial of total degree d: A function f: n, f() f is a polynomial in the components of Any 1D direction f(s + tr) is a polynomial of maimum degree d in t. Eamples: f(, y) := + y + y is of total degree 2. In diagonal direction, we obtain f(t[1/ 2, 1/ 2] T ) = t 2. f(, y) := c c 02 y 2 + c 11 y + c 10 + c 01 y + c 00 is a quadratic polynomial in two variables 54
55 Quadratic Polynomials In general, any quadratic polynomial in n variables can be written as: T A + b T + c A is an n n matri, b is an ndim. vector, c is a number Matri A can always be chosen to be symmetric If it isn t, we can substitute by 0.5 (A + A T ), not changing the polynomial 55
56 56 Eample Eample: ) ( ) ( ] [ ] [ ) ( T T T y y y y y y y y y y y y y f y f
57 57 Quadratic Polynomials Specifying quadratic polynomials: T A + b T + c b shifts the function in space (if A has full rank): c is an additive constant c c c A A A A A A 2 T (A sym.) T T T T = b
58 Some Properties Important properties Multivariate polynomials form a vector space We can add them componentwise: y 2 + 4y y y 2 + 1y y + 5 = y 2 + 5y y + 9 In vector notation: T A 1 + b 1T + c 1 + ( T A 2 + b 2T + c 2 ) = T (A 1 + A 2 ) + (b 1 + b 2 ) T + (c 1 + c 2 ) 58
59 Quadratic Polynomials Quadrics Zero level set of a quadratic polynomial: quadric Shape depends on eigenvalues of A b shifts the object in space c sets the level 59
60 60 Shapes of Quadrics Shape analysis: A is symmetric A can be diagonalized with orthogonal eigenvectors Q contains the principal ais of the quadric The eigenvalues determine the quadratic growth (up, down, speed of growth) n n Q Q Q Q A 1 T 1 T T T
61 61 Shapes of Quadratic Polynomials 1 = 1, 2 = 1 1 = 1, 2 = 1 1 = 1, 2 = 0
62 62 The IsoLines: Quadrics elliptic hyperbolic degenerate case 1 > 0, 2 > 0 1 < 0, 2 > 0 1 = 0, 2 0
63 Quadratic Optimization Quadratic Optimization Minimize quadratic objective function T A + b T + c Required: A > 0 (only positive eigenvalues) It s a paraboloid with a unique minimum The verte (critical point) can be determined by simply solving a linear system Necessary and sufficient condition 2A = b 63
64 Condition Number How stable is the solution? Depends on Matri A good bad 64
65 Regularization Regularization Sums of positive semidefinite matrices are positive semidefinite Add regularizing quadric Eample Fill in the valleys Bias in the solution Original: T A + b T + c Regularized: T (A + I ) + b T + c A + I 65
66 66 Rayleigh Quotient Relation to eigenvalues: Min/ma eigenvalues of a symmetric A epressed as constraint quadratic optimization: T A T min min A T ma ma min A T 1 The other way round eigenvalues solve a certain type of constrained, (nonconve) optimization problem. ma T A T 1
67 67 Coordinate Transformations One more interesting property: Given a positive definite symmetric ( SPD ) matri M (all eigenvalues positive) Such a matri can always be written as square of another matri: n T T T D D T D T D T T D D T 1 2 T TDT M
68 SPD Quadrics main ais T T M Identity I Interpretation: Start with a unit positive quadric T. Scale the main ais (diagonal of D) M TDT T D 2 Rotate to a different coordinate system (columns of T) Recovering main ais from M: Compute eigensystem ( principal component analysis ) T 68
69 69 Why should I care? What are quadrics good for? logprobability of Gaussian models Estimation in Gaussian probabilistic models......is quadratic optimization....is solving of linear systems of equations. Quadratic optimization easy to use & solve feasible :) Approimate more comple models locally p Gaussian normal distribution, ( ) 1 ep 2π 2 2 2
70 Groups and Transformations
71 Groups Definition: A set G with operation is called a group (G, ), iff: Closed: : G G G (always maps back to G) Associativity: (f g) h = f (g h) Neutral element: there eists id G such that for any g G: g id = g Inverse element: For each g G there eists g 1 G such that g g 1 = g 1 g = id Abelian Groups The group is commutative iff always f g = g f 71
72 Eamples of Groups Eamples: G = invertible matrices, = composition (matri mult.) G = invertible affine transformation of R d, = composition (matri form: homogeneos coordinates) G = bijections of a set S to itself, = composition G = smooth C k bijections of a set S to itself, = composition G = global symmetry transforms of a shape, = composition G = permutation of a discrete set, = composition 72
73 Eamples of Groups Eamples: G = invertible matrices, = composition (matri mult.) G = invertible affine transformation of R d Subgroups: G = similarity transform (translation, rotation, mirroring, scaling 0) E(d): G = rigid motions (translation, rotation, mirroring) SE(d): G = rigid motions (translation, rotation) O(d): G = orthogonal matri (rotation, mirroring) (columns/rows orthonormal) SO(d): G = orthogonal matri (rotation) (columns/rows orthonormal, determinant 1) G = translations (the only commutative group out of these) 73
74 Eamples of Groups Eamples: G = global symmetry transforms of a 2D shape 74
75 Eamples of Groups Eamples: G = global symmetry transforms of a 3D shape (etended to infinity) (etended to infinity) 75
76 Outlook More details on this later Symmetry groups Structural regularity Crystalographic groups and regular lattices 76
Geometric Modeling Summer Semester 2010 Mathematical Tools (1)
Geometric Modeling Summer Semester 2010 Mathematical Tools (1) Recap: Linear Algebra Today... Topics: Mathematical Background Linear algebra Analysis & differential geometry Numerical techniques Geometric
More informationGeometric Modeling Summer Semester 2012 Linear Algebra & Function Spaces
Geometric Modeling Summer Semester 2012 Linear Algebra & Function Spaces (Recap) Announcement Room change: On Thursday, April 26th, room 024 is occupied. The lecture will be moved to room 021, E1 4 (the
More informationMobile Robotics 1. A Compact Course on Linear Algebra. Giorgio Grisetti
Mobile Robotics 1 A Compact Course on Linear Algebra Giorgio Grisetti SA1 Vectors Arrays of numbers They represent a point in a n dimensional space 2 Vectors: Scalar Product ScalarVector Product Changes
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 WeiTa Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More informationMath 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination
Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column
More informationReview problems for MA 54, Fall 2004.
Review problems for MA 54, Fall 2004. Below are the review problems for the final. They are mostly homework problems, or very similar. If you are comfortable doing these problems, you should be fine on
More informationIntroduction to Mobile Robotics Compact Course on Linear Algebra. Wolfram Burgard, Cyrill Stachniss, Kai Arras, Maren Bennewitz
Introduction to Mobile Robotics Compact Course on Linear Algebra Wolfram Burgard, Cyrill Stachniss, Kai Arras, Maren Bennewitz Vectors Arrays of numbers Vectors represent a point in a n dimensional space
More informationReview of some mathematical tools
MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical
More informationApplied Linear Algebra in Geoscience Using MATLAB
Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data TwoDimensional Plots Programming in
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More informationComputational Methods. Eigenvalues and Singular Values
Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations
More informationDSGA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DSGA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationBackground Mathematics (2/2) 1. David Barber
Background Mathematics (2/2) 1 David Barber University College London Modified by Samson Cheung (sccheung@ieee.org) 1 These slides accompany the book Bayesian Reasoning and Machine Learning. The book and
More informationIntroduction to Mobile Robotics Compact Course on Linear Algebra. Wolfram Burgard, Bastian Steder
Introduction to Mobile Robotics Compact Course on Linear Algebra Wolfram Burgard, Bastian Steder Reference Book Thrun, Burgard, and Fox: Probabilistic Robotics Vectors Arrays of numbers Vectors represent
More informationCourse Notes: Week 1
Course Notes: Week 1 Math 270C: Applied Numerical Linear Algebra 1 Lecture 1: Introduction (3/28/11) We will focus on iterative methods for solving linear systems of equations (and some discussion of eigenvalues
More informationMATHEMATICS COMPREHENSIVE EXAM: INCLASS COMPONENT
MATHEMATICS COMPREHENSIVE EXAM: INCLASS COMPONENT The following is the list of questions for the oral exam. At the same time, these questions represent all topics for the written exam. The procedure for
More informationApplied Linear Algebra
Applied Linear Algebra Peter J. Olver School of Mathematics University of Minnesota Minneapolis, MN 55455 olver@math.umn.edu http://www.math.umn.edu/ olver Chehrzad Shakiban Department of Mathematics University
More informationConceptual Questions for Review
Conceptual Questions for Review Chapter 1 1.1 Which vectors are linear combinations of v = (3, 1) and w = (4, 3)? 1.2 Compare the dot product of v = (3, 1) and w = (4, 3) to the product of their lengths.
More information14 Singular Value Decomposition
14 Singular Value Decomposition For any highdimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationLinear Algebra. Min Yan
Linear Algebra Min Yan January 2, 2018 2 Contents 1 Vector Space 7 1.1 Definition................................. 7 1.1.1 Axioms of Vector Space..................... 7 1.1.2 Consequence of Axiom......................
More informationMATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP)
MATH 20F: LINEAR ALGEBRA LECTURE B00 (T KEMP) Definition 01 If T (x) = Ax is a linear transformation from R n to R m then Nul (T ) = {x R n : T (x) = 0} = Nul (A) Ran (T ) = {Ax R m : x R n } = {b R m
More information33A Linear Algebra and Applications: Practice Final Exam  Solutions
33A Linear Algebra and Applications: Practice Final Eam  Solutions Question Consider a plane V in R 3 with a basis given by v = and v =. Suppose, y are both in V. (a) [3 points] If [ ] B =, find. (b)
More informationLINEAR ALGEBRA KNOWLEDGE SURVEY
LINEAR ALGEBRA KNOWLEDGE SURVEY Instructions: This is a Knowledge Survey. For this assignment, I am only interested in your level of confidence about your ability to do the tasks on the following pages.
More informationIntroduction to Mobile Robotics Compact Course on Linear Algebra. Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Diego Tipaldi, Luciano Spinello
Introduction to Mobile Robotics Compact Course on Linear Algebra Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Diego Tipaldi, Luciano Spinello Vectors Arrays of numbers Vectors represent a point
More informationLecture II: Linear Algebra Revisited
Lecture II: Linear Algebra Revisited Overview Vector spaces, Hilbert & Banach Spaces, etrics & Norms atrices, Eigenvalues, Orthogonal Transformations, Singular Values Operators, Operator Norms, Function
More information4.2. ORTHOGONALITY 161
4.2. ORTHOGONALITY 161 Definition 4.2.9 An affine space (E, E ) is a Euclidean affine space iff its underlying vector space E is a Euclidean vector space. Given any two points a, b E, we define the distance
More informationUNIT 6: The singular value decomposition.
UNIT 6: The singular value decomposition. María Barbero Liñán Universidad Carlos III de Madrid Bachelor in Statistics and Business Mathematical methods II 20112012 A square matrix is symmetric if A T
More informationhomogeneous 71 hyperplane 10 hyperplane 34 hyperplane 69 identity map 171 identity map 186 identity map 206 identity matrix 110 identity matrix 45
address 12 adjoint matrix 118 alternating 112 alternating 203 angle 159 angle 33 angle 60 area 120 associative 180 augmented matrix 11 axes 5 Axiom of Choice 153 basis 178 basis 210 basis 74 basis test
More informationLECTURE 7. Least Squares and Variants. Optimization Models EE 127 / EE 227AT. Outline. Least Squares. Notes. Notes. Notes. Notes.
Optimization Models EE 127 / EE 227AT Laurent El Ghaoui EECS department UC Berkeley Spring 2015 Sp 15 1 / 23 LECTURE 7 Least Squares and Variants If others would but reflect on mathematical truths as deeply
More informationMath 302 Outcome Statements Winter 2013
Math 302 Outcome Statements Winter 2013 1 Rectangular Space Coordinates; Vectors in the ThreeDimensional Space (a) Cartesian coordinates of a point (b) sphere (c) symmetry about a point, a line, and a
More informationLINEAR ALGEBRA 1, 2012I PARTIAL EXAM 3 SOLUTIONS TO PRACTICE PROBLEMS
LINEAR ALGEBRA, I PARTIAL EXAM SOLUTIONS TO PRACTICE PROBLEMS Problem (a) For each of the two matrices below, (i) determine whether it is diagonalizable, (ii) determine whether it is orthogonally diagonalizable,
More informationNumerical Methods I NonSquare and Sparse Linear Systems
Numerical Methods I NonSquare and Sparse Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATHGA 2011.003 / CSCIGA 2945.003, Fall 2014 September 25th, 2014 A. Donev (Courant
More informationk is a product of elementary matrices.
Mathematics, Spring Lecture (Wilson) Final Eam May, ANSWERS Problem (5 points) (a) There are three kinds of elementary row operations and associated elementary matrices. Describe what each kind of operation
More informationThe value of a problem is not so much coming up with the answer as in the ideas and attempted ideas it forces on the would be solver I.N.
Math 410 Homework Problems In the following pages you will find all of the homework problems for the semester. Homework should be written out neatly and stapled and turned in at the beginning of class
More informationMatrices and Vectors. Definition of Matrix. An MxN matrix A is a twodimensional array of numbers A =
30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a twodimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can
More informationStat 159/259: Linear Algebra Notes
Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the
More informationCSE 554 Lecture 7: Alignment
CSE 554 Lecture 7: Alignment Fall 2012 CSE554 Alignment Slide 1 Review Fairing (smoothing) Relocating vertices to achieve a smoother appearance Method: centroid averaging Simplification Reducing vertex
More informationFinal Exam, Linear Algebra, Fall, 2003, W. Stephen Wilson
Final Exam, Linear Algebra, Fall, 2003, W. Stephen Wilson Name: TA Name and section: NO CALCULATORS, SHOW ALL WORK, NO OTHER PAPERS ON DESK. There is very little actual work to be done on this exam if
More information7. Symmetric Matrices and Quadratic Forms
Linear Algebra 7. Symmetric Matrices and Quadratic Forms CSIE NCU 1 7. Symmetric Matrices and Quadratic Forms 7.1 Diagonalization of symmetric matrices 2 7.2 Quadratic forms.. 9 7.4 The singular value
More informationLinear Algebra Final Exam Review
Linear Algebra Final Exam Review. Let A be invertible. Show that, if v, v, v 3 are linearly independent vectors, so are Av, Av, Av 3. NOTE: It should be clear from your answer that you know the definition.
More informationPreliminary Examination, Numerical Analysis, August 2016
Preliminary Examination, Numerical Analysis, August 2016 Instructions: This exam is closed books and notes. The time allowed is three hours and you need to work on any three out of questions 14 and any
More informationLinear Algebra Review
Linear Algebra Review CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Linear Algebra Review 1 / 16 Midterm Exam Tuesday Feb
More informationReduction to the associated homogeneous system via a particular solution
June PURDUE UNIVERSITY Study Guide for the Credit Exam in (MA 5) Linear Algebra This study guide describes briefly the course materials to be covered in MA 5. In order to be qualified for the credit, one
More informationWe wish the reader success in future encounters with the concepts of linear algebra.
Afterword Our path through linear algebra has emphasized spaces of vectors in dimension 2, 3, and 4 as a means of introducing concepts which go forward to IRn for arbitrary n. But linear algebra does not
More informationLinear Algebra Primer
Linear Algebra Primer David Doria daviddoria@gmail.com Wednesday 3 rd December, 2008 Contents Why is it called Linear Algebra? 4 2 What is a Matrix? 4 2. Input and Output.....................................
More informationWe use the overhead arrow to denote a column vector, i.e., a number with a direction. For example, in threespace, we write
1 MATH FACTS 11 Vectors 111 Definition We use the overhead arrow to denote a column vector, ie, a number with a direction For example, in threespace, we write The elements of a vector have a graphical
More informationLinear LeastSquares Data Fitting
CHAPTER 6 Linear LeastSquares Data Fitting 61 Introduction Recall that in chapter 3 we were discussing linear systems of equations, written in shorthand in the form Ax = b In chapter 3, we just considered
More informationElementary linear algebra
Chapter 1 Elementary linear algebra 1.1 Vector spaces Vector spaces owe their importance to the fact that so many models arising in the solutions of specific problems turn out to be vector spaces. The
More informationLeast Squares with Examples in Signal Processing 1. 2 Overdetermined equations. 1 Notation. The sum of squares of x is denoted by x 2 2, i.e.
Least Squares with Eamples in Signal Processing Ivan Selesnick March 7, 3 NYUPoly These notes address (approimate) solutions to linear equations by least squares We deal with the easy case wherein the
More informationCS 143 Linear Algebra Review
CS 143 Linear Algebra Review Stefan Roth September 29, 2003 Introductory Remarks This review does not aim at mathematical rigor very much, but instead at ease of understanding and conciseness. Please see
More informationMATH 31  ADDITIONAL PRACTICE PROBLEMS FOR FINAL
MATH 3  ADDITIONAL PRACTICE PROBLEMS FOR FINAL MAIN TOPICS FOR THE FINAL EXAM:. Vectors. Dot product. Cross product. Geometric applications. 2. Row reduction. Null space, column space, row space, left
More informationMath Subject GRE Questions
Math Subject GRE Questions Calculus and Differential Equations 1. If f() = e e, then [f ()] 2 [f()] 2 equals (a) 4 (b) 4e 2 (c) 2e (d) 2 (e) 2e 2. An integrating factor for the ordinary differential equation
More informationLecture 6. Numerical methods. Approximation of functions
Lecture 6 Numerical methods Approximation of functions Lecture 6 OUTLINE 1. Approximation and interpolation 2. Leastsquare method basis functions design matrix residual weighted least squares normal equation
More informationDuke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014
Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014 Linear Algebra A Brief Reminder Purpose. The purpose of this document
More informationIterative Methods for Solving A x = b
Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http
More informationCambridge University Press The Mathematics of Signal Processing Steven B. Damelin and Willard Miller Excerpt More information
Introduction Consider a linear system y = Φx where Φ can be taken as an m n matrix acting on Euclidean space or more generally, a linear operator on a Hilbert space. We call the vector x a signal or input,
More informationLinear Algebra review Powers of a diagonalizable matrix Spectral decomposition
Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2016 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing
More informationPreface to Second Edition... vii. Preface to First Edition...
Contents Preface to Second Edition..................................... vii Preface to First Edition....................................... ix Part I Linear Algebra 1 Basic Vector/Matrix Structure and
More informationLinear Algebra: Matrix Eigenvalue Problems
CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given
More informationMATH 583A REVIEW SESSION #1
MATH 583A REVIEW SESSION #1 BOJAN DURICKOVIC 1. Vector Spaces Very quick review of the basic linear algebra concepts (see any linear algebra textbook): (finite dimensional) vector space (or linear space),
More informationCOMP 558 lecture 18 Nov. 15, 2010
Least squares We have seen several least squares problems thus far, and we will see more in the upcoming lectures. For this reason it is good to have a more general picture of these problems and how to
More informationNumerical Methods I Eigenvalue Problems
Numerical Methods I Eigenvalue Problems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATHGA 2011.003 / CSCIGA 2945.003, Fall 2014 October 2nd, 2014 A. Donev (Courant Institute) Lecture
More informationLinear Algebra & Geometry why is linear algebra useful in computer vision?
Linear Algebra & Geometry why is linear algebra useful in computer vision? References: Any book on linear algebra! [HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia
More information8. Diagonalization.
8. Diagonalization 8.1. Matrix Representations of Linear Transformations Matrix of A Linear Operator with Respect to A Basis We know that every linear transformation T: R n R m has an associated standard
More informationAPPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the ndimensional column vector with components x 1 x 2.
APPENDIX A Background Mathematics A. Linear Algebra A.. Vector algebra Let x denote the ndimensional column vector with components 0 x x 2 B C @. A x n Definition 6 (scalar product). The scalar product
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Classical Iterations We have a problem, We assume that the matrix comes from a discretization of a PDE. The best and most popular model problem is, The matrix will be as large
More informationUpon successful completion of MATH 220, the student will be able to:
MATH 220 Matrices Upon successful completion of MATH 220, the student will be able to: 1. Identify a system of linear equations (or linear system) and describe its solution set 2. Write down the coefficient
More informationLinear Algebra review Powers of a diagonalizable matrix Spectral decomposition
Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2018 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing
More information1. Foundations of Numerics from Advanced Mathematics. Linear Algebra
Foundations of Numerics from Advanced Mathematics Linear Algebra Linear Algebra, October 23, 22 Linear Algebra Mathematical Structures a mathematical structure consists of one or several sets and one or
More informationImage Registration Lecture 2: Vectors and Matrices
Image Registration Lecture 2: Vectors and Matrices Prof. Charlene Tsai Lecture Overview Vectors Matrices Basics Orthogonal matrices Singular Value Decomposition (SVD) 2 1 Preliminary Comments Some of this
More informationMath Linear Algebra II. 1. Inner Products and Norms
Math 342  Linear Algebra II Notes 1. Inner Products and Norms One knows from a basic introduction to vectors in R n Math 254 at OSU) that the length of a vector x = x 1 x 2... x n ) T R n, denoted x,
More informationGlossary of Linear Algebra Terms. Prepared by Vince Zaccone For Campus Learning Assistance Services at UCSB
Glossary of Linear Algebra Terms Basis (for a subspace) A linearly independent set of vectors that spans the space Basic Variable A variable in a linear system that corresponds to a pivot column in the
More informationReview of Some Concepts from Linear Algebra: Part 2
Review of Some Concepts from Linear Algebra: Part 2 Department of Mathematics Boise State University January 16, 2019 Math 566 Linear Algebra Review: Part 2 January 16, 2019 1 / 22 Vector spaces A set
More informationLinear Algebra Review. Vectors
Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors
More informationMATH 1120 (LINEAR ALGEBRA 1), FINAL EXAM FALL 2011 SOLUTIONS TO PRACTICE VERSION
MATH (LINEAR ALGEBRA ) FINAL EXAM FALL SOLUTIONS TO PRACTICE VERSION Problem (a) For each matrix below (i) find a basis for its column space (ii) find a basis for its row space (iii) determine whether
More informationApplied Linear Algebra in Geoscience Using MATLAB
Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data TwoDimensional Plots Programming in
More informationLecture Notes 1: Vector spaces
Optimizationbased data analysis Fall 2017 Lecture Notes 1: Vector spaces In this chapter we review certain basic concepts of linear algebra, highlighting their application to signal processing. 1 Vector
More information1 9/5 Matrices, vectors, and their applications
1 9/5 Matrices, vectors, and their applications Algebra: study of objects and operations on them. Linear algebra: object: matrices and vectors. operations: addition, multiplication etc. Algorithms/Geometric
More informationand let s calculate the image of some vectors under the transformation T.
Chapter 5 Eigenvalues and Eigenvectors 5. Eigenvalues and Eigenvectors Let T : R n R n be a linear transformation. Then T can be represented by a matrix (the standard matrix), and we can write T ( v) =
More informationChapter 6. Nonlinear Equations. 6.1 The Problem of Nonlinear Rootfinding. 6.2 Rate of Convergence
Chapter 6 Nonlinear Equations 6. The Problem of Nonlinear Rootfinding In this module we consider the problem of using numerical techniques to find the roots of nonlinear equations, f () =. Initially we
More informationLinear Equations in Linear Algebra
1 Linear Equations in Linear Algebra 1.1 SYSTEMS OF LINEAR EQUATIONS LINEAR EQUATION,, 1 n A linear equation in the variables equation that can be written in the form a a a b 1 1 2 2 n n a a is an where
More informationI. Multiple Choice Questions (Answer any eight)
Name of the student : Roll No : CS65: Linear Algebra and Random Processes Exam  Course Instructor : Prashanth L.A. Date : Sep24, 27 Duration : 5 minutes INSTRUCTIONS: The test will be evaluated ONLY
More informationMathematical foundations  linear algebra
Mathematical foundations  linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More informationLinear Algebra for Machine Learning. Sargur N. Srihari
Linear Algebra for Machine Learning Sargur N. srihari@cedar.buffalo.edu 1 Overview Linear Algebra is based on continuous math rather than discrete math Computer scientists have little experience with it
More informationAlgebra C Numerical Linear Algebra Sample Exam Problems
Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finitedimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 19: More on Arnoldi Iteration; Lanczos Iteration Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis I 1 / 17 Outline 1
More informationBASIC ALGORITHMS IN LINEAR ALGEBRA. Matrices and Applications of Gaussian Elimination. A 2 x. A T m x. A 1 x A T 1. A m x
BASIC ALGORITHMS IN LINEAR ALGEBRA STEVEN DALE CUTKOSKY Matrices and Applications of Gaussian Elimination Systems of Equations Suppose that A is an n n matrix with coefficents in a field F, and x = (x,,
More informationGRE Subject test preparation Spring 2016 Topic: Abstract Algebra, Linear Algebra, Number Theory.
GRE Subject test preparation Spring 2016 Topic: Abstract Algebra, Linear Algebra, Number Theory. Linear Algebra Standard matrix manipulation to compute the kernel, intersection of subspaces, column spaces,
More informationMathematics 1. Part II: Linear Algebra. Exercises and problems
Bachelor Degree in Informatics Engineering Barcelona School of Informatics Mathematics Part II: Linear Algebra Eercises and problems February 5 Departament de Matemàtica Aplicada Universitat Politècnica
More informationLinear Algebra & Geometry why is linear algebra useful in computer vision?
Linear Algebra & Geometry why is linear algebra useful in computer vision? References: Any book on linear algebra! [HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia
More informationLecture 2: Linear Algebra Review
EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 26 of the web textbook 1 2.1 Vectors 2.1.1
More informationPaul Heckbert. Computer Science Department Carnegie Mellon University. 26 Sept B  Introduction to Scientific Computing 1
Paul Heckbert Computer Science Department Carnegie Mellon University 26 Sept. 2 5859B  Introduction to Scientific Computing aerospace: simulate subsonic & supersonic air flow around full aircraft, no
More informationMath 224, Fall 2007 Exam 3 Thursday, December 6, 2007
Math 224, Fall 2007 Exam 3 Thursday, December 6, 2007 You have 1 hour and 20 minutes. No notes, books, or other references. You are permitted to use Maple during this exam, but you must start with a blank
More informationMaths for Signals and Systems Linear Algebra in Engineering
Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE
More informationSUMMARY OF MATH 1600
SUMMARY OF MATH 1600 Note: The following list is intended as a study guide for the final exam. It is a continuation of the study guide for the midterm. It does not claim to be a comprehensive list. You
More informationMath 411 Preliminaries
Math 411 Preliminaries Provide a list of preliminary vocabulary and concepts Preliminary Basic Netwon s method, Taylor series expansion (for single and multiple variables), Eigenvalue, Eigenvector, Vector
More information15 Singular Value Decomposition
15 Singular Value Decomposition For any highdimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationANSWERS (5 points) Let A be a 2 2 matrix such that A =. Compute A. 2
MATH 7 Final Exam Sample Problems Spring 7 ANSWERS ) ) ). 5 points) Let A be a matrix such that A =. Compute A. ) A = A ) = ) = ). 5 points) State ) the definition of norm, ) the CauchySchwartz inequality
More informationB553 Lecture 5: Matrix Algebra Review
B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations
More information