Statistical Geometry Processing Winter Semester 2011/2012

Save this PDF as:
Size: px
Start display at page:

Transcription

1 Statistical Geometry Processing Winter Semester 2011/2012 Linear Algebra, Function Spaces & Inverse Problems

2 Vector and Function Spaces

3 3 Vectors vectors are arrows in space classically: 2 or 3 dim. Euclidian space

4 4 Vector Operations w v v + w Adding Vectors: Concatenation

5 5 Vector Operations 2.0 v 1.5 v v -1.0 v Scalar Multiplication: Scaling vectors (incl. mirroring)

6 6 You can combine it... v 2w + v w Linear Combinations: This is basically all you can do. n r v i1 i i

7 Vector Spaces Vector space: Set of vectors V Based on field F (we use only F = ) Two operations: Adding vectors u = v + w (u, v, w V) Scaling vectors w = v (u V, F) Vector space aioms: 7

8 8 Additional Tools More concepts: Subspaces, linear spans, bases Scalar product Angle, length, orthogonality Gram-Schmidt orthogonalization Cross product (R 3 ) Linear maps Matrices Eigenvalues & eigenvectors Quadratic forms (Check your old math books)

9 Eample Spaces Function spaces: Space of all functions f: Space of all smooth C k functions f: Space of all functions f: [0..1] 5 8 etc... + =

10 Function Spaces Intuition: Start with a finite dimensional vector Increase sampling density towards infinity Real numbers: uncountable amount of dimensions [f 1,f 2,...,f 9 ] T [f 1,f 2,...,f 18 ] T f() d = 9 d = 18 d = 10

11 Dot Product on Function Spaces Scalar products For square-integrable functions f, g: n, the standard scalar product is defined as: It measures an abstract norm and angle between function (not in a geometric sense) Orthogonal functions: f g : f ( ) g( ) d Do not influence each other in linear combinations. Adding one to the other does not change the value in the other ones direction. 11

12 Approimation of Function Spaces Finite dimensional subspaces: Function spaces with infinite dimension are hard to represented on a computer For numerical purpose, finite-dimensional subspaces are used to approimate the larger space Two basic approaches 12

13 Approimation of Function Spaces Task: Given: Infinite-dimensional function space V. Task: Find f V with a certain property. Recipe: Finite Differences Sample function f on discrete grid Approimate property discretely Derivatives => finite differences Integrals => Finite sums Optimization: Find best discrete function 13

14 14 Approimation of Function Spaces actual solution function space basis approimate solution Recipe: Finite Elements Choose basis functions b 1,..., b d V Find f = d i=1 λ i b i that matches the property best f is described by ( 1,..., d )

15 15 Eamples 2 2 1,8 1,6 1,5 1,4 1 1,2 1 0,8 0,5 0 0,6 0,4 0,2-0, ,5 1 1,5 2-1,5 0,5 1 1, Monomial basis Fourier basis B-spline basis, Gaussian basis

16 16 Best Match Linear combination matches best Solution 1: Least squares minimization n f λ i b i d min R i=1 2 Solution 2: Galerkin method Both are equivalent i = 1.. n: f λ i b i n i=1, b i = 0

17 Optimality Criterion Given: Subspace W V An element v V Then we get: V W v w w W minimizes the quadratic error w v 2 (i.e. the Euclidean distance) if and only if: the residual w v is orthogonal to W Least squares = minimal Euclidean distance 17

18 Formal Derivation Least-squares n 2 E f = f λ i b i d R i=1 n n n = f 2 2 λ i f b i + λ i λ j b i b j R i=1 i=1 i=1 Setting derivatives to zero: d E f = 2 λ 1 f, b 1 λ n f, b n + λ 1,, λ n b i, b j Result: j = 1.. n: f λ i b i n i=1, b j = 0 18

19 Linear Maps

20 Linear Maps A Function f: V W between vector spaces V, W is linear if and only if: v 1,v 2 V: f (v 1 +v 2 ) = f (v 1 ) + f (v 2 ) vv, F: f (v) = f (v) Constructing linear mappings: A linear map is uniquely determined if we specify a mapping value for each basis vector of V. 20

21 21 Matri Representation Finite dimensional spaces Linear maps can be represented as matrices For each basis vector v i of V, we specify the mapped vector w i. Then, the map f is given by: n n n v v v v f f w w v )... ( 1 1 1

22 22 Matri Representation This can be written as matri-vector product: The columns are the images of the basis vectors (for which the coordinates of v are given) n n v v v f 1 1 ) ( w w

23 Linear Systems of Equations Problem: Invert an affine map Given: M = b We know M, b Looking for Solution Set of solutions: always an affine subspace of n, or the empty set. Point, line, plane, hyperplane... Innumerous algorithms for solving linear systems 23

24 Solvers for Linear Systems Algorithms for solving linear systems of equations: Gaussian elimination: O(n 3 ) operations for n n matrices We can do better, in particular for special cases: Band matrices: constant bandwidth Sparse matrices: constant number of non-zero entries per row Store only non-zero entries Instead of (3.5, 0, 0, 0, 7, 0, 0), store [(1:3.5), (5:7)] 24

25 Solvers for Linear Systems Algorithms for solving linear systems of n equations: Band matrices, O(1) bandwidth: Modified O(n) elimination algorithm. Iterative Gauss-Seidel solver converges for diagonally dominant matrices Typically: O(n) iterations, each costs O(n) for a sparse matri. Conjugate Gradient solver Only symmetric, positive definite matrices Guaranteed: O(n) iterations Typically good solution after O( n) iterations. More details on iterative solvers: J. R. Shewchuk: An Introduction to the Conjugate Gradient Method Without the Agonizing Pain,

26 Eigenvectors & Eigenvalues Definition: Linear map M, non-zero vector with M = an is eigenvalue of M is the corresponding eigenvector. 26

27 Eample Intuition: In the direction of an eigenvector, the linear map acts like a scaling Eample: two eigenvalues (0.5 and 2) Two eigenvectors Standard basis contains no eigenvectors 27

28 Eigenvectors & Eigenvalues Diagonalization: In case an n n matri M has n linear independent eigenvectors, we can diagonalize M by transforming to this coordinate system: M = TDT

29 Spectral Theorem Spectral Theorem: If M is a symmetric n n matri of real numbers (i.e. M = M T ), there eists an orthogonal set of n eigenvectors. This means, every (real) symmetric matri can be diagonalized: M = TDT T with an orthogonal matri T. 29

30 Computation Simple algorithm Power iteration for symmetric matrices Computes largest eigenvalue even for large matrices Algorithm: Start with a random vector (maybe multiple tries) Repeatedly multiply with matri Normalize vector after each step Repeat until ration before / after normalization converges (this is the eigenvalue) Intuition: Largest eigenvalue = dominant component/direction 30

31 31 Powers of Matrices What happens: A symmetric matri can be written as: M TDT T 1 T n T Taking it to the k-th power yields: M k TDT T TDT T k 1 T T k T T... TDT TD T T T Bottom line: Eigenvalue analysis key to understanding powers of matrices. k n

32 Improvements Improvements to the power method: Find smallest? use inverse matri. Find all (for a symmetric matri)? run repeatedly, orthogonalize current estimate to already known eigenvectors in each iteration (Gram Schmidt) How long does it take? ratio to net smaller eigenvalue, gap increases eponentially. There are more sophisticated algorithms based on this idea. 32

33 Generalization: SVD Singular value decomposition: Let M be an arbitrary real matri (may be rectangular) Then M can be written as: M = U D V T The matrices U, V are orthogonal D is a diagonal matri (might contain zeros) The diagonal entries are called singular values. U and V are different in general. For diagonalizable matrices, they are the same, and the singular values are the eigenvalues. 33

34 34 Singular Value Decomposition Singular value decomposition M U D = orthogonal V T orthogonal

35 Singular Value Decomposition Singular value decomposition Can be used to solve linear systems of equations For full rank, square M: M = U D V T M -1 = (U D V T ) -1 = (V T ) -1 D -1 (U -1 ) = V D -1 U T Good numerical properties (numerically stable) More epensive than iterative solvers The OpenCV library provides a very good implementation of the SVD 35

36 Linear Inverse Problems

37 Inverse Problems Settings A (physical) process f takes place It transforms the original input into an output b Task: recover from b Eamples: 3D structure from photographs Tomography: values from line integrals 3D geometry from a noisy 3D scan 37

38 Linear Inverse Problems Assumption: f is linear and finite dimensional f () = b M f = b Inversion of f is said to be an ill-posed problem, if one of the following three conditions hold: There is no solution There is more than one solution There is eactly one solution, but the SVD contains very small singular values. 38

39 39 Ill posed Problems Ratio: Small singular values amplify errors Assume ineact input Measurement noise Numerical noise Reminder: M -1 = V D -1 U T does not hurt does not hurt (orthogonal) (orthogonal) this one is decisive Orthogonal transforms preserve norm of, so V and U do not cause problems

40 40 Ill posed Problems Ratio: Small singular values amplify errors Reminder: = M -1 b = (V D -1 U T )b Say D looks like that: D : Any input noise in b in the direction of the fourth right singular vector will be amplified by If our measurement precision is less than that, the result will be unusable. Does not depend on how we invert the matri. Condition number: ma / min

41 41 Ill Posed Problems Two problems: (1) Mapping destroys information goes below noise level cannot be recovered by any means (2) Inverse mapping amplifies noise D : yields garbage solution even remaining information not recovered etremely large random solutions are obtained We can do something about problem #2

42 Regularization Regularization Avoiding destructive noise caused by inversion Various techniques Goal: ignore the misleading information Subspace inversion: Ignore subspace with small singular values needs an SVD, risk of ringing Additional assumptions: smoothness (or something similar) make compound problem (f -1 + assumptions) well posed We will look at this in detail later 42

43 43 Illustration of the Problem f g forward f g problem original function smoothed function

44 44 Illustration of the Problem f inverse f g problem reconstructed function smoothed function

45 45 Illustration of the Problem f inverse f g problem regularized reconstructed function smoothed function

46 46 Illustration of the Problem f g original function G G -1 regularization Frequency Domain Frequency Domain

47 47 Illustration of the Problem f g forward f g problem original function smoothed function * =

48 Illustration of the Problem f g forward f g problem original function * = smoothed function solution (from 2 digits) correct 48

49 Analysis Matri 1,4 1,2 1 0,8 0,6 0,4 0, Spectrum 0,6 0,6 0,4 0,4 0,2 0-0,2 4 3,2 3,2 0,2 0-0,2 0,2 0,2-0,4-0,4-0, Dominant Eigenvectors -0, Smallest Eigenvectors 49

50 50 Digging Deeper... Let s look at this eample again: Convolution operation: f () = R Shift-invariant linear operator f g continuous f(t) g t dt discrete

51 Convolution Theorem Fourier Transform Function space: f: 0.. 2π R (suff. smooth) Fourier basis: 1, cos k, sin k k = 1,2, Properties The Fourier basis is an orthogonal basis (standard scalar prod.) The Fourier basis diagonalizes all shift-invariant linear operators 51

52 Convolution Theorem This means: Let F: N R be the Fourier transform (FT) of f: [0.. 2π] R Then: f g = FT 1 (F G) Consequences Fourier spectrum of convolution operator ( filter ) determines well-posedness of inverse operation (deconvolution) Certain frequencies might be lost In our eample: high frequencies 52

54 Multivariate Polynomials A multi-variate polynomial of total degree d: A function f: n, f() f is a polynomial in the components of Any 1D direction f(s + tr) is a polynomial of maimum degree d in t. Eamples: f(, y) := + y + y is of total degree 2. In diagonal direction, we obtain f(t[1/ 2, 1/ 2] T ) = t 2. f(, y) := c c 02 y 2 + c 11 y + c 10 + c 01 y + c 00 is a quadratic polynomial in two variables 54

55 Quadratic Polynomials In general, any quadratic polynomial in n variables can be written as: T A + b T + c A is an n n matri, b is an n-dim. vector, c is a number Matri A can always be chosen to be symmetric If it isn t, we can substitute by 0.5 (A + A T ), not changing the polynomial 55

56 56 Eample Eample: ) ( ) ( ] [ ] [ ) ( T T T y y y y y y y y y y y y y f y f

57 57 Quadratic Polynomials Specifying quadratic polynomials: T A + b T + c b shifts the function in space (if A has full rank): c is an additive constant c c c A A A A A A 2 T (A sym.) T T T T = b

58 Some Properties Important properties Multivariate polynomials form a vector space We can add them component-wise: y 2 + 4y y y 2 + 1y y + 5 = y 2 + 5y y + 9 In vector notation: T A 1 + b 1T + c 1 + ( T A 2 + b 2T + c 2 ) = T (A 1 + A 2 ) + (b 1 + b 2 ) T + (c 1 + c 2 ) 58

59 Quadratic Polynomials Quadrics Zero level set of a quadratic polynomial: quadric Shape depends on eigenvalues of A b shifts the object in space c sets the level 59

60 60 Shapes of Quadrics Shape analysis: A is symmetric A can be diagonalized with orthogonal eigenvectors Q contains the principal ais of the quadric The eigenvalues determine the quadratic growth (up, down, speed of growth) n n Q Q Q Q A 1 T 1 T T T

61 61 Shapes of Quadratic Polynomials 1 = 1, 2 = 1 1 = 1, 2 = -1 1 = 1, 2 = 0

62 62 The Iso-Lines: Quadrics elliptic hyperbolic degenerate case 1 > 0, 2 > 0 1 < 0, 2 > 0 1 = 0, 2 0

63 Quadratic Optimization Quadratic Optimization Minimize quadratic objective function T A + b T + c Required: A > 0 (only positive eigenvalues) It s a paraboloid with a unique minimum The verte (critical point) can be determined by simply solving a linear system Necessary and sufficient condition 2A = b 63

64 Condition Number How stable is the solution? Depends on Matri A good bad 64

65 Regularization Regularization Sums of positive semi-definite matrices are positive semi-definite Add regularizing quadric Eample Fill in the valleys Bias in the solution Original: T A + b T + c Regularized: T (A + I ) + b T + c A + I 65

66 66 Rayleigh Quotient Relation to eigenvalues: Min/ma eigenvalues of a symmetric A epressed as constraint quadratic optimization: T A T min min A T ma ma min A T 1 The other way round eigenvalues solve a certain type of constrained, (non-conve) optimization problem. ma T A T 1

67 67 Coordinate Transformations One more interesting property: Given a positive definite symmetric ( SPD ) matri M (all eigenvalues positive) Such a matri can always be written as square of another matri: n T T T D D T D T D T T D D T 1 2 T TDT M

68 SPD Quadrics main ais T T M Identity I Interpretation: Start with a unit positive quadric T. Scale the main ais (diagonal of D) M TDT T D 2 Rotate to a different coordinate system (columns of T) Recovering main ais from M: Compute eigensystem ( principal component analysis ) T 68

69 69 Why should I care? What are quadrics good for? log-probability of Gaussian models Estimation in Gaussian probabilistic models......is quadratic optimization....is solving of linear systems of equations. Quadratic optimization easy to use & solve feasible :-) Approimate more comple models locally p Gaussian normal distribution, ( ) 1 ep 2π 2 2 2

70 Groups and Transformations

71 Groups Definition: A set G with operation is called a group (G, ), iff: Closed: : G G G (always maps back to G) Associativity: (f g) h = f (g h) Neutral element: there eists id G such that for any g G: g id = g Inverse element: For each g G there eists g 1 G such that g g 1 = g 1 g = id Abelian Groups The group is commutative iff always f g = g f 71

72 Eamples of Groups Eamples: G = invertible matrices, = composition (matri mult.) G = invertible affine transformation of R d, = composition (matri form: homogeneos coordinates) G = bijections of a set S to itself, = composition G = smooth C k bijections of a set S to itself, = composition G = global symmetry transforms of a shape, = composition G = permutation of a discrete set, = composition 72

73 Eamples of Groups Eamples: G = invertible matrices, = composition (matri mult.) G = invertible affine transformation of R d Subgroups: G = similarity transform (translation, rotation, mirroring, scaling 0) E(d): G = rigid motions (translation, rotation, mirroring) SE(d): G = rigid motions (translation, rotation) O(d): G = orthogonal matri (rotation, mirroring) (columns/rows orthonormal) SO(d): G = orthogonal matri (rotation) (columns/rows orthonormal, determinant 1) G = translations (the only commutative group out of these) 73

74 Eamples of Groups Eamples: G = global symmetry transforms of a 2D shape 74

75 Eamples of Groups Eamples: G = global symmetry transforms of a 3D shape (etended to infinity) (etended to infinity) 75

76 Outlook More details on this later Symmetry groups Structural regularity Crystalographic groups and regular lattices 76

Geometric Modeling Summer Semester 2010 Mathematical Tools (1)

Geometric Modeling Summer Semester 2010 Mathematical Tools (1) Recap: Linear Algebra Today... Topics: Mathematical Background Linear algebra Analysis & differential geometry Numerical techniques Geometric

Geometric Modeling Summer Semester 2012 Linear Algebra & Function Spaces

Geometric Modeling Summer Semester 2012 Linear Algebra & Function Spaces (Recap) Announcement Room change: On Thursday, April 26th, room 024 is occupied. The lecture will be moved to room 021, E1 4 (the

Mobile Robotics 1. A Compact Course on Linear Algebra. Giorgio Grisetti

Mobile Robotics 1 A Compact Course on Linear Algebra Giorgio Grisetti SA-1 Vectors Arrays of numbers They represent a point in a n dimensional space 2 Vectors: Scalar Product Scalar-Vector Product Changes

Chapter 3 Transformations

Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases

Math 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination

Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column

Review problems for MA 54, Fall 2004.

Review problems for MA 54, Fall 2004. Below are the review problems for the final. They are mostly homework problems, or very similar. If you are comfortable doing these problems, you should be fine on

Introduction to Mobile Robotics Compact Course on Linear Algebra. Wolfram Burgard, Cyrill Stachniss, Kai Arras, Maren Bennewitz

Introduction to Mobile Robotics Compact Course on Linear Algebra Wolfram Burgard, Cyrill Stachniss, Kai Arras, Maren Bennewitz Vectors Arrays of numbers Vectors represent a point in a n dimensional space

Review of some mathematical tools

MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical

Applied Linear Algebra in Geoscience Using MATLAB

Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots Programming in

Linear Algebra Massoud Malek

CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product

Computational Methods. Eigenvalues and Singular Values

Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations

DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.

DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1

Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012

Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.

Background Mathematics (2/2) 1. David Barber

Background Mathematics (2/2) 1 David Barber University College London Modified by Samson Cheung (sccheung@ieee.org) 1 These slides accompany the book Bayesian Reasoning and Machine Learning. The book and

Introduction to Mobile Robotics Compact Course on Linear Algebra. Wolfram Burgard, Bastian Steder

Introduction to Mobile Robotics Compact Course on Linear Algebra Wolfram Burgard, Bastian Steder Reference Book Thrun, Burgard, and Fox: Probabilistic Robotics Vectors Arrays of numbers Vectors represent

Course Notes: Week 1

Course Notes: Week 1 Math 270C: Applied Numerical Linear Algebra 1 Lecture 1: Introduction (3/28/11) We will focus on iterative methods for solving linear systems of equations (and some discussion of eigenvalues

MATHEMATICS COMPREHENSIVE EXAM: IN-CLASS COMPONENT

MATHEMATICS COMPREHENSIVE EXAM: IN-CLASS COMPONENT The following is the list of questions for the oral exam. At the same time, these questions represent all topics for the written exam. The procedure for

Applied Linear Algebra

Applied Linear Algebra Peter J. Olver School of Mathematics University of Minnesota Minneapolis, MN 55455 olver@math.umn.edu http://www.math.umn.edu/ olver Chehrzad Shakiban Department of Mathematics University

Conceptual Questions for Review

Conceptual Questions for Review Chapter 1 1.1 Which vectors are linear combinations of v = (3, 1) and w = (4, 3)? 1.2 Compare the dot product of v = (3, 1) and w = (4, 3) to the product of their lengths.

14 Singular Value Decomposition

14 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing

Linear Algebra. Min Yan

Linear Algebra Min Yan January 2, 2018 2 Contents 1 Vector Space 7 1.1 Definition................................. 7 1.1.1 Axioms of Vector Space..................... 7 1.1.2 Consequence of Axiom......................

MATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP)

MATH 20F: LINEAR ALGEBRA LECTURE B00 (T KEMP) Definition 01 If T (x) = Ax is a linear transformation from R n to R m then Nul (T ) = {x R n : T (x) = 0} = Nul (A) Ran (T ) = {Ax R m : x R n } = {b R m

33A Linear Algebra and Applications: Practice Final Exam - Solutions

33A Linear Algebra and Applications: Practice Final Eam - Solutions Question Consider a plane V in R 3 with a basis given by v = and v =. Suppose, y are both in V. (a) [3 points] If [ ] B =, find. (b)

LINEAR ALGEBRA KNOWLEDGE SURVEY

LINEAR ALGEBRA KNOWLEDGE SURVEY Instructions: This is a Knowledge Survey. For this assignment, I am only interested in your level of confidence about your ability to do the tasks on the following pages.

Introduction to Mobile Robotics Compact Course on Linear Algebra. Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Diego Tipaldi, Luciano Spinello

Introduction to Mobile Robotics Compact Course on Linear Algebra Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Diego Tipaldi, Luciano Spinello Vectors Arrays of numbers Vectors represent a point

Lecture II: Linear Algebra Revisited

Lecture II: Linear Algebra Revisited Overview Vector spaces, Hilbert & Banach Spaces, etrics & Norms atrices, Eigenvalues, Orthogonal Transformations, Singular Values Operators, Operator Norms, Function

4.2. ORTHOGONALITY 161

4.2. ORTHOGONALITY 161 Definition 4.2.9 An affine space (E, E ) is a Euclidean affine space iff its underlying vector space E is a Euclidean vector space. Given any two points a, b E, we define the distance

UNIT 6: The singular value decomposition.

UNIT 6: The singular value decomposition. María Barbero Liñán Universidad Carlos III de Madrid Bachelor in Statistics and Business Mathematical methods II 2011-2012 A square matrix is symmetric if A T

homogeneous 71 hyperplane 10 hyperplane 34 hyperplane 69 identity map 171 identity map 186 identity map 206 identity matrix 110 identity matrix 45

address 12 adjoint matrix 118 alternating 112 alternating 203 angle 159 angle 33 angle 60 area 120 associative 180 augmented matrix 11 axes 5 Axiom of Choice 153 basis 178 basis 210 basis 74 basis test

LECTURE 7. Least Squares and Variants. Optimization Models EE 127 / EE 227AT. Outline. Least Squares. Notes. Notes. Notes. Notes.

Optimization Models EE 127 / EE 227AT Laurent El Ghaoui EECS department UC Berkeley Spring 2015 Sp 15 1 / 23 LECTURE 7 Least Squares and Variants If others would but reflect on mathematical truths as deeply

Math 302 Outcome Statements Winter 2013

Math 302 Outcome Statements Winter 2013 1 Rectangular Space Coordinates; Vectors in the Three-Dimensional Space (a) Cartesian coordinates of a point (b) sphere (c) symmetry about a point, a line, and a

LINEAR ALGEBRA 1, 2012-I PARTIAL EXAM 3 SOLUTIONS TO PRACTICE PROBLEMS

LINEAR ALGEBRA, -I PARTIAL EXAM SOLUTIONS TO PRACTICE PROBLEMS Problem (a) For each of the two matrices below, (i) determine whether it is diagonalizable, (ii) determine whether it is orthogonally diagonalizable,

Numerical Methods I Non-Square and Sparse Linear Systems

Numerical Methods I Non-Square and Sparse Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 25th, 2014 A. Donev (Courant

k is a product of elementary matrices.

Mathematics, Spring Lecture (Wilson) Final Eam May, ANSWERS Problem (5 points) (a) There are three kinds of elementary row operations and associated elementary matrices. Describe what each kind of operation

The value of a problem is not so much coming up with the answer as in the ideas and attempted ideas it forces on the would be solver I.N.

Math 410 Homework Problems In the following pages you will find all of the homework problems for the semester. Homework should be written out neatly and stapled and turned in at the beginning of class

Matrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =

30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can

Stat 159/259: Linear Algebra Notes

Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the

CSE 554 Lecture 7: Alignment

CSE 554 Lecture 7: Alignment Fall 2012 CSE554 Alignment Slide 1 Review Fairing (smoothing) Relocating vertices to achieve a smoother appearance Method: centroid averaging Simplification Reducing vertex

Final Exam, Linear Algebra, Fall, 2003, W. Stephen Wilson

Final Exam, Linear Algebra, Fall, 2003, W. Stephen Wilson Name: TA Name and section: NO CALCULATORS, SHOW ALL WORK, NO OTHER PAPERS ON DESK. There is very little actual work to be done on this exam if

7. Symmetric Matrices and Quadratic Forms

Linear Algebra 7. Symmetric Matrices and Quadratic Forms CSIE NCU 1 7. Symmetric Matrices and Quadratic Forms 7.1 Diagonalization of symmetric matrices 2 7.2 Quadratic forms.. 9 7.4 The singular value

Linear Algebra- Final Exam Review

Linear Algebra- Final Exam Review. Let A be invertible. Show that, if v, v, v 3 are linearly independent vectors, so are Av, Av, Av 3. NOTE: It should be clear from your answer that you know the definition.

Preliminary Examination, Numerical Analysis, August 2016

Preliminary Examination, Numerical Analysis, August 2016 Instructions: This exam is closed books and notes. The time allowed is three hours and you need to work on any three out of questions 1-4 and any

Linear Algebra Review

Linear Algebra Review CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Linear Algebra Review 1 / 16 Midterm Exam Tuesday Feb

Reduction to the associated homogeneous system via a particular solution

June PURDUE UNIVERSITY Study Guide for the Credit Exam in (MA 5) Linear Algebra This study guide describes briefly the course materials to be covered in MA 5. In order to be qualified for the credit, one

We wish the reader success in future encounters with the concepts of linear algebra.

Afterword Our path through linear algebra has emphasized spaces of vectors in dimension 2, 3, and 4 as a means of introducing concepts which go forward to IRn for arbitrary n. But linear algebra does not

Linear Algebra Primer

Linear Algebra Primer David Doria daviddoria@gmail.com Wednesday 3 rd December, 2008 Contents Why is it called Linear Algebra? 4 2 What is a Matrix? 4 2. Input and Output.....................................

We use the overhead arrow to denote a column vector, i.e., a number with a direction. For example, in three-space, we write

1 MATH FACTS 11 Vectors 111 Definition We use the overhead arrow to denote a column vector, ie, a number with a direction For example, in three-space, we write The elements of a vector have a graphical

Linear Least-Squares Data Fitting

CHAPTER 6 Linear Least-Squares Data Fitting 61 Introduction Recall that in chapter 3 we were discussing linear systems of equations, written in shorthand in the form Ax = b In chapter 3, we just considered

Elementary linear algebra

Chapter 1 Elementary linear algebra 1.1 Vector spaces Vector spaces owe their importance to the fact that so many models arising in the solutions of specific problems turn out to be vector spaces. The

Least Squares with Examples in Signal Processing 1. 2 Overdetermined equations. 1 Notation. The sum of squares of x is denoted by x 2 2, i.e.

Least Squares with Eamples in Signal Processing Ivan Selesnick March 7, 3 NYU-Poly These notes address (approimate) solutions to linear equations by least squares We deal with the easy case wherein the

CS 143 Linear Algebra Review

CS 143 Linear Algebra Review Stefan Roth September 29, 2003 Introductory Remarks This review does not aim at mathematical rigor very much, but instead at ease of understanding and conciseness. Please see

MATH 31 - ADDITIONAL PRACTICE PROBLEMS FOR FINAL

MATH 3 - ADDITIONAL PRACTICE PROBLEMS FOR FINAL MAIN TOPICS FOR THE FINAL EXAM:. Vectors. Dot product. Cross product. Geometric applications. 2. Row reduction. Null space, column space, row space, left

Math Subject GRE Questions

Math Subject GRE Questions Calculus and Differential Equations 1. If f() = e e, then [f ()] 2 [f()] 2 equals (a) 4 (b) 4e 2 (c) 2e (d) 2 (e) 2e 2. An integrating factor for the ordinary differential equation

Lecture 6. Numerical methods. Approximation of functions

Lecture 6 Numerical methods Approximation of functions Lecture 6 OUTLINE 1. Approximation and interpolation 2. Least-square method basis functions design matrix residual weighted least squares normal equation

Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014

Duke University, Department of Electrical and Computer Engineering Optimization for Scientists and Engineers c Alex Bronstein, 2014 Linear Algebra A Brief Reminder Purpose. The purpose of this document

Iterative Methods for Solving A x = b

Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http

Cambridge University Press The Mathematics of Signal Processing Steven B. Damelin and Willard Miller Excerpt More information

Introduction Consider a linear system y = Φx where Φ can be taken as an m n matrix acting on Euclidean space or more generally, a linear operator on a Hilbert space. We call the vector x a signal or input,

Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition

Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2016 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing

Preface to Second Edition... vii. Preface to First Edition...

Contents Preface to Second Edition..................................... vii Preface to First Edition....................................... ix Part I Linear Algebra 1 Basic Vector/Matrix Structure and

Linear Algebra: Matrix Eigenvalue Problems

CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given

MATH 583A REVIEW SESSION #1

MATH 583A REVIEW SESSION #1 BOJAN DURICKOVIC 1. Vector Spaces Very quick review of the basic linear algebra concepts (see any linear algebra textbook): (finite dimensional) vector space (or linear space),

COMP 558 lecture 18 Nov. 15, 2010

Least squares We have seen several least squares problems thus far, and we will see more in the upcoming lectures. For this reason it is good to have a more general picture of these problems and how to

Numerical Methods I Eigenvalue Problems

Numerical Methods I Eigenvalue Problems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 2nd, 2014 A. Donev (Courant Institute) Lecture

Linear Algebra & Geometry why is linear algebra useful in computer vision?

Linear Algebra & Geometry why is linear algebra useful in computer vision? References: -Any book on linear algebra! -[HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia

8. Diagonalization.

8. Diagonalization 8.1. Matrix Representations of Linear Transformations Matrix of A Linear Operator with Respect to A Basis We know that every linear transformation T: R n R m has an associated standard

APPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the n-dimensional column vector with components x 1 x 2.

APPENDIX A Background Mathematics A. Linear Algebra A.. Vector algebra Let x denote the n-dimensional column vector with components 0 x x 2 B C @. A x n Definition 6 (scalar product). The scalar product

The Conjugate Gradient Method Classical Iterations We have a problem, We assume that the matrix comes from a discretization of a PDE. The best and most popular model problem is, The matrix will be as large

Upon successful completion of MATH 220, the student will be able to:

MATH 220 Matrices Upon successful completion of MATH 220, the student will be able to: 1. Identify a system of linear equations (or linear system) and describe its solution set 2. Write down the coefficient

Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition

Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2018 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing

1. Foundations of Numerics from Advanced Mathematics. Linear Algebra

Foundations of Numerics from Advanced Mathematics Linear Algebra Linear Algebra, October 23, 22 Linear Algebra Mathematical Structures a mathematical structure consists of one or several sets and one or

Image Registration Lecture 2: Vectors and Matrices

Image Registration Lecture 2: Vectors and Matrices Prof. Charlene Tsai Lecture Overview Vectors Matrices Basics Orthogonal matrices Singular Value Decomposition (SVD) 2 1 Preliminary Comments Some of this

Math Linear Algebra II. 1. Inner Products and Norms

Math 342 - Linear Algebra II Notes 1. Inner Products and Norms One knows from a basic introduction to vectors in R n Math 254 at OSU) that the length of a vector x = x 1 x 2... x n ) T R n, denoted x,

Glossary of Linear Algebra Terms. Prepared by Vince Zaccone For Campus Learning Assistance Services at UCSB

Glossary of Linear Algebra Terms Basis (for a subspace) A linearly independent set of vectors that spans the space Basic Variable A variable in a linear system that corresponds to a pivot column in the

Review of Some Concepts from Linear Algebra: Part 2

Review of Some Concepts from Linear Algebra: Part 2 Department of Mathematics Boise State University January 16, 2019 Math 566 Linear Algebra Review: Part 2 January 16, 2019 1 / 22 Vector spaces A set

Linear Algebra Review. Vectors

Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors

MATH 1120 (LINEAR ALGEBRA 1), FINAL EXAM FALL 2011 SOLUTIONS TO PRACTICE VERSION

MATH (LINEAR ALGEBRA ) FINAL EXAM FALL SOLUTIONS TO PRACTICE VERSION Problem (a) For each matrix below (i) find a basis for its column space (ii) find a basis for its row space (iii) determine whether

Applied Linear Algebra in Geoscience Using MATLAB

Applied Linear Algebra in Geoscience Using MATLAB Contents Getting Started Creating Arrays Mathematical Operations with Arrays Using Script Files and Managing Data Two-Dimensional Plots Programming in

Lecture Notes 1: Vector spaces

Optimization-based data analysis Fall 2017 Lecture Notes 1: Vector spaces In this chapter we review certain basic concepts of linear algebra, highlighting their application to signal processing. 1 Vector

1 9/5 Matrices, vectors, and their applications

1 9/5 Matrices, vectors, and their applications Algebra: study of objects and operations on them. Linear algebra: object: matrices and vectors. operations: addition, multiplication etc. Algorithms/Geometric

and let s calculate the image of some vectors under the transformation T.

Chapter 5 Eigenvalues and Eigenvectors 5. Eigenvalues and Eigenvectors Let T : R n R n be a linear transformation. Then T can be represented by a matrix (the standard matrix), and we can write T ( v) =

Chapter 6. Nonlinear Equations. 6.1 The Problem of Nonlinear Root-finding. 6.2 Rate of Convergence

Chapter 6 Nonlinear Equations 6. The Problem of Nonlinear Root-finding In this module we consider the problem of using numerical techniques to find the roots of nonlinear equations, f () =. Initially we

Linear Equations in Linear Algebra

1 Linear Equations in Linear Algebra 1.1 SYSTEMS OF LINEAR EQUATIONS LINEAR EQUATION,, 1 n A linear equation in the variables equation that can be written in the form a a a b 1 1 2 2 n n a a is an where

I. Multiple Choice Questions (Answer any eight)

Name of the student : Roll No : CS65: Linear Algebra and Random Processes Exam - Course Instructor : Prashanth L.A. Date : Sep-24, 27 Duration : 5 minutes INSTRUCTIONS: The test will be evaluated ONLY

Mathematical foundations - linear algebra

Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar

Linear Algebra for Machine Learning. Sargur N. Srihari

Linear Algebra for Machine Learning Sargur N. srihari@cedar.buffalo.edu 1 Overview Linear Algebra is based on continuous math rather than discrete math Computer scientists have little experience with it

Algebra C Numerical Linear Algebra Sample Exam Problems

Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finite-dimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric

AMS526: Numerical Analysis I (Numerical Linear Algebra)

AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 19: More on Arnoldi Iteration; Lanczos Iteration Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis I 1 / 17 Outline 1

BASIC ALGORITHMS IN LINEAR ALGEBRA. Matrices and Applications of Gaussian Elimination. A 2 x. A T m x. A 1 x A T 1. A m x

BASIC ALGORITHMS IN LINEAR ALGEBRA STEVEN DALE CUTKOSKY Matrices and Applications of Gaussian Elimination Systems of Equations Suppose that A is an n n matrix with coefficents in a field F, and x = (x,,

GRE Subject test preparation Spring 2016 Topic: Abstract Algebra, Linear Algebra, Number Theory.

GRE Subject test preparation Spring 2016 Topic: Abstract Algebra, Linear Algebra, Number Theory. Linear Algebra Standard matrix manipulation to compute the kernel, intersection of subspaces, column spaces,

Mathematics 1. Part II: Linear Algebra. Exercises and problems

Bachelor Degree in Informatics Engineering Barcelona School of Informatics Mathematics Part II: Linear Algebra Eercises and problems February 5 Departament de Matemàtica Aplicada Universitat Politècnica

Linear Algebra & Geometry why is linear algebra useful in computer vision?

Linear Algebra & Geometry why is linear algebra useful in computer vision? References: -Any book on linear algebra! -[HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia

Lecture 2: Linear Algebra Review

EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 2-6 of the web textbook 1 2.1 Vectors 2.1.1

Paul Heckbert. Computer Science Department Carnegie Mellon University. 26 Sept B - Introduction to Scientific Computing 1

Paul Heckbert Computer Science Department Carnegie Mellon University 26 Sept. 2 5-859B - Introduction to Scientific Computing aerospace: simulate subsonic & supersonic air flow around full aircraft, no

Math 224, Fall 2007 Exam 3 Thursday, December 6, 2007

Math 224, Fall 2007 Exam 3 Thursday, December 6, 2007 You have 1 hour and 20 minutes. No notes, books, or other references. You are permitted to use Maple during this exam, but you must start with a blank

Maths for Signals and Systems Linear Algebra in Engineering

Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE

SUMMARY OF MATH 1600

SUMMARY OF MATH 1600 Note: The following list is intended as a study guide for the final exam. It is a continuation of the study guide for the midterm. It does not claim to be a comprehensive list. You

Math 411 Preliminaries

Math 411 Preliminaries Provide a list of preliminary vocabulary and concepts Preliminary Basic Netwon s method, Taylor series expansion (for single and multiple variables), Eigenvalue, Eigenvector, Vector

15 Singular Value Decomposition

15 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing