Lecture 4: Linear Algebra Review, Part III

Transcription

1 Lecture 4: Linear Algebra Review, Part III Brian Borchers February 1, Vector Norms Although the conventional Euclidean length is most commonly used, there are alternative ways to measure the length of a vector. Definition 1 Any measure of the length of a vector that satisfies the following four properties can be used as a norm. 1. For any vector x, x 0.. For any vector x and any scalar s, sx = s x. 3. For any vectors x and y, x + y x + y. 4. x =0 if and only if x =0. Definition The p norm of a vector in R n is defined by x p =( x 1 p + x p x n p ) 1/p It can be shown that for any p, the p norm satisfies the conditions of Definition 1. Note that the conventional Euclidean length is just the norm. Two other p norms are commonly used. The 1 norm is the sum of the absolute values of the entries in x. Theinfinity norm is the maximum of the absolute values of the entries in x. Example 1 Let x = 1. 3 Then x 1 =6 x = 14 x =3 These alternative norms can be useful in finding approximate solutions to over determined linear systems of equations. To minimize the maximum of the errors, we minimize Ax b. To minimize the sum of the absolute values of the errors, we minimize Ax b 1. Unfortunately, these minimization problems can be much harder to solve then then the least squares problem. 1

2 Matrix Norms Definition 3 Any measure of the size or length of an m by n matrix that satisfies the following four properties can be used as a matrix norm. 1. For any matrix A, A 0.. For any matrix A and any scalar s, sa = s A. 3. For any matrices A and B, A + B A + B. 4. A =0 if and only if A =0. 5. For any two matrices A and B of compatible sizes, AB A B. Definition 4 The p norm of a matrix A is defined by A p =max Ax p. x p=1 In this formula, x p and Ax p are vector p norms, while A p is the matrix p norm of A. Solving the maximization problem to determine a matrix p norm could be extremely difficult. Fortunately, there are simpler formulas for the most commonly used matrix p norms. A 1 =max j A = m A ij. i=1 λ max (A T A) where λ max (A T A) is the largest eigenvalue of A T A. n A =max A ij. i Definition 5 The Frobenius norm of an m by n matrix is given by m n A F = j=1 i=1 j=1 Definition 6 A matrix norm and a vector norm are compatible if Ax A x The matrix p norm is (by its definition) compatible with the vector p norm from which it was derived. It can also be shown that the Frobenius norm of a matrix is compatible with the vector norm. Thus the Frobenius norm is sometimes used with the vector norm. In practice, the Frobenius norm, 1 norm, and infinity norm of a matrix are easy to compute, while the norm of a matrix can be difficult to compute. For this reason, the norm is seldom used. A ij

3 3 The Condition Number of a Linear System of Equations Suppose that we want to solve a system of n equations in n variables Ax = b. Suppose further that because of measurement errors in b, we actually solve Aˆx = ˆb. Can we get a bound on x ˆx in terms of b ˆb? Ax = b Aˆx = ˆb. A(x ˆx) =b ˆb. (x ˆx) =A 1 (b ˆb). x ˆx = A 1 (b ˆb). x ˆx A 1 b ˆb. This formula provides an absolute bound on the error in the solution. It is also worthwhile to compute a relative error bound. x ˆx b A 1 b ˆb. b x ˆx Ax A 1 b ˆb. b x ˆx b ˆb A x A 1 b. x ˆx A A 1 b ˆb x b. The relative error in b is measured by b ˆb b. The relative error in x is measured by x ˆx. x The constant cond(a) = A A 1 3

4 is called the condition number of A. Note that nothing that we did in the calculation of the condition number depends on which particular norm we used the condition number can be computed using the 1 norm, norm, infinity norm, or even the Frobenius norm. The condition number provides an upper bound on how inaccurate the solution to a system of equations might be because of errors in the right hand side. In some cases, the condition number greatly overestimates the error in the solution. As a practical matter, it s wise to assume that the error is of roughly the size predicted by the condition number. In practice, floating point arithmetic only allows us to store numbers to about 16 digits of precision. If the condition number is greater than 10 16, then by the above inequality, there may be no accurate digits in the computer solution to the system of equations. Systems of equations with very large condition numbers are called ill-conditioned. It is important to understand that ill-conditioning is a property of the system of equations and not the algorithm used to solve the system of equations. Illconditioning can t be fixed simply by using a better algorithm. Instead, we must either increase the precision of our computer or find a different, better conditioned system of equations to solve. 4 Using MATLAB The following session demonstrates some of the features of the MATLAB software package. < M A T L A B > Copyright The MathWorks, Inc. Version (R11) Jan To get started, type one of these: helpwin, helpdesk, or demo. For product information, type tour or visit The simplest way to use MATLAB is as a calculator >> 3/ >> ans^ 4

5 .500 MATLAB lets you store numbers in variables. >> a=1.5 a = >> b= b = >> a*b 3 Note that variable names are case sensitive >> A??? Undefined function or variable A. MATLAB has many built in functions. The help command can be used to learn about MATLAB s built in commands. >> help elfun Elementary math functions. Trigonometric. sin - Sine. sinh - Hyperbolic sine. asin - Inverse sine. asinh - Inverse hyperbolic sine. 5

6 cos cosh acos acosh tan tanh atan atan atanh sec sech asec asech csc csch acsc acsch cot coth acot acoth - Cosine. - Hyperbolic cosine. - Inverse cosine. - Inverse hyperbolic cosine. - Tangent. - Hyperbolic tangent. - Inverse tangent. - Four quadrant inverse tangent. - Inverse hyperbolic tangent. - Secant. - Hyperbolic secant. - Inverse secant. - Inverse hyperbolic secant. - Cosecant. - Hyperbolic cosecant. - Inverse cosecant. - Inverse hyperbolic cosecant. - Cotangent. - Hyperbolic cotangent. - Inverse cotangent. - Inverse hyperbolic cotangent. Exponential. exp - Exponential. log - Natural logarithm. log10 - Common (base 10) logarithm. log - Base logarithm and dissect floating point number. pow - Base power and scale floating point number. sqrt - Square root. nextpow - Next higher power of. Complex. abs angle complex conj imag real unwrap isreal cplxpair - Absolute value. - Phase angle. - Construct complex data from real and imaginary parts. - Complex conjugate. - Complex imaginary part. - Complex real part. - Unwrap phase angle. - True for real array. - Sort numbers into complex conjugate pairs. Rounding and remainder. fix - Round towards zero. floor - Round towards minus infinity. ceil - Round towards plus infinity. 6

7 round mod rem sign - Round towards nearest integer. - Modulus (signed remainder after division). - Remainder after division. - Signum. >> help specfun Specialized math functions. Specialized math functions. airy - Airy functions. besselj - Bessel function of the first kind. bessely - Bessel function of the second kind. besselh - Bessel functions of the third kind (Hankel function). besseli - Modified Bessel function of the first kind. besselk - Modified Bessel function of the second kind. beta - Beta function. betainc - Incomplete beta function. betaln - Logarithm of beta function. ellipj - Jacobi elliptic functions. ellipke - Complete elliptic integral. erf - Error function. erfc - Complementary error function. erfcx - Scaled complementary error function. erfinv - Inverse error function. expint - Exponential integral function. gamma - Gamma function. gammainc - Incomplete gamma function. gammaln - Logarithm of gamma function. legendre - Associated Legendre function. cross - Vector cross product. Number theoretic functions. factor - Prime factors. isprime - True for prime numbers. primes - Generate list of prime numbers. gcd - Greatest common divisor. lcm - Least common multiple. rat - Rational approximation. rats - Rational output. perms - All possible permutations. nchoosek - All combinations of N elements taken K at a time. factorial - Factorial function. Coordinate transforms. cartsph - Transform Cartesian to spherical coordinates. 7

8 cartpol polcart sphcart hsvrgb rgbhsv - Transform Cartesian to polar coordinates. - Transform polar to Cartesian coordinates. - Transform spherical to Cartesian coordinates. - Convert hue-saturation-value colors to red-green-blue. - Convert red-green-blue colors to hue-saturation-value. >> help log LOG Natural logarithm. LOG(X) is the natural logarithm of the elements of X. Complex results are produced if X is not positive. See also LOG, LOG10, EXP, LOGM. Overloaded methods help sym/log.m Of course, MATLAB can also do calculations with matrices and vectors. matrices are entered using [] s and ; s as follows: >> A=[1 3; 4 5 6; 7 8 9] A = The eye command generates an identity matrix of specified size. >> eye(3) The zeros and ones commands generate matrices of all zeros or all ones. >> zeros(,3) 8

9 >> ones(3,) If you don t want to see the result printed out, add a ; to the end of the command. >> Z=zeros(100,100); We can build up larger matrices from smaller matrices. >> A=[1 ; 3 4] A = >> B=[5 6; 7 8] B = >> [A B] >> C=[A; B] C = 1 9

10 We can refer to particular elements within a matrix using subscripts. >> C(1,) >> C(3,1) 5 The notation ":" refers to an entire row or column of a matrix. >> C(1,:) 1 >> C(:,) We can also refer to a range of rows or columns. >> C(:4,)

11 8 We can also perform arithmetic on matrices. >> A+B >> A*B >> A >> inv(a) We can solve a system of equations in a variety of ways. >> b=[1 ] b = 1 We can use the rref command to find the RREF of the augmented matrix. 11

12 >> rref([a b]) We can also use the \ operator, which finds the least squares solution To Ax=b, and works even when the system is over determined and has no exact solutions. >> x=a\b x = MATLAB has a built in command to compute the rank of a matrix. >> rank(a) We can also use the rref command to find bases for N(A) and R(A). >> A=[ ; 1 7 3; ] A = >> rref(a)

13 We get a basis for R(A) by taking the pivot columns from A. >> BRA=A(:,1:) BRA = From the RREF, we can see that the solutions to Ax=0 have x1=-*x3-x4 x=-3*x3-x4 x3, x4 free. From this we can get the basis vectors for N(A). >> BNA=[ ; ] BNA = We can use the QR factorization to make these orthogonal bases. >> [q,r]=qr(bra) q = r =

14 >> BRAO=q(:,1:) BRAO = >> [q,r]=qr(bna) q = r = >> BNAO=q(:,1:) BNAO = MATLAB also has built in functions for computing matrix and vector norms. >> help norm NORM Matrix or vector norm. For matrices... NORM(X) is the largest singular value of X, max(svd(x)). NORM(X,) is the same as NORM(X). NORM(X,1) is the 1-norm of X, the largest column sum, 14

15 = max(sum(abs((x)))). NORM(X,inf) is the infinity norm of X, the largest row sum, = max(sum(abs((x )))). NORM(X, fro ) is the Frobenius norm, sqrt(sum(diag(x *X))). NORM(X,P) is available for matrix X only if P is 1,, inf or fro. For vectors... NORM(V,P) = sum(abs(v).^p)^(1/p). NORM(V) = norm(v,). NORM(V,inf) = max(abs(v)). NORM(V,-inf) = min(abs(v)). See also COND, CONDEST, NORMEST. Overloaded methods help ss/norm.m help lti/norm.m help frd/norm.m >> x=[1 1 ] x = 1 1 >> norm(x) >> norm(x,1) 6 >> norm(x, inf ) 15

16 >> norm(a).4477 >> norm(a,).4477 >> norm(a,1) 3 >> norm(a, inf ) 30 >> norm(a, fro ).4499 >> Computing dot products is easy- just multiply x *y >> x=[1 3] x = 1 3 >> y=[1 0 1] y = 16

17 1 0 1 >> x *y 4 We can also compute orthogonal projections. >> p=(x *y)*y/(y *y) p = 0 MATLAB also has a programming language with functions, if, while, etc. The following examples show some user defined functions. The actual code for these functions appears in the next section. >> help fact f=fact(n) n An integer, >= 0. f n! >> fact(0) 1 >> fact(1) 17

18 1 >> fact() >> fact(3) 6 >> fact(7) 5040 The basisn function computes a basis for the null space of A. >> basisn(a) The basisr function computes a basis for the range of A. >> basisr(a) >> quit

19 5 MATLAB Programming Examples The code for the fact function is f=fact(n) n An integer, >= 0. f n! function f=fact(n) if (n == 0), f=1; else f=n*fact(n-1); end; The basisn function finds a basis for N(A). B=basisn(A) A an m by n matrix. B an n by p matrix whose columns are a basis for N(A) function B=basisn(A) [m,n]=size(a); First, find the rref(a). R=rref(A); Find r=rank(a). This tells us how many free variables there are. While we re at it, find the pivot columns. Initialize the rank to 0. r=0; Initialize the list of pivot cols to nothing. pcols=[]; Loop through the rows, looking only at nonzero rows. 19

20 for i=1:m, if (norm(r(i,:)) ~= 0), increment the count of pivot rows by 1. r=r+1; Find the indices of all nonzeros in row i. ind=find(r(i,:)); ind(1) contains the index of the first nonzero in row i. Add it to the list of pivot columns. pcols=[pcols ind(1)]; end; end; Compute the number of free variables p=n-r; Initialize B to zeros. B=zeros(n,p); Loop through the columns of R, looking for non-pivot columns. For each non-pivot column, update the basis. Use k to keep track of which column of the basis we re working on. k=0; for j=1:n, if (isempty(find(pcols==j))), This column corresponds to a free variable. k=k+1; for i=1:r, B(pcols(i),k)=-R(i,j); end; B(j,k)=1; end; end; 0

21 The basisr function finds a basis for R(A). B=basisr(A) A an m by n matrix. B an m by p matrix whose columns are a basis for R(A) function B=basisr(A) [m,n]=size(a); First, find the rref(a). R=rref(A); Find r=rank(a). This tells us how many free variables there are. While we re at it, find the pivot columns. Initialize the rank to 0. r=0; Initialize the list of pivot cols to nothing. pcols=[]; Loop through the rows, looking only at nonzero rows. for i=1:m, if (norm(r(i,:)) ~= 0), increment the count of pivot rows by 1. r=r+1; Find the indices of all nonzeros in row i. ind=find(r(i,:)); ind(1) contains the index of the first nonzero in row i. Add it to the list of pivot columns. pcols=[pcols ind(1)]; end; end; 1

22 Take the pivot columns out of A. B=A(:,pcols);