Exercises * on Linear Algebra

Transcription

1 Exercises * on Linear Algebra Laurenz Wiskott Institut für Neuroinformatik Ruhr-Universität Bochum, Germany, EU 4 February 7 Contents Vector spaces 4. Definition Linear combinations Linear (in)dependence Exercise: Linear independence Exercise: Linear independence Exercise: Linear independence Exercise: Linear independence Exercise: Linear independence Exercise: Linear independence Exercise: Linear independence Exercise: Linear independence in C Basis systems Exercise: Vector space of the functions sin(x + φ) Laurenz Wiskott (homepage This work (except for all figures from other sources, if present) is licensed under the Creative Commons Attribution-ShareAlike 4. International License. To view a copy of this license, visit Figures from other sources have their own copyright, which is generally indicated. Do not distribute parts of these lecture notes showing figures with non-free copyrights (here usually figures I have the rights to publish but you don t, like my own published figures). Several of my exercises (not necessarily on this topic) were inspired by papers and textbooks by other authors. Unfortunately, I did not document that well, because initially I did not intend to make the exercises publicly available, and now I cannot trace it back anymore. So I cannot give as much credit as I would like to. The concrete versions of the exercises are certainly my own work, though. * These exercises complement my corresponding lecture notes available at Teaching/Material/, where you can also find other teaching material such as programming exercises. The table of contents of the lecture notes is reproduced here to give an orientation when the exercises can be reasonably solved. For best learning effect I recommend to first seriously try to solve the exercises yourself before looking into the solutions.

2 .4. Exercise: Basis systems Exercise: Dimension of a vector space Exercise: Dimension of a vector space Representation wrt a basis Exercise: Representation of vectors w.r.t. a basis Exercise: Representation of vectors w.r.t. a basis Euclidean vector spaces 7. Inner product Exercise: Inner product for functions Exercise: Representation of an inner product Norm Exercise: City-block metric Exercise: Ellipse w.r.t. the city-block metric Exercise: From norm to inner product Exercise: From norm to inner product (concrete) Angle Exercise: Angle with respect to an inner product Exercise: Angle with respect to an inner product Exercise: Angle with respect to an inner product Orthonormal basis systems 9 3. Definition Exercise: Pythagoras theorem Exercise: Linear independence of orthogonal vectors Exercise: Product of matrices of basis vectors Representation wrt an orthonormal basis Exercise: Writing vectors in terms of an orthonormal basis Inner product Exercise: Norm of a vector Exercise: Writing polynomials in terms of an orthonormal basis and simplified inner product Projection Exercise: Projection Exercise: Is P a projection matrix

3 3.4.3 Exercise: Symmetry of a projection matrix Change of basis Exercise: Change of basis Exercise: Change of basis Exercise: Change of basis Schmidt orthogonalization process Exercise: Gram-Schmidt orthonormalization Exercise: Gram-Schmidt orthonormalization Exercise: Gram-Schmidt orthonormalization of polynomials Matrices Exercise: Matrix as a sum of a symmetric and an antisymmetric matrix Matrix multiplication Matrices as linear transformations Exercise: Antisymmetric matrices yield orthogonal vectors Exercise: Matrices that preserve the length of all vectors Exercise: Derivative as a matrix operation Exercise: Derivative as a matrix operation Exercise: Derivative as a matrix operation Rank of a matrix Determinant Exercise: Determinants Exercise: Determinant Exercise: Determinant Inversion Trace Exercise: Trace and determinant of a symmetric matrix Orthogonal matrices Diagonal matrices Exercise: Matrices as transformations Exercise: Matrices as transformations Exercise: Matrices with certain properties Eigenvalue equation for symmetric matrices Exercise: Eigenvectors of a matrix

4 4.9. Exercise: Eigenvalue problem Exercise: Eigenvectors of a matrix Exercise: Eigenvectors of a matrix of type v i v T i Exercise: Eigenvectors of a symmetric matrix are orthogonal General eigenvectors Exercise: Matrices with given eigenvectors and -values Exercise: From eigenvalues to matrices Exercise: Generalized eigenvalue problem Complex eigenvalues Exercise: Complex eigenvalues Nonquadratic matrices Quadratic forms Vector spaces. Definition. Linear combinations.3 Linear (in)dependence.3. Exercise: Linear independence Are the following vectors linearly independent? Do they form a basis of the given vector space V? (a) v = 3, v = 8, V = R 3. (b) f (x) = x + 3x +, f (x) = 3x + 6x, f 3 (x) = x +, vector space of polynomials of degree..3. Exercise: Linear independence Are the following vectors linearly independent? Do they form a basis of the given vector space V? (a) v = ( ) ( ) 3, v =, V = R 4. (b) f (x) = x, f (x) = 3 + 4x, V = vector space of polynomials of degree..3.3 Exercise: Linear independence Are the following vectors linearly independent? Do they form a basis of the given vector space V? 4

5 ( ) ( ) 3 (a) v =, v =, V = R. (b) f (x) = x x 3, f (x) = x + 3x 5, f 3 (x) = x, V = vector space of polynomials of degree..3.4 Exercise: Linear independence Are the following vectors linearly independent? Do they form a basis of the given vector space V? 3 (a) v =, v =, V = R 3. 4 (b) f (x) = 3x, f (x) = x, f 3 (x) =, V = vector space of polynomials of degree..3.5 Exercise: Linear independence Are the following vectors linearly independent? Do they form a basis of the given vector space V? (a) v = 3, v =, V = R 3. (b) f (x) = x, f (x) = x + 3, f 3 (x) = x, V = vector space of polynomials of degree..3.6 Exercise: Linear independence Are the following vectors linearly independent? Do they form a basis of the given vector space V? (a) v = ( ) ( ), v =, V = R. (b) f (x) = 3x +x, f (x) = x 3, f 3 (x) = x, f 4 (x) = 4x +5x+3, V = vector space of polynomials of degree..3.7 Exercise: Linear independence Are the following vectors linearly independent? Do they form a basis of the given vector space V? (a) v = ( ) ( ) ( ) 4, v =, v 3 =, V = R 7. (b) f (x) = sin(x), f (x) = sin(x + π/4), f 3 (x) = sin(x + π/), V = L{sin(αx), cos(αx) : α R}. Hint: sin(x ± y) = sin(x) cos(y) ± cos(x) sin(y)..3.8 Exercise: Linear independence in C 3 Are the following vectors in C 3 over the field C linearly independent? Do they form a basis? r =, r = +i, r 3 = i. () +i i 5

6 .4 Basis systems.4. Exercise: Vector space of the functions sin(x + φ) Show that the set of functions V = {f(x) = A sin(x + φ) A R, φ [, π]} generates a vector space over the field of real numbers. Find a basis for V and determine its dimension. Hint: The addition theorems for trigonometric functions are helpful for the solution, in particular sin(x+y) = sin(x) cos(y) + sin(y) cos(x)..4. Exercise: Basis systems. Find two different basis systems for the vector space of the polynomials of degree 3.. Find a basis for the vector space of symmetric 3 3 matrices..4.3 Exercise: Dimension of a vector space Determine the dimension of the following vector spaces. (a) Vector space of real symmetric n n matrices (for a given n). (b) Vector space of mixed polynomials in x and y (e.g. f(x, y) = x y + x 3y + 5) that have a maximal degree of n in x and y (i.e. for each term x nx y ny in the polynomial n x + n y n must hold)..4.4 Exercise: Dimension of a vector space Determine the dimension of the following vector spaces.. Vector space of real antisymmetric n n matrices (for a given n). A matrix M is antisymmetric if M T = M. M T is the transpose of M.. Vector space of the series that converge to zero..5 Representation wrt a basis.5. Exercise: Representation of vectors w.r.t. a basis Write the vectors w.r.t. the given basis. ( ) ( ) ( 3 (a) Vectors: v =, v =, v 3 3 = 5 e e The subscript e indicates the canonical basis. (b) Vector: f(x) = 3x + 3 x ; Basis: x, x, (c) Vector: g(x) = (x + )(x ); Basis: x 3, x +, x,. ) ; Basis: b = e ( ), b = e ( ) e 6

7 .5. Exercise: Representation of vectors w.r.t. a basis Write the vectors w.r.t. the given basis. (a) Vector: v = 3 ; Basis: b = e The subscript e indicates the canonical basis., b = e (b) Vector: h(x) = 3x x 3; Basis: x x +, x +, x, b 3 = e e Euclidean vector spaces. Inner product.. Exercise: Inner product for functions Consider the space of real continuous functions defined on [, ] for which [f(x)] dx exists. Let the inner product be (f, g) := with an arbitrary positive weighting function w(x)f(x)g(x) dx, () < w(x) <. (). Prove that () is indeed an inner product.. Show whether () is an inner product also for non-continuous functions. 3. Show whether () is an inner product for continuous functions even if the weighting function is positive only in the inner of the interval, i.e. if w(x) > x (, ) but w(±) =... Exercise: Representation of an inner product Let V be an N-dimensional vector space over R and let {b i } with i =,..., N be a basis. Let x = ( x,..., x N ) T b and ỹ = (ỹ,..., ỹ N ) T b be the representations of two vectors x, y V with respect to the basis {b i}. Show that: (x, y) = x T Aỹ. () where A is an N N-matrix.. Norm.. Exercise: City-block metric The norm of the city-block metric is defined as: x CB := i x i Prove that this actually is a norm. 7

8 .. Exercise: Ellipse w.r.t. the city-block metric What does an ellipse in a two-dimensional space with city block metric look like? An Ellipse is the set of all points x whose sum of the distances x a and x b equals r, given the two focal points a and b and a radius r. Examine the following cases: (a) a = (, ) T, b = (, ) T, and r = 4, (b) a = (, ) T, b = (, ) T, and r = Exercise: From norm to inner product Every inner product (, ) defines a norm by x = (x, x). Show that a norm can also define an inner product over the field R (if it exists, which is the case if the parallelogram law x + y + x y = ( x + y ) holds). Hint: Make the ansatz (x + y, x + y) =... and derive a formula for the inner product given a norm...4 Exercise: From norm to inner product (concrete) Given a norm x, a corresponding inner product can be derived with (x, y) := ( x + y x y ). () Derive the corresponding inner product for the norm g := w(x)g(x) dx, () with w(x) being some arbitrary strictly positive function..3 Angle.3. Exercise: Angle with respect to an inner product Draw the following vectors and calculate the angle between them with respect to the given inner product.. v = 4 and v = 5 with the standard Euclidean inner product.. f (x) = x 3 + and f (x) = 3x with the inner product (f, g) := f(x)g(x) dx..3. Exercise: Angle with respect to an inner product Draw the following vectors and calculate the angle between them with respect to the given inner product. (a) v = ( ) ( ) and v = with the standard Euclidean inner product. 3 (b) f (x) = arctan(x) and f (x) = cos(x) with the inner product (f, g) := exp( x )f(x)g(x) dx. 8

9 .3.3 Exercise: Angle with respect to an inner product Draw the following vectors and calculate the angle between them with respect to the given inner product. ( (a) v = ) ( and v = ) with the inner product (x, y) := x T ( ) y. (b) f (x) = 3x and f (x) = x with the inner product (f, g) := f(x)g(x) dx. 3 Orthonormal basis systems 3. Definition 3.. Exercise: Pythagoras theorem Prove the generalized Pythagoras theorem: Let v i, i {,..., N} be pairwise orthogonal vectors. Then holds. N N v i = v i. i= i= 3.. Exercise: Linear independence of orthogonal vectors Show that N pairwise orthogonal vectors (not permitting the zero vector) are always linearly independent Exercise: Product of matrices of basis vectors Let {b i }, i =,..., N, be an orthonormal basis and N indicate the N-dimensional identity matrix.. Show that (b, b,..., b N ) T (b, b,..., b N ) = N. Does the result also hold if one only takes the first N basis vectors? If not, try to interpret the resulting matrix.. Show that (b, b,..., b N )(b, b,..., b N ) T = N. Does the result also hold if one only takes the first N basis vectors? If not, try to interpret the resulting matrix. 3. Representation wrt an orthonormal basis 3.. Exercise: Writing vectors in terms of an orthonormal basis Given the orthonormal basis b = 6, b = 6 3, b 3 = 3 6. () 9

10 . Write the vectors v and v in terms of the orthonormal basis b i. v =, v =. () 3. What is the matrix with which you could transform any vector given in terms of the Euclidean basis into a representation in terms of the orthonormal basis b i? 3.3 Inner product 3.3. Exercise: Norm of a vector Let b i, i =,..., N, be an orthonormal basis. Then we have (b i, b j ) = δ ij and N v = v i b i with v i := (v, b i ) v. () i= Show that N v = vi. () i= 3.3. Exercise: Writing polynomials in terms of an orthonormal basis and simplified inner product The normalized Legendre polynomials L = /, L = 3/ x, L = 5/8 ( + 3x ) form an orthonormal basis of the vector space of polynomials of degree with respect to the inner product (f, g) = f(x)g(x) dx.. Write the following polynomials in terms of the basis L, L, L : Verify the result. f (x) = + x, f (x) = 3 x.. Calculate the inner product (f, f ) first directly with the integral and then based on the coefficients of the vectors written in terms of the basis L, L, L. 3.4 Projection 3.4. Exercise: Projection. Project the vector v = (,, ) T onto the space orthogonal to the vector b = (,, ) T.. Construct a 3 3-matrix P that realizes the projection onto the subspace orthogonal to b, so that v = Pv for any vector v. 3. Calculate the product of P with itself, i.e. PP.

11 3.4. Exercise: Is P a projection matrix Determine whether matrix is a projection matrix or not. P = 5 ( 4 ) () Exercise: Symmetry of a projection matrix Prove that the matrix P of an orthogonal projection is always symmetric. 3.5 Change of basis 3.5. Exercise: Change of basis Let {a i } and {b i } be two orthonormal bases in R 3 : a = 3, a =, a 3 = 6 b =, b =, b 3 =. 6 3 Determine the matrices B b a and B a b for the transformations from basis a to basis b and vice versa. Are there similarities between the two matrices? What happens if you multiply the two matrices? Extra question: What would change if the basis would not be orthonormal? Extra question: How can you generalize this concept of change of basis to vector spaces of polynomials of degree?, 3.5. Exercise: Change of basis Consider {a i } and {b i }, where a = b = 6, a =, b =, a 3 =, b 3 = 3,.. Are {a i } and {b i } an orthonormal basis of R 3? If not, make them orthonormal.. Find the transformation matrix B b a. 3. Find the inverse of matrix B b a.

12 3.5.3 Exercise: Change of basis Let {a i } and {b i } be two orthonormal bases in R : a = ( ), a = ( ), b = ( 5 Write the vector v = ( 3 which is given in terms of the basis a, in terms of basis b. ) a ), b = 5 ( ). (), () 3.6 Schmidt orthogonalization process 3.6. Exercise: Gram-Schmidt orthonormalization Construct an orthonormal basis for the space spanned by the vectors v =, v =, v 3 = Exercise: Gram-Schmidt orthonormalization Find an orthonormal basis for the spaces spanned by the following sets of vectors... v =, v =, v 3 =. () v =, v =. () Exercise: Gram-Schmidt orthonormalization of polynomials Construct an orthonormal basis for the space of polynomials of degree in R given the inner product and the norm induced by this inner product. (g, h) = g(x)h(x) dx () Extra question: Would the result change if we usee a different inner product, e.g. with an integral on the interval [, +] instead of [, ]? Extra question: Seeing this basis, what does it mean to project a polynomial of degree onto the space of polynomials of degree?

13 4 Matrices 4.. Exercise: Matrix as a sum of a symmetric and an antisymmetric matrix Prove that any square matrix M can be written as a sum of a symmetric matrix M + and an antisymmetric matrix M, i.e. M = M + + M with (M + ) T = M + and (M ) T = M. Hint: Construct a symmetric matrix and an antisymmetric matrix from M. 4. Matrix multiplication 4. Matrices as linear transformations 4.. Exercise: Antisymmetric matrices yield orthogonal vectors. Show that multiplying a vector v R N with an antisymmetric N N-matrix A yields a vector orthogonal to v. In other words A T = A = (v, Av) = v R N. (). Show the converse. If a matrix A transforms any vector v such that it becomes orthogonal to v, then A is antisymmetric. In other words (v, Av) = v R N = A T = A. () 4.. Exercise: Matrices that preserve the length of all vectors Let A be a matrix that preserves the length of any vector under its transformation, i.e. Av = v v R N. () Show that A must be an orthogonal matrix. Hint: For a square matrix M we have v T Mv = v R N M = M T. () 4..3 Exercise: Derivative as a matrix operation Taking the derivative of a function is a linear operation. Find a matrix that realizes a derivative on the vector spaces spanned by the following function sets F. Use the given functions as a basis with respect to which you represent the vectors. Determine the rank of each matrix. (a) F = {sin(x), cos(x)}. () (b) F = {, x +, x }. () (c) F = {exp(x), exp(x)}. (3) 3

14 4..4 Exercise: Derivative as a matrix operation Taking the derivative of a function is a linear operation. Find a matrix that realizes a derivative on the vector spaces spanned by the following function sets F. Use the given functions as a basis with respect to which you represent the vectors. Determine the rank of each matrix. (a) F = {sin(x), cos(x)}. () (b) F = {, x +, x }. () 4..5 Exercise: Derivative as a matrix operation Taking the derivative of a function is a linear operation. Find a matrix that realizes a derivative on the vector space spanned by the function set F = {sin(x), cos(x), x sin(x), x cos(x)}. Use the given functions as a basis. Determine the rank of the matrix. 4.3 Rank of a matrix 4.4 Determinant 4.4. Exercise: Determinants Calculate the determinants of the following matrices. (a) (b) (c) ( ) cos(φ) sin(φ) M = sin(φ) cos(φ) () M = () 3 M 3 = (3) 4.4. Exercise: Determinant Calculate the determinant of matrix M = () Exercise: Determinant Calculate the determinant of matrix M = () 4

15 4.5 Inversion Trace 4.6. Exercise: Trace and determinant of a symmetric matrix. Show that the trace of a symmetric matrix equals the sum of its eigenvalues.. Show that the determinant of a symmetric matrix equals the product of its eigenvalues. Hint: For two square matrices A and B we have AB = A B. Extra question: To what extent does this result generalize to real valued rotation matrices (without a flip)? Extra question: What happens if you combine a rotation and a flip? What can you say about the eigenvalues? Does the rule above still hold? 4.7 Orthogonal matrices 4.8 Diagonal matrices 4.8. Exercise: Matrices as transformations Describe with words the transformations realized by the following matrices. Estimate the values of the corresponding determinants without actually calculating them, only based on the intuitive understanding of the transformations. (a) (b) (c) M 3 = M = M = 3 () cos(φ) sin(φ) sin(φ) cos(φ) () (3) 4.8. Exercise: Matrices as transformations Describe with words the transformation realized by matrix cos(φ) sin(φ) M = sin(φ) cos(φ). () Exercise: Matrices with certain properties Let A be a set of 3 3 matrices A with one of the following conditions. For each set (i) Give a non-trivial example element for each of the sets. 5

16 (ii) What is the dimensionality of each set, i.e. what is the number of degrees of freedom (DOF, the number of variables that you can vary independently) the matrices of each set have? (iii) Which sets do not form a vector space? (If you think it is a vector space, no reasoning is required.). A T = A. Extra question: How does the number of degrees of freedom scale with the dimensionality of the matrix?. A has rank. Extra question: How does the number of degrees of freedom scale with the dimensionality of the matrix? 3. A T = A. Extra question: How does the number of degrees of freedom scale with the dimensionality of the matrix? 4.9 Eigenvalue equation for symmetric matrices 4.9. Exercise: Eigenvectors of a matrix Determine the eigenvalues and eigenvectors for matrix M = 3. () Hint: Guess and verify Exercise: Eigenvalue problem Determine the eigenvalues and eigenvectors of the following symmetric matrices. ( ) ( ) 7 6 A = B = 6 Show that the eigenvectors are orthogonal to each other. () Exercise: Eigenvectors of a matrix Determine the eigenvalues and eigenvectors for matrix 3 M := 4 3. () Exercise: Eigenvectors of a matrix of type v i v T i. Let {v i }, i {,..., m} be a set of pairwise orthogonal vectors in R n with m < n. Find a set of eigenvectors and eigenvalues of the matrix m A = v i v T i. (). Interpret the transformation realized by A if the vectors are normalized to length. i= 6

17 4.9.5 Exercise: Eigenvectors of a symmetric matrix are orthogonal Prove that the eigenvectors of a symmetric matrix are orthogonal, if their eigenvalues are different. Proceed as follows:. Let A be a symmetric N-dimensional matrix, i.e. A = A T. Show first that (v, Aw) = (Av, w) for any vectors v, w R N, with (, ) indicating the Euclidean inner product.. Let {a i } be the eigenvectors of the matrix A with the eigenvalues λ i. Show with the help of part one that (a i, a j ) = if λ i λ j. Hint: λ i (a i, a j ) =... Extra question: What can you say about the product of two symmetric matrics? 4. General eigenvectors 4.. Exercise: Matrices with given eigenvectors and -values. Construct a matrix M that has the following right-eigenvectors r i (not normalized!) and eigenvalues λ i. r = (, ) T, λ =, r = (, ) T, λ =. () Verify your result.. Determine the left-eigenvectors and the corresponding eigenvalues of matrix M. Verify your result. Extra question: What can you say about the eigenvalues of an upper triangular matrix? 4.. Exercise: From eigenvalues to matrices To determine the eigenvalues of a given matrix, one has to find the roots (D: Nullstellen) of the characteristic polynomial. Here we want to go the other way around, at least for -matrices.. Construct a parameterized -matrix A that has two given eigenvalues λ and λ. Matrix A should not just be an example, such as A := diag(λ, λ ), but should be written such that all possible matrices with the two eigenvalues could be realized with it. Note that a real -matrix has initially four free parameters, two of which are constrained by the two given eigenvalues. Thus you have to introduce two additional parameters α and β and write matrix A in these four parameters. I suggest to use the auxiliary variable λ := (λ + λ )/ and the two parameters α := a λ and β := a /a. A should be real but the eigenvalues may be complex.. Interpret the result. What happens with the eigenvectors if you set β = and vary α? What happens if you set α = and vary β? 4..3 Exercise: Generalized eigenvalue problem Consider the generalized eigenvalue problem Au i = λ i Bu i () with some real N N matrices A and B. The λ i are the right-eigenvalues and the u i are the (non-zero) right-eigenvectors. To find corresponding left-eigenvalues µ and left-eigenvectors v i, one has to solve the equation v T i A = µ i v T i B () 7

18 . Show that left- and right-eigenvalues are identical.. Show that v T j Au i = as well as v T j Bu i = for λ i λ j. Hint: Consider () and () simultaneously with different eigenvalues. 3. Show that for symmetric A and B the right-eigenvectors are also left-eigenvectors. 4. Show that for symmetric A and B we have u T j Au i = as well as u T j Bu i = for λ i λ j. 5. For symmetric A and B it is convenient to normalize the eigenvectors such that u T i Bu i =. Assume the eigenvectors form a basis, i.e. they are complete, and you want to represent an arbitrary vector y wrt this basis, i.e. y = i α i u i (3) Which constraint on the α i follows from the constraint y T By =? 6. Assume A and B are symmetric and you want to minimize (or maiximze) y T Ay under the constraint y T By =. What is the solution? Hint: Use ansatz (3) and assume u T i Bu i =. 4. Complex eigenvalues 4.. Exercise: Complex eigenvalues Determine the (complex) eigenvalues and eigenvectors of the rotation matrix ( ) cos φ sin φ D(φ) =. sin φ cos φ () 4. Nonquadratic matrices Quadratic forms + 8