Final Exam. 1 True or False (15 Points)

Transcription

1 Final Exam Submit by Oct. 16, :59pm EST Please submit early, and update your submission if you want to make changes. Do not wait to the last minute to submit: we reserve the right not to accept late submissions. Please look over the exam before starting, and ask questions during the class period; we will answer questions as quickly as we can during class, but later in the day it may take substantially longer. For the true or false questions: you may print out this page, or answer on a separate page. If you do the latter, please make sure you label your answers clearly for which question they belong to. 1 True or False (15 Points) 1. (circle one: T F ) For any finite-dimensional vector space, the size of its basis is equal to its dimension. 2. (circle one: T F ) For any finite-dimensional vector space, there is always a unique basis. 3. (circle one: T F ) The perceptron algorithm always converges to a binary classifier that has zero error rate. 4. (circle one: T F ) {} and {{}} are the same set. 5. (circle one: T F ) {} is a subset of {{}}. 6. (circle one: T F ) {} is an element of {{}}. 7. (circle one: T F ) Every vector space contains a zero vector. 8. (circle one: T F ) A vector space can have more than one zero vector. 9. (circle one: T F ) A system of n linear equations with n unknowns has at least one solution. 10. (circle one: T F ) A system of n linear equations with n unknowns has at most one solution. 11. (circle one: T F ) Matrices can be used to represent any translation or rotation on the vector space R n. 12. (circle one: T F ) Any combination of rotations on the vector space R n can be combined into a single matrix. 13. (circle one: T F ) Every square matrix is invertible. 14. (circle one: T F ) Every symmetric matrix is invertible. 15. (circle one: T F ) Matrix multiplication is associative and commutative. 1

2 2 Linear algebra (15 Points) 1. Consider the vector space of all 2 by 2 matrices with real entries, denoted R 2 2. In this space, we define vector addition to be componentwise matrix addition, and we define scalar multiplication to be componentwise scalar multiplication. What is the dimension of this vector space? 2. In the same space, consider the following vectors: [ ] [ ] [ ] b 1 =, b =, b = 0 1 Are they sufficient to form a basis for R 2 2? If not, which vector(s) should be added? 3. Now consider the space of symmetric 2 2 matrices, with the same addition and scalar multiplication operations. Write down a basis for this space. 3 Transformations (15 Points) Write the matrices that correspond to the following transformations: 1. M : R 2 R 3 defined by M(x, y) = (2x + 3y, x y, 9x) 2. M : P 5 P 5 defined by (Mf)(t) = f (t), where P n is the space of degree-n polynomials with the standard basis (1, t, t 2,..., t n ) 4 Matrix Inverses (20 Points) In this question, all matrices are square. Be sure to justify every step in your proofs: indicate which previous statements the step follows from, and what rule(s) you re using to make this step. 1. We know that if A is an invertible matrix, then there exists a matrix A 1 such that AA 1 = I and A 1 A = I. Assume we know that there exist matrices B and C, such that BA = I and AC = I. Prove that B = C. (Hint: try beginning with the product BAC.) 2. If matrices X and Y are invertible, prove that (XY ) 1 = Y 1 X 1. 5 Kernel Perceptron (20 Points) As discussed in class, we can generalize the perceptron algorithm to work on any Hilbert space (complete inner product space) the result is called a kernel perceptron. In this question we ll work out how to implement the kernel percepton when our feature vector φ(x) is in R q with q large (so that it s impractical to work with vectors like φ(x) explicitly). To do so, we will assume that we have a subroutine to 2

3 compute φ(x) φ(x ) for any vectors x, x efficiently (i.e., without ever building a vector in R q explicitly); this operation is called a kernel function. Assume we have observed input vectors x i R p with labels y i { 1, 1} for i = 1... n. Also assume we have a feature function φ : R p R q. Then the ordinary perceptron algorithm works as follows: we start with weight vector w = 0 and bias b = 0. We run through the data repeatedly. As long as our predictions are correct, we leave w and b alone. If we make a mistake, we update (if the true label is positive, y i = 1), or w w + φ(x i ) b b + 1 w w φ(x i ) b b 1 (if the true label is negative, y i = 1). If on some iteration w φ(x i ) + b = 0, we consider this example to be a mistake, no matter what the correct sign y i is. Because of the form of the perceptron update, we can see that the weights are always a linear combination of the labeled training data, i.e., w = n a j y j φ(x j ). (1) j=1 The only way we ever use the weight vector w is to make predictions: ŷ i = sign(w φ(x i ) + b) (2) ( n ) = sign a j y j φ(x j ) φ(x i ) + b (3) j=1 So, we never need to store w itself, but only the coefficient vector a = (a 1, a 2,...). The predictions depend on the data only through dot products of the form K ij = φ(x j ) φ(x i ) which we assumed above that we can compute efficiently. K R n n is called the kernel matrix or Gram matrix; for our purposes, we can compute it once before starting the algorithm. The kernel perceptron is this version of the algorithm: the one that stores a and b instead of w and b. For this problem, we will use one-dimensional observations x R and a quadratic feature function φ(x) = (x, x 2 ). (Yes, we know that q = 2 is not actually too large to work with.) You are given four data points with observed x values of 1, 0.1, 0.1, 1, and labels (y values) of 1, 1, 1, Give an expression for the value of φ(x i ) φ(x j ) in terms of x i and x j. 2. Compute the kernel matrix K for this data. 3

4 3. Starting with parameters a = (0, 0, 0, 0) and b = 0, simulate two full passes of the kernel perceptron through the data (in the order given), reporting the prediction for each data point and the value of a and b after processing each data point. For a trivial update (one which doesn t change the parameters), please just write same instead of repeating the values of a and b. 4. What is the error rate of the final parameters a and b on this data set? That is, what fraction of examples are classified incorrectly? 5. What is the lowest error rate that we could achieve on this data with an ordinary linear perceptron ŷ = sign(wx + b)? Hint: for part 3, you should only need to make nontrivial updates on approximately half of the steps. 6 Differentials (15 Points) Consider constant vectors a, b, c, d, e and a variable matrix X. Suppose y = a ln(exp(b T Xc) + exp(d T Xe)). Derive an expression for the differential dy in terms of dx. Show your work. If you computed the ordinary derivative dy, what would be the number of modes dx of the resulting tensor? A Extra credit: Probability (10 Points) In this problem you will work out a probability distribution for statistics about demographics and likes for traveling. First, denote X as a random variable indicating gender and Y as a random variable indicating a like for traveling. Suppose we have: Male Female P (X) = 3/5 2/5 Like Traveling Dislike Traveling P (Y ) = 2/3 1/3 P (Z X): Male Female Age /6 1/12 Age /6 1/4 Age /6 2/3 1. Construct the joint probability table P (X, Y ) assuming that these two random variables are independent. 4

5 2. Now suppose you are given additional statistics: a conditional distribution of Z, age, given X, gender. With this extra information (along with the assumptions in part 1): Compute the joint table P (X, Z). What is the probability of a person being female given that the age is 40 50? What is the probability that the age is 20 30? 5