COS 424: Interacting with Data. Written Exercises

Transcription

1 COS 424: Interacting with Data Hoework #4 Spring 2007 Regression Due: Wednesday, April 18 Written Exercises See the course website for iportant inforation about collaboration and late policies, as well as where and when to turn in assignents. Be sure to show your work and justify your answers. Proble E-1 Consider a ixture of Gaussians odel defined by K eans µ 1,...,µ K, variance σ 2, and proportions π = π 1,...,π K. In such a odel, each (real-valued) X n is generated as follows: First, one of the ixture coponents Z n {1,...,K} is chosen at rando according to π (so that Z n = z with probability π z ). Then, given that Z n = z, X n is chosen according to a Gaussian distribution with ean µ z and variance σ 2. Note that only X n is visible; Z n is hidden. We assue that σ > 0 is known and fixed. a. Give a graphical odel depiction of this process, including the paraeters. b. Given data X 1,...,X N, describe in detail the EM algorith for estiating µ 1,...,µ K and π. c. Argue that as σ 2 0, this algorith approaches the K-eans algorith. d. Argue directly that as σ 2 0, the EM objective approaches the K-eans objective. Proble E-2 As is usual for linear regression, suppose we are given training data (x 1,y 1 ),...,(x,y ) where y i R and x i R n (with coponents x ij ). In this proble, we seek linear odels of the for ˆf(x) = w 0 + w x where w 0 is the scalar intercept ter, and w = w 1,...,w n is a (colun) vector of weights over the n inputs. Consider the proble in ridge regression of iniizing (w 0 + w x i y i ) 2 + λ w 2 2. (1) Here, as in Hastie et al. (but unlike in class), we include an explicit intercept ter w 0, but oit this ter fro the regression penalty. a. Suppose for this part only that x ij = 0 for all j. Let X be the n atrix of all inputs in which the i-th row is equal to (the transpose of) x i, and let y be the (colun) vector whose i-th entry is y i. Show that the solution of (1) is given by where I is the n n identity atrix. ŵ 0 = 1 y i ŵ = (X X + λi) 1 X y

2 b. Returning to the general case (in which the input vectors do not su to zero), let a j = 1 x ij and define x i by x ij = x ij a j. Note that, after centering in this fashion, the new input vectors su to zero so that the technique in the last part can be applied. Show that iniizing (1) is equivalent to iniizing (w 0 + w x i y i ) 2 + λ w 2 2. (2) In other words, if ŵ 0,ŵ is the solution that iniizes (1), and ŵ 0,ŵ is the solution that iniizes (2), show that ŵ 0 +ŵ x = ŵ 0 +ŵ x for any x and its transfor x. Moreover, given a solution ŵ 0,ŵ to (2), show explicitly how to transfor it directly into a solution ŵ 0,ŵ to (1). c. Suppose that the inputs are both centered and scaled. In other words, suppose we instead define x i by x ij = (x ij a j )/s j for soe constants s j. Show that the iniization probles (1) and (2) need no longer be equivalent (in the sense described above). Show nevertheless how a solution ŵ 0,ŵ to (2) can be transfored back into ŵ 0,ŵ, not necessarily a solution to (1), but for which ŵ 0 + ŵ x = ŵ 0 + ŵ x for any x and its transfor x. Proble E-3 In this proble, we will show that ridge regression can be kernelized by showing that the iniization criterion can be rewritten in ters only of inner products of input vectors. Assue a training set as in the last proble. Let w R n be any vector, and let α 1,...,α be chosen to iniize F(α 1,...,α ) = w 2 α i x i. Let v = α i x i. a. Prove that v x i = w x i for all i. (Hint: consider the partial derivatives of F.) b. Prove that v 2 w 2 with equality if and only if v = w. c. Consider iniizing (w x i y i ) 2 + λ w 2 2 (3) where λ 0. Show that this iniization proble ust always have a solution w which is a linear cobination of x 1,...,x. (You do not need to show that every solution has this for, just that there always exists at least one solution in this for.) d. Substituting i α ix i for w, rewrite (3) so that the input vectors x i only appear as inner products with other input vectors, thus showing that this iniization proble can be kernelized. 2 2

3 Proble E-4 In class, we showed that if X and Y are rando variables with Y real-valued, then E [( ˆf(X) Y ) 2] is iniized over all choices of the function ˆf when ˆf(x) = E[Y x], that is, when ˆf(x) is the expected value of Y for a given x. In this proble, we will see what happens if we instead use [ E ˆf(X) ] Y. In particular, we will see that this expectation of the absolute loss is iniized over all choices of ˆf when ˆf(x) is equal to the edian of Y given x. (We say that is a edian of a real-valued rando variable Z if Pr[Z ] 1/2 and Pr[Z ] 1/2. Note that the edian is not always unique.) To siplify the proble, we fix x (as we did in class) and also assue that Y is concentrated on a finite set of values. Thus, the proble can be reforulated as follows: Let c 1 < c 2 < < c l be the finite set of values in Y s range, and let Y be equal to c j with probability p j (where we assue without loss of generality that the p j s are all strictly positive). Prove that [ E ˆf ] Y is iniized over ˆf R if and only if ˆf is equal to a edian of Y. (Hint: First deterine the value ˆf [ that iniizes E ˆf ] Y when ˆf is restricted to lie in the confined range [c i, c i+1 ]. Then use your answer to prove the general result.) 3

4 Experients and prograing in R Data and code: The data, helper code and function teplates for the probles below can be obtained by following the links on the webpage for this assignent. Once downloaded into a suitable location, just type source("helper.r") to load all of the datasets described below (other than co2 which is already built in to R). In every case, this will load training and test sets as appropriately naed data fraes. The quantity to be predicted will always be the last colun of these data fraes. What to turn in: You should turn in the filestubs.r with all of the functions filled in. Also turn in any other files that you ay have created and used to coplete this assignent, for instance containing additional R functions; be sure to include the R code that you used to coplete each of the probles. These should all be subitted electronically using oodle, as should your predictions for optional Proble R-5, if you choose to coplete it. Your written answers to the various probles, including any plots that you generated, should be subitted in hard copy together with your written exercises. Proble R-1 Ipleent the ridge regression algorith as described in Section of Hastie et al. (as well as Proble E-2 above). Use exactly the ethod they describe in which an iplicit intercept ter is always included in the regression, but is always oitted fro the penalty ter used by ridge regression; use the centering technique they describe for this purpose. Also, each input (a.k.a feature or diension or variable) should be scaled to have unit variance on the training set. In other words, if the training exaples are x 1,...,x where x ij is the j-th input value of x i, then each x ij should be replaced by (x ij a j )/s j where and a j = 1 x ij s j = 1 (x ij a j ) 2. To ake predictions on test exaples, you will need to scale and center the in the sae fashion (using the sae constants a j and s j coputed only on the training set). Call your procedure ridge(); the paraeters and return values for this procedure are detailed in the provided file called stubs.r. Test your procedure to be sure it is working. For instance, you can try it on sall data atrices for which the answer can be coputed by hand. Write a short paragraph explaining what steps you took to be sure your code is working properly (this is crucial since you will be using this function throughout the rest of the assignent). Proble R-2 The co2 tie series dataset records atospheric concentrations of carbon dioxide onthly fro 1959 to This dataset should already be built in to standard distributions of R 4

5 as the tie-series object co2. (Be careful not to confuse it with the CO2 dataset, also built in to R.) The goal is to estiate future CO 2 concentrations as a function of the year t. We will experient with a nuber of bases, or sets of functions. For instance, the basis {t,t 2 } eans that we are seeking an approxiation of CO 2 concentration of the for w 0 + w 1 t + w 2 t 2, where t is the date in years, possibly with a fractional part (as returned, for instance, by tie(co2)). (On this assignent, we always iplicitly include the intercept ter.) Write a function called run.co2() that runs your ridge regression code on data collected through a specified cut-off year for a variety of bases. Details of what is required for this procedure are given in stubs.r. a. Run your ridge regression procedure on each basis given below with the regression paraeter λ set equal to 10 6 : {t} {t,t 2 } {t,t 2,...,t 20 } {t,t 2,...,t 50 } {t,t 2,cos(2πt)} {t,t 2,sin(2πt)} {t,t 2,sin(2πt),cos(2πt)} Use all data fro the start of the recording period in 1959 through the end of 1974 as training data; use the data collected fro the beginning of 1975 through the end of the recording period in 1997 as a test set. For each basis below, record (and turn in) the root ean squared error (RMSE) on the test set, where the RMSE of a odel ˆf on a set (x 1,y 1 ),...,(x,y ) is given by 1 ( ˆf(x i ) y i ) 2. The RMSE on a test set is considered a reasonable easure of the accuracy of the predictions. Also ake a plot showing actual CO 2 concentrations as a function of the year, superiposed with the concentrations (both training and test) predicted by the odel that you built for that basis. (You ight wish to use colors to distinguish predictions in the training set fro predictions in the test set.) b. Experient with other choices of the paraeter λ and/or other bases. Turn in the results of these experients. c. Briefly discuss the results, including the following questions: Which odel gives the best predictions? How do the results ake sense and fit your expections (and how do they not)? If it were the beginning of 1975 and you only had data through that date, which odel would have seeed the ost reasonable to have chosen for predicting future CO 2 concentrarions, and how accurate would that odel actually have been? How does working with this kind of tie-series data differ fro i.i.d. data? 5

6 Proble R-3 The housing dataset was constructed fro the 1990 US census. Here, the goal is to predict the edian price of a house in a given region based on deographic coposition and the state of the housing arket in that region. Detailed descriptions of the variables used in this and the other datasets are available on the ain webpage for this assignent. After following the instructions above, this dataset will be loaded into the variables house.train and house.test. The abalone dataset is concerned with predicting the age of abalone (a shellfish) fro physical easureents. The age of abalone can be deterined by cutting the shell through the cone, staining it, and counting the nuber of rings through a icroscope a tedious and tie-consuing task. Here, we instead attept to predict the age using other easureents, which are easier to obtain, such as sex, length, diaeter, etc. After following the instructions above, this dataset will be loaded into the variables abalone.train and abalone.test. a. On the housing dataset, run ridge regression on the provided training and testing sets for a variety of values of the regularization paraeter λ. Explore this paraeter using exponentially varying values such as...,4,2,1,0.5,0.25,... Plot test RMSE as a function of log 10 (λ). Your goal, of course, is to iniize the test RMSE, and you ay need to do soe exploring to find the interesting range of λ where this quantity is sallest. (It is not necessary that your ridge() function work for very sall values of λ due to nuerical liitations of R when solving linear equations. This ay liit the range of values you can search.) b. Repeat the experient in the last part for the abalone dataset. c. Briefly discuss the results above, including the following questions: How sensitive is the perforance on the test set to the choice of λ for each of the datasets? In other words, if we are slightly off in our choice of λ, how uch will this affect test perforance? How do the plots you ade for housing and abalone differ qualitatively, or how are they siilar? Speculate on possible explanations for these differences or siilarities, and what these ean in practice when choosing λ. d. Using the housing test set, ake a scatterplot coparing predicted edian housing prices to actual housing prices. In other words, ake a scatterplot with one point for each exaple in the test set where the x-coordinate is the actual edian housing price, and the y-coordinate is the price predicted by your ridge() procedure when run on the housing dataset with the best choice of λ (as deterined above). Briefly discuss the quality of the fit as reflected on this scatterplot. e. Manually explore the weight coefficients returned by your ridge() procedure when run on the housing dataset with the best choice of λ (as deterined above). Discuss what can or cannot be inferred fro these coefficients regarding the factors that do or do not influence the edian price of houses in a region. 6

7 Proble R-4 Ipleent k-fold cross validation as a eans for estiating test error using only the training set. Your procedure should be called cross.val(), and should call ridge() k ties in the standard anner to obtain k estiates of the test error which are averaged and returned by the function. Details of what is required are given in the stubs.r file. a. Run 10-fold cross validation on the housing training set for the sae set of λ values you used in part (a) of the last proble. On a single plot, show the test RMSE as a function of log 10 (λ) as estiated using cross validation. On the sae plot, also show your results fro part (a) of the last proble so that the estiated and actual test error nubers can be copared. b. Repeat the experient in the last part for the abalone dataset, coparing your results to those obtained in part (b) of Proble R-3. c. Discuss these plots. How good a job is cross validation doing as a ethod for estiating test error? How good a job is cross validation doing as a ethod for selecting the best choice of λ? Proble R-5 (optional and for extra credit) The coputer-activity dataset records various perforance easures fro a Sun Sparcstation, such as the bytes read or written fro syste eory. The goal is to predict the percentage of tie that the cpu is running in user ode. After following the instructions above, this dataset will be loaded into the variables cop.train and cop.test. Note that we are not providing target values for the test set (all of these have been set to zero). For this part of the assignent, we are asking you to use whatever techniques you wish to coe up with the ost accurate predictions you can for the test set. After the deadline, we will copare your subitted results to the actual test values, and we will post the results on-line. For this part of the hoework, you can use soe of the techniques explored on this proble set. Or you can try different ethods such as feature selection, lasso, nearestneighbors, decision trees, kernel regression, and ore. You can use any of the functions built in to R, provided you understand what they are doing. You are welcoe and encouraged to try out your own ideas. (But please do not use any ethods that go against the spirit of this assignent, such as searching for this dataset on the web.) You should save and subit a file with your nae (or a pseudony), a one-sentence description of the technique used, and a list of predictions on test exaples. A function called save.prediction.file() has been provided for this purpose (see helper.r for details on how to use it). The saved file should be subitted via oodle. We will use the inforation you provide to generate a public, on-line copilation of the results. If you wish to reain anonyous, you do not need to provide your real nae in the file you generate, but can instead use a pseudony of your choice. Also, write and turn in a brief description of the approach you followed to generate your predictions, including plots you ight have used along the way (for instance, for choosing λ if using ridge regression), as well as an explanation of why you tried what you did. Finally, be sure to turn in any code that you wrote for this part. 7