18.S096: Homework Problem Set 1 (revised)

Transcription

1 8.S096: Homework Problem Set (revised) Topics i Mathematics of Data Sciece (Fall 05) Afoso S. Badeira Due o October 6, 05 Exteded to: October 8, 05 This homework problem set is due o October 6, at the start of class Try ot to look up the aswers, you ll lear much more if you try to thik about the problems without lookig up the solutios. You ca work i groups but each studet must write his/her ow solutio based o his/her ow uderstadig of the problem. If you eed to impose extra coditios o a problem to make it easier, state explicitly that you have doe so. Solutios where extra coditios were assumed will also be graded (probably scored as a partial aswer).. Liear Algebra Problem. Show the resut we used i class: If M R is a symmetric matrix ad d the max U R d U T U=I d d Tr ( U T MU (+) where λ is the largest k-th eigevalue of M. k ) d (+) = λ (M), k k=

2 . Estimators Problem. Give x,, x i.i.d. samples from a distributio X with mea µ ad covariace Σ, show that µ = T x k, ad Σ = (x k µ ) (x k µ ), k= k= are ubiased estimators for µ ad Σ, i.e., show that E [µ ] = µ ad E [Σ ] = Σ..3 Radom Matrices Recall the defiitio of a stadard gaussia Wiger Matrix W : a symmetric radom matrix W R whose diagoal ad upper-diagoal etries are idepedet W ii N (0, ) ad, for i < j, W ij N (0, ). This radom ma- uder orthogoal cojugatio: U T W U W for trix emsemble is ivariat ay U O(). Also, the distributio of the eigevalues of W coverges to the so-called semicircular law with support [, ] dsc(x) = 4 x [,] (x). (try it i draw a histogram of the distributio of the eigevalues of W for, say = 500.) MATLAB I the ext problem, you will show that the largest eigevalue of W has expected value at most. For that, we will make use of Slepia s Compariso Lemma. Slepia s Compariso Lemma is a crucial tool to compare Gaussia Processes. A Gaussia process is a family of gaussia radom variables idexed by some set T, more precisely is a family of gaussia radom variables {X t } t T (if T is fiite this is simply a gaussia vector). Give a gaussia process X t, a particular quatity of iterest is E [max t T Xt]. Ituitively, if we have two Gaussia processes X t ad [ Y t with mea zero E [X t ] = E [Y t ] = 0, for all t T ad same variaces E Xt ] [ ] = E Y t the the process that has the least correlatios should have a larger maximum (thik the maximum etry of vector with i.i.d. gaussia etries versus oe always with the same Note that, a priori, there could be a very large eigevalue ad it would still ot cotradict the semicircular law, sice it does ot predict what happes to a vaishig fractio of the eigevalues.

3 gaussia etry). A simple versio of Slepia s Lemma makes this ituitio precise: I the coditios above, if for all t, t T the E [X t X t ] E [Y t Y t ], [ ] [ ] E max X t E max Y t. A slightly more geeral versio of it asks that the two Gaussia processes X t ad Y t have mea zero E [X t ] = E [Y t ] = 0, for all t T but ot ecessarily the same variaces. I that case it says that: If or all t, t T E [X t X t ] E [Y t Y t ], () the ] E [max X t [ E ] max Y t. Problem.3 We will use Slepia s Compariso Lemma to show that. Note that Eλ max (W ). λ max (W ) = max v T W v, v: v = which meas that, if we take for uit-orm v, Y v := v T W v we have that [ ] λ max (W ) = E max Y v, v S. Use Slepia to compare Y v with X v defied as where g N (0, I ) X v = v T g, 3. Use Jese s iequality to upperboud E [max v S X v]. Although ituitive i some sese, this is a delicate statemet about Gaussia radom variables, it turs out ot to hold for other distributios. 3

4 Problem.4 I this problem you ll derive the limit of the largest eigevalue of a rak perturbatio of a Wiger matrix. For this problem, you do t have to justify all of the steps rigorously. You ca use the same level of rigor that was used i class to derive the aalogue result for sample covariace matrices. Derivig this pheomea rigorously would take cosiderably more work ad is outside of the scope of this homework. Cosider the matrix M = W + βvv T for v = ad W a stadard Gaussia Wiger matrix. The purpose of this homework problem is to uderstad the behavior of λ max (M). Because W is ivariat to orthogoal cojugatio we ca focus o uderstadig ( ) λ max W + βe e T. Use the same techiques as used i class to derive the behavior of this quatity. (Hit: at some poit, you ll probably have to ite ate 4 x gr y x dx. You ca use the fact that, for y >, 4 x y x (y dx = π ) y 4 (you ca also use a itegrator software, such as Mathematica, for this)..4 Diffusio Maps ad other embeddigs Problem.5 The rig graph o odes is a graph where ode < k < is coected to ode k ad k + ad ode is coected to ode. Derive the two-dimesioal diffusio map embeddig for the rig graph (if the eigevectors are complex valued, try creatig real valued oes usig multiplicity of the eigevalues). Is it a reasoable embeddig of this graph i two dimesios? Problem.6 (Multidimesioal Scalig Revised) Suppose you wat to represet data poits i R d ad all you are give is estimates for their Euclidea distaces δ ij x i x j. Multiimesioal scalig attempts to fid a d dimesios that agrees, as much as possible, with these estimates. Orgaizig X = [x,..., x ] ad cosider the matrix whose etries are δ ij.. Show that, if δ ij = x i x j the there is a choice of x i (ote that the solutio is ot uique, as a traslatio of the poits will preserve the pairwise distaces, e.g.) for which X T X = H H, 4

5 where H = I T.. If the goal is to fid poits i R d, how would you do it (keep part of the questio i mid)? (The procedure you have just derived is kow as Multidimesioal Scalig) This motivates a way to embed a graph i d dimesios. Give two odes we take δ ij to be the square of some atural distace o a graph such as, for example, the geodesic distace (the distace of the shortest path betwee the odes) ad the use the ideas above to fid a embeddig i R d for which Euclidea distaces most resemble geodesic distaces o the graph. This is the motivatio behid a dimesio reductio techique called ISOMAP (J. B. Teebaum, V. de Silva, ad J. C. Lagford, Sciece 000). 5

6 MIT OpeCourseWare 8.S096 Topics i Mathematics of Data Sciece Fall 05 For iformatio about citig these materials or our Terms of Use, visit: