2. High dimensional data

Transcription

1 /8/00. Hgh mensons. Hgh mensonal ata Conser representng a ocument by a vector each component of whch correspons to the number of occurrences of a partcular wor n the ocument. The Englsh language has on the orer of 5,000 wors. Thus, a ocument s represente by a 5,000 mensonal vector. Normalze the vectors so that they are all of unt length. If two ocuments are smlar, the ot prouct of ther corresponng vectors wll be close to one. If the ocuments are not smlar, then the ot prouct wll be close to zero. Search engnes represent both the content of web pages an also queres by vectors. To respon to a query, the search engne taes the ot prouct of the query vector wth all ocument vectors to locate the most relevant ocuments. aarvar 0 abacus 0 anttrust 4 ocument CEO 7 Mcrosoft 6 wnows 4 Fgure.: A ocument an ts corresponng vector. The vector space representaton of ocuments gves rse to hgh mensonal ata. Another example arses n etermnng pars of proucts purchase at the same tme. If there are 8 0,000 proucts for sale n a grocery store, the number of pars s0. Recorng the number 8 of tmes customers buy a partcular par results n a0 mensonal vector. Our ntuton has been forme n low mensons an s often msleang when conserng hgh mensonal ata. Conser placng 00 ponts unformly at ranom n a unt square. Unformly at ranom, means that each coornate s generate nepenently an selecte unformly at ranom from the nterval [0, ]. If we select a pont an measure the stance to all other ponts, we wll see a strbuton of stances. If we ncrease the menson an generate the ponts unformly at ranom n a 00-mensonal unt hypercube, the strbuton of stances becomes concentrate about an average stance. The reason for ths s the followng. Let x an y be ponts n a -mensonal space. Then

2 /8/00. Hgh mensons st( x y) x y Snce the strbuton of each x an each y s unform over [0,], the strbuton of each x y s of boune varance an by Hoeffng s nequalty the strbuton of x y s concentrate about ts expecte value.. The hgh mensonal sphere One of the nterestng facts about a unt raus sphere n hgh mensons s that as the menson ncreases the volume of the sphere goes to zero. Ths has mportant mplcatons. Also the volume of the sphere s essentally all contane n a thn slce at the equator. The volume s also essentally all contane n a narrow annulus at the surface. There s essentally no nteror volume. Smlarly the surface area s essentally all at the equator... Volume of the unt hyper sphere an unt hyper cube Conser the fference between the volume of a unt hypercube an the volume of a unt raus hyper sphere as the menson,, of the space ncreases. As the menson of the hypercube ncreases, ts volume s always one an the maxmum possble stance between two ponts grows as. In contrast, as the menson of a hyper sphere ncreases, ts volume goes to zero an the maxmum possble stance between two ponts stays at two. Note that for =, the unt square centere at the orgn les completely nse the unt raus crcle. The stance from the orgn to a vertex of the square s an thus the square les nse the crcle. At =4 the stance from the orgn to a vertex of a unt hypercube centere at the orgn s an thus the vertex les on the surface of the unt 4-sphere centere at the orgn. As the menson ncreases the stance from the orgn to a vertex of the hypercube ncreases as an for large the vertces of the hypercube le far outse the unt sphere. Fgure. llustrates conceptually a hypercube an a hyper sphere. The vertces of the hyper cube are at stance from the orgn an thus le outse the unt sphere. On the other han the m ponts of each face of the cube are only stance from the orgn an thus are nse the sphere. Almost all the volume of the cube s locate outse the sphere.

3 /8/00. Hgh mensons 3 Almost all volume of the hypercube unt sphere vertex of hyper cube Fgure.: Conceptual rawng of hyper sphere an hyper cube = =4 Arbtrary Fgure.3: Illustraton of the relatonshp between the hyper sphere an hyper cube n, 4, an mensons.. Volume of a hyper sphere For fxe menson, the volume of a hyper sphere as a functon of ts raus grows as r. For fxe raus, the volume of a hyper sphere s a functon of the menson of the space. What s nterestng s that the volume of a unt hyper sphere goes to zero as the menson of the sphere ncreases. To calculate the volume of a hyper sphere, one can ntegrate n ether Cartesan or polar coornates. In Cartesan coornates the volume of a unt hyper sphere s gven by x x x x x x V x x x x x x x x x

4 /8/00. Hgh mensons 4 Snce the lmts of the ntegrals are qute complex t s easer to use polar coornates. Then V r r S r0 where S s the sol angle extene by the sphere. Snce the varables r an o not nteract, S V r r r0 S The queston remans, how o we etermne the surface area A Conser a fferent ntegral x x x I e x x x? Inclung the exponental allows us to ntegrate to nfnty rather then stoppng at the surface of a hyper sphere. Ths allows us to ntegrate easly n both Cartesan coornates an polar coornates. Integratng n both Cartesan an polar coornates allows us to solve for the surface area of the unt hyper sphere. Frst calculate I() by ntegraton n Cartesan coornates. x I e x Next calculate I() by ntegratng n polar coornates. Snce each se of the fferental element s r, the volume of the fferental element s r r r r. Thus The ntegral r I e r r S 0 A s the ntegral over all sol angles an gves us the surface area, n S r A(), of a unt hyper sphere. Thus, I A e r r. Evaluatng the remanng ntegral gves 0 S where the gamma functon r t e r r e t t 0 0 x s a generalzaton of the factoral functon for non ntegers values of x. x x x,, an x x!.. For nteger x,

5 /8/00. Hgh mensons 5 Returnng to the ntegral A. Therefore, the volume of a unt hyper sphere s A V To chec the formula for the volume of a hyper cube note that V 3 3 an 4 V whch are correct volumes for the unt hyper spheres n two an three mensons. Note that snce lm V ( ) 0. s an exponental n an grows as the factoral of,..3 Most of the volume s near the equator Conser a hgh mensonal unt sphere an fx the North Pole on the x axs at x. Dve the sphere n half by ntersectng t wth the plane x 0. The ntersecton of the plane wth the sphere forms a regon of one lower menson, namelyx x, x 0 calle the equator. The ntersecton s a sphere of menson - an has volume V. In three mensons ths regon s a crcle, n four mensons the regon s a three mensonal sphere, etc. In general, the ntersecton s a sphere of menson - an has volume V. It turns out that essentally all of the mass of the upper hemsphere sphere les between the plane x 0 an a parallel plane, x, that s slghtly hgher. To see ths, calculate the volume of the porton of the sphere above the slce lyng between x 0 an x t 0. Let 0 T x x, x t be the porton of the sphere above the slce. To calculate the volume of T, ntegrate over t from t 0 to. The ncremental volume wll be a s of wth t whose face s a sphere of menson - of some raus epenng on t. The raus of the s s t Thus an therefore the surface area of the s s t V.

6 /8/00. Hgh mensons 6 Vol T t V t V t t t0 t0 Note that V enotes the volume of the mensonal unt sphere. We use Vol to enote Vol T for the volume of the regon T. the volume of other sets such as The above ntegral s ffcult to ntegrate so we use some approxmatons. Frst, we use the x approxmaton xe for all real x an change the upper boun on the ntegral to be nfnty. Ths gves t Snce t for t t0, an ntegral of the form t t0 t0 e 0 t t whch has value t Vol T V e t 0 t e t 0 t0 t e t can be upper boune by t0. Thus, an upper boun on the volume of T s t T e 0 V Vol (.) Next we lower boun the volume of the entre upper hemsphere. Tang the rato of the upper boun on the volume above the slce at t 0 to the lower boun on the volume of the entre hemsphere gves us an upper boun on the fracton of the volume above the slce. Snce we beleve that most of the volume s nx x, x, we use the approxmaton V t t V t t 0 0 m Usng the nequalty m for 0 Snce t n the range 0, V t t V t t 0 0 we can replace the t by n the ntegral an thus

7 /8/00. Hgh mensons 7 V t t V t t 0 0 V t (.) 0 V If we compute the rato of the upper boun on the volume of T, Eq. (.), to the lower boun on the volume of the hemsphere, Eq. (.), we see that the volume above the s x x, x t 0 s less than t e 0 of the total volume of the hemsphere. Lemma.: For any c 0 x x x c c e c Vol, V Proof: Substtute c for t 0 n the above. Note that we have shown that essentally all the mass of the sphere les n a narrow slce at the equator. Note that we selecte a unt vector n the x recton an efne the equator to be the ntersecton of a plane perpencular to the unt vector an the sphere. However, we coul have selecte an arbtrary pont on the surface of the sphere an consere the vector from the center of the sphere to that pont an then efne the equator usng the plane through the center perpencular to ths arbtrary vector. Essentally all the mass of the sphere les n a narrow slce about ths equator also..4 Most of the volume of a sphere s n a narrow annulus The area of a crcle s r. Note that one fourth of the area of the crcle s wthn stance one half from the center of the crcle. However, n mensonal space, for the sphere B 0, of raus centere at the orgn B e 4V Vol 0, V V. prove /. Thus, over one fourth of the volume of the mensonal sphere s wthn stance of the surface of the sphere prove /.

8 /8/00. Hgh mensons 8 c Lemma.: Proof: Substtute c c B e Vol 0, Vol for all c. n the above scusson..5 Most of the surface area of a sphere s near the equator Just as a two mensonal crcle has an area an a crcumference an a three mensonal sphere has a volume an a surface area, a mensonal sphere has a volume an a surface area. The surface area of the hyper sphere s the setx x. The crcumference at the equator s the set S x x, x 0. The surface area of the sphere s a menson lower than the volume an the crcumference at the equator s two mensons lower than the volume of the sphere. Just as wth volume, essentally all the surface area of the sphere s near the equator. To see ths, we calculate the surface area of the slce of the sphere between x 0 an x t 0. Let S x x, x t0. To calculate the surface area of S, ntegrate over t from t 0 to. The ncremental surface unt wll be a ban of wth t whose ege s a - mensonal sphere of some raus epenng on t. At x tthe raus of the ege s the - mensonal crcumference of the ege s Thus t V. Vol S V t t t0 t an therefore Agan the above ntegral s ffcult to ntegrate an we wll use the same approxmatons as n the earler secton on volume. Ths leas to the equaton t S e 0 V Vol (3) Next we lower boun the surface area of the entre upper hemsphere.

9 /8/00. Hgh mensons 9 V t t V t t (4) 0 0 V If we compare the upper boun on S, Eq. (3), wth the lower boun on the surface area of the hemsphere, Eq. (4), we see that the surface area above the ban x x,0 x t0s less than t e 0 of the total surface area. Lemma.3: For any c 0 x x x c c e c Vol, V From the fact that the volume of the sphere s the ntegral of the surface area of a sphere V V r 0 we see that surface area s the ervatve of the volume wth respect to the raus r. In two mensons the volume of a crcle s r an the crcumference s r. In three mensons the volume s r an the surface area s 4 r..6 Generatng ponts unformly at ranom on a sphere We now conser how to generate ponts unformly at ranom on the surface of a hyper sphere. Frst, conser generatng ranom ponts on a crcle of unt raus by the followng metho. Inepenently generate each coornate unformly at ranom from the nterval,. Ths prouces ponts strbute unformly at ranom over a square that s large enough to completely contan the unt crcle. If we then project each pont onto the unt crcle, the strbuton wll not be unform snce more ponts fall on a lne from the orgn to a vertex of the square, than fall on a lne from the orgn to the mpont of an ege ue to the fference n length. To solve ths problem, scar all ponts outse the unt crcle an project the remanng ponts onto the crcle. One mght generalze ths technque n the obvous way to hgher mensons. However, the rato of the volume of a mensonal unt sphere to the volume of a mensonal unt cube ecreases raply mang the process mpractcal for hgh mensons snce almost no ponts wll le nse the sphere. The soluton s to generate Gaussan varables. The probablty strbuton for a pont x, x,, x s gven by

10 /8/00. Hgh mensons 0 P x, x,, x e x x x an s sphercally symmetrc. Thus, normalzng the vector x, x,, x to a unt vector gves a strbuton that s unform over the sphere..7 Dstance between ranom ponts on a unt mensonal sphere If we pc ranom ponts on the surface of a raus one hyper sphere, the stances woul agan become more concentrate as the menson ncreases an woul approach a stance of square root two. To see ths, ranomly generate ponts on a -mensonal sphere. Rotate the coornate system so that one of the ponts s at the North Pole. Snce all of the surface area of a hgh mensonal sphere s n a narrow ban about the equator the remanng ponts are all near the equator an the stance of each of these ponts to the pont at the North Pole s about. Fgure.4: Two ranomly chosen ponts n hgh menson are almost surely orthogonal.

11 /8/00. Hgh mensons Dstance between two ponts on two fferent spheres n hgh menson Gven two unt raus spheres n hgh menson wth centers P an Q separate by a stance, what s the stance between a ranomly chosen pont x on the surface of the frst sphere an a ranomly chosen pont y on the surface of the secon sphere? We can wrte y xas ( P x) ( Q P) ( y Q). We clam that the three segments wll be par-wse nearly orthogonal. To see ths, frst Q Ps a fxe (not ranom) vector an by the fact that most of the surface area of a sphere s close to (any) equator, P x an yqare nearly orthogonal to Q P. Further, snce x an y are nepenent, we can pc them n any orer; so pc x frst. Then, when y s pce, both P x an Q Pare fxe vectors. Now, there s very lttle surface area of sphere far away from the equator perpencular to each of P x an Q P separately. But then wth a factor of two, we get by the unon boun that there s lttle surface area far from ether equator; thus we get mutual orthogonalty. Thus, by Pythagoras Theorem we have x y P x Q P y Q. REWRITE ABOVE PARAGRAPH x Y P Q Fgure.5: Dstance between a par of ranom ponts from two fferent Gaussans.8 Gaussans n hgh menson A one mensonal Gaussan has ts mass close to the orgn. However, as we ncrease the menson somethng fferent happens. The -mensonal sphercal Gaussan wth zero mean an varance has ensty functon

12 /8/00. Hgh mensons px ( ) exp x / Although the value of the Gaussan s maxmum at the orgn, there s very lttle volume there. Integratng the probablty ensty over a unt sphere centere at the orgn, yels zero mass snce the volume of the sphere s zero. In fact, one woul nee to ncrease the raus of the sphere to ( ) before one woul have a nonzero volume an hence a nonzero probablty mass. If one ncreases the raus beyon, the ntegral ceases to ncrease even though the volume ncreases snce the probablty ensty s roppng off at a much hgher rate. Thus, the natural scale for the Gaussan s n unts of. Expecte square stance of pont from center of a Gaussan Conser a -mensonal Gaussan centere at the orgn wth varance. For a pont x [ x, x,, x ] chosen at ranom from the Gaussan, what s the expecte square magntue of x? x x E x x x E x e x For large, the value of the square magntue of x s tghtly concentrate about ts mean. We wll call the square root of the expecte square stance (namely the Gaussan. ) the ``raus of In the rest of ths secton, we conser sphercal Gaussans wth ; all results can be scale up by. The probablty mass of a Gaussan as a functon of the stance from ts center s gven by r e r IS THIS CORRECT OR IS THERE A CONSTANT C? where r s the stance from the center an s the menson of the space. The probablty mass functon has ts maxmum at whch can be seen as follows r r r r e r r ( ) e r r e 0 r. Calculaton of wth of annulus / r The functon e r rops off fast away from ts maxmum. In fact, most of the mass of the Gaussan wll be contane n a narrow annulus of wth O( ). Conser the rato of the probablty mass as a functon of r for r. where the probablty mass s maxmze, an r.

13 /8/00. Hgh mensons 3 e e e For large, e. Thus, the rato of probablty mass rops off as for a large constant nepenent of, the annulus between ra an contans most of the mass. So, as gets large, we have e. So wth of the annulus. USE O NOTATION WE DO NOT KNOW raus of the sphercal Gaussan CONSTANT Thus, smlar to the stuaton for the hyper-sphere, most of the mass s concentrate n a thn annulus (for the sphere, the rato was /, rather than /.) Separatng Gaussans Conser two sphercal unt varance Gaussans. The stance between two ponts generate by the same Gaussan s. If two ponts come from fferent Gaussans separate by, then the stance between them s. Here we have mae an approxmaton that the ponts le on a sphere of raus an thus there s some approxmaton error n the stances. Let c boun the approxmaton error. Then nees to be large enough so that c Snce n orer to etermne whether two ponts are from the same or fferent Gaussans. Ths requres that to be of orer or 4 n orer to etermne f two ponts were generate by the same or fferent Gaussans. Thus, mxtures of sphercal Gaussans can be separate prove ther centers are separate by more than Algorthm: Calculate all par wse stances between ponts. The cluster of smallest par wse stances must come from a sngle Gaussan. Remove these ponts an repeat the process. In Chapter 4, we wll see an algorthm to separate a mxture of sphercal Gaussans. Fttng a sngle sphercal Gaussan to Data Gven a set of sample ponts, x, x,, x n, n a mensonal space, we wsh to fn the sphercal Gaussan that best fts the ponts. Let F be the unnown Gaussan. The probablty of pcng these very ponts when we sample accorng to F s gven by 4.

14 /8/00. Hgh mensons 4 where the normalzng constant c s ce e x u x u x u x n n x. Note that c s really nepenent of μ an s equal x to e x. The Maxmum Lelhoo Estmator (MLE) of F gven the samples x, x,, x n s the F that maxmzes the above probablty. Lemma.4 : Letx, x,, xn be a set of ponts n -space. Then x u x u xn u n s x x xn mnmze when μ s the centro of the ponts x, x,, x n, namely. Proof: Settng the ervatve of x u x u x u to zero yels x x xn Solvng for u gves. n x u x u x u 0. n Thus, n the maxmum lelhoo estmate for F, s set to the centro. Next we wll show that σ s set to the stanar evaton of the sample. Substtute v an a xu xu xn u n the sample probablty formula. Ths gves n x e av x v e x Now, a s fxe an v s to be etermne. Tang logs, the expresson to maxmze s m vx av mln e x x To fn the maxmum, fferentate wth respect to ν, set the ervatve to zero, an solve for. The ervatve s vx x e x a m e vx x x x.

15 /8/00. Hgh mensons 5 Settng y vx n the ervatve, yels m a y y e y y v e y y y. Snce the rato of the two ntegrals s the expecte stance square of a mensonal sphercal Gaussan of stanar evaton to ts center an ths s nown to be we get a a m m v It s easy to see that the maxmum occurs when a m. Note that ths quantty s the square root average stance square of the samples to ther mean n a coornate recton, whch s the sample stanar evaton. Thus we get the followng Lemma. Lemma.5: The maxmum lelhoo sphercal Gaussan for a set of samples s the one wth center equal to the sample mean an stanar evaton equal to the stanar evaton of the sample. ADD NOTE THAT USING AVERAGE VALUE FOR MEAN BRINGS IN DEPENDENCY.9 The Ranom Projecton Theorem an the Nearest Neghbor problem Many problems often nvolve hgh mensonal ata. One such problem s the Nearest Neghbor problem n whch we are gven a set of n ponts n mensons. The ponts are processe an store n a atabase. Presente wth a set of query ponts, for each query, report the nearest pont from the atabase. Varatons of the problem where we have to report all nearby ponts are also of nterest. The am s often to mnmze the query response tme, perhaps at the cost of some extra pre-processng tme. One place the problem arses s n web search. Web pages are represente n the vector space moel as ponts n hgh mensonal space. As a web-crawler scovers web pages, t processes them nto some ata structure. A query q s also a pont n the hgh mensonal space. We wsh to fn the ponts closest to the pont q qucly. Here, we llustrate how fnng the nearest neghbour can be mae effcent by frst projectng the ponts n the atabase to a ranomly chosen lower mensonal space. The central result s a theorem that asserts that stances between pars of ponts are preserve (up to a nown scale factor) by a ranom projecton onto a subspace prove the menson of the subspace s not too low. Clearly one coul not project a 3-mensonal object onto a -mensonal subspace an preserve stances between most pars of ponts. We begn by provng that projectng any fxe -mensonal vector nto a ranom - mensonal subspace of R, results n a vector of length very close to tmes the length of the orgnal vector. The projecton s one by ranomly choosng a bass whose frst axes span the subspace we are projectng onto. Snce the subspace s ranom as s the bass

16 /8/00. Hgh mensons 6 for the subspace the square length of each component of the vector n the new coornate system shoul be equal an be / tmes the whole. Snce the projecton eeps only the frst coornates, the sum of the square value of the projecton s coornates woul be tmes the whole. The theorem states that ranom subspaces behave ncely. In fact, t asserts that the probablty that the length square of the projecton evates from falls off exponentally wth the evaton. To show that the probablty falls off exponentally fast t woul be convenent f the subspace was fxe an the vector was ranom. Thus we observe that projectng a fxe vector onto a ranom subspace s the same as projectng a ranom vector onto a fxe sub space. Let v be a fxe (not ranom) vector n R an V be a ranom -mensonal subspace of R. The length of the projecton of v onto V s the same ranom varable as the length of a ranom vector z of the same length as v projecte onto the subspace U spanne by the frst unt vectors of the coornate system. Let z ( z, z,, z ). The expecte value of. z s clearly. We wll show that the value of z s tghtly concentrate aroun Theorem (The Ranom Projecton Theorem): Let z be a ranom unt length vector n - mensons an z be the vector of ts frst components. For0 ò 6 Pr z ò e. Proof: We nee the followng fact. If x s a normally strbute real ranom varable wth zero mean an varance one, that s p( x) e x, then x tx tx tx 0 0 ( t) x E[ e ] e p( x) x e e x e x Now for t, tx Ee [ ] t t. One way of pcng a ranom vector z of length s to pc nepenent Gaussan ranom varables x, x,, x, each wth mean 0 an varance an tae z x/ x. Ths yels the ranom vector z of length one. Thus, when x /, wth òwe have

17 /8/00. Hgh mensons 7 Prob z Prob x x x x x x. x x x x x x. Prob 0 Thus for any t 0 Prob z Prob t x x x 0 x x x Prob t x x x x x x e Applyng Marov s nequalty whch states that Prob y E y Prob z E t x x x x x x e t E x x x t e e x x x E e t( ) x E tx e t t. where t s restrcte so that t. Now select t to mnmze the probablty. Let Mnmzng g s the same as maxmzng gt () t t t t f() t The maxmum of f(t) occurs for t 0. It s easy to chec that t0. Set ( ) t t0, then ( ) ( ) (ln ) Prob z e ( ) x usng xe for all real x. Now by power seres expanson, we have ln ln( ) (/ ) ò ò ò from whch the lemma follows for the case / z. The proof for the case when z / s smlar an s omtte.

18 /8/00. Hgh mensons 8 The Ranom Projecton Theorem enables us to argue (usng the unon boun) that the projecton to orer log n mensons preserves all parwse stances between a set of n ponts consstng of the atabase an the query ponts, so that we get the answers rght for all the queres. Ths s the content of the Johnson-Lnenstrauss lemma. Theorem (Johnson-Lnenstrauss lemma): For any 0 an any nteger n, let be a postve nteger such that 64ln n. ò Then for any set P of n ponts n R, there s a map f : R R such that for all u an v n P, ( ) u v f ( u) f ( v) ( ) u v Further ths map can be foun n ranomze polynomal tme. Proof: If the theorem s trval. Let S be a ranom -mensonal subspace an let f(u) be the projecton of u onto S. Let r f ( u) f ( v). Applyng the above Ranom Projecton theorem, for any fxe u an v, the probablty that r s outse the range ( ) u v,( ) u v s at most. By the unon boun the probablty that any 3 n n par has a large storton s less than 3. n n For the nearest neghbor problem, f the atabase has n ponts n t an we expect n queres urng the lfetme, then tae n n n an project the atabase to a ranom mensonal 64ln n space, where,. On recevng a query, project the query to the same subspace an ò compute nearby atabase ponts. The theorem says that wth hgh probablty, ths wll yel the rght answer, whatever the query. In general, nearest neghbor algorthms frst fn a set of canate nearby ponts an then choose the nearest pont from the set of canates. Suppose the number of canates s m. Wthout the projecton, worng n the whole mensonal space woul have taen tme m to compare the query pont to each canate. But wth the projecton, we tae only tme to project the query to the subspace an then m tme to compare t aganst the canates. Snce <<, ths saves tme. We o not go nto the etals of how to ensure that m s not too large here. Exercse: (Overlap of spheres) Let X a be a ranom sample from the unt sphere n - mensons wth the orgn as center. (a) What s the mean of ths ranom varable? The mean, enote EX ( ), s vector, whose th component s the mean of the th component of the ranom sample

19 /8/00. Hgh mensons 9 (b) What s the varance of X (agan component-wse)? (c) Show that for any unt length vector u, the varance of the real-value ranom varable u T X s u E( X ). Usng ths, compute the varance an stanar evaton of u T X. () Gven two spheres n space, both of raus one whose centers are stance a apart. Show that the volume of ther ntersecton s at most 4e a ( ) a tmes the volume of each one. [Hnt: See pcture an also use Lemma.] (e) From (), conclue that f the nter-center separaton of the two spheres s (raus / ), then they share very small mass. Theoretcally, at ths separaton, gven ranomly generate ponts from the two strbutons, one nse each sphere, t s possble to tell whch sphere contans whch pont,.e., classfy them nto two clusters so that each s exactly the set of ponts generate from one sphere. The actual separaton requres an effcent algorthm to acheve ths. Note that the nter-center separaton requre n (e) goes to zero as gets larger prove the raus of the spheres remans the same. So t s easer tell apart spheres (of the same ra) n hgher mensons. (f) Derve the requre separaton for a par of mensonal sphercal Gaussans, both wth the same stanar evaton. Soluton: (a) EX ( ) 0 for all, so EX ( ) 0. (b) Var( X ) E( X ) E( X ) by symmetry. Let V ( ) enote the volume of the unt sphere an A ( ) enotng the surface area of the sphere of raus one. The nfntesmal volume of an annulus of wth r at raus r has volume A( ) r r. So we have

20 /8/00. Hgh mensons 0 A( ) E( X ) A( ) r r r. V ( ) r0 V ( )( ) Thus, Var ( X ). (c) The proof s by nucton on. It s clear for. Var ( u X ) E(( u X ) ), snce the mean s 0. Now, E(( u X) ) E( u X ) E( uu j X X j ) j If the X ha been nepenent, then the secon term woul be zero. But they are obvously not. So we tae each secton of the sphere cut by a hyperplane of the form X constant, frst ntegrate over ths secton, then ntegrate over all sectons. In probablty notaton, ths s tang the ``contonal expectaton contone on (each value of) X an then tang the expectaton over all values of X. Dong ths, we get E( u u X X ) E u X u X E u u X X j j j j j j;, j EX u X EX, X3, X, 3, u X X EX EX X X u X u j X j X j;, j [Notaton: E( Y X) s some functon f of X ; t s really short-han for wrtng f ( a) E( Y X a).] Now, for every fxe value of X, E( X X) 0 for, so the frst term s zero. Snce a secton of the sphere s just a sphere, the secon term s zero by nucton on. () Loong at the pcture, by symmetry, we see that the volume of the ntersecton of the two spheres s just twce the volume of the secton of the frst sphere gven by: { x : x ; x a / } f we assume wthout loss of generalty that the center of the secon sphere s at ( a, 0, 0, 0). (e) Smple. (f) If a sphercal Gaussan has stanar evaton n each recton, then ts raus (really the square root of the average square stance from the mean) s. Its projecton on any lne s agan a Gaussan wth stanar evaton (as we show n Chapter 4 (or 5)??). Let a>0 an let the centers be 0 an ( a,0,0, 0) wthout loss of generalty. To fn the share mass, we can use the projecton onto the x axs an ntegrate to get that the share mass s x / ( xa) / Mn e, e. x We boun ths by usng

21 /8/00. Hgh mensons x / ( xa) / Mn e, e x a/ x x a e a ( xa) / x / e x e x xa/ x / x / e x e x xa/ xa/ a 8 where n the last step, we are able to ntegrate xe n close form. So agan, as soon as the nter-center separaton goes beyon a few stanar evatons, the share mass goes own exponentally Exercses Exercse.: Let x an y be ranom varables wth unform strbuton n [0,]. What s the expecte value E(x)? E(x )? E(x-y)? an E((x-y)^)? Exercse.: What s the strbuton of the stance between two ponts chosen unformly at ranom n the nterval [0,]? In the unt square? In the unt hypercube n 00 mensons? Exercse.3: Integrate usng polar coornates the area of the porton of a crcle n a cone of 45. Exercse.4: For what value of s the volume, V(), of a -mensonal hyper sphere maxmum? Exercse.5: How oes the volume of a hyper sphere of raus two behave as the menson of the space ncreases? What f the raus was larger than two but constant nepenent of? What functon of woul the raus nee to be for a hyper sphere of raus r to have approxmately constant volume as the menson ncreases? Exercse: (a) What s the volume of a hyper sphere of raus r n -mensons? (b) What s the surface area of a hyper sphere of raus r n mensons? (c) What s the relatonshp between the volume an the surface area of a hyper sphere of raus r n mensons? () Why oes the relatonshp etermne n (c) hol? (e) Geometrcally what s the secon ervatve wth respect to the raus of the volume of a hypersphere. Exercse.6: Conser vertces of a hyper cube centere at the orgn of wth two. Vertces are the ponts,,,. Place a unt raus hyper sphere at each vertex. Each sphere fts n a hyper cube of wth two an thus no two spheres ntersect. Prove that the volume of cx

22 /8/00. Hgh mensons all of the spheres s a vanshng fracton of the hyper cube as the menson goes to zero. That s, a pont of the hyper cube pce at ranom wll not fall nto any sphere. Exercse.7: How large must be for the annulus to contan 99% of the volume of the mensonal sphere. Exercse.8: Create a hstogram of all stances between pars of 00 ponts on a sphere n 3-mensons an 00-mensons. Exercse.9: (a) Wrte a computer program that generates n ponts unformly strbute over the surface of a -mensonal sphere. (b) Create a ranom lne through the orgn an project the ponts onto the lne. Plot the strbuton of ponts on the lne. (c) What oes your result from part b say about the surface area of the sphere n relaton to the lne,.e., where s the surface area concentrate relatve to the lne? Exercse.0: If one generates ponts wth each coornate a unt varance Gaussan, the ponts wll approxmately le on the surface of a sphere of raus are projecte onto a ranom lne through the orgn?. What s the strbuton when the ponts Exercse.: Quantfy the stance between two ranom ponts on the surfaces of two unt raus hyperspheres whose centers are separate by. I.e., prove that the probablty that the stance s more than a away s at most some (exponentally fallng) functon of a. Exercse.: Project the surface area of a sphere of raus n mensons on to a lne through the center. For,3, erve an explct formula for how the projecte surface area changes as we move along the lne. For large, argue (ntutvely) that the projecte surface area shoul behave le a Gaussan. Exercse.3: In menson 00 what percentage of the surface area of a sphere s wthn stance /0 of the equatoral zone. Here fx the North an South Poles an as for two planes perpencular to the axs from the North to South Pole, what percentage of the stance to the pole must the planes be to contan 95% of the surface area? Exercse.4: Project the vertces of a unt hypercube wth a vertex at the orgn onto a lne from 0,0,,0 to,,,. Argue that the ``ensty of the number of projecte ponts (per unt stance) vares roughly as a Gaussan wth varance O () wth the m-pont as center. Exercse.5: Place two unt spheres n -mensons, one at (-,0,0,,0 ) an the other at (,0,0,,0). Gve an upper boun on the probablty that a ranom lne through the orgn wll ntersect the spheres?

23 /8/00. Hgh mensons 3 Exercse.6: Gven two unt varance Gaussans n hgh mensonal space whose centers are one unt apart, by how much o ther annul at raus of wth ò 0, small, overlap? Exercse.7: How many ponts o you nee n hgh mensonal space to easly etect clusters? How o you formulate ths problem an evelop an answer? Exercse.8: Place n ponts at ranom on a -mensonal unt sphere. Assume s large. Pc a ranom vector an let t efne two parallel hyper planes. How far apart can the hyper planes be move an stll have no ponts between them? Exercse.9: Generate a 000 ponts at vertces of a 000 mensonal cube. Select two ponts an j at ranom an fn a path from to j by the followng algorthm. Start at an go to the closest pont havng the property that st, j an st, j are both less than st,. Then contnue the path by the same algorthm from j to. What s the expecte length of the path? Exercse.0: If one has 000 ponts n two mensons that are wthn a unt box, one mght vew them as steppng stones n a pon. Select two ponts an j at ranom an fn a path from to j by the followng algorthm. Start at an go to closest pont havng the property that st(,j) an st(,j) are both less than st(,j). Then contnue the path by the same algorthm from to j. A computer smulaton suggests that on average the path wll be of length 34. If one repeats the experment for 000 ponts n 000 mensons on average the path wll consst of only 5 hops. Exercse.: Conser a set of vectors n a hgh mensonal space. Assume the vectors have been normalze so that ther lengths are one. Thus, the ponts le on a unt sphere. Select two ponts at ranom. Assume one s at the North pole. As the menson of the space ncreases the probablty that the other pont s close to the equator goes to one. To see ths note that the rato of the area of a cone wth axs at the North pole of fxe angle say 45 to the area of a hemsphere goes to zero as the menson ncreases. Exercse.: What s the expecte stance between two ponts selecte at ranom nse a -mensonal unt cube? For two ponts selecte at ranom nse a -mensonal unt hyper sphere? What s cosne of the angle between them? Exercse.3: Conser two ranom 0- vectors n hgh menson. What s the angle between them? What s probablty that angle s less than 45? Exercse.4: Project the surface area of a -mensonal unt hyper sphere onto one of ts axes. What s the strbuton of projecte area on the axs?

24 /8/00. Hgh mensons 4 Exercse.5: Where o ponts generate by a heavy tale, hgh mensonal strbuton le? For the Gaussan strbuton ponts le n an annulus because the probablty strbuton falls off qucly as the volume ncreases. Exercse.6: Gven a cluster of ponts n -mensons how many ponts o we nee to average to accurately etermne a center? Exercse.7: Show that the maxmum of f() t s attane at t 0. ( ) Hnt: Maxmze the logarthm of f() t by fferentatng. Exercse.8: Gven the probablty strbuton ten ponts estmate u an. 3 e x5 3 generate ten ponts. From the Exercse.9: Calculate V() by a recursve proceure V()=cV(-). Develop exercse. References