2 Statistical Principles

Size: px

Start display at page:

Download "2 Statistical Principles"

Brianna Washington
5 years ago
Views:

1 2 Statistical Priciples This lecture will serve two mai goals. First we will itroduce ad the tool of radom hash fuctios. Secod we itroduce a radomized/probabilistic view of algorithms ad data aalysis. This will iclude revisitig ideas about cocetratio of measure, ad also probably approximate correct (PAC error bouds. We will study these properties through three pheomeo of radom processes: Birthday Paradox: To measure the expected collisio of radom evets. A radom group of 23 people has about a 50% chace of havig a pair with the same birthday. Coupo Collectors: radom variable. To measure the expectatio for seeig all possible outcomes of a discrete Cosider a lottery where o each trial you receive oe of possible coupos at radom. It takes i expectatio about ( l trials to collect them all. Cetral Limit Theorem: To boud the leakage from the expected value of repeated radom trials. Cosider drawig ages (i 0 to 125 from a ukow distributio of people, how far is the sample average from the true average age. From aother perspective, these describe the effects of radom variatio. The first describes collisio evets, the secod coverig evets, ad the third cocetratio evets Idepedet ad Idetically Distributed What is data? A simplified view is a set X = {x 1, x 2,..., x }, ad i particular where each x i D(θ; that is each item is iid from some (probably ukow distributio D(θ. The abbreviatio iid meas idepedetly ad idetically distributed. I this cotext this meas each x i is (idetical draw from the same distributio (i this case D(θ ad (idepedet that the value of x i is ot affected by ay other value x j (for i j. This will ot hold for all data sets. But its ot ucommo for some aspect of the data to fit this model, or at least be close eough to this model. Ad for may problems without such a assumptio, its impossible to try to recover the true uderlyig pheomeo that is the data miig objective. This is represeted by the parameter(s θ of the distributio D(θ. I other settigs, we will create a (radomized algorithm that geerates data which precisely follows this patter (we ca do this, sice we desig the algorithm!. Such a algorithm desig is beeficial sice we kow a lot about iid data, this is the focus of most of the the classical part of the field of statistics! I both cases, this is idicative of the very powerful big data paradigm: Create a complex/accurate estimate by combiig may small ad simple observatios. Model. For the settig of this lecture there is a commo model of radom elemets draw from a discrete uiverse. The uiverse has possible objects; we represet this as [] ad let i [] represet oe elemet (idexed by i from {1, 2, 3,..., } i this uiverse. The objects may be IP addresses, days of the year, words i a dictioary, but for otatioal ad implemetatio simplicity, we ca always have each elemet (IP address, day, word map to a distict iteger i where 0 < i. The we study the properties of drawig k items uiformly at radom from [] with replacemet. 1

2 2.1 (Radom Hash Fuctios A key tool, perhaps the tool, i probabilistic algorithms is a hash fuctio. There are two mai types of hash fuctios. The secod (which we defie i Lecture 5 is locality-preservig. But ofte oe wats a hash fuctios which is ucorrelated, like i a hash table data structure. A radom hash fuctio h H maps from oe set A to aother B (usually B = [m] so that coditioed o the radom choice h H, the locatio h(x B for ay x A is equally likely. Ad for ay two (or more strogly, but harder to achieve, all x x A the h(x ad h(x are idepedet (coditioed o the radom choice of h H. For a fixed h H, the map h(x is determiistic, so o matter how may times we call h o argumet x, it always returs the same result. The full idepedece requiremet is ofte hard to achieve i theory, but i practice this differece is usually safe to igore. For the purposes of this class, most built-i hash fuctios should work sufficietly well. Assume we wat to costruct a hash fuctio h : Σ k [m] where m is a power of two (so we ca represet it as log 2 m bits ad Σ k is a strig of k of characters (lets say a strig of umbers Σ = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}. So x Σ k ca also be iterpreted as a k-digit umber. 1. SHA-1: See This secure hash fuctio takes i a strig of bits so Σ = {0, 1} ad k is the umber of bits (but may frot eds exists that ca let Σ be all ASCII characters, ad determiistically outputs a set of 160 bits, so m = If oe desired a smaller value of m, oe ca simply use ay cosistet subset of the bits. To create a family of hash fuctios H parameterized by some seed a (the salt, oe ca simply cocateate a to all iputs, to obtai a hash h a H. So h a (x = SHA-1(cocat(a, x. Fuctioality for this (or other similar hash fuctios should be built i or available as stadard packages for may programmig laguages. These built i fuctios ofte ca take i strigs ad other various iputs which are the iterally coverted to bits. 2. Multiplicative Hashig: h a (x = m frac(x a a is a real umber (it should be large with biary represetatio a good mix of 0s ad 1s, frac( takes the fractioal part of a umber, e.g. frac( = 0.234, ad takes the iteger part of a umber, roudig dow so = 15. Ca sometimes be more efficietly implemeted as (xa/2 q mod m where q is essetially replacig the frac( operatio ad determiig the umber of bits precisio. If you wat somethig simple ad fast to implemet yourself, this is a fie choice. I this case the radomess is the choice of a. Oce that is chose (radomly, the the fuctio h a ( is determiistic. 3. Modular Hashig: h(x = x mod m (This is ot recommeded, but is a commo first approach. It is listed here to advise agaist it. This roughly evely distributes a umber x to [m], but umbers that are both the same mod m always hash to the same locatio. Oe ca alleviate this problem by usig a radom large prime m < m. This will leave bi {m + 1, m + 1,..., m 1, m} always empty, but has less regularity. 2.2 Birthday Paradox First, let us cosider the famous situatio of birthdays. Lets make formal the settig. Cosider a room of k people, chose at radom from the populatio, ad assume each perso is equally likely to have ay birthday (excludig February 29th, so there are = 365 possible birthdays.

3 The probability that ay two (i.e. k = 2 people (ALICE ad BOB have the same birthday is 1/ = 1/ The birthday of ALICE could be aythig, but oce it is kow by ALICE, the BOB has probability 1/365 of matchig it. To measure that at least oe pair of people have the same birthday, it is easier to measure the probability that o pair is the same. For k = 2 the aswer is 1 1/ ad for = 365 that is about For a geeral umber k (say k = 23 there are ( k 2 = k (k 1/2 (read as k choose 2 pairs. For k = 23, the ( ( 23 2 = 253. Note that k 2 = Θ(k 2. We eed for each of these evets that the birthdays do ot match. Assumig idepedece we have (1 1/ (k 2 or = Ad the probability there is a match is thus 1 mius this umber just over 50%. 1 (1 1/ (k 2 or = 0.532, What are the problems with this? First, the birthdays may ot be idepedetly distributed. More people are bor i sprig. There may be o-egligible occurrece of twis. Sometimes this is really a problem, but ofte it is egligible. Other times this aalysis will describe a algorithm we create, ad we ca cotrol idepedece. Secod, what happes whe k = +1, the we should always have some pair with the same birthday. But for k = 366 ad = 365 the 1 (1 1/ (k 2 = 1 (364/365 ( = 1 ( = < 1. Yes, it is very small, but it is less tha 1, ad hece must be wrog. Really, the probability should be ( 1 1 ( 2 ( 3... = 1 k 1 ( i.

4 Iductively, i the first roud (the secod perso (i = 2 there is a ( 1/ chace of havig o collisio. If this is true, we ca go to the ext roud, where there are the two distict items see, ad so the third perso has ( 2/ chace of havig a distict birthday. I geeral, iductively, after the ith roud, there is a ( i/ chace of o collisio (if there were o collisios already, sice there are i distict evets already witessed. As a simple saity check, i the ( + 1th term ( /( = 0/ = 0; thus the probability of some collisio of birthdays is 1 0 = 1. Take away message. There are collisios i radom data! More precisely, if you have equi-probability radom evets, the expect after about k = 2 evets to get a collisio. Note , a bit more tha 23. Note that (1 + α t t e α for large eough t. So settig k = 2 the 1 (1 1/ (k 2 1 (1 1/ 1 e 1.63 This is ot exactly 1/2, ad we used a buch of tricks, but it shows roughly what happes. This is fairly accurate, but has oticeable variace. Note for = 365 ad k = 18 the ad whe k = 28 the 1 (1 1/ (k 2 = 1 (364/ (1 1/ (k 2 = 1 (364/ This meas that if you keep addig (radom people to the room, the first matchig of birthdays happes 30% (= 64% 34% of the time betwee the 18th ad 28th perso. Whe k = 50 people are i the room, the 1 (1 1/ (k 2 = 1 (364/ , ad so oly about 3.5% percet of the time are there o pair with the same birthday. 2.3 Coupo Collectors Lets ow formalize the famous coupo lottery. There are types of coupos, ad we participate i a series of idepedet trials, ad o each trial we have equal probability (1/ of gettig each coupo. We wat to collect all toys available i a McDoald s Happy Meal. How may trials (k should we expect to partake i before we collect all coupos? Let r i be the expected umber of trials we eed to take before receivig exactly i distict coupos. Let r 0 = 0, ad set t i = r i r i 1 to measure the expected umber of trials betwee gettig i 1 distict coupos ad i distict coupos. Clearly, r 1 = t 1 = 1, ad it has o variace. Our first trials always yields a ew coupo. The the expected umber of trials to get all coupos is T = t i. To measure t i we will defie p i as the probability that we get a ew coupo after already havig i 1 distict coupos. Thus t i = 1/p i. Ad p i = ( i + 1/. We are ow ready for some algebra: T = t i = i + 1 = 1 i.

5 Now we just eed to boud the quatity (1/i. This is kow at the th Harmoic Number H. It is kow that H = γ + l + o(1/ where l( is the atural log (that is l e = 1 ad γ is the Euler-Masheroi costat. Thus we eed, i expectatio, trials to obtai all distict coupos. k = T = H (γ + l Extesios. What if some coupos are more likely tha others. McDoalds offers three toys: Alvi, Simo, ad Theodore, ad for every 10 toys, there are 6 Alvis, 3 Simos, ad 1 Theodore. How may trials do we expect before we collect them all? I this case, there are = 3 probabilities {p 1 = 6/10, p 2 = 3/10, p 3 = 1/10} so that p i = 1. The aalysis ad tight bouds here is a bit more complicated, but the key isight is that it is domiated by the smallest probability evet. Let p = mi i p i. The we eed about radom trials to obtai all coupos. k ( 1 p (γ + l These properties ca be geeralized to a family of evets from a cotiuous domai. Here there ca be evets with arbitrarily small probability of occurrig, ad so the umber of trials we eed to get all evets becomes arbitrarily large (followig the above o-uiform aalysis. So typically we set some probability ε [0, 1]. (Typically we cosider ε as somethig like {0.01, 0.001} so 1/ε somethig like {100, 1000}. Now we wat to cosider ay set of evets with combied probability greater tha ε. (We ca t cosider all such subsets, but we ca restrict to all, say, cotiguous sets itervals if the evets have a atural orderig. The we eed k 1 ε log 1 ε radom trials to have at least oe radom trial i ay subset with probability at least ε. Such a set is called a ε-et.

6 Take away message. It takes about l trials to get all items at radom from a set of size, ot. That is we eed a extra about l factor to probabilistically guaratee we hit all evets. Whe probabilities are ot equal, it is the smallest probability item that domiates everythig! To hit all (icely shaped regios of size ε we eed about (1/ε log(1/ε samples, eve if they ca be covered by 1/ε items. 2.4 Probably Approximately Correct (PAC The above discussios are startig to hit at a probability approximate correct (PAC boud. There are radom evets that produce a estimate ˆX (which is a radom variable that is probably close to its expected value µ = E[ ˆX]. The tricky part of about describig this is that we do t wat to say ˆX is always close to µ. Because it is a radom process, sometimes thigs go (horribly! wrog. So we wat to provide a term δ (0, 1 which is a probability of failure. Sometimes (with say δ = 0.01 probability of failure, or 1% of the time X is ot close to µ. Moreover, we also do t wat to talk about the probability that X = µ. Sice this may occur with very small probability (we eed exactly the right umber of happy meas, maybe 1000 to collect all 100 or so coupos. Rather we would like to allow some error threshold, usig a parameter ε > 0. So we eed two parameters to describe the error δ (probably ad ε (approximate. Ultimately most bouds look of the form: Pr[ ˆX µ ε] δ. That is, the probability that ˆX (which is some radom variable, ofte a sum of iid radom variables is further tha ε to its expected value µ, is at most δ. Or equivaletly (perhaps more optimistically Pr[ ˆX µ < ε] > 1 δ. I like to thik of ε as the error tolerace ad δ as the probability of failure i.e., that we exceed the error tolerace. However, ofte these bouds will allow us to write the required sample size i terms of ε ad δ. This allows us to trade these two terms off for ay fixed kow ; we ca allow less error tolerace if we are willig to allow more probability of failure, ad vice-versa. 2.5 Cetral Limit Theorem ad Cheroff-Hoeffdig Boud Whe dealig with moder big data sets, a very commo theme is reducig the set through a radom process. These geerally work by makig may simple estimates of the full data set, ad the judgig them as a whole. Perhaps magically, these may simple estimates ca provide a very accurate ad small represetatio of the large data set. I statistics, this pheomeo is usually referred to as the Cetral Limit Theorem. The key tool i showig how may of these simple estimates are eeded for a fixed accuracy trade-off, ad formulatig the Cetral Limit Theorem as a PAC boud, is the Cheroff-Hoeffdig iequality. We cosider a specific form of the Cheroff-Hoeffdig iequality. It is ot the strogest form of the boud, but is for may applicatios asymptotically equivalet, ad it also fairly straight-forward to use. It is more similar to the form of Azuma s iequality which deals with Martigales that have more complicated depedece structure.

7 Theorem Cosider a set of r idepedet idetically distributed (iid radom variables {X 1,..., X r } such that X i for each i [r]. Let A = 1 r r X i (average of X i s. The for ay α > 0 ( rα 2 Pr[ A E[A] > α] 2 exp 2 2. This requires us to kow two thigs about the radom variables. First, we should kow the expected value of E[X i ] for all X i ; sice they are iid, its the same for all of them. Secod, we eed to kow a upper boud o the rage of these radom variables. For istace, if its someoes age, we ca probably boud = 125 sice o oe is older tha 125 years old, ad all ages are positive so 0 > = 125. This tells us how far we ca reasoably expect our sample estimate A of the average age i a set to be from the true average. Thus, we may ot eve eed to kow E[A], the true expected age, but just wat to kow how likely it is that we are very far from it. For istace, if we fid A = 30, ad r = 1000, the the probability that E[A] [20, 40] (so α = 10 is at most 2 exp( /( So (at most about 8% of the time we would have a estimate ot withi 10 years. (Note, this souds very pessimistic, sice it could cotai the high variace case where p fractio of the the perso is 125 ad (1 p fractio they are 125. Its importat to try to get a tight estimate o the actual rage of deviatio. More geeral form. This is a slightly more geeral form that I ofte fid very useful as well. Theorem Cosider a set of r idepedet radom variables {X 1,..., X r }. If we kow a i X i b i, the let i = b i a i. Let M = r X i. The for ay α > 0 ( 2α 2 Pr[ M E[M] > α] 2 exp r. 2 i For the same problem above, sice we ca use ow each i = 125 reveals a probability of at most that the estimate is ot withi 10 years of the true value. Hashig. These bouds map back to the hashig problem, to uderstad the case whe all of the buckets have roughly the same umber of items that fall i them. This requires roughly k = (1/ε 2 log(1/δ to be hashed so that each bucket is withi εk/ of all other buckets, with probability at least 1 δ.

CS 330 Discussion - Probability

CS 330 Discussion - Probability CS 330 Discussio - Probability March 24 2017 1 Fudametals of Probability 11 Radom Variables ad Evets A radom variable X is oe whose value is o-determiistic For example, suppose we flip a coi ad set X =