Concentration function and other stuff

Concentration function and other stuff Sabrina Sixta Tuesday, June 16, 2014 Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 1 / 13

Table of Contents Outline 1 Chernoff Bound and Law of Large Numbers 2 Concentration Phenomenon 3 Blowing-up Lemma 4 Concentration function 5 Calculate value of concentration function for Hamming cube 6 Improvement on Chernoff Bounds 7 Hoeffding Inequality Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 2 / 13

Chernoff Bound and Law of Large Numbers Warming up to Chernoff Bound Let Ω = {A, B} be a two-element partition of a finite metric space X = (X, d) such that (1) d(a, B) = 1 and (2) there exists a bijection i : A B where d(a, i(a)) = 1 for all a A. Let f : X R be a 1-Lipschitz function. Then the function E[f Ω] : X R is 1-Lipschitz too. Or more relevent to us, E[f Ω](a) E[f Ω](b) 1. Let f : {0, 1} n R be a 1-Lipschitz function relative to the Hamming distance, d and let f 1, f 2,..., f n be a corresponding martingale with respect to the standard refining partition. Then for every i 1, 2,... n, d i 1 2. Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 3 / 13

Chernoff Bound and Law of Large Numbers Chernoff Bound and Law of Large Numbers Chernoff Bound Let f : {0, 1} n R be a 1-Lipschitz function relative to the normalized Hamming distance, d. Then for every ɛ > 0, µ # {x {0, 1} n : f (x) E[f ] ɛ} 2e 2nɛ2. Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 4 / 13

Chernoff Bound and Law of Large Numbers Chernoff Bound and Law of Large Numbers Chernoff Bound Let f : {0, 1} n R be a 1-Lipschitz function relative to the normalized Hamming distance, d. Then for every ɛ > 0, µ # {x {0, 1} n : f (x) E[f ] ɛ} 2e 2nɛ2. Law of Large Numbers Applying the normalized weight function, w, to the Chernoff bound, we get for ɛ > 0, µ # {x {0, 1} n : w 1 2 ɛ} 2e 2nɛ2. Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 4 / 13

Chernoff Bound and Law of Large Numbers Chernoff Bound and Law of Large Numbers cont. Figure: Actual concentration values for the Hamming Cube, {0, 1} 10 vs. the Chernoff Bound. µ # {x {0, 1} n : w 1 2 n 2k 2n } = 2 2 n k i=0 ( ) n. i Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 5 / 13

Concentration Phenomenon Concentration Phenomenon Azuma s inequality and the Chernoff bounds are both part of a larger group of inequalities that manifest the phenomenon of concentration of measure on structures of high dimension. That is, on a typical mathematical structure of high dimension (like the Hamming cube, {0, 1} n ) every nice function (1-Lipschitz functions) concentrates near one value. Bounds like Chernoff s are called concentration inequalities. Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 6 / 13

Concentration Phenomenon Concentration Phenomenon Cont. Figure: Orthogonal projection of 1000 random points on a unit Euclidean d-cube (clockwise, d=3,10,100,1000). Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 7 / 13

Blowing-up Lemma Warming up to Blowing-up Lemma ɛ-neighbourhood Let A X, where (X, d) is a metric space and let ɛ > 0. The ɛ-neighbourhood of A in X is, A ɛ = {x X : there exists a A, d(x, a) < ɛ}. Extended Chernoff inequality Let f : {0, 1} n R be a 1-Lipschitz function relative to the normalized Hamming distance. Then for every ɛ > 0 µ # {x {0, 1} n : f (x) E[f ] ɛ} e 2nɛ2 and µ # {x {0, 1} n : f (x) E[f ] ɛ} e 2nɛ2. Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 8 / 13

Blowing-up Lemma Blowing-up Lemma Blowing-up lemma Let A {0, 1} n be a set of n-bit strings such that µ # (A) 1 2, then for ɛ > 0, the ɛ-neighbourhood for A, with respect to the normalized Hamming distance, satisfies the following inequality, µ # (A ɛ ) 1 e ɛ2 n/2. Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 9 / 13

Concentration function Concentration function The concentration function of the Hamming cube {0, 1} n, denoted α(n, ɛ) or α({0, 1} n, ɛ) on R +, is { 1 α(ɛ) = 2 if ɛ = 0 1 min{µ # (B ɛ ) : B {0, 1} n, µ # (B) 1 2 } if ɛ > 0. (1) Some properties of the concentration function on the Hamming cube, {0, 1} n : non-increasing, left-continuous, not right-continuous. The concentration function of the Hamming cube {0, 1} n satisfies α({0, 1} n, ɛ) e ɛ2 n/2. Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 10 / 13

Calculate value of concentration function for Hamming cube Calculate value of concentration function for Hamming cube Hamming ball A set B {0, 1} n is called a Hamming ball if for σ {0, 1} n and k = 1, 2,..., n, B k (σ) B B k+1 (σ). Harper isopetrimetric theorem For every A in the metric space ({0, 1} n, d) and each k = 1, 2,..., n there is a Hamming ball B with #(B) = #(A) and #(B k ) #(A k ). For a Hamming cube of odd dimension, {0, 1} 2m+1 equipped with the normalized distance d, the value of the concentration function for ɛ > 0 is where n = 2m + 1. m+nɛ α(ɛ) = 1 2 n k=0 ( n k ) n m nɛ = 2 n k=0 ( ) n k Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 11 / 13

Improvement on Chernoff Bounds Improvement on Chernoff Bounds The concentration function of the Hamming cube, {0, 1} n, satisfies the following inequality α({0, 1} n, ɛ) e 2ne2. Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 12 / 13

Hoeffding Inequality Hoeffding Inequality Rademach s Cube: { 1, 1} n = {σ 1, σ 2,..., σ n : σ i { 1, 1} for i 1, 0,..., n} Hoeffding Inequality This inequality is another modification of Azuma s inequality for the Rademach s Cube. Let a = (a 1, a 2,..., a n ) R n then for c > 0, µ # {η { 1, 1} n : n i=1 ( η i a i > c} < 2exp c 2 a 2 2 ). Sabrina Sixta () Concentration function and other stuff Tuesday, June 16, 2014 13 / 13