Section 26. Characteristic Functions. Po-Ning Chen, Professor. Institute of Communications Engineering. National Chiao Tung University

Transcription

1 Section 26 Characteristic Functions Po-Ning Chen, Professor Institute of Communications Engineering National Chiao Tung University Hsin Chu, Taiwan 3, R.O.C.

2 Characteristic function 26- Definition (characteristic function) The characteristic function of a random variable X is defined for real t by: ϕ(t) e itx df X (x) cos(tx)df X (x)+i sin(tx)df X (x). The characteristic function ϕ(t) M(it), where M(t) is the moment generating function of random variable X. The characteristic function is the (inverse) Fourier transform of distribution function. The characteristic function always exist, because distribution function is always integrable. It is named the characteristic function since it completely characterizes the distribution.

3 Fundamental properties regarding characteristic functions26-2 Fundamental properties of characteristic functions The characteristic function of sum of two independent random variables is the product of individual characteristic functions. The characteristic function uniquely determines distribution function. From the pointwise convergence of characteristic function follows the indistribution convergence of random variables.

4 Moments and characteristic function 26-3 ϕ(). ϕ(t) for all t R. ϕ(t) isuniformly continuous. Definition (uniform continuity) A function f(t) isuniformly continuous in A, if for any δ>, there exists h> such that sup f(x) f(y) <δ. {(x,y) A 2 : x y h}

5 Moments and characteristic function 26-4 Proof: ϕ(t + h) ϕ(t) ( e i(t+h)x df X (x) (e ihx )e itx df X (x) (e ihx ) e itx dfx (x) (e ihx ) dfx (x), e itx df X (x) (cos(hx) ) 2 +sin 2 (hx)df X (x) 2 2cos(hx)dFX (x) 2 sin(hx/2) df X (x) ) 2E [ sin(hx/2) ] where the bound is independent of t..

6 Taylor expansion for characteristic functions 26-5 Theorem (Taylor s formula with remainder) Assume that f( ) hasa continuous derivative of order n + in some open interval containing a. Then n f (k) (a) f(x) (x a) k + x (x t) n f (n+) (t)dt. k! n! k By the Taylor s formula with remainder about x, n e ix (e ix ) (k) x x k + x (x t) n (e it ) (n+) dt k! n! k n i k k! xk + x (x t) ( n i n+ e it) dt n! k n (ix) k x + in+ (x t) n e it dt. k! n! k a

7 Taylor expansion for characteristic functions 26-6 Integration by part for the below expression yields that: x (x t) n e it (x t)n x x dt e it (x t) n ( + ) ie it dt n n which implies that: x This summarizes to: e ix xn n + i n x x (x t) n e it dt (x t) n dt + i n (x t) n (e it )dt i n n (ix) k k k! i n+ x n! i n (n )! x (x t) n e it dt x x (x t) n e it dt, (x t) n e it dt, (x t) n (e it )dt

8 Taylor expansion for characteristic functions 26-7 Therefore, for x, eix n (ix) k k! k i n+ x (x t) n e it dt n! i n x (x t) n (e it )dt (n )! x (x t) n e it dt n! (n )! x n! 2 (n )! x n+ (n +)! 2x n n! x (x t) n dt x (x t) n (e it ) dt (x t) n dt, (since e it e it +2)

9 Taylor expansion for characteristic functions 26-8 and for x<, n (ix) k eix k! k i n+ x (x t) n e it dt n! i n x (x t) n (e it )dt (n )! n! x (n )! n! x 2 (n )! ( x) n+ (n +)! 2( x) n n! (x t) n e it dt x (t x) n dt x (x t) n (e it ) dt (t x) n dt, (since e it e it +2)

10 Taylor expansion for characteristic functions 26-9 So to speak, eix n k (ix) k k! min { } x n+ (n +)!, 2 x n. n! Lemma If X has a moment of order n, then n ϕ(t) (it) k [ { }] E[X k tx n+ ] k! E min (n +)!, 2 tx n n! k { t n+ E[ X n+ ] (if the moment of order (n + ) also exists) min, 2 t n E[ X n } ]. (n +)! n! Proof: n ϕ(t) k (it) k E[X k ] k! R n e itx (it) k df X (x) x k df X (x) k! k R n (itx) k eitx k! df X(x). R k

11 Taylor expansion for characteristic functions 26- Concern: When does the characteristic function have Taylor expansion? Corollary If for any t, then Theorem If then t n E[ X n ] lim n n! ϕ(t) E[e t X ] ϕ(t), (it) k E[X k ]. k! k k t k k! E[ X k ] <, (it) k E[X k ]. k! k

12 Taylor expansion for characteristic functions 26- Corollary If for some t, then ϕ(t ) k (it ) k E[X k ], k!. for every t t t, ϕ(t) k (it) k E[X k ]. k! 2. ϕ (k) () i k E[X k ] for every k,, 2,... Property 2, as a direct consequence of Property, is seemingly an analogous property to the moment generating function. However, moments can be determined by the characteristic function in a much weaker sense! (Notably, unlike the moment generation function which may have a big jump at the origin, the characteristic function is always uniformly continuous.)

13 Taylor expansion for characteristic functions 26-2 Lemma If E[ X k ] <, then Proof: ϕ (k) () i k E[X k ].. k : [ ϕ(t + h) ϕ(t) E[iXe itx ] h E h ei(t+h)x itx] h eitx ixe [ ] E e itx (eihx ihx) h [ e itx E e ihx ] ihx h [ min{(/2) hx 2 ], 2 hx } E h E [ min{(/2)h X 2, 2 X } ]. By bounded convergence theorem, min{(/2)h X 2, 2 X } 2 X, where X is integrable, and lim h min{(/2)h x 2, 2 x } almost everywhere jointly imply lim h E [ min{(/2)h X 2, 2 X } ].

14 Taylor expansion for characteristic functions 26-3 Theorem (bounded convergence theorem) If (i) f n g almost everywhere, (ii) g is integrable, and (iii) f n f almost everywhere, then f is integrable and f n dµ fdµ. Hence, we obtain: and ϕ (t) lim h ϕ(t + h) ϕ(t) h ϕ () E[iX]. E[(iX)e itx ],

15 Taylor expansion for characteristic functions k 2: ϕ (t + h) ϕ (t) h E[(iX) 2 e itx ] E[(iX)e i(t+h)x ] E[(iX)e itx ] E[(iX) 2 e itx ] h [ ] E (ix)e itx eihx ihx h [ E X e itx e ihx ] ihx h E [ min{(/2)h X 3, 2 X 2 } ], which again gives by bounded convergence theorem that ϕ (2) (t) E[(iX) 2 e itx ], and ϕ (2) () E[(iX) 2 ]. 3. Induction: The lemma can be proved by repeating the above process by induction (proving that the lemma holds for k n + under the premise that the lemma is valid for k n).

16 Taylor expansion for characteristic functions 26-5 The above lemma does not imply that the derivatives of ϕ(t) alwaysexist.on the contrary, ϕ(t) may be non-differentiable even if it is uniformly continuous. The above lemma actually tells us that the more moments X has, the more derivatives ϕ has. Brainstorming: Can this property be applied to Fourier Transform pair? Certainly, if the function is integrable. Brainstorming: In concept, the tail (rate-of-decaying) behavior of the distribution determines the smoothness (order of differentiability) of ϕ.

17 Riemann-Lebesgue theorem 26-6 Theorem 26. (Riemann-Lebesgue theorem) If X has a density, then ϕ X (t) t. Every Lebesgue integrable function can be approximated by Riemann integrable functions of two kinds. Theorem 7. Suppose that R f(x) dx < (which indicates its Lebesgue integrability) and ε>fixed.. There exists a step function g(x) k i x ii (ai,b i ](x) (which indicates its Riemann integrability), where each a i and b i are finite real numbers, such that f(x) g(x) dx < ε. R 2. There exists a continuous integrable h with bounded support (namely, which indicates its Riemann integrability) such that f(x) h(x) dx < ε. R Notably, a bounded measurable function is Riemann integrable on [a, b] if it is continuous in [a, b].

18 Riemann-Lebesgue theorem 26-7 Proof: From Theorem 7., there is a step function g(x) k j x ji (aj,b j ](x)such that f(x) g(x) dx < ε, R where f(x) is the density of X, and satisfies R f(x)dx <. Therefore, f(x)e itx dx g(x)e itx dx (f(x) g(x)) e itx dx R R R f(x) g(x) dx < ε. The proof is completed by noting that ε is arbitrary and k g(x)e itx dx x j I (aj,b j ](x) e itx dx R R j j a j R k bj x j e itx dx k j x j ( e itb j e ita j it ) t.

19 Independent and Inversion 26-8 The characteristic function of sum of independent random variables is the product of individual characteristic functions. The characteristic function of X is the complex conjugate of the characteristic function of X.

20 Inversion and uniqueness theorem 26-9 How to prove that The characteristic function uniquely determines the probability distribution. This is the task of Uniqueness Theorem. Please again think of the counterpart theorem for Fourier transformation. Also, remember that the cdf of a random variable is sufficient to define its statistic properties.

21 Inversion and uniqueness theorem 26-2 Example 8.4 Prove S(T ) Proof: T sin(x) x dx T T sin(x) x [ ] sin(x) e ux du dx [ T ] e ux sin(x)dx du π 2. dx T (By Fubini s theorem) [ e ut (u sin(t )+cos(t)) ] du +u 2 +u 2du e ut +u2(usin(t )+cos(t))du tan (u) π 2 e ut +u 2 cos(t φ u)du, where cos(φ u )/ +u 2 e ut +u 2 cos(t φ u)du.

22 Inversion and uniqueness theorem 26-2 Finally, e ut cos(t φ u)du +u 2 T e ut +u 2 cos(t φ u) du e ut du, because T. +u 2 for u Theorem (Fubini s theorem) Suppose X and Y can be expressed as X i U i and Y i V i for some {U i } i and {V i} i with µ(u i) < and ν(v i ) <. Then, if either ( ) f(x, y) ν(dy) µ(dx) < or X ( Y X Y ( ) f(x, y) µ(dx) ν(dy) <, Y X ) ( ) f(x, y)ν(dy) µ(dx) f(x, y)µ(dx) ν(dy). Y X

23 Inversion and uniqueness theorem Checking: T sin(x)e ux du T sin(x) dx T<. x

24 Inversion and uniqueness theorem Indeed, S(T ) S(π).

25 Inversion and uniqueness theorem Observation T T e itx it dt 2sgn(x)S(T x ), where sgn (x), if x>;, if x ;, if x<. Proof: T T T e itx it dt e itx T it dt + e itx it dt T e itx T e itx dt + it it dt T e itx e itx dt it T sin(tx) 2 dt t T sin (t x ) 2sgn(x) dt, t T x sin (t ) 2sgn(x) dt 2sgn(x)S(T x ). t

26 Inversion and uniqueness theorem Theorem 26.2 (Uniqueness theorem) For any a and b with Pr[X a] Pr[X b] anda<b, Pr[a <X b] lim T 2π T T e ita e itb ϕ X (t)dt, it where ϕ X (t) is the characteristic function of random variable X. Proof: Let I T denote the quantity inside the limit, namely, I T 2π 2π 2π π T T T T e ita e itb ϕ X (t)dt it e ita e itb ( ) e itx df X (x) dt it ( T e it(x a) e it(x b) ) dt df X (x) T it [ ] sgn(x a)s(t x a ) sgn(x b)s(t x b ) dfx (x). The absolute value of the above integrand, namely [ sgn(x a)s(t x a ) sgn(x b)s(t x b ) ], is bounded above by 2S(π), which is integrable with

27 Inversion and uniqueness theorem respect to measure df X (x). Hence, by bounded convergence theorem, lim T I T π π [ ] lim sgn(x a)s(t x a ) sgn(x b)s(t x b ) dfx (x) T ψ a,b (x)df X (x),

28 Inversion and uniqueness theorem where ψ a,b (x) Consequently, lim [sgn(x a)s(t x a ) sgn(x b)s(t x b )] T lim T lim T I T S(T x b ) S(T x a ), for x<a; S(T x b ), for x a; S(T x a )+S(T x b ), for a<x<b; S(T x a ), for x b; S(T x a ) S(T x b ), for x>b, for x<a; π, for x a; 2 π, for a<x<b; π, for x b; 2, for x>b. π ψ a,b (x)df X (x) 2 Pr[X a]+pr[a<x<b]+ Pr[X b]. 2

29 Inversion and uniqueness theorem The proof is completed by noting that Pr[X a] Pr[X b]. Theorem 26.2 (Uniqueness theorem) For any a and b with a<b, 2 Pr[X a]+pr[a<x<b]+ Pr[X b] lim 2 T 2π T where ϕ X (t) is the characteristic function of random variable X. T e ita e itb ϕ X (t)dt, it Remarks: We can tell more from the proof (than from the statement of the theorem). For example, Theorem If ϕ X(t) dt <, thennopointmassexists. I.e., Pr[X a] for every a R. Proof:

30 Inversion and uniqueness theorem { e ix min! x, }! 2 x { e ix ( + ix) min 2! x 2, }! 2 x ( eix +ix + ) { 2 (ix)2 min 3! x 3, } 2! 2 x 2. 2 Pr[X a]+pr[a<x<b]+ e ita e itb Pr[X b] lim ϕ X (t)dt 2 T 2π T it T lim sup e ita e itb T 2π T it ϕ X(t) dt e it(b a) 2π t(b a) b a ϕ X(t) dt b a ϕ X (t) dt. 2π T Thus, when taking a b, we immediately obtain Pr[X a].

31 Inversion and uniqueness theorem 26-3 Another example of We can tell more from the proof. is: Theorem If ϕ X(t) dt <, then the random variable X has density f, and its density (can be made to) satisfies:. f X (x) ϕ X (t)e itx dt; 2π 2. f X (x) F X (x) for every x R; 3. f X (x) is uniformly continuous for x R. Proof: First, from slide 26-28, we know that random variable X has no point mass. Then, by Theorem 26.2, we have that for every x (as Pr[X x] and Pr[X x + h] ) F X (x + h) F X (x) h lim T 2π e itx e it(x+h) I [ T,T] ϕ X (t)dt. ith As e itx e it(x+h) I [ T,T] ϕ X (t) ith e ith th ϕ X(t) ϕ X (t), which is integrable with respect to Lebesgue measure, dominated convergence

32 Inversion and uniqueness theorem 26-3 theorem implies that F X (x + h) F X (x) h 2π e itx e it(x+h) ϕ X (t)dt. ith Using dominated convergence theorem again with respect to h yields: F X (x + h) F X (x) lim h h 2π 2π 2π We can similarly show that: F X (x) F X (x h) lim h h Accordingly, F X (x) 2π e itx e it(x+h) lim ϕ X (t)dt h ith ϕ X (t) ( e it(x+h) e itx ) lim dt it h h ϕ X (t)e itx dt. 2π ϕ X (t)e itx dt. ϕ X (t)e itx dt.

33 Inversion and uniqueness theorem Claim: F X (x) is uniformly continuous in x R. Proof: F X(x + h) F X(x) 2π 2π 2π ϕ X(t) dt 2π ϕ X (t)e it(x+h) dt ϕ X (t)e itx (e iht )dt ϕ X (t) e itx e iht dt where f U (t) ϕ X (t) ( ϕ X (t) dt π where the bound is independent of x. ϕ X (t)e itx dt e iht fu (t)dt / ϕ X (t) dt ) E[ sin(hu/2) ].

34 Inversion and uniqueness theorem The proof is then completed by quoting the following theorem. In other words, since F satisfies that b a F (x)dx F (b) F (a) for every a, b R, it is surely a density of X. Theorem (See Section 7) If F is a function with continuous derivative F, then b F (x)dx F (b) F (a). a Note that the density is not unique. Any function g satisfying b for every a, b Ris a density of X. a g(x)dx F (b) F (a)

35 Inversion and uniqueness theorem Summary of properties of cdf F F is differentiable almost everywhere. Theorem (H. L. Royden, Real Analysis, 3rd edition, pp., 988) Let g be an increasing real-valued function on the interval [a, b]. Then, g is differentiable almost everywhere. The derivative g is measurable, and b a g (x)dx g(b) g(a). By this theorem, together with the fact that the number of intervals that F is increasing is countable, cdf F is almost everywhere differentiable. F is not necessarily differentiable on every x. F may not have density.

36 Inversion and uniqueness theorem If F has density f, thenf f almost everywhere. If F has continuous density f, thenf f everywhere. By definition of density, h x+h x f(x)dx F (x + h) F (x). h This can be used to show the above two statements. If F is continuous, then F (can be made to) be the density of F. Remarks The tail behavior of the characteristic function determines the smoothness of f X. The tail behavior of the density (i.e., f X ) determines the smoothness of ϕ X.

37 Characteristic functions of some known distributions Standard normal: First, f X (x) 2π e x2 /2 for <x<. ϕ X (t) e t2 /2. ϕ X (t) dt Hence, f X (x) can be recovered from 2π continuous. e t2 /2 dt 2π<. ϕ X(t)e itx dt, and is also uniformly As ϕ X (t) decays exponentially fast at t, f X (x) has all orders of derivative and is very, very smooth in its shape.

38 Characteristic functions of some known distributions Furthermore, t n E[ X n ] n! t n2n/2 Γ((n +)/2) π n! t 2j 2 j (j )! (2j )! for n 2j odd; 2π t 2j 2 j Γ((2j +)/2) (2j)! for n 2j even π { max, } (2 t 2 ) j for n 2j oddandn 2j even t j! { max, }( ) 2e t 2 j since j! > (j/e) j t j n Stirling s approximation ( n 2nπ e ) n <n! < 2nπ ( n e ) n ( + ). 2n Gamma function Γ(x +)x Γ(x) forx>andγ(/2) π.

39 Characteristic functions of some known distributions implies that ϕ X (t) (it) k E[X k ] k! (it) k ( 2 (k 2)/2 ( + ( ) k ) )Γ((k +)/2) k! π ( ) j ( 2 j ) Γ((2j +)/2) t 2j (2j)! π ( ) j ( ) t2 e t2 /2. j! 2 k k j j

40 Characteristic functions of some known distributions Uniform: First, f X (x) for <x<. ϕ X (t) eit. it sin(t/2) sin(s) ϕ X (t) dt dt 4 ds. t/2 s Hence, f X (x) cannot be recovered from 2π ϕ X(t)e itx dt, and is not uniformly continuous as it has jumps at x andx. In addition, lim n implies that t n E[ X n ] n! lim n t n (/(n +)) n! (it) k ϕ X (t) E[X k ] k! k (it) k ( ) k! k + k (it) k+ ( it (k +)! k lim n ) eit. it t n (n +)!

41 Characteristic functions of some known distributions 26-4 Exponential: First, f X (x) e x for <x<. ϕ X (t) it. it dt dt 2 +t 2 ϕ X(t)e itx dt, and is not uni- Hence, f X (x) cannot be recovered from 2π formly continuous as it has a jump at x. In addition, for t <, t n E[ X n ] lim n n! ( E[e tx ] e t x e x dx implies that for t <, (it) k ϕ X (t) E[X k ] k! k lim n t n n! n! k dt. +t 2 lim n t n e ( t )x dx ) t < (it) k (k!) k! k (it) k it.

42 Characteristic functions of some known distributions 26-4 f X (x) Double exponential (Bilateral exponential): 2 e x for <x<. ϕ X (t) +t 2. First, +t 2 dt π<. Hence, f X (x) can be recovered from 2π ϕ X(t)e itx dt, and also is uniformly continuous. But it is not differentiable at x. (Check tail of ϕ(t)!) In addition, for t <, t n E[ X n ] lim n n! ( E[e tx ] implies that for t <, (it) k ϕ X (t) E[X k ] k! k e t x 2 e x dx t n (n!) lim n n! (it) k k k! lim n t n e ( t )x dx t < ( ) 2 ( + ( )k )k! ) (it) 2j +t 2. j

43 Characteristic functions of some known distributions Cauchy: First, f X (x) π( + x 2 ) ϕ X (t) e t. for <x<. e t dt 2 <. Hence, f X (x) can be recovered from 2π ϕ X(t)e itx dt, and is uniformly continuous and differentiable (for ϕ X (t) decays very fast at t and hence f X (x) isverysmooth). In addition, t n E[ X n ] lim n n! {, for t > ; indeterminate, for t, (due to the indeterminate of ) and ϕ X (t) exhibits no Taylor-expansion expression.

44 Characteristic functions of some known distributions Triangular: Notably, f X (x) x for <x<. ( ) cos(t) ϕ X (t) 2. Hence, f X (x) can be recovered from 2π t 2 2 cos(t) t 2 dt 2π<. ϕ X(t)e itx dt, and also is uniformly continuous. But it is not differentiable at x,,. (Check tail of ϕ(t)!) In addition, t n E[ X n ] lim n n! implies that (it) k ϕ X (t) k! k lim n t n (2/[(n +2)(n +)]) n! E[X k ] k lim n (it) k ( ) +( ) k 2 k! (k +)(k +2) 2 t n (n +2)! j ( ) j (t 2 ) j (2j +2)! 2 cos(t) t 2.

45 Characteristic functions of some known distributions No name: Notably, f X (x) ( ) cos(x) for <x<. π x 2 ϕ X (t) ( t )I (,) (t). ( t )I(,) (t) dt <. Hence, f X (x) can be recovered from 2π ϕ X(t)e itx dt, and also is uniformly continuous and differentiable (for ϕ X (t) decays very, very fast at t,and hence f X (x) has all orders of derivatives and is very, very smooth). In addition, E[ X n ] for n 3; hence, t n E[ X n ] lim n n! {, for t > ; indeterminate, for t, and ϕ X (t) exhibits no Taylor-expansion expression.

46 Continuity theorem Theorem 26.3 (Continuity theorem) Proof: X n X if, and only if, ϕ Xn (t) ϕ X (t) for every t R.. X n X implies ϕ Xn (t) n ϕ X (t). Theorem 25.8 (A rephrased version) The following two conditions are equivalent. F n F ; lim f(x)df n (x) f(x)df (x) for every bounded, continuous n R R real function f. From Theorem 25.8, X n X implies that cos(tx)df Xn (x) lim n R R cos(tx)df X (x) and lim sin(tx)df Xn (x) sin(tx)df X (x). n R R These two jointly give that ϕ Xn (t) n ϕ X (t).

47 Continuity theorem ϕ Xn (t) n ϕ X (t) implies X n X. We will first show that ϕ Xn (t) n ϕ X (t) implies {X n } n (or equivalently, {F n } n }) is tight. Then, by Helly s Theorem, there exists a subsequence {X nk } k such that X nk Y,whereY is a random variable with legitimate cdf F Y ( ). Theorem 25.9 (Helly s theorem) For every sequence {F n } n of distribution functions, there exists a subsequence {F nk } k and a non-decreasing, right-continuous function F (not necessarily a cdf) such that lim F n k k (x) F (x) for every continuous points of F. Theorem 25. (rephrased version) Tightness of {F nk } k is a necessary and sufficient condition for the limit F ( ) in Helly s theorem to be a cdf. Definition (tightness) A sequence of cdf s is said to be tight if for any ε>, there exist x and y such that F n (x) <εand F n (y) > ε for all sufficiently large n.

48 Continuity theorem By Fubini s theorem, u u u ( ϕ Xn (t)) dt u u u u ( ) ( e itx )df n (x) dt ( u ) ( e itx )dt df n (x) ( u sin(ux) ) 2 df n (x) ux (This shows that u u u ( ϕ X n (t)) dt is a non-negative real number.) ( 2 sin(ux) ) df n (x) (since sin(ux) ) [ x 2/u] ux ux ( 2 ) df n (x) [ x 2/u] ux df n (x) [ x 2/u] [ Pr X n 2 ]. u

49 Continuity theorem Checking for Fubini s theorem: u u e itx df n (x)dx u u 2dx 4u<. With the above inequality, we next prove that the sequence of probability measures {F Xn } n is tight. Fix an ε>. Since ϕ X (t) is (uniformly) continuous and ϕ X (), there exists u (small enough) such that u u u ( ϕ X (t))dt < ε 2. Continuity at t means that for ε/4 >, there exists u > such that ϕ X (t) <ε/4for u t u. By bounded convergence theorem, u u u ( ϕ Xn (t))dt n u u u ( ϕ X (t))dt.

50 Continuity theorem Hence, there exists N such that for n>n, u u u ( ϕ Xn (t))dt < ε, which by the previously obtained inequality confirms that [ Pr X n > 2 ] <εfor all n N, u namely, {F Xn } n is tight. Now suppose that X nk Y for some Y. Then, by the first part of the proof, we obtain that for every t, However, we are given that for every t, ϕ Xnk (t) k ϕ Y (t). ϕ Xnk (t) k ϕ X (t). These two limits should coincide in every t, i.e., ϕ X (t) ϕ Y (t). We can then infer by the Uniqueness Theorem that X nk X.

51 Continuity theorem 26-5 Theorem 26.2 (Uniqueness theorem) For any a and b with Pr[X a] Pr[X b] anda<b, Pr[a <X b] lim T 2π T T e ita e itb ϕ X (t)dt, it where ϕ X (t) is the characteristic function of random variable X. In fact, we know by ϕ Xn (t) n ϕ X (t) and the Uniqueness Theorem that for every subsequence {n j } j satisfying that {X nj } j converges in distribution to some limiting random variable, this limiting random variable should be X. Finally, we will prove X n X by contradiction. Suppose X n does not converge in distribution to X. Then, there exists x with Pr[X x] suchthat Pr[X n x] Pr[X x], which implies the existence of subsequence {n j } j such that lim inf Pr[Xnj x] Pr[X x] >. j

52 Continuity theorem 26-5 However, for this {X nj } j, we can find a further subsequence such that X nj X. Thiscontradictsto k lim inf Pr[Xnj x] Pr[X x] >. j We therefore obtain the desired contradiction.

53 Continuity theorem The next two corollaries can be proved similarly using the above proof. But their statements could be more useful in applications. Corollary Suppose a sequence of characteristic functions {ϕ n (t)} n has limits in every t, namely lim t ϕ n (t) exists for every t. Define g(t) lim t ϕ n (t). Then if g(t) is continuous at t, then there exists a probability measure µ such that µ n µ, and µ has characteristic function g where µ n is the probability measure corresponding to characteristic function ϕ n ( ). Proof: Notably, in the second part of the proof (i.e, the proof of the tightness of {µ n } n ), the only condition required is that g(t) (i.e., ϕ X(t) in the previous proof) is continuous at t andg(). So the same proof can be used to show this corollary.

54 Continuity theorem In the proof, the continuity of g was used to establish tightness. Hence, if {F X n} n is assumed tight in the first place, then the corollary apparently holds. Corollary 2 Suppose a sequence of characteristic functions {ϕ n (t)} n has limits in every t, namely lim t ϕ n (t) exists for every t. Define g(t) lim ϕ Xn (t). t Then if {µ n } n is tight, then there exists a probability measure µ such that µ n µ, and µ has characteristic function g where µ n is the probability measure corresponding to characteristic function ϕ n ( ). Observation If {X n } n is uniformly bounded, then {F X n } n is tight.

55 Continuity theorem Example 26.2 Let X n be uniformly distributed over ( n, n). Then ϕ Xn (t) n n e itx sin(nt) dx. 2n nt Hence, the limit of ϕ Xn (t) exists, and is equal to: { sin(nt), if t ; g(t) lim ϕ Xn (t) lim n n nt lim lim cos(nt), if t n t In this example, {F Xn } n is not tight, and as expected, g( ) is not continuous at t. {, if t ;, if t Notably, the probability measures of X n s converges vaguely to all-zero measure. Actually, g(t) cannot be obtained by integrating e itx with respect to the true allzero-measure for x R, but a (bizarre) measure that is zero everywhere and integrates to over the entire range. Hence, I put double quotation mark here to indicate it is not the usual all-zero measure.

56 Applying Fourier series to probabilities Theorem Suppose the support of the distribution of random variable X is contained in [, 2π]. Then b Pr[a <X b] lim σ m (t)dt, m a if Pr[X a] Pr[X b] and<a<b<2π, where σ m (t) 2π sin 2 [m(x t)/2] 2πm sin 2 [(x t)/2] df X(x). Proof: By Fubini s theorem, b b ( 2π sin 2 ) [m(x t)/2] σ m (t)dt a a 2πm sin 2 [(x t)/2] df X(x) dt 2π ( b sin 2 ) [m(x t)/2] 2πm a sin 2 [(x t)/2] dt df X (x) 2π ( b sin 2 ) [m(t x)/2] 2πm a sin 2 [(t x)/2] dt df X (x) 2π ( b x sin 2 ) (ms/2) 2πm sin 2 (s/2) ds df X (x). a x

57 Applying Fourier series to probabilities By dominated convergence theorem, for any δ (, 2π), sin 2 (ms/2) lim m 2πm sin 2 (s/2) ds 2π sin 2 (s/2) δ< s <2π δ< s <2π ( lim m It can be shown (cf. the next slide) that for integer m, π sin 2 (ms/2) 2πm π sin 2 (s/2) ds ( π m ) l e isk ds m 2πm π 2πm l k l l l m l Hence, m 2πm l 2πm s δ k l 2sin(kπ) k m sin 2 (ms/2) m sin 2 ds, (s/2) which implies that (note: 2π <a x<b x<2π) lim m ( b x 2πm a x sin 2 ) (ms/2) sin 2 (s/2) ds l k l, if a x>;, if a x< <b x;, if b x< sin 2 ) (ms/2) ds. m l k l sin(kπ) kπ π π. e isk ds {, if x<aor x>b;, if a<x<b.

58 Applying Fourier series to probabilities m l l k l e isk m l e is e is e is e isl ( e is(2l+) ) e is ( m l e ism ( e ism ) 2 e is ( e is ) 2 e isl m l e is(l+) ) ( e ism e eis ( e ism ) ) is e ( is e is(m ) ( e ism ) eis ( e ism ) e is e is sin2 (ms/2) sin 2 (s/2). ) As Pr[X a] Pr[X b], we obtain from dominated convergence theorem that b 2π ( b x sin 2 ) (ms/2) lim σ m (t)dt lim m a m 2πm a x sin 2 (s/2) ds df X (x) Pr[a<X<b].

59 Applying Fourier series to probabilities Discussions: Following from the previous theorem, the Fourier coefficients are defined by: c m 2π e imx df X (x) form, ±, ±2,. {c m } m,±,±2,... canbeviewedasthevalues of the characteristic function for integer arguments. In other words, c m ϕ X (m), where Notably, σ m (t) 2πm 2πm 2π 2π m ϕ X (t) 2π e itx df X (x). sin 2 [m(x t)/2] sin 2 [(x t)/2] df X(x) l l k l e i(x t)k df X (x) m 2πm l l c k e itk. k l

60 Applying Fourier series to probabilities Hence, the distribution of a random variable with support contained in [, 2π] can be uniquely determined by the samples of ϕ X (t), namely {ϕ X (m)} m,±,±2,.... Trivial Extension: The distribution of a random variable with bounded support can be uniquely determined by the samples of its characteristic function (if with a properly selected sampling period). A drawback of using samples to determine the distribution with bounded support is that the probability masses at two end points are undetermined. In other words, using the above theorem as an example, we can have two distinct distributions with equal positive Pr[X ]+Pr[X 2π] >, but their samples are identical. Example Pr[X ]Pr[Y 2π]. Then ϕ X (t) E[e itx ]andϕ Y (t) E[e ity ]e i2πt. However, for every integer m, ϕ X (m) ϕ Y (m) e i2πm. Observation Distributions with the same Pr[X ]+Pr[X 2π] are indistinguishable using the approach of Fourier series.

61 Applying Fourier series to probabilities 26-6 Corollary Suppose the support of the distribution of random variable X is contained in [, 2π]. Then for <a<b<2π, 2 Pr[X a]+pr[a<x<b]+ Pr[X b] lim 2 m b a σ m (t)dt. Proof: Since sin 2 (ms/2)/sin 2 (s/2) is a bounded even function, Hence, 2πm 2πm π π sin 2 (ms/2) sin 2 (s/2) ds 2πm sin 2 (ms/2) sin 2 (s/2) ds π 2πm π sin 2 (ms/2) sin 2 (s/2) ds. sin 2 (ms/2) sin 2 (s/2) ds 2,

62 Applying Fourier series to probabilities 26-6 which implies that lim m ( b x 2πm a x Accordingly, the corollary holds. sin 2 ) (ms/2) sin 2 (s/2) ds, if a x>; 2, if a x ;, if a x< <b x;, if b x ; 2, if b x<, if x<aor x>b; 2, if x a or x b;, if a<x<b. Theorem Suppose {X n } n have supports in [, 2π], and suppose the mth Fourier coefficient c m (n) ofx n converges to c m.then X n X, where X is the distribution determined through {c m } m integer. The distribution of X is unique except possiblely in the way the mass on {, 2π} is split on the points of and 2π. Proof: Note that a sequence of random variables is tight, if their supports are uniformly bounded.

63 Uniformly distributed modulo Give a sequence of real numbers x,x 2,x 3,... Let the probability measure µ n put masses /n at points 2π (x k x k ). Let the probability measure µ be uniformly distributed over (, 2π]. Hence, and c m c m (n) 2π 2π e imx µ(dx) 2π e imx µ n (dx) n 2π n e i2πm(x k x k ), k e imx dx {, if m ;, if m. Therefore, µ n µ, if for m, n n k e i2πm(x k x k ) n. This is named Weyl s criterion.

64 Uniformly distributed modulo Now if x k kθ, whereθ is irrational, then e i2πmθ form,and n e i2πm(x k x k ) n e i2πm(kθ kθ ) n n k n k n k e i2πmkθ ei2πmθn ei2πmθ n e i2πmθ ei2πmθeiπmθn (e iπmθn e iπmθn ) n e iπmθ (e iπmθ e iπmθ ) iπmθ(n+) sin(πmθn) e n sin(πmθ) n.

65 Fourier Transform Question: Is (inverse) Fourier transform always continuous, just like the characteristic function? Answer: No. Think of the (inverse) Fourier transform of sin(x)/x., if t > ; sin(x) π x eitx dx, if t ; 2 π, if t <. If f(x) dx <, then the (inverse) Fourier transform is uniformly continuous. e i(t+h)x f(x)dx e itx f(x)dx (e ihx )e itx f(x)dx (e ihx ) e itx f(x) dx (e ihx ) f(x) dx ( ) f(x) dx (e ihx ) df (x), where df (x) f(x) dx/ ( f(x) dx).

66 Fourier Transform Question: Does (inverse) Fourier transform has Taylor expansion? Answer: This section learns us that if t n R. lim x n f(x) dx,or n n! 2. e t x f(x) dx <, R then Fourier transform has Taylor expansion at t. We also learn that even if Fourier transform does not equal its infinite Taylor sum, we can still express it in partial Taylor sum up to order n, if x n f(x) dx <. R

67 Fourier Transform Question: Does (inverse) Fourier transform has derivatives? Answer: If x n f(x) dx <, R then Fourier transform has kth derivatives at t forallk n. More can be established through the lessen from this section!