Understanding the asymptotic behaviour of empirical Bayes methods

Transcription

1 Undersanding he asympoic behaviour of empirical Bayes mehods Boond Szabo, Aad van der Vaar and Harry van Zanen EURANDOM,

2 Conens 2/20 Moivaion Nonparameric Bayesian saisics Signal in Whie noise model Empirical Bayes mehod Addiional quesions

3 Applicaion 3/20 Applicaion of empirical Bayes mehod in nonpararmeric saisics: Genomic daa analysis Conexual region classificaion High dimensional classificaions Qualiy assurance Alhough i is widely used in pracice, i does no have full heoreical underpinning.

4 Bayes mehod 4/20 Mahemaical Saisics: We observe daa X and we have a family of possible disribuions {P θ : θ Θ}. The goal is o make inference abou θ based on he observed daa X. Frequenis mehod: Belief: The daa has disribuion P θ0. (There is a rue, unknown θ 0.) Mehod: We esimae θ 0 wih he help of he daa. Bayesian mehod: Belief: The parameer θ is sampled from a prior disribuion Π. Nex he daa X is generaed from he disribuion P θ. Mehod: We use he condiional disribuion of θ given X, he poserior disribuion, o make inference. Mixed mehod: We sudy he Bayesian procedures from a frequenis poin of view.

5 Parameric vs Nonparameric Bayes 5/20 Parameric Model: Bernsein-Von Mises Thoerem: Under some regulariy assumpions he poserior disribuion will ypically concenrae around he rue parameer wih rae 1/ n, independenly of he choice of he prior disribuion. Nonparameric Model: Problem: The poserior disribuion asympoically depends on he choice of he prior. Even seemingly good choice of he prior disribuion can lead o a sub-opimal poserior: Won concenrae a all. Concenrae, bu wih a slower rae han he opimal.

6 Signal in Whie noise model 6/20 Goal: Esimae he unknown funcion f 0 L 2 [0, 1]: dx = f 0 ()d + 1 n dw, where X is he observed funcion. f0 primiive of f0 observed funcion f0() F0() X()

7 Gaussian sequence Model I. 7/20 Les ake an orhonormal basis of L 2 [0, 1], for example e k = 2sin(kπ). The Fourier expansion of f 0 : f 0 = θ 0,k e k k=1 The observed noisy version of he Fourier coefficiens: X k = 1 0 e k ()dx The noise in he Fourier coefficiens: Z k = 1 0 e k ()dw, are i.i.d. N(0, 1) random variables.

8 Gaussian sequence Model II. 8/20 Model: Gaussian sequence model: X k = θ 0,k + 1 n Z k, k = 1, 2,... where Z k s are iid sandard Gaussian disribued and θ 0 = (θ 0,1, θ 0,2,...) l 2. Goal: Make inference abou θ 0 wih he help of he observed sequence X = (X 1, X 2,...). Assumpion: Le θ 0 Θ β = {θ : θ 2 k C 2 k 1 2β } for a (unknown) β, C > 0.

9 Bayes approach 9/20 Bayes mehod: Endow he unknown parameer θ 0 wih a prior disribuion: 1) Π τ = N (0, τ 2 k 1 2α ), k=1 wih some fixed α > 0. Poserior disribuion pus mos of is mass on a θ 0 cenered ball, where he radius depends on τ. Goal: Choose τ such ha he poserior disribuion conracs around θ 0 in he opimal minimax rae n β 1+2β.

10 Oracle 10/20 Opimal choice τ n n α β 1+2β. By choosing τ wih smaller or bigger order we ge a sub-opimal conracion rae. Problem: τ n depends on β oracle scaling. Soluion: Use adapive Bayesian mehod and le he daa decide τ: Full Bayes mehod. Empirical Bayes mehod.

11 Full Bayes mehod 11/20 Mehod: We endow he hyperparameer τ wih a hyper prior λ. So we ge a wo level prior disribuion, he hierarchical prior disribuion: Π = λ(τ)π τ dτ. T Then we use his hierarchical prior in he Bayes mehod. Advanage: We don mix he Frequenis and Bayesian echnique Disadvanage: Difficul o compue in many cases

12 Empirical Bayes mehod 12/20 Mehod: We esimae τ in a frequenis way: Le ˆτ maximize τ p θ (x)dπ τ (θ). Θ Then we use he empirical Bayes prior Πˆτ (by subsiuing ˆτ ino he prior) in he Bayes mehod. Advanage: Compuaional convenien widely used in pracice Disadvanages: No heoreical underpinning Mixing of frequenis and Bayesian echniques.

13 Figures I. 13/20 ha{f} log likelihood funcion ha{f}() log likelihood(au) au Figure: EB poserior mean and he log-likelihood funcion. (θ 2 0,k = 2cos(k)k 3 )

14 Figures II. 14/20 ha{f} f() Figure: EB poserior mean, poserior mean wih ˆτ n /300 and 300ˆτ n.

15 Figures III. 15/20 ha{f} ha{f} ha{f} ha{f}() ha{f}() ha{f}() ha{f} ha{f} ha{f} ha{f}() ha{f}() ha{f}() Figure: EB poserior mean for β = 1 and α = 1/5, 1/3, 1/2, 1, 2 and 5.

16 Resuls 16/20 We can show ha here exiss a cu-off poin a β = α + 1/2. For α + 1/2 > β: he empirical Bayes mehod mimics he oracle, so he empirical Bayes poserior concenraes around θ 0 in he opimal minimax rae. For α + 1/2 = β: he EB mehod achieves he opimal rae up o a log n facor. For α + 1/2 < β: he EB mehod gives a significanly worse resul han he oracle.

17 Quesions 17/20 Is here a prior, which mimics he oracle for all β > 0? Does he hierarchical Bayes mehod give he same resul? Wha happens for analyic funcions, where θ 2 0,k Ce kq, for q > 0? Does he empirical Bayes mehod work in inverse problems and linear funcional esimaions? (wih Barek Knapik) Oher, more complex nonparameric saisical models.

18 Universal prior - Aemp 1 18/20 Smooher priors give beer asympoic resuls choose prior wih exponenial decay: Π τ = N(0, τ 2 e kp ), k=1 p>0. There exiss an opimal oracle scaling rae. BUT! The EB mehod will perform sub-opimally.

19 Universal prior - Aemp 2 19/20 We choose he prior: Π τ = N(0, e τ 2 k p ), k=1 p>0. There exiss an opimal oracle scaling rae. AND! The EB mehod will perform opimally up o a log n facor.

20 Main messages 20/20 Empirical Bayes mehod can yield adapive, rae-opimal procedures. Too rough priors yield sub-opimal rae. Smooher priors perform ypically beer, bu be we have o be careful. The resuls can be applied in hierarchical Bayes mehod. Many aspecs are sill unclear.