Approximating high-dimensional posteriors with nuisance parameters via integrated rotated Gaussian approximation (IRGA)

Transcription

1 Approximating high-dimensional posteriors with nuisance parameters via integrated rotated Gaussian approximation (IRGA) Willem van den Boom Department of Statistics and Applied Probability National University of Singapore Bayesian Computation for High-Dimensional Statistical Models Institute of Mathematical Sciences National University of Singapore September 10, 2018

2 Collaborators David B. Dunson Galen Reeves Department of Statistical Science Duke University 2

3 Outline IRGA with Bayesian variable selection as an example Approximation accuracy IRGA in a more general setup 3

4 IRGA with Bayesian variable selection as an example 4

5 Standard linear model n x p matrix y = X + n observations p unknown parameters Gaussian errors N(0, 2 I) The entries of β are conditionally independent given hyperparameters θ: py p( ) = p( j ) j=1 5

6 Example: Bayesian Variable Selection Entries of β conditionally independent given hyperparameters θ py p( ) = p( j ) j=1 Mixed discrete-continuous distribution for marginal prior probability equal to zero distribution of nonzero p( j ) =(1 ) ( j )+ N ( j 0, ), =(, ) 6

7 Inference Goal: compute posterior marginal distribution of first entry Z p( 1 y) = p( y)d p 2 e.g. for P(β 1 0 y) where p( y) = p( )p(y ) R p( )p(y )d Treat β p 2 as nuisance parameters Number of possible subsets with number of variables p {j β j 0} grows exponentially Posterior is a mixture of 2 p Gaussians Intractable for large p. Therefore, approximations are used for scalable inference. 7

8 Approximation methods Most popular approximation methods fit into two groups: Sampling based: MCMC, Gibbs sampling, Stochastic Search Variable Selection (George & McCulloch, 1993), or Bayesian Adaptive Sampling (Clyde et al., 2011). Exact solution asymptotically Hard to establish convergence due to exponential and discrete nature of the posterior Deterministic: Variational Bayes (Carbonetto & Stephens, 2012, Ormerod et al., 2017), Expectation propagation (Hernández-Lobato et al., 2015), or EM variable selection (Ročková & George, 2014). Often unclear what quality of approximation is achieved IRGA provides interpretable approximations to posterior marginals and can overcome some of these issues. 8

9 Overview of IRGA Goal: Marginal posterior, such as p(β 1 y) = π(β y) dβ p 2 Achieved by approximately integrating outβ p 2 Based on a data rotation Transparent approximation allows for theoretical analysis 9

10 Overview of IRGA 1. Rotate the data to isolate the parameter of interest Introduce an auxiliary variable which summarizes the influence of the nuisance parameters on β p 2 β 1 2. Use any means possible to compute/estimate the posterior mean and posterior variance of the auxiliary variable 3. Apply a Gaussian approximation to the auxiliary variable and solve the one-dimensional integration problem to obtain the posterior approximation for β 1 β 1 10

11 2 6 1: Rotate Apply rotation matrix 4 Q to the data which 5 zeros out all but the first entry in the first column of the data ỹ = Qy and X = QX Only the first observation depends on 1: px ỹ 1 = x 1,1 1 + x 1,j j + 1 j= x 1,1 x 1,2 x 1,p 0 x 2,2 x 2,p ỹ = x n,2 x n,p N(0, 2 I) ζ auxiliary variable captures influence of nuisance parameters 11

12 1: Rotate p unknown parameters n observations (the data) 1 y 1 2 y 2 3 y 3 4 y n p 12

13 1: Rotate y 1 2 y 2 3 y p y n 13

14 1: Rotate y 1 y 2 2 y y n p 14

15 1: Rotate ỹ 1 ỹ 2 2 ỹ ỹ n p 15

16 1: Rotate ζ ỹ 1 ỹ 2 ỹ ζ = p j=2 x 1,j β j auxiliary variable encapsulates influence of nuisance parameters 1 ỹ n p 16

17 2: Estimate / compute Compute the posterior mean and variance of the auxiliary variable: E[ζ ỹ n 2 ] Cov[ζ ỹn 2 ] Vector Approximate Message Passing (VAMP, Rangan et al., 2017) LASSO Variational Bayes [your favorite method] The quantities are independent of the target parameter β 1 17

18 3: Approximate Apply Gaussian approximation to auxiliary variable to compute posterior approximation prior distribution p(β 1 y) p(ỹ 1 ζ, β 1 ) p(β 1 ) p(ζ ỹ p 2 ) dζ Gaussian by assumption on noise replace with Gaussian using mean and variance from previous step Approximation p(ζ ỹ p ) can be accurate even if the prior 2 and posterior are highly non-gaussian: Most low-dimensional projections of high-dimensional distributions are close to Gaussian (projection pursuit, Diaconis & Freedman, 1984). p Additionally, CLT or Bernstein-von Mises: ζ = x 1,j β j j=2 18

19 Overview of IRGA 1. Rotate the data to isolate the parameter of interest Introduce an auxiliary variable ζ the nuisance parameters β p on β 2 1 which summarizes the influence of 2. Use any means possible to compute/estimate the posterior mean and posterior variance of p(ζ ỹ p ) 2 3. Apply a Gaussian approximation to p(ζ ỹ p ) 2 and solve the onedimensional integration problem to obtain the approximation for p(β 1 y) β 1 19

20 Approximation accuracy 20

21 21 Gaussian approximation accuracy Spike and slab σ 2 = 1 Theoretical Quantiles Sample Quantiles Spike and slab σ 2 = 0.5 Theoretical Quantiles Sample Quantiles Horseshoe σ 2 = 1 Theoretical Quantiles Sample Quantiles Horseshoe σ 2 = 0.5 Theoretical Quantiles Sample Quantiles Q-Q plots of from simulated data with X equal to 3,571 highly correlated SNPs from Friedman et al. (2010). p(ζ ỹ n 2 )

22 Quadratic Wasserstein distance W 2 {p 1 (x 1 ), p 2 (x 2 )} = inf E ( x 1 x 2 2 ) 1 2 where the infimum is over all joint distributions on such that x 1 p 1 ( ) and x 2 p 2 ( ). (x 1, x 2 ) 22

23 Gaussian approximation accuracy Theorem If p(β p is nearly isotropic and its distance from its mean 2 ỹp) 2 concentrates, then the Wasserstein distance W 2 {p(ζ ỹ n 2 ), p(ζ ỹn 2 ) } is small for most vectors ( x 1,j ) p. j=2 Isotropy is when independent draws from too correlated. p(β p 2 ỹp 2 ) are not Isotropic Non-isotropic 23

24 Kullback-Leibler divergence D {p 1 (x 1 ) p 2 (x 2 )} = p 1 (x) log p 1 (x) p 2 (x) dx 24

25 Posterior approximation accuracy Theorem E [ D {p(β 1 y) IRGA approximation Gaussian approximation p(β 1 y)} ỹ n 2] 1 2σ 2 W2 {p(ζ ỹ 2 n 2 ), p(ζ ỹn 2 ) } where y β N (Xβ, σ 2 I n ) with β p(β) 25

26 Posterior approximation accuracy Theorem E [ D {p(β 1 y) IRGA approximation Gaussian approximation p(β 1 y)} ỹ n 2] 1 2σ 2 W2 {p(ζ ỹ 2 n 2 ), p(ζ ỹn 2 ) } where y β N (Xβ, σ 2 I n ) with β p(β) 26

27 Application Error in posterior inclusion prob Difference from Gibbs estimate Predictor x Integrated rotated Gaussian approximation Expectation propagation (Hernández-Lobato et al., 2015) Variational Bayes (Carbonetto & Stephens, 2012) Diabetes data (Efron et al., 2004) 27

28 IRGA in a more general setup 28

29 Standard linear model n x p matrix y = X + p unknown parameters 29

30 <latexit sha1_base64="4vfe4dnx9/wwclgiw3s9xapw2dy=">aaacbxicbvdlssnafj3uv42vqktddbahiprebn0irufcvrapbekztg/aozmhmxohhg7c+ctuxcji1n9w5984abpq1goxezjnxmbu8wloplksb6owsli0vfjcndfwnza3sts7trklgkkdrjwsby9i4cyehmkkqzswqakpq8sbxmv+6wgezff4p0yxuahph8xnlcgtduv7i3xhth0pfmhh+lpyf6sba7fkplplvtwaam8toydllkpelx05vygmayskcijlx7zi5azekey5je0nkratoir96ggakgckm06ugondrfswhwldocit9fdgsgipr4gnjwoibnlwy8t/ve6i/hm3zwgckajp9ce/4vhfoise95gaqvhie0if03/fdeaeouohz+oq7nmt50nzpgpbvfv2tfy7zomooj10gcrirmeohm5qhtuqry/ogb2in+pjedhejy/pamhid3brhxifp43clha=</latexit> <latexit sha1_base64="4vfe4dnx9/wwclgiw3s9xapw2dy=">aaacbxicbvdlssnafj3uv42vqktddbahiprebn0irufcvrapbekztg/aozmhmxohhg7c+ctuxcji1n9w5984abpq1goxezjnxmbu8wloplksb6owsli0vfjcndfwnza3sts7trklgkkdrjwsby9i4cyehmkkqzswqakpq8sbxmv+6wgezff4p0yxuahph8xnlcgtduv7i3xhth0pfmhh+lpyf6sba7fkplplvtwaam8toydllkpelx05vygmayskcijlx7zi5azekey5je0nkratoir96ggakgckm06ugondrfswhwldocit9fdgsgipr4gnjwoibnlwy8t/ve6i/hm3zwgckajp9ce/4vhfoise95gaqvhie0if03/fdeaeouohz+oq7nmt50nzpgpbvfv2tfy7zomooj10gcrirmeohm5qhtuqry/ogb2in+pjedhejy/pamhid3brhxifp43clha=</latexit> <latexit sha1_base64="4vfe4dnx9/wwclgiw3s9xapw2dy=">aaacbxicbvdlssnafj3uv42vqktddbahiprebn0irufcvrapbekztg/aozmhmxohhg7c+ctuxcji1n9w5984abpq1goxezjnxmbu8wloplksb6owsli0vfjcndfwnza3sts7trklgkkdrjwsby9i4cyehmkkqzswqakpq8sbxmv+6wgezff4p0yxuahph8xnlcgtduv7i3xhth0pfmhh+lpyf6sba7fkplplvtwaam8toydllkpelx05vygmayskcijlx7zi5azekey5je0nkratoir96ggakgckm06ugondrfswhwldocit9fdgsgipr4gnjwoibnlwy8t/ve6i/hm3zwgckajp9ce/4vhfoise95gaqvhie0if03/fdeaeouohz+oq7nmt50nzpgpbvfv2tfy7zomooj10gcrirmeohm5qhtuqry/ogb2in+pjedhejy/pamhid3brhxifp43clha=</latexit> <latexit sha1_base64="4vfe4dnx9/wwclgiw3s9xapw2dy=">aaacbxicbvdlssnafj3uv42vqktddbahiprebn0irufcvrapbekztg/aozmhmxohhg7c+ctuxcji1n9w5984abpq1goxezjnxmbu8wloplksb6owsli0vfjcndfwnza3sts7trklgkkdrjwsby9i4cyehmkkqzswqakpq8sbxmv+6wgezff4p0yxuahph8xnlcgtduv7i3xhth0pfmhh+lpyf6sba7fkplplvtwaam8toydllkpelx05vygmayskcijlx7zi5azekey5je0nkratoir96ggakgckm06ugondrfswhwldocit9fdgsgipr4gnjwoibnlwy8t/ve6i/hm3zwgckajp9ce/4vhfoise95gaqvhie0if03/fdeaeouohz+oq7nmt50nzpgpbvfv2tfy7zomooj10gcrirmeohm5qhtuqry/ogb2in+pjedhejy/pamhid3brhxifp43clha=</latexit> <latexit sha1_base64="f5smxrusgpqnj32wfaywghhhei4=">aaacenicbvc7sgnbfl3rm66vqkxnybdeiuza+aaxkihlfncesjhmtibjknnzdwzwcevat7dxv2wsvcxsroz8gyepiizeuha451zupteiovpacx6sqemz2bn5zik9uls8sppdw79rusij9ujei1koskkccepppjktx5limoc0fltpenrpnkrfingtozgthrgpwimrra1vy+6ehye/xlovbolv9zyvvmyhveiui3wdjurgnhpytr/qjhchihfyar8/aecxlv326xfjqio04vipiuveuppiqrnhtgv7iaixjm3cpbudbtalq2k/uxdtg6aogpe0lttqs6mtkq6v6osbcfzuvonaj/xpqys6cvbnmygttquzlgokhjm0vqehopouan4xabpjzk2itlderjs32uyj7njksedt5q/z7qwbk5zcodkwcvuway7sqweuoageehiez3ifn+vjerhery+bdcoazmzan7k+fgg9np/t</latexit> <latexit sha1_base64="9hskttzqesa0izxf6o1+tbaq3dm=">aaacenicbvdlssnafj3uv42vqks3g0uqfyvx4wpeoiauqxhbagkztkft0mkkzkyeevipbvwc/8gncxuxbly5ef/gsdtfbb1w4xdoudx7rh8xkpvl/ri5qemz2bn8vlmwuls8ulhdu5zhldbxcmhcuforjixy4iiqgklfgqdaz6tqd08zvxphhkqhv1k9ihgbanpaohgpttuko2eh0a2q6vh+cpnejnxvncas3qbqvegopm1fq2t1c04cewikxx/muft4bvyahs+3gei4ifxhhqss21akvaqjrtejqengkkqid1gb1dxksc/2kn6mfg5ppglboddnfeyzoxmjcqtsbb52zjfkcs0j/9pqswrtewnluawix4nfrzhbntz7egxsqbbipq0qfltfcnehcysvfqopn2cpr54ezm7pogrf2mxycrhuhmyatbanblahyuacviadmlght+afvbopxrpxzrwprdljolmo/ptx+qutuafh</latexit> <latexit sha1_base64="9hskttzqesa0izxf6o1+tbaq3dm=">aaacenicbvdlssnafj3uv42vqks3g0uqfyvx4wpeoiauqxhbagkztkft0mkkzkyeevipbvwc/8gncxuxbly5ef/gsdtfbb1w4xdoudx7rh8xkpvl/ri5qemz2bn8vlmwuls8ulhdu5zhldbxcmhcuforjixy4iiqgklfgqdaz6tqd08zvxphhkqhv1k9ihgbanpaohgpttuko2eh0a2q6vh+cpnejnxvncas3qbqvegopm1fq2t1c04cewikxx/muft4bvyahs+3gei4ifxhhqss21akvaqjrtejqengkkqid1gb1dxksc/2kn6mfg5ppglboddnfeyzoxmjcqtsbb52zjfkcs0j/9pqswrtewnluawix4nfrzhbntz7egxsqbbipq0qfltfcnehcysvfqopn2cpr54ezm7pogrf2mxycrhuhmyatbanblahyuacviadmlght+afvbopxrpxzrwprdljolmo/ptx+qutuafh</latexit> <latexit sha1_base64="6bcbuzr6v+q9ehf2nba8s4thee4=">aaacenicbvbns8naen34wetx1aoxxskih5j48enufmrjfwmltsyb7bzdutne3ylqqv6df/+kfw8qxj1589+4axuorq8ghu/nmdmviaxxyns/1tz8wulscmglulq2vrfz2tq+01gikhnpjclvcihmgkvmagfbgrfijaweqwf9i9yvpzklesrvyrazpyrdytucejbsq3r4eya9keavcnkb7d6vhvcqafyqyq+iscs0l+2kpqsejc6ylneytvbp22thnamzbcqi1k3hjsfpiqjobcukxqjztgifdfntuenmyj8d/pthfao0csdspitgoto5kzjq60eymm78rj3t5ej/xjobzomfchknwcqdleokaptv84bwmytgqqwmivrxcyumpaiibrnj0ytgtl88s9yjymnfuxbk1fnxggw0i/bqaxlqmaqik1rdlqlocb2gn/rupvuv1of1owqds8yzo+gprk9fudoebg==</latexit> Linear model with nuisance parameter n x p matrix n x q matrix y = X + F (Z)+ p < n unknown parameters nuisance parameter F : R n q! R n 30

31 <latexit sha1_base64="dynxuojsdpvdu8vb72qkenvwvqa=">aaacuxiclzfnb9qweiadukbdvpzy5gj1bsqirjkquqneoqkpai9vov26yp2uhk+tdws7kt1bwln5kftgv8hj5tapopbkll49m+pxjnnscgtr9dsih6w8fpr4da335omz5y/6l9e/26iyji9yiqsztqnlumg+aggsj0vdquolp0+vvjtx85/cwfhom1iupfe01yitjijh0/4v962+ommltjsyyzjyofsh+c2xqnleyc6odmc1ktwd4nrya1kxbnnb8l5db5s/3m2rfvn6xnglbxw0lt0grurzidumtdid/nfl03/f7/shsr7ry9ofrmoofb5v4s4muketaf+azapwka6bswrtji5ksbw1ijjkdy9ulpeuxdgct7zvvhgbuhb1nx7jyqxnhffha27pzqphlbullfrmzjr7n9bav8umfwqfeyd0wqhxbnkoqysgajf/igfccazy4q1lrvi3yjanhjlwv93zs4jvjnzfjlahe8p4685g/3o3jvx0gm2gtrsjxbspdtejgiew7azjkav5+clmw3l4uuwng67mfbql0p4bktznfa==</latexit> <latexit sha1_base64="dynxuojsdpvdu8vb72qkenvwvqa=">aaacuxiclzfnb9qweiadukbdvpzy5gj1bsqirjkquqneoqkpai9vov26yp2uhk+tdws7kt1bwln5kftgv8hj5tapopbkll49m+pxjnnscgtr9dsih6w8fpr4da335omz5y/6l9e/26iyji9yiqsztqnlumg+aggsj0vdquolp0+vvjtx85/cwfhom1iupfe01yitjijh0/4v962+ommltjsyyzjyofsh+c2xqnleyc6odmc1ktwd4nrya1kxbnnb8l5db5s/3m2rfvn6xnglbxw0lt0grurzidumtdid/nfl03/f7/shsr7ry9ofrmoofb5v4s4muketaf+azapwka6bswrtji5ksbw1ijjkdy9ulpeuxdgct7zvvhgbuhb1nx7jyqxnhffha27pzqphlbullfrmzjr7n9bav8umfwqfeyd0wqhxbnkoqysgajf/igfccazy4q1lrvi3yjanhjlwv93zs4jvjnzfjlahe8p4685g/3o3jvx0gm2gtrsjxbspdtejgiew7azjkav5+clmw3l4uuwng67mfbql0p4bktznfa==</latexit> <latexit sha1_base64="dynxuojsdpvdu8vb72qkenvwvqa=">aaacuxiclzfnb9qweiadukbdvpzy5gj1bsqirjkquqneoqkpai9vov26yp2uhk+tdws7kt1bwln5kftgv8hj5tapopbkll49m+pxjnnscgtr9dsih6w8fpr4da335omz5y/6l9e/26iyji9yiqsztqnlumg+aggsj0vdquolp0+vvjtx85/cwfhom1iupfe01yitjijh0/4v962+ommltjsyyzjyofsh+c2xqnleyc6odmc1ktwd4nrya1kxbnnb8l5db5s/3m2rfvn6xnglbxw0lt0grurzidumtdid/nfl03/f7/shsr7ry9ofrmoofb5v4s4muketaf+azapwka6bswrtji5ksbw1ijjkdy9ulpeuxdgct7zvvhgbuhb1nx7jyqxnhffha27pzqphlbullfrmzjr7n9bav8umfwqfeyd0wqhxbnkoqysgajf/igfccazy4q1lrvi3yjanhjlwv93zs4jvjnzfjlahe8p4685g/3o3jvx0gm2gtrsjxbspdtejgiew7azjkav5+clmw3l4uuwng67mfbql0p4bktznfa==</latexit> <latexit sha1_base64="dynxuojsdpvdu8vb72qkenvwvqa=">aaacuxiclzfnb9qweiadukbdvpzy5gj1bsqirjkquqneoqkpai9vov26yp2uhk+tdws7kt1bwln5kftgv8hj5tapopbkll49m+pxjnnscgtr9dsih6w8fpr4da335omz5y/6l9e/26iyji9yiqsztqnlumg+aggsj0vdquolp0+vvjtx85/cwfhom1iupfe01yitjijh0/4v962+ommltjsyyzjyofsh+c2xqnleyc6odmc1ktwd4nrya1kxbnnb8l5db5s/3m2rfvn6xnglbxw0lt0grurzidumtdid/nfl03/f7/shsr7ry9ofrmoofb5v4s4muketaf+azapwka6bswrtji5ksbw1ijjkdy9ulpeuxdgct7zvvhgbuhb1nx7jyqxnhffha27pzqphlbullfrmzjr7n9bav8umfwqfeyd0wqhxbnkoqysgajf/igfccazy4q1lrvi3yjanhjlwv93zs4jvjnzfjlahe8p4685g/3o3jvx0gm2gtrsjxbspdtejgiew7azjkav5+clmw3l4uuwng67mfbql0p4bktznfa==</latexit> <latexit sha1_base64="vgixoijal4spmy24zpirblyelfk=">aaab63icbva9swnbej2lxzf+rs0voqycvbizubugjwucngkkr9jb7cvl9vao3tkhhimtbsxubp0p+r12/gb/hjupqhmfddzem2fmxpairtfxvqzc0vlk6lp+vbcxubw9u9zdu9nxqijzacxi1qiizojl5ifhwrqjyiqkbksh/euxx79nsvny3uigyx5eupkhnbi0ktd6yejaxzjtdiawf4k7i6xk4aj2/xg0qraln61otnoisascan10nqt9jcjkvlbhozvqlhdaj13wnfssigk/mxw7te+m0rhdwjmsae/u3xmzibqerihpjaj29lw3fv/zmimgf37gzziik3s6keyfjbe9/tzucmuoioehhcpubrvpjyhc0ertmcg48y8veu+sffl2ayamk5gidwdwdkfgwjlu4aaq4aefdk/waq+wtj6tn+t92pqzzjp78afwxw8+nji3</latexit> sha1_base64="xmnlatczj/0h8crjzuzzz+slrua=">aaab63icbvbnt8jaej36ififevtssew8kdalein68yijfrjoyhazwobtttmdmidhn3jxomarf8ib/8yfeldwjzo8vdetmxlrjouhz/t2vlbx1jc2s1vl7z3dvf3kwegdsxpnmecpthurygalubiqiimttcnlionnahgz9zupqi1i1t2nmgwt1lcifpyrlyloexlrvqpezzvbxsz+qapqongtfhv6kc8tvmqlm6btexmfy6zjcimtcic3mde+zh1sw6pygiycz46dukdw6blxqm0pcmfq74kxs4wzjzhttbgnzki3ff/z2jnfl+fyqcwnvhy+km6ls6k7/dztcy2c5mgsxrwwt7p8wdtjzpmp2xd8xzexsxbeu6r5d161fl2kuyjjoiez8oec6naldqiag4bneiu3rzkvzrvzmw9dcyqzi/gd5/mhvaioja==</latexit> <latexit sha1_base64="s1pfrennkgbghukbb0henxrw2t4=">aaab63icbzc7sgnbfibpxlumt6ilfonbsaq7nmoxtlfmwdwbzamzk9lkyozsmnnwceuewczcxdz38dns7hwuj5dce38y+pj/c5hztphkydb1v5zcyura+kzxs7s1vbo7v94/uddjphn3wsit3qqp4vio7qnayvup5jqojw+gw5tj3nzg2ohe3eeo5ufm+0peglg0lt8jodjuuejw3animnhzqnsopxrfafdvlj87vyrlmvfijdwm7bkpbjnvkjjk41inmzylbej7vg1r0zibij8ooyan1umrknh2ksrt93dhtmnjrnfok2oka7oytcz/snag0wwqc5vmybwbfrrlkmbcjputntccorxzoewloytha6opq3ufkj2ct7jymvjn1auq1/aqtwuyqqhhcajn4mef1oaw6uadawgp8awvjnkenffnbvzacoy9h/bhzvsppx2qug==</latexit> <latexit sha1_base64="86la2fyceuu8lzvatfuadn2atbs=">aaab63icbzc7sgnbfibpeo3xfrvuzdaivmhxru2cnpyjucaqlgf2mkmgzm4um2efsks0trfqsfud8hx2pomv4ersaoipax//fw5zzgktkqy67peztlyyurae28hvbm3v7bb29u9nngrgfrblwnddarguivsoupj6ojmnqslryf9mnnceudyivnc4shgq0a4shceowstvhhxpq1b0s+5ezbg8grtlr6pq9+pxqniqfdbbmusjrpbjakzdcxmmmqprmmmh+wzqeejzn3z5w6kietdbnhl2se6t0yadwnunkezc3x0zjywzrkgtjcj2zhw2nv/lgil2lonmqcrfrtj0o04qcczkvdlpc80zyoefyrswsxlwo5oytpfj2yn48ysvgn9euip5va9yvoapcnaij3aghlxagw6haj4wepael/dqkofzexpep6vlzqznap7i+fgbglysia==</latexit> <latexit sha1_base64="86la2fyceuu8lzvatfuadn2atbs=">aaab63icbzc7sgnbfibpeo3xfrvuzdaivmhxru2cnpyjucaqlgf2mkmgzm4um2efsks0trfqsfud8hx2pomv4ersaoipax//fw5zzgktkqy67peztlyyurae28hvbm3v7bb29u9nngrgfrblwnddarguivsoupj6ojmnqslryf9mnnceudyivnc4shgq0a4shceowstvhhxpq1b0s+5ezbg8grtlr6pq9+pxqniqfdbbmusjrpbjakzdcxmmmqprmmmh+wzqeejzn3z5w6kietdbnhl2se6t0yadwnunkezc3x0zjywzrkgtjcj2zhw2nv/lgil2lonmqcrfrtj0o04qcczkvdlpc80zyoefyrswsxlwo5oytpfj2yn48ysvgn9euip5va9yvoapcnaij3aghlxagw6haj4wepael/dqkofzexpep6vlzqznap7i+fgbglysia==</latexit> <latexit sha1_base64="jpanwujdoqhzfi77wyjqcntcajk=">aaab63icbvbnt8jaej3if+ix6tflizhxrfov6o3oxsmmvkigidtlchu222z3akiiv8glbzve/upe/dcu0iocl5nk5b2zzmylmikmed63u1pb39jckm9xdnb39g+qh0epjs01x4cnmtxtibmuqmfagis2m40siss2othtzg89otyivq80zjbm2ecjwhbgvgq6erlrvwte3zvdxsv+qwpqonmrfnx7kc8tvmqlm6bjexmfe6zjcintsjc3mde+ygpswkpygiaczi+dumdw6btxqm0pcufq74kjs4wzj5httbgnzbi3e//zojnfv+feqcwnvhyxkm6ls6k7+9ztc42c5ngsxrwwt7p8ydtjzpop2bd85zdxsxbrv677936tcvokuyytoivz8oesgnahtqiag4bneiu3rzkvzrvzswgtocxmmfyb8/kdl8iodq==</latexit> <latexit sha1_base64="s1pfrennkgbghukbb0henxrw2t4=">aaab63icbzc7sgnbfibpxlumt6ilfonbsaq7nmoxtlfmwdwbzamzk9lkyozsmnnwceuewczcxdz38dns7hwuj5dce38y+pj/c5hztphkydb1v5zcyura+kzxs7s1vbo7v94/uddjphn3wsit3qqp4vio7qnayvup5jqojw+gw5tj3nzg2ohe3eeo5ufm+0peglg0lt8jodjuuejw3animnhzqnsopxrfafdvlj87vyrlmvfijdwm7bkpbjnvkjjk41inmzylbej7vg1r0zibij8ooyan1umrknh2ksrt93dhtmnjrnfok2oka7oytcz/snag0wwqc5vmybwbfrrlkmbcjputntccorxzoewloytha6opq3ufkj2ct7jymvjn1auq1/aqtwuyqqhhcajn4mef1oaw6uadawgp8awvjnkenffnbvzacoy9h/bhzvsppx2qug==</latexit> <latexit sha1_base64="86la2fyceuu8lzvatfuadn2atbs=">aaab63icbzc7sgnbfibpeo3xfrvuzdaivmhxru2cnpyjucaqlgf2mkmgzm4um2efsks0trfqsfud8hx2pomv4ersaoipax//fw5zzgktkqy67peztlyyurae28hvbm3v7bb29u9nngrgfrblwnddarguivsoupj6ojmnqslryf9mnnceudyivnc4shgq0a4shceowstvhhxpq1b0s+5ezbg8grtlr6pq9+pxqniqfdbbmusjrpbjakzdcxmmmqprmmmh+wzqeejzn3z5w6kietdbnhl2se6t0yadwnunkezc3x0zjywzrkgtjcj2zhw2nv/lgil2lonmqcrfrtj0o04qcczkvdlpc80zyoefyrswsxlwo5oytpfj2yn48ysvgn9euip5va9yvoapcnaij3aghlxagw6haj4wepael/dqkofzexpep6vlzqznap7i+fgbglysia==</latexit> <latexit sha1_base64="86la2fyceuu8lzvatfuadn2atbs=">aaab63icbzc7sgnbfibpeo3xfrvuzdaivmhxru2cnpyjucaqlgf2mkmgzm4um2efsks0trfqsfud8hx2pomv4ersaoipax//fw5zzgktkqy67peztlyyurae28hvbm3v7bb29u9nngrgfrblwnddarguivsoupj6ojmnqslryf9mnnceudyivnc4shgq0a4shceowstvhhxpq1b0s+5ezbg8grtlr6pq9+pxqniqfdbbmusjrpbjakzdcxmmmqprmmmh+wzqeejzn3z5w6kietdbnhl2se6t0yadwnunkezc3x0zjywzrkgtjcj2zhw2nv/lgil2lonmqcrfrtj0o04qcczkvdlpc80zyoefyrswsxlwo5oytpfj2yn48ysvgn9euip5va9yvoapcnaij3aghlxagw6haj4wepael/dqkofzexpep6vlzqznap7i+fgbglysia==</latexit> <latexit sha1_base64="jpanwujdoqhzfi77wyjqcntcajk=">aaab63icbvbnt8jaej3if+ix6tflizhxrfov6o3oxsmmvkigidtlchu222z3akiiv8glbzve/upe/dcu0iocl5nk5b2zzmylmikmed63u1pb39jckm9xdnb39g+qh0epjs01x4cnmtxtibmuqmfagis2m40siss2othtzg89otyivq80zjbm2ecjwhbgvgq6erlrvwte3zvdxsv+qwpqonmrfnx7kc8tvmqlm6bjexmfe6zjcintsjc3mde+ygpswkpygiaczi+dumdw6btxqm0pcufq74kjs4wzj5httbgnzbi3e//zojnfv+feqcwnvhyxkm6ls6k7+9ztc42c5ngsxrwwt7p8ydtjzpop2bd85zdxsxbrv677936tcvokuyytoivz8oesgnahtqiag4bneiu3rzkvzrvzswgtocxmmfyb8/kdl8iodq==</latexit> <latexit sha1_base64="39wqq+cbicqvo/g1cciy5gfbvo4=">aaab63icbvc7tsnaefyhvwivacuujyikqsimaboigspewgqpsalz5zyccj5bd2ukyoubacga0fipfacdhz/c5vfawkgrjwz2tbstpliydn0vp7c0vlk6vlwvbwxube+ud/dutzjpxn2wyetfhdrwkrt3uadkd6nmna4lb4adq7hfvofaietd4ddlqux7skscubss337gsdvlilt1jyclxjursu3wo/enapvo+bpdtvgwc4vmumnanptikfongkk+kruzw1pkbrthw5yqgnmt5jnjr+tykl0sjdqwqjjrf0/kndzmgie2m6byn/pewpzpa2uynqe5ugmgxlhpoiitbbmy/px0heym5dasyrswtxlwp5oytpmubaje/mulxd+txls9hg3jeqyowgecwql4cay1uiy6+mbawcm8w4ujncfn1xmbthac2cw+/ihz/gnhvzdr</latexit> <latexit sha1_base64="vgixoijal4spmy24zpirblyelfk=">aaab63icbva9swnbej2lxzf+rs0voqycvbizubugjwucngkkr9jb7cvl9vao3tkhhimtbsxubp0p+r12/gb/hjupqhmfddzem2fmxpairtfxvqzc0vlk6lp+vbcxubw9u9zdu9nxqijzacxi1qiizojl5ifhwrqjyiqkbksh/euxx79nsvny3uigyx5eupkhnbi0ktd6yejaxzjtdiawf4k7i6xk4aj2/xg0qraln61otnoisascan10nqt9jcjkvlbhozvqlhdaj13wnfssigk/mxw7te+m0rhdwjmsae/u3xmzibqerihpjaj29lw3fv/zmimgf37gzziik3s6keyfjbe9/tzucmuoioehhcpubrvpjyhc0ertmcg48y8veu+sffl2ayamk5gidwdwdkfgwjlu4aaq4aefdk/waq+wtj6tn+t92pqzzjp78afwxw8+nji3</latexit> sha1_base64="xmnlatczj/0h8crjzuzzz+slrua=">aaab63icbvbnt8jaej36ififevtssew8kdalein68yijfrjoyhazwobtttmdmidhn3jxomarf8ib/8yfeldwjzo8vdetmxlrjouhz/t2vlbx1jc2s1vl7z3dvf3kwegdsxpnmecpthurygalubiqiimttcnlionnahgz9zupqi1i1t2nmgwt1lcifpyrlyloexlrvqpezzvbxsz+qapqongtfhv6kc8tvmqlm6btexmfy6zjcimtcic3mde+zh1sw6pygiycz46dukdw6blxqm0pcmfq74kxs4wzjzhttbgnzki3ff/z2jnfl+fyqcwnvhy+km6ls6k7/dztcy2c5mgsxrwwt7p8wdtjzpmp2xd8xzexsxbeu6r5d161fl2kuyjjoiez8oec6naldqiag4bneiu3rzkvzrvzmw9dcyqzi/gd5/mhvaioja==</latexit> Overview of IRGA Apply rotation matrix Q = (R, S) where R (n x p) is an orthonormal basis for the column space of X and S (n x (n-p)) for the orthogonal complement. Then, the likelihood for Q T y factorizes as R T y,f N R T X + R T F (Z), 2 I p S T y,f N S T F (Z), 2 I n p Only the first distribution involves Auxiliary variable <latexit sha1_base64="39wqq+cbicqvo/g1cciy5gfbvo4=">aaab63icbvc7tsnaefyhvwivacuujyikqsimaboigspewgqpsalz5zyccj5bd2ukyoubacga0fipfacdhz/c5vfawkgrjwz2tbstpliydn0vp7c0vlk6vlwvbwxube+ud/dutzjpxn2wyetfhdrwkrt3uadkd6nmna4lb4adq7hfvofaietd4ddlqux7skscubss337gsdvlilt1jyclxjursu3wo/enapvo+bpdtvgwc4vmumnanptikfongkk+kruzw1pkbrthw5yqgnmt5jnjr+tykl0sjdqwqjjrf0/kndzmgie2m6byn/pewpzpa2uynqe5ugmgxlhpoiitbbmy/px0heym5dasyrswtxlwp5oytpmubaje/mulxd+txls9hg3jeqyowgecwql4cay1uiy6+mbawcm8w4ujncfn1xmbthac2cw+/ihz/gnhvzdr</latexit> captures influence of F on Approximate p(ζ S T y) by a Gaussian Compute p(β y) p(r T y β, ζ) p(β) p(ζ S T y) dζ 31

32 JBOZXCXMwAFdFRz6ECX5EvK8=">AAACoXicbVHbjtMwEHXCbSm3Ao9IaESFtCuhKomQ4HEFiwQPoALb7UpxWjmu01prJ8GeIIWQ/+I7eONvcNMg6JaRRj4+x55jz6SlkhaD4JfnX7l67fqNg5uDW7fv3L03vP/gzBaV4WLKC1WY85RZoWQupihRifPSCKZTJWbpxeuNPvsqjJVFfop1KRLNVrnMJGfoqMXwxxuqRIYxnHQrbcpDmgpkVMsl1EcU6HeXa4ZN2e4qRq7WSFvo9s3ndn4KNWzZZOCKfQGaGcabsG0iauVKs3nUwmwRzSPovcCZfXMld2ocPfvrty/CH9/FcBSMgy5gH4Q9GJE+JovhT7oseKVFjlwxa+MwKDFpmEHJlWgHtLKiZPyCrUTsYM60sEnTdbiFp45ZQlYYlzlCx/57o2Ha2lqn7qRmuLaXtQ35Py2uMHuZNDIvKxQ53xpllQIsYDMuWEojOKraAcaNdG8Fvmaur+iGOnBNCC9/eR+cReMwGIcfn4+OX/XtOCCPyBNySELyghyTt2RCpoR7j70T7733wR/57/yJ/2l71Pf6Ow/JTvjxb/+CzEQ=</latexit> JBOZXCXMwAFdFRz6ECX5EvK8=">AAACoXicbVHbjtMwEHXCbSm3Ao9IaESFtCuhKomQ4HEFiwQPoALb7UpxWjmu01prJ8GeIIWQ/+I7eONvcNMg6JaRRj4+x55jz6SlkhaD4JfnX7l67fqNg5uDW7fv3L03vP/gzBaV4WLKC1WY85RZoWQupihRifPSCKZTJWbpxeuNPvsqjJVFfop1KRLNVrnMJGfoqMXwxxuqRIYxnHQrbcpDmgpkVMsl1EcU6HeXa4ZN2e4qRq7WSFvo9s3ndn4KNWzZZOCKfQGaGcabsG0iauVKs3nUwmwRzSPovcCZfXMld2ocPfvrty/CH9/FcBSMgy5gH4Q9GJE+JovhT7oseKVFjlwxa+MwKDFpmEHJlWgHtLKiZPyCrUTsYM60sEnTdbiFp45ZQlYYlzlCx/57o2Ha2lqn7qRmuLaXtQ35Py2uMHuZNDIvKxQ53xpllQIsYDMuWEojOKraAcaNdG8Fvmaur+iGOnBNCC9/eR+cReMwGIcfn4+OX/XtOCCPyBNySELyghyTt2RCpoR7j70T7733wR/57/yJ/2l71Pf6Ow/JTvjxb/+CzEQ=</latexit> JBOZXCXMwAFdFRz6ECX5EvK8=">AAACoXicbVHbjtMwEHXCbSm3Ao9IaESFtCuhKomQ4HEFiwQPoALb7UpxWjmu01prJ8GeIIWQ/+I7eONvcNMg6JaRRj4+x55jz6SlkhaD4JfnX7l67fqNg5uDW7fv3L03vP/gzBaV4WLKC1WY85RZoWQupihRifPSCKZTJWbpxeuNPvsqjJVFfop1KRLNVrnMJGfoqMXwxxuqRIYxnHQrbcpDmgpkVMsl1EcU6HeXa4ZN2e4qRq7WSFvo9s3ndn4KNWzZZOCKfQGaGcabsG0iauVKs3nUwmwRzSPovcCZfXMld2ocPfvrty/CH9/FcBSMgy5gH4Q9GJE+JovhT7oseKVFjlwxa+MwKDFpmEHJlWgHtLKiZPyCrUTsYM60sEnTdbiFp45ZQlYYlzlCx/57o2Ha2lqn7qRmuLaXtQ35Py2uMHuZNDIvKxQ53xpllQIsYDMuWEojOKraAcaNdG8Fvmaur+iGOnBNCC9/eR+cReMwGIcfn4+OX/XtOCCPyBNySELyghyTt2RCpoR7j70T7733wR/57/yJ/2l71Pf6Ow/JTvjxb/+CzEQ=</latexit> JBOZXCXMwAFdFRz6ECX5EvK8=">AAACoXicbVHbjtMwEHXCbSm3Ao9IaESFtCuhKomQ4HEFiwQPoALb7UpxWjmu01prJ8GeIIWQ/+I7eONvcNMg6JaRRj4+x55jz6SlkhaD4JfnX7l67fqNg5uDW7fv3L03vP/gzBaV4WLKC1WY85RZoWQupihRifPSCKZTJWbpxeuNPvsqjJVFfop1KRLNVrnMJGfoqMXwxxuqRIYxnHQrbcpDmgpkVMsl1EcU6HeXa4ZN2e4qRq7WSFvo9s3ndn4KNWzZZOCKfQGaGcabsG0iauVKs3nUwmwRzSPovcCZfXMld2ocPfvrty/CH9/FcBSMgy5gH4Q9GJE+JovhT7oseKVFjlwxa+MwKDFpmEHJlWgHtLKiZPyCrUTsYM60sEnTdbiFp45ZQlYYlzlCx/57o2Ha2lqn7qRmuLaXtQ35Py2uMHuZNDIvKxQ53xpllQIsYDMuWEojOKraAcaNdG8Fvmaur+iGOnBNCC9/eR+cReMwGIcfn4+OX/XtOCCPyBNySELyghyTt2RCpoR7j70T7733wR/57/yJ/2l71Pf6Ow/JTvjxb/+CzEQ=</latexit> <latexit sha1_base64="n5zgcu6hmy5da0/qqa9eay68ez0=">aaacmnicbvbdsxtbfj21h+pq29g+9mvoecyvsbsk6pu0ig0fikjtqznpudu5mwzozc4zd4ww5d/54h/pq6h0wyqv/rfoyh6q9sda4zx7uxnovmrlkul+rusphj56vlyygq+tp3n6rlhx/ksvkiexiwtdug4ghrwy2cffgrulqzczxups5p3mpz5f51vhv9ckxl6bkvw5kkbbgjq+tbgwashfhgtbb8irewsgsqrdf54kjtmjou7off6gh2x9e73nw9jiwpc2/ziwsxbqncyxhtsassuzg98n6yi02qkhg8ypmsxkzdcs1ob9l01k6tfgsemn01huhkuqjzdcxqawdpp+pc885ztbgfk8cofz4np1340ajpctk4xjwrx/15uj//n6few7/vrzsik08uzqxmlobz8vyifkosq9cqsku+gvxi7bgarqcxxkso9gvk867dzekz1629x/t2hjhb1kr9gws9ko22cf2chrmmno2e92wf5e59hv6dk6uhldihy7l9gtrh+vadmaqm0=</latexit> <latexit sha1_base64="n5zgcu6hmy5da0/qqa9eay68ez0=">aaacmnicbvbdsxtbfj21h+pq29g+9mvoecyvsbsk6pu0ig0fikjtqznpudu5mwzozc4zd4ww5d/54h/pq6h0wyqv/rfoyh6q9sda4zx7uxnovmrlkul+rusphj56vlyygq+tp3n6rlhx/ksvkiexiwtdug4ghrwy2cffgrulqzczxups5p3mpz5f51vhv9ckxl6bkvw5kkbbgjq+tbgwashfhgtbb8irewsgsqrdf54kjtmjou7off6gh2x9e73nw9jiwpc2/ziwsxbqncyxhtsassuzg98n6yi02qkhg8ypmsxkzdcs1ob9l01k6tfgsemn01huhkuqjzdcxqawdpp+pc885ztbgfk8cofz4np1340ajpctk4xjwrx/15uj//n6few7/vrzsik08uzqxmlobz8vyifkosq9cqsku+gvxi7bgarqcxxkso9gvk867dzekz1629x/t2hjhb1kr9gws9ko22cf2chrmmno2e92wf5e59hv6dk6uhldihy7l9gtrh+vadmaqm0=</latexit> <latexit sha1_base64="n5zgcu6hmy5da0/qqa9eay68ez0=">aaacmnicbvbdsxtbfj21h+pq29g+9mvoecyvsbsk6pu0ig0fikjtqznpudu5mwzozc4zd4ww5d/54h/pq6h0wyqv/rfoyh6q9sda4zx7uxnovmrlkul+rusphj56vlyygq+tp3n6rlhx/ksvkiexiwtdug4ghrwy2cffgrulqzczxups5p3mpz5f51vhv9ckxl6bkvw5kkbbgjq+tbgwashfhgtbb8irewsgsqrdf54kjtmjou7off6gh2x9e73nw9jiwpc2/ziwsxbqncyxhtsassuzg98n6yi02qkhg8ypmsxkzdcs1ob9l01k6tfgsemn01huhkuqjzdcxqawdpp+pc885ztbgfk8cofz4np1340ajpctk4xjwrx/15uj//n6few7/vrzsik08uzqxmlobz8vyifkosq9cqsku+gvxi7bgarqcxxkso9gvk867dzekz1629x/t2hjhb1kr9gws9ko22cf2chrmmno2e92wf5e59hv6dk6uhldihy7l9gtrh+vadmaqm0=</latexit> <latexit sha1_base64="n5zgcu6hmy5da0/qqa9eay68ez0=">aaacmnicbvbdsxtbfj21h+pq29g+9mvoecyvsbsk6pu0ig0fikjtqznpudu5mwzozc4zd4ww5d/54h/pq6h0wyqv/rfoyh6q9sda4zx7uxnovmrlkul+rusphj56vlyygq+tp3n6rlhx/ksvkiexiwtdug4ghrwy2cffgrulqzczxups5p3mpz5f51vhv9ckxl6bkvw5kkbbgjq+tbgwashfhgtbb8irewsgsqrdf54kjtmjou7off6gh2x9e73nw9jiwpc2/ziwsxbqncyxhtsassuzg98n6yi02qkhg8ypmsxkzdcs1ob9l01k6tfgsemn01huhkuqjzdcxqawdpp+pc885ztbgfk8cofz4np1340ajpctk4xjwrx/15uj//n6few7/vrzsik08uzqxmlobz8vyifkosq9cqsku+gvxi7bgarqcxxkso9gvk867dzekz1629x/t2hjhb1kr9gws9ko22cf2chrmmno2e92wf5e59hv6dk6uhldihy7l9gtrh+vadmaqm0=</latexit> <latexit sha1_base64="acp/iepfd0w5i/saqobwiiubvow=">aaacbhicbvdlsgmxfm3uv62vuzfdbivqgpqzexrzfirlcvybnafk0kwbmmrckhfk6cknv+lghsju/qh3/o1powttpxdh5jx7yb0nkoxq43nftm5tfwnzk79d2nnd2z9wd49aokkvjk2cser1iqqjo4i0dtwmdkqiieemtkprzcxvpxclasluzviskkobodhfyfip5xblquqmoqtxykaph3lxrshyvdjzs17vmwouej8jjzch0xo/gn6cu06ewqxp3fu9acijuozirqafinveijxca9k1vcboddizhzgfp1bpwzhrtosbc/x3xarxrcc8sp0cmafe9mbif143nffvokfcpoyivpgothk0czwlavtuewzy2bkefbw7qjxecmfjcyvyepzlk1dj67zqe1x/7qjuu87iyimioafl4inluao3oagaainh8axewzvz5lw4787hojxnzdph4a+czx+mbzyz</latexit> <latexit sha1_base64="acp/iepfd0w5i/saqobwiiubvow=">aaacbhicbvdlsgmxfm3uv62vuzfdbivqgpqzexrzfirlcvybnafk0kwbmmrckhfk6cknv+lghsju/qh3/o1powttpxdh5jx7yb0nkoxq43nftm5tfwnzk79d2nnd2z9wd49aokkvjk2cser1iqqjo4i0dtwmdkqiieemtkprzcxvpxclasluzviskkobodhfyfip5xblquqmoqtxykaph3lxrshyvdjzs17vmwouej8jjzch0xo/gn6cu06ewqxp3fu9acijuozirqafinveijxca9k1vcboddizhzgfp1bpwzhrtosbc/x3xarxrcc8sp0cmafe9mbif143nffvokfcpoyivpgothk0czwlavtuewzy2bkefbw7qjxecmfjcyvyepzlk1dj67zqe1x/7qjuu87iyimioafl4inluao3oagaainh8axewzvz5lw4787hojxnzdph4a+czx+mbzyz</latexit> <latexit sha1_base64="acp/iepfd0w5i/saqobwiiubvow=">aaacbhicbvdlsgmxfm3uv62vuzfdbivqgpqzexrzfirlcvybnafk0kwbmmrckhfk6cknv+lghsju/qh3/o1powttpxdh5jx7yb0nkoxq43nftm5tfwnzk79d2nnd2z9wd49aokkvjk2cser1iqqjo4i0dtwmdkqiieemtkprzcxvpxclasluzviskkobodhfyfip5xblquqmoqtxykaph3lxrshyvdjzs17vmwouej8jjzch0xo/gn6cu06ewqxp3fu9acijuozirqafinveijxca9k1vcboddizhzgfp1bpwzhrtosbc/x3xarxrcc8sp0cmafe9mbif143nffvokfcpoyivpgothk0czwlavtuewzy2bkefbw7qjxecmfjcyvyepzlk1dj67zqe1x/7qjuu87iyimioafl4inluao3oagaainh8axewzvz5lw4787hojxnzdph4a+czx+mbzyz</latexit> <latexit sha1_base64="acp/iepfd0w5i/saqobwiiubvow=">aaacbhicbvdlsgmxfm3uv62vuzfdbivqgpqzexrzfirlcvybnafk0kwbmmrckhfk6cknv+lghsju/qh3/o1powttpxdh5jx7yb0nkoxq43nftm5tfwnzk79d2nnd2z9wd49aokkvjk2cser1iqqjo4i0dtwmdkqiieemtkprzcxvpxclasluzviskkobodhfyfip5xblquqmoqtxykaph3lxrshyvdjzs17vmwouej8jjzch0xo/gn6cu06ewqxp3fu9acijuozirqafinveijxca9k1vcboddizhzgfp1bpwzhrtosbc/x3xarxrcc8sp0cmafe9mbif143nffvokfcpoyivpgothk0czwlavtuewzy2bkefbw7qjxecmfjcyvyepzlk1dj67zqe1x/7qjuu87iyimioafl4inluao3oagaainh8axewzvz5lw4787hojxnzdph4a+czx+mbzyz</latexit> Posterior approximation accuracy Theorem IRGA approximation Gaussian approximation E D {p( y) k ˆp( y)} S T y apple W 2 2 p( S T y), ˆp( S T y) where y,f N X + F (Z), 2 I n with (,F) p( )p(f ) 32

33 Advantages of IRGA Approximation employed is transparent and allows for analysis yielding theoretical guarantees Can leverage your favorite method (e.g. lasso, approximate message passing, etc.) to produce accurate approximations of marginal posteriors 33

34 Related papers van den Boom, W., Reeves, G., and Dunson, D. B. (2015). Scalable approximations of marginal posteriors in variable selection. arxiv: van den Boom, W., and Dunson, D. B., and Reeves, G. (2015). Quantifying uncertainty in variable selection with arbitrary matrices. In IEEE 6th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing. van den Boom, W. (2018). Tailored Scalable Dimensionality Reduction. PhD thesis, Department of Statistical Science, Duke University. 34

35 References Carbonetto, P. & Stephens, M. (2012). Scalable variational inference for Bayesian variable selection in regression, and its accuracy in genetic association studies. Bayesian Analysis, 7: Clyde, M. A., Ghosh, J. & Littman, M. L. (2011). Bayesian adaptive sampling for variable selection and model averaging. Journal of Computational and Graphical Statistics, 20: Diaconis, P. & Freedman, D. (1984). Asymptotics of graphical projection pursuit. The Annals of Statistics, 12: Friedman, J., Hastie, T. & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1): George, E. I. & McCulloch, R. E. (1993). Variable selection via Gibbs sampling. Journal of the American Statistical Association, 85: Hernández-Lobato, J. M., Hernández-Lobato, D. & Suárez, A. (2015). Expectation propagation in linear regression models with spike-and-slab priors. Machine Learning, 99: Ormerod, J. T., You, C. & Müller S. (2017). A variational Bayes approach to variable selection. Electronic Journal of Statistics, 11: Ročková, V. & George, E. I. (2014). EMVS: The EM approach to Bayesian variable selection. Journal of the American Statistical Association, 109: