Uncertainty Quantification for Inverse Problems. November 7, 2011
|
|
- Jasmine Patterson
- 5 years ago
- Views:
Transcription
1 Uncertainty Quantification for Inverse Problems November 7, 2011
2 Outline UQ and inverse problems Review: least-squares Review: Gaussian Bayesian linear model Parametric reductions for IP Bias, variance and MSE for Tikhonov Resampling methods Bayesian inversion Exploratory data analysis (by example)
3 Introduction Inverse problem: to recover an unknown object from indirect noisy observations Example: Deblurring of a signal f Forward problem: µ(x) = 1 A(x t)f (t) dt 0 Noisy data: y i = µ(x i ) + ε i, i = 1,..., n Inverse problem: recover f (t) for t [0, 1]
4 General Model A = (A i ), A i : H IR, f H y i = A i [f ] + ε i, i = 1,..., n The recovery of f from A[f ] is ill-posed Errors ε i modeled as random Estimate f is an H-valued random variable Basic Question (UQ) How good is the estimate f? What is the distribution of f? Summarize characteristics of the distribution of f E.g., how likely is it that f will be far from f?
5 Review: least-squares y = Aβ + ε A is n m with n > m and A t A has stable inverse E(ε) = 0, Var(ε) = σ 2 I Least-squares estimate of β: β = arg min y Ab 2 = (A t A) 1 A t y b β is an unbiased estimator of β: E β ( β ) = β β
6 Covariance matrix of β: Var β ( β ) = σ 2 (A t A) 1 If ε is Gaussian N(0, σ 2 I), then β is the MLE and β N( β, σ 2 (A t A) 1 ) Unbiased estimator of σ 2 : σ 2 = y A β 2 /(n m) = y ŷ 2 /(n dof(ŷ)), where ŷ = H β, H = A (A t A) 1 A t = hat-matrix
7 Confidence regions 1 α confidence interval for β i : I i (α) : βi ± t α/2,n m σ ( (A t A) 1 ) i,i C.I. s are pre-data Pointwise vs simultaneous 1 α coverage: pointwise: P[ β i I i (α) ] = 1 α simultaneous: P[ β i I i (α) i ] = 1 α Easy to find 1 α joint confidence region R α for β which gives simultaneous coverage i
8 Residuals r = y ŷ = (I H)y E(r) = 0, Var(r) = σ 2 (I H) If ε N(0, σ 2 I), then: r N(0, σ 2 (I H)) do not behave like original errors ε Corrected residuals r i = r i / 1 H ii Corrected residuals may be used for model validation Other types of residuals (e.g., recursive)
9 Review: Gaussian Bayesian linear model y β N(Aβ, σ 2 I), β N(µ, τ 2 Σ) µ, σ, τ, Σ known Goal: update prior distribution of β in light of data y. Compute posterior distribution of β given y: β y N(µ y, Σ y ) Σ y = σ 2 ( A t A + (σ 2 /τ 2 )Σ 1 ) 1 µ y = ( A t A + (σ 2 /τ 2 )Σ 1 ) 1 (A t y + (σ 2 /τ 2 )Σ 1 µ) = weighted average of µ ls and µ
10 Summarizing posterior: µ y = E(µ y) = mean and mode of posterior (MAP) Σ y = Var(µ y) = covariance matrix of posterior 1 α credible interval for β i : (µ y ) i ± z α/2 ( Σ y ) i,i Can also construct 1 α joint credible region for β
11 Some properties µ y minimizes E δ(y) β 2 among all δ(y) µ y β ls as σ 2 /τ 2 0, µ y µ as σ 2 /τ 2 µ y can be used as a frequentist estimate of β but: µ y is biased its variance is NOT obtained by sampling from the posterior Warning: a 1 α credible regions MAY NOT have 1 α frequentist coverage If µ, σ, τ or Σ unknown use hierarchical or empirical Bayesian and MCMC methods
12 Parametric reductions for IP Transform infinite to finite dimensional problem: i.e., A[f ] Aa Assumptions y = A[f ] + ε = Aa + δ + ε Function representation: f (x) = φ(x) t a + η(x), A[η] = 0 Penalized least-squares estimate: â = arg min y Ab 2 + λ 2 b t Sb b = (A t A + λ 2 S) 1 A t y f (x) = φ(x)tâ
13 Example: y i = 1 0 A(x i t)f (t) dt + ε i Quadrature approximation: 1 0 A(x i t)f (t) dt (1/m) Discretized system: m A(x i t j )f (t j ) j=1 y = Af + δ + ε, A ij = A(x i t j ), f i = f (x i ) Regularized estimate: f = arg min g IR m y Ag 2 + λ 2 Dg 2 = (A t A + λ 2 D t D) 1 A t y Ly f (xi ) = f i = e t i Ly
14 Example: A i continuous on Hilbert space H. Then: orthonormal φ k H, orthonormal v k IR n and non-decreasing λ k 0 such that A[φ k ] = λ k v k, f = f 0 + f 1, with f 0 Null(A), f 1 Null(A) f 0 not constrained by data, f 1 = n i=1 a kφ k Discretized system: y = V a + ε, V = (λ 1 v 1 λ n v n ) Regularized estimate: â = arg min y V b 2 + λ 2 b 2 Ly b IR n f (x) = φ(x) t Ly, φ(x) = (φ i (x))
15 Example: A i continuous on RKHS H = W m ([0, 1]). Then: H = N m 1 Hm and φ j H and κ j H such that 1 y A[f ] 2 + λ 2 (f (m) 1 ) 2 = (y i κ i, f ) 2 + λ 2 P Hm f 2 0 i f = j a j φ j + η, η span{φ k } Discretized system: Regularized estimate: y = Xa + δ + ε â = arg min y Xb 2 + λ 2 b t Pb Ly b IR n f (x) = φ(x) t Ly, φ(x) = (φ i (x))
16 To summarize: y = Aa + δ + ε f (x) = φ(x) t a + η(x), A[η] = 0 â = arg min b IR n y Xb 2 + λ 2 b t Pb = (A t A + λ 2 P) 1 A t y f (x) = φ(x)tâ
17 Bias, variance and MSE of f Is f (x) close to f (x) on average? Bias( f (x) ) = E f (x) f (x) = φ(x) t B λ a+φ(x) t G λ A t δ η(x) Pointwise sampling variability: Pointwise MSE: Var( f (x) ) = σ 2 AG λ φ(x) 2 MSE( f (x) ) = Bias( f (x) ) 2 + Var( f (x) ) Integrated MSE: IMSE( f (x) ) = E f (x) f (x) 2 dx = MSE( f (x) ) dx
18 What about the bias of f? Upper bounds are sometimes possible Geometric information can be obtained from bounds Optimization approach: find max/min of bias s.t. f S Average bias: choose a prior for f and compute mean bias over prior Example: f in RKHS H, f (x) = ρ x, f Bias( f (x) ) = A x ρ x, f Bias( f (x) ) A x ρ x f
19 Confidence intervals If ε Gaussian and Bias( f (x) ) B(x), then f (x) ± (zα/2 σ AG λ φ(x) + B(x)) is a 1 α C.I. for f (x) Simultaneous 1 α C.I. s for E( f (x)), x S: find β such that [ ] P sup Z t V (x) β x S α Z i iid N(0, 1), V (x) = K G λ φ(x)/ K G λ φ(x) and use results from maxima of Gaussian processes theory
20 Residuals r = y Aâ = y ŷ Moments: E(r) = A Bias( â ) + δ Var(r) = σ 2 (I H λ ) 2 Note: Bias( A â ) = A Bias( â ) may be small even if Bias( â ) is significant Corrected residuals: r i = r i /(1 (H λ ) ii )
21 Estimating σ and λ λ can be chosen using same data y (with GCV, L-curve, discrepancy principle, etc.) or independent training data This selection introduces an additional error If δ 0, σ 2 can be estimated as in least-squares: σ 2 = y ŷ 2 /(n dof(ŷ)) otherwise one may consider y = µ + ε, µ = A[f ] and use methods from nonparametric regression
22 Resampling methods Idea: If f and σ are good estimates, then one should be able to generate synthetic data consistent with actual observations If f is a good estimate make synthetic data y 1 = A[ f ] + ε i,..., y b = A[ f ] + ε b with ε i from same distribution as ε. Use same estimation procedure to get f i from yi : y 1 f 1,..., y b f b Approximate distribution of f with that of f 1,..., f b
23 Generating ε similar to ε Parametric resampling: ε N(0, σ 2 I) Nonparametric resampling: sample ε with replacement from corrected residuals Problems: Possibly badly biased and computational intensive Residuals may need to be corrected also for correlation structure A bad f may lead to misleading results Training sets: Let {f 1,..., f k } be a training set (e.g., historical data) For each f j use resampling to estimate MSE( f j ) Study variability of MSE( f j )/ f j
24 Bayesian inversion Hierarchical model (simple Gaussian case): y a, θ N(Aa + δ, σ 2 I), a θ F a ( θ), θ F θ θ = (δ, σ, τ) It all reduces to sampling from posterior F(a y) use MCMC methods Possible problems: Selection of priors, improper priors Convergence of MCMC Computational cost of MCMC in high dimensions Interpretation of results
25 Warning Parameters can be tuned so that Tikhonov estimate = f = E(f y) but this does not mean that the uncertainty of f is obtained from the posterior of f given y
26 Warning Bayesian or frequentist inversions can be used for uncertainty quantification but: Uncertainties in each have different interpretations that should be understood Both make assumptions (sometimes stringent) that should be revealed The validity of the assumptions should be explored exploratory analysis and model validation
27 Exploratory data analysis (by example) Example: l 2 and l 1 estimates data = y = Af + ε = AW β + ε fl 2 = arg min g y Ag λ 2 Dg 2 2 λ : from GCV fl 1 = W β, β = arg min b y AW b λ b 1 λ : from discrepancy principle E(ε) = 0, Var(ε) = σ 2 I σ = smoothing spline estimate
28 Example: y, Af & f
29 Tools: robust simulation summaries Sample mean and variance of x 1,..., x m : Recursive formulas: X = (x x m )/m S 2 = 1 (x i m X) 2 i X n+1 = X n + 1 n + 1 (x n+1 X n ) T n+1 = T n + n n + 1 (x n+1 X n ) 2, S 2 n = T n /n Not robust
30 Robust measures Median and median absolute deviation from median (MAD) X = median{x 1,..., x m } { MAD = median x 1 X m,..., x m X } m Approximate recursive formulas with good asymptotics
31 Stability of f l 2
32 Stability of f l 1
33 CI for bias of A f l 2 Good estimate of f good estimate of Af y: direct observation of Af ; less sensitive to λ 1 α CIs for bias of A f ( ŷ = Hy ): ŷ i y i 1 H ii ± z α/2 σ 1 + (H2 ) ii (H ii ) 2 (1 H ii ) 2
34 95% CIs for bias
35 Coverage of 95% CIs for bias
36 Bias bounds for f l 2 For fixed λ: Var( f l 2) = σ 2 G(λ) 1 A t A G(λ) 1 Bias( f l 2) = E( f l 2 ) f = B(λ), G(λ) = A t A + λ 2 D t D, B(λ) = λ 2 G(λ) 1 D t Df. Median bias: Bias M ( f l 2 ) = median( f l 2 ) f For λ estimated: Bias M ( f l 2 ) median[ B( λ) ]
37 Hölder s inequality B( λ) i λ 2 U p,i ( λ) Df q B( λ) i λ 2 U w p,i( λ) β q U p,i ( λ) = DG( λ) 1 e i p or U w p,i( λ) = W D t DG( λ) 1 e i p Plots of U p,i ( λ) and U w p,i ( λ) vs x i
38
39 Assessing fit Compare y to A f + ε Parametric/nonparametric bootstrap samples y : (i) (parametric) ε i N(0, σ 2 I), or (nonparametric) from corrected residuals r c (ii) y i = A f + ε i Compare statistics of y and {y 1,..., y B } (blue,red) Compare statistics of r c and N(0, σ 2 I) (black) Compare parametric vs nonparametric results Example: Simulations for A f l 2 with: y (1) y (2) y (3) good with skewed noise biased
40 Example T 1 (y) = min(y) T 2 (y) = max(y) T k (y) = 1 n ( y mean(y) ) k /std(y) k n (k = 3, 4) i=1 T 5 (y) = sample median (y) T 6 (y) = MAD(y) T 7 (y) = 1st sample quartile (y) T 8 (y) = 3rd sample quartile (y) T 9 (y) = # runs above/below median(y) P j = P ( T j (y ) T o j ), T o j = T j (y)
41
42
43 Assessing fit: Bayesian case Hierarchical model: y f, σ, γ N(Af, σ 2 I) f γ exp ( f t D t Df /2γ 2 ) (σ, γ) π with E(f y) = ) 1 (A t A + σ2 γ 2 Dt D A t y Posterior mean = f l 2 for γ = σ/λ Sample (f, σ ) from posterior of (f, σ) y and simulated data y = Af + σ ε with ε N(0, I) Explore consistency of y with simulated y
44 Simulating f Modify simulations from y i = A f + ε i to y i = Af i + ε i, f i similar to f Frequentist simulation of f? One option: f i from historical data (training set) Check relative MSE for the different f i
45 Bayesian or frequentist? Results have different interpretations Frequentists can use Bayesian methods to derive procedures; Bayesians can use frequentist methods to evaluate procedures Consistent results for small problems and lots of data Possibly very different results for complex large-scale problems Theorems do not address relation of theory to reality
46 Basic Bayesian methods: Readings Carlin, JB & Louis, TA, 2008, Bayesian Methods for Data Analysis (3rd Edn.), Chapman & Hall Gelman, A, Carlin, JB, Stern, HS & Rubin, DB, 2003, Bayesian Data Analysis (2nd Edn.), Chapman & Hall Bayesian methods for inverse problems: Calvetti, D. & Somersalo, E, 2007, Introduction to Bayesian Scientific Computing: Ten Lectures on Subjective Computing, Springer Kaipio, J & Somersalo, E, 2004, Statistical and Computational Inverse Problems, Springer Stuart, A.M., Inverse Problems: A Bayesian Perspective, 2010, Acta Numerica, DOI: /S Discussion of frequentist vs Bayesian statistics Freedman, D, Some issues in the foundation of statistics, 1995, Foundations of Sciences, 1, Tutorial on frequentist and Bayesian methods for inverse problems Stark, PB & Tenorio, L, 2011, A primer of frequentist and Bayesian inference in inverse problems. In Computational Methods for Large-Scale Inverse Problems and Quantification of Uncertainty, pp.9 32, Wiley Statistics and inverse problems Evans, SN & Stark, PB, 2003, Inverse problems as statistics, Inverse Problems, 18, R55 Marzouk, Y & Tenorio, L, 2012, Uncertainty Quantification for Inverse Problems O Sullivan, F, 1986, A statistical perspective on ill-posed inverse problems, Statistical Science, 1, Tenorio, L, Andersson, F, de Hoop, M & Ma, P, 2011, Data analysis tools for uncertainty quantification of inverse problems, Inverse Problems, 27, Wahba, G, 1990, Spline Smoothing for Observational Data, SIAM Special section in Inverse Problems: Volume 24, Number 3, June 2008 Mathematical statistics Casella, G & Berger, RL, 2001, Statistical Inference, Duxbury Press Schervish, ML, 1996, Theory of Statistics, Springer
Statistical Measures of Uncertainty in Inverse Problems
Statistical Measures of Uncertainty in Inverse Problems Workshop on Uncertainty in Inverse Problems Institute for Mathematics and Its Applications Minneapolis, MN 19-26 April 2002 P.B. Stark Department
More informationAugmented Tikhonov Regularization
Augmented Tikhonov Regularization Bangti JIN Universität Bremen, Zentrum für Technomathematik Seminar, November 14, 2008 Outline 1 Background 2 Bayesian inference 3 Augmented Tikhonov regularization 4
More informationUsing prior knowledge in frequentist tests
Using prior knowledge in frequentist tests Use of Bayesian posteriors with informative priors as optimal frequentist tests Christian Bartels Email: h.christian.bartels@gmail.com Allschwil, 5. April 2017
More informationStatistics: Learning models from data
DS-GA 1002 Lecture notes 5 October 19, 2015 Statistics: Learning models from data Learning models from data that are assumed to be generated probabilistically from a certain unknown distribution is a crucial
More informationAccounting for Complex Sample Designs via Mixture Models
Accounting for Complex Sample Designs via Finite Normal Mixture Models 1 1 University of Michigan School of Public Health August 2009 Talk Outline 1 2 Accommodating Sampling Weights in Mixture Models 3
More informationCan we do statistical inference in a non-asymptotic way? 1
Can we do statistical inference in a non-asymptotic way? 1 Guang Cheng 2 Statistics@Purdue www.science.purdue.edu/bigdata/ ONR Review Meeting@Duke Oct 11, 2017 1 Acknowledge NSF, ONR and Simons Foundation.
More informationPlausible Values for Latent Variables Using Mplus
Plausible Values for Latent Variables Using Mplus Tihomir Asparouhov and Bengt Muthén August 21, 2010 1 1 Introduction Plausible values are imputed values for latent variables. All latent variables can
More informationPattern Recognition and Machine Learning. Bishop Chapter 2: Probability Distributions
Pattern Recognition and Machine Learning Chapter 2: Probability Distributions Cécile Amblard Alex Kläser Jakob Verbeek October 11, 27 Probability Distributions: General Density Estimation: given a finite
More informationPart 4: Multi-parameter and normal models
Part 4: Multi-parameter and normal models 1 The normal model Perhaps the most useful (or utilized) probability model for data analysis is the normal distribution There are several reasons for this, e.g.,
More information41903: Introduction to Nonparametrics
41903: Notes 5 Introduction Nonparametrics fundamentally about fitting flexible models: want model that is flexible enough to accommodate important patterns but not so flexible it overspecializes to specific
More informationRegression, Ridge Regression, Lasso
Regression, Ridge Regression, Lasso Fabio G. Cozman - fgcozman@usp.br October 2, 2018 A general definition Regression studies the relationship between a response variable Y and covariates X 1,..., X n.
More informationDensity Estimation. Seungjin Choi
Density Estimation Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
More informationBayesian Inference. Chapter 1. Introduction and basic concepts
Bayesian Inference Chapter 1. Introduction and basic concepts M. Concepción Ausín Department of Statistics Universidad Carlos III de Madrid Master in Business Administration and Quantitative Methods Master
More informationTime Series and Forecasting Lecture 4 NonLinear Time Series
Time Series and Forecasting Lecture 4 NonLinear Time Series Bruce E. Hansen Summer School in Economics and Econometrics University of Crete July 23-27, 2012 Bruce Hansen (University of Wisconsin) Foundations
More informationF & B Approaches to a simple model
A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 215 http://www.astro.cornell.edu/~cordes/a6523 Lecture 11 Applications: Model comparison Challenges in large-scale surveys
More informationRemedial Measures for Multiple Linear Regression Models
Remedial Measures for Multiple Linear Regression Models Yang Feng http://www.stat.columbia.edu/~yangfeng Yang Feng (Columbia University) Remedial Measures for Multiple Linear Regression Models 1 / 25 Outline
More informationGAUSSIAN PROCESS REGRESSION
GAUSSIAN PROCESS REGRESSION CSE 515T Spring 2015 1. BACKGROUND The kernel trick again... The Kernel Trick Consider again the linear regression model: y(x) = φ(x) w + ε, with prior p(w) = N (w; 0, Σ). The
More informationFractional Imputation in Survey Sampling: A Comparative Review
Fractional Imputation in Survey Sampling: A Comparative Review Shu Yang Jae-Kwang Kim Iowa State University Joint Statistical Meetings, August 2015 Outline Introduction Fractional imputation Features Numerical
More informationModeling Real Estate Data using Quantile Regression
Modeling Real Estate Data using Semiparametric Quantile Regression Department of Statistics University of Innsbruck September 9th, 2011 Overview 1 Application: 2 3 4 Hedonic regression data for house prices
More informationThe Bayesian approach to inverse problems
The Bayesian approach to inverse problems Youssef Marzouk Department of Aeronautics and Astronautics Center for Computational Engineering Massachusetts Institute of Technology ymarz@mit.edu, http://uqgroup.mit.edu
More informationStatistics & Data Sciences: First Year Prelim Exam May 2018
Statistics & Data Sciences: First Year Prelim Exam May 2018 Instructions: 1. Do not turn this page until instructed to do so. 2. Start each new question on a new sheet of paper. 3. This is a closed book
More informationBagging During Markov Chain Monte Carlo for Smoother Predictions
Bagging During Markov Chain Monte Carlo for Smoother Predictions Herbert K. H. Lee University of California, Santa Cruz Abstract: Making good predictions from noisy data is a challenging problem. Methods
More informationAn Introduction to Bayesian Linear Regression
An Introduction to Bayesian Linear Regression APPM 5720: Bayesian Computation Fall 2018 A SIMPLE LINEAR MODEL Suppose that we observe explanatory variables x 1, x 2,..., x n and dependent variables y 1,
More informationPractical Bayesian Optimization of Machine Learning. Learning Algorithms
Practical Bayesian Optimization of Machine Learning Algorithms CS 294 University of California, Berkeley Tuesday, April 20, 2016 Motivation Machine Learning Algorithms (MLA s) have hyperparameters that
More informationMeasurement Error and Linear Regression of Astronomical Data. Brandon Kelly Penn State Summer School in Astrostatistics, June 2007
Measurement Error and Linear Regression of Astronomical Data Brandon Kelly Penn State Summer School in Astrostatistics, June 2007 Classical Regression Model Collect n data points, denote i th pair as (η
More informationBayesian Aggregation for Extraordinarily Large Dataset
Bayesian Aggregation for Extraordinarily Large Dataset Guang Cheng 1 Department of Statistics Purdue University www.science.purdue.edu/bigdata Department Seminar Statistics@LSE May 19, 2017 1 A Joint Work
More informationRandom Fields in Bayesian Inference: Effects of the Random Field Discretization
Random Fields in Bayesian Inference: Effects of the Random Field Discretization Felipe Uribe a, Iason Papaioannou a, Wolfgang Betz a, Elisabeth Ullmann b, Daniel Straub a a Engineering Risk Analysis Group,
More informationStatistics 203: Introduction to Regression and Analysis of Variance Course review
Statistics 203: Introduction to Regression and Analysis of Variance Course review Jonathan Taylor - p. 1/?? Today Review / overview of what we learned. - p. 2/?? General themes in regression models Specifying
More informationStatistics - Lecture One. Outline. Charlotte Wickham 1. Basic ideas about estimation
Statistics - Lecture One Charlotte Wickham wickham@stat.berkeley.edu http://www.stat.berkeley.edu/~wickham/ Outline 1. Basic ideas about estimation 2. Method of Moments 3. Maximum Likelihood 4. Confidence
More informationBetter Bootstrap Confidence Intervals
by Bradley Efron University of Washington, Department of Statistics April 12, 2012 An example Suppose we wish to make inference on some parameter θ T (F ) (e.g. θ = E F X ), based on data We might suppose
More informationStat 5101 Lecture Notes
Stat 5101 Lecture Notes Charles J. Geyer Copyright 1998, 1999, 2000, 2001 by Charles J. Geyer May 7, 2001 ii Stat 5101 (Geyer) Course Notes Contents 1 Random Variables and Change of Variables 1 1.1 Random
More informationLabor-Supply Shifts and Economic Fluctuations. Technical Appendix
Labor-Supply Shifts and Economic Fluctuations Technical Appendix Yongsung Chang Department of Economics University of Pennsylvania Frank Schorfheide Department of Economics University of Pennsylvania January
More informationLECTURE 5 NOTES. n t. t Γ(a)Γ(b) pt+a 1 (1 p) n t+b 1. The marginal density of t is. Γ(t + a)γ(n t + b) Γ(n + a + b)
LECTURE 5 NOTES 1. Bayesian point estimators. In the conventional (frequentist) approach to statistical inference, the parameter θ Θ is considered a fixed quantity. In the Bayesian approach, it is considered
More informationFall 2017 STAT 532 Homework Peter Hoff. 1. Let P be a probability measure on a collection of sets A.
1. Let P be a probability measure on a collection of sets A. (a) For each n N, let H n be a set in A such that H n H n+1. Show that P (H n ) monotonically converges to P ( k=1 H k) as n. (b) For each n
More informationA note on multiple imputation for general purpose estimation
A note on multiple imputation for general purpose estimation Shu Yang Jae Kwang Kim SSC meeting June 16, 2015 Shu Yang, Jae Kwang Kim Multiple Imputation June 16, 2015 1 / 32 Introduction Basic Setup Assume
More informationSpatially Adaptive Smoothing Splines
Spatially Adaptive Smoothing Splines Paul Speckman University of Missouri-Columbia speckman@statmissouriedu September 11, 23 Banff 9/7/3 Ordinary Simple Spline Smoothing Observe y i = f(t i ) + ε i, =
More informationChapter 8: Sampling distributions of estimators Sections
Chapter 8: Sampling distributions of estimators Sections 8.1 Sampling distribution of a statistic 8.2 The Chi-square distributions 8.3 Joint Distribution of the sample mean and sample variance Skip: p.
More informationSimple Regression Model Setup Estimation Inference Prediction. Model Diagnostic. Multiple Regression. Model Setup and Estimation.
Statistical Computation Math 475 Jimin Ding Department of Mathematics Washington University in St. Louis www.math.wustl.edu/ jmding/math475/index.html October 10, 2013 Ridge Part IV October 10, 2013 1
More informationGenerative Models and Stochastic Algorithms for Population Average Estimation and Image Analysis
Generative Models and Stochastic Algorithms for Population Average Estimation and Image Analysis Stéphanie Allassonnière CIS, JHU July, 15th 28 Context : Computational Anatomy Context and motivations :
More informationConfidence Estimation Methods for Neural Networks: A Practical Comparison
, 6-8 000, Confidence Estimation Methods for : A Practical Comparison G. Papadopoulos, P.J. Edwards, A.F. Murray Department of Electronics and Electrical Engineering, University of Edinburgh Abstract.
More informationUNIVERSITÄT POTSDAM Institut für Mathematik
UNIVERSITÄT POTSDAM Institut für Mathematik Testing the Acceleration Function in Life Time Models Hannelore Liero Matthias Liero Mathematische Statistik und Wahrscheinlichkeitstheorie Universität Potsdam
More informationEffective Computation for Odds Ratio Estimation in Nonparametric Logistic Regression
Communications of the Korean Statistical Society 2009, Vol. 16, No. 4, 713 722 Effective Computation for Odds Ratio Estimation in Nonparametric Logistic Regression Young-Ju Kim 1,a a Department of Information
More informationEfficient MCMC Sampling for Hierarchical Bayesian Inverse Problems
Efficient MCMC Sampling for Hierarchical Bayesian Inverse Problems Andrew Brown 1,2, Arvind Saibaba 3, Sarah Vallélian 2,3 CCNS Transition Workshop SAMSI May 5, 2016 Supported by SAMSI Visiting Research
More informationConvergence of Square Root Ensemble Kalman Filters in the Large Ensemble Limit
Convergence of Square Root Ensemble Kalman Filters in the Large Ensemble Limit Evan Kwiatkowski, Jan Mandel University of Colorado Denver December 11, 2014 OUTLINE 2 Data Assimilation Bayesian Estimation
More informationGaussian Processes. Le Song. Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012
Gaussian Processes Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 01 Pictorial view of embedding distribution Transform the entire distribution to expected features Feature space Feature
More informationSTATS 200: Introduction to Statistical Inference. Lecture 29: Course review
STATS 200: Introduction to Statistical Inference Lecture 29: Course review Course review We started in Lecture 1 with a fundamental assumption: Data is a realization of a random process. The goal throughout
More informationIntroduction to Bayesian methods in inverse problems
Introduction to Bayesian methods in inverse problems Ville Kolehmainen 1 1 Department of Applied Physics, University of Eastern Finland, Kuopio, Finland March 4 2013 Manchester, UK. Contents Introduction
More informationDS-GA 1002 Lecture notes 11 Fall Bayesian statistics
DS-GA 100 Lecture notes 11 Fall 016 Bayesian statistics In the frequentist paradigm we model the data as realizations from a distribution that depends on deterministic parameters. In contrast, in Bayesian
More informationInference For High Dimensional M-estimates. Fixed Design Results
: Fixed Design Results Lihua Lei Advisors: Peter J. Bickel, Michael I. Jordan joint work with Peter J. Bickel and Noureddine El Karoui Dec. 8, 2016 1/57 Table of Contents 1 Background 2 Main Results and
More informationLecture Notes 15 Prediction Chapters 13, 22, 20.4.
Lecture Notes 15 Prediction Chapters 13, 22, 20.4. 1 Introduction Prediction is covered in detail in 36-707, 36-701, 36-715, 10/36-702. Here, we will just give an introduction. We observe training data
More informationMachine Learning CSE546 Sham Kakade University of Washington. Oct 4, What about continuous variables?
Linear Regression Machine Learning CSE546 Sham Kakade University of Washington Oct 4, 2016 1 What about continuous variables? Billionaire says: If I am measuring a continuous variable, what can you do
More informationMachine Learning CSE546 Carlos Guestrin University of Washington. September 30, What about continuous variables?
Linear Regression Machine Learning CSE546 Carlos Guestrin University of Washington September 30, 2014 1 What about continuous variables? n Billionaire says: If I am measuring a continuous variable, what
More informationA Magiv CV Theory for Large-Margin Classifiers
A Magiv CV Theory for Large-Margin Classifiers Hui Zou School of Statistics, University of Minnesota June 30, 2018 Joint work with Boxiang Wang Outline 1 Background 2 Magic CV formula 3 Magic support vector
More information9. Robust regression
9. Robust regression Least squares regression........................................................ 2 Problems with LS regression..................................................... 3 Robust regression............................................................
More informationApplied Econometrics (QEM)
Applied Econometrics (QEM) The Simple Linear Regression Model based on Prinicples of Econometrics Jakub Mućk Department of Quantitative Economics Jakub Mućk Applied Econometrics (QEM) Meeting #2 The Simple
More informationGaussian processes. Chuong B. Do (updated by Honglak Lee) November 22, 2008
Gaussian processes Chuong B Do (updated by Honglak Lee) November 22, 2008 Many of the classical machine learning algorithms that we talked about during the first half of this course fit the following pattern:
More informationSTAT 100C: Linear models
STAT 100C: Linear models Arash A. Amini June 9, 2018 1 / 56 Table of Contents Multiple linear regression Linear model setup Estimation of β Geometric interpretation Estimation of σ 2 Hat matrix Gram matrix
More informationNonparametric Drift Estimation for Stochastic Differential Equations
Nonparametric Drift Estimation for Stochastic Differential Equations Gareth Roberts 1 Department of Statistics University of Warwick Brazilian Bayesian meeting, March 2010 Joint work with O. Papaspiliopoulos,
More informationBayesian data analysis in practice: Three simple examples
Bayesian data analysis in practice: Three simple examples Martin P. Tingley Introduction These notes cover three examples I presented at Climatea on 5 October 0. Matlab code is available by request to
More informationFractional Hot Deck Imputation for Robust Inference Under Item Nonresponse in Survey Sampling
Fractional Hot Deck Imputation for Robust Inference Under Item Nonresponse in Survey Sampling Jae-Kwang Kim 1 Iowa State University June 26, 2013 1 Joint work with Shu Yang Introduction 1 Introduction
More informationLecture 2: Statistical Decision Theory (Part I)
Lecture 2: Statistical Decision Theory (Part I) Hao Helen Zhang Hao Helen Zhang Lecture 2: Statistical Decision Theory (Part I) 1 / 35 Outline of This Note Part I: Statistics Decision Theory (from Statistical
More informationRegression Models - Introduction
Regression Models - Introduction In regression models there are two types of variables that are studied: A dependent variable, Y, also called response variable. It is modeled as random. An independent
More informationLinear Models for Regression
Linear Models for Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr
More informationBayesian inverse problems with Laplacian noise
Bayesian inverse problems with Laplacian noise Remo Kretschmann Faculty of Mathematics, University of Duisburg-Essen Applied Inverse Problems 2017, M27 Hangzhou, 1 June 2017 1 / 33 Outline 1 Inverse heat
More informationBayesian Calibration of Simulators with Structured Discretization Uncertainty
Bayesian Calibration of Simulators with Structured Discretization Uncertainty Oksana A. Chkrebtii Department of Statistics, The Ohio State University Joint work with Matthew T. Pratola (Statistics, The
More informationSimple Linear Regression (Part 3)
Chapter 1 Simple Linear Regression (Part 3) 1 Write an Estimated model Statisticians/Econometricians usually write an estimated model together with some inference statistics, the following are some formats
More informationBayesian linear regression
Bayesian linear regression Linear regression is the basis of most statistical modeling. The model is Y i = X T i β + ε i, where Y i is the continuous response X i = (X i1,..., X ip ) T is the corresponding
More informationCSci 8980: Advanced Topics in Graphical Models Gaussian Processes
CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee November 15, 2007 Gaussian Processes Outline Gaussian Processes Outline Parametric Bayesian Regression Gaussian
More informationNonparameteric Regression:
Nonparameteric Regression: Nadaraya-Watson Kernel Regression & Gaussian Process Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro,
More informationLecture 8: Information Theory and Statistics
Lecture 8: Information Theory and Statistics Part II: Hypothesis Testing and I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 23, 2015 1 / 50 I-Hsiang
More informationGaussian Process Regression
Gaussian Process Regression 4F1 Pattern Recognition, 21 Carl Edward Rasmussen Department of Engineering, University of Cambridge November 11th - 16th, 21 Rasmussen (Engineering, Cambridge) Gaussian Process
More informationVariational Bayesian Inference for Parametric and Non-Parametric Regression with Missing Predictor Data
for Parametric and Non-Parametric Regression with Missing Predictor Data August 23, 2010 Introduction Bayesian inference For parametric regression: long history (e.g. Box and Tiao, 1973; Gelman, Carlin,
More informationUniform Inference for Conditional Factor Models with Instrumental and Idiosyncratic Betas
Uniform Inference for Conditional Factor Models with Instrumental and Idiosyncratic Betas Yuan Xiye Yang Rutgers University Dec 27 Greater NY Econometrics Overview main results Introduction Consider a
More informationDiscrete Ill Posed and Rank Deficient Problems. Alistair Boyle, Feb 2009, SYS5906: Directed Studies Inverse Problems 1
Discrete Ill Posed and Rank Deficient Problems Alistair Boyle, Feb 2009, SYS5906: Directed Studies Inverse Problems 1 Definitions Overview Inversion, SVD, Picard Condition, Rank Deficient, Ill-Posed Classical
More information2 Statistical Estimation: Basic Concepts
Technion Israel Institute of Technology, Department of Electrical Engineering Estimation and Identification in Dynamical Systems (048825) Lecture Notes, Fall 2009, Prof. N. Shimkin 2 Statistical Estimation:
More informationStatistical Inference
Statistical Inference Liu Yang Florida State University October 27, 2016 Liu Yang, Libo Wang (Florida State University) Statistical Inference October 27, 2016 1 / 27 Outline The Bayesian Lasso Trevor Park
More informationDISCUSSION: COVERAGE OF BAYESIAN CREDIBLE SETS. By Subhashis Ghosal North Carolina State University
Submitted to the Annals of Statistics DISCUSSION: COVERAGE OF BAYESIAN CREDIBLE SETS By Subhashis Ghosal North Carolina State University First I like to congratulate the authors Botond Szabó, Aad van der
More informationConvergence rates of spectral methods for statistical inverse learning problems
Convergence rates of spectral methods for statistical inverse learning problems G. Blanchard Universtität Potsdam UCL/Gatsby unit, 04/11/2015 Joint work with N. Mücke (U. Potsdam); N. Krämer (U. München)
More informationy(x) = x w + ε(x), (1)
Linear regression We are ready to consider our first machine-learning problem: linear regression. Suppose that e are interested in the values of a function y(x): R d R, here x is a d-dimensional vector-valued
More informationInverse problems in statistics
Inverse problems in statistics Laurent Cavalier (Université Aix-Marseille 1, France) Yale, May 2 2011 p. 1/35 Introduction There exist many fields where inverse problems appear Astronomy (Hubble satellite).
More informationA Modern Look at Classical Multivariate Techniques
A Modern Look at Classical Multivariate Techniques Yoonkyung Lee Department of Statistics The Ohio State University March 16-20, 2015 The 13th School of Probability and Statistics CIMAT, Guanajuato, Mexico
More informationPARAMETER ESTIMATION: BAYESIAN APPROACH. These notes summarize the lectures on Bayesian parameter estimation.
PARAMETER ESTIMATION: BAYESIAN APPROACH. These notes summarize the lectures on Bayesian parameter estimation.. Beta Distribution We ll start by learning about the Beta distribution, since we end up using
More informationOutline Lecture 2 2(32)
Outline Lecture (3), Lecture Linear Regression and Classification it is our firm belief that an understanding of linear models is essential for understanding nonlinear ones Thomas Schön Division of Automatic
More informationBayesian Learning. HT2015: SC4 Statistical Data Mining and Machine Learning. Maximum Likelihood Principle. The Bayesian Learning Framework
HT5: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Maximum Likelihood Principle A generative model for
More informationLecture 7 Introduction to Statistical Decision Theory
Lecture 7 Introduction to Statistical Decision Theory I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 20, 2016 1 / 55 I-Hsiang Wang IT Lecture 7
More informationBayesian Inference: Concept and Practice
Inference: Concept and Practice fundamentals Johan A. Elkink School of Politics & International Relations University College Dublin 5 June 2017 1 2 3 Bayes theorem In order to estimate the parameters of
More informationWhat s New in Econometrics. Lecture 13
What s New in Econometrics Lecture 13 Weak Instruments and Many Instruments Guido Imbens NBER Summer Institute, 2007 Outline 1. Introduction 2. Motivation 3. Weak Instruments 4. Many Weak) Instruments
More informationReproducing Kernel Hilbert Spaces Class 03, 15 February 2006 Andrea Caponnetto
Reproducing Kernel Hilbert Spaces 9.520 Class 03, 15 February 2006 Andrea Caponnetto About this class Goal To introduce a particularly useful family of hypothesis spaces called Reproducing Kernel Hilbert
More informationChapter 9. Non-Parametric Density Function Estimation
9-1 Density Estimation Version 1.2 Chapter 9 Non-Parametric Density Function Estimation 9.1. Introduction We have discussed several estimation techniques: method of moments, maximum likelihood, and least
More informationInterval Estimation III: Fisher's Information & Bootstrapping
Interval Estimation III: Fisher's Information & Bootstrapping Frequentist Confidence Interval Will consider four approaches to estimating confidence interval Standard Error (+/- 1.96 se) Likelihood Profile
More informationEcon 2148, fall 2017 Gaussian process priors, reproducing kernel Hilbert spaces, and Splines
Econ 2148, fall 2017 Gaussian process priors, reproducing kernel Hilbert spaces, and Splines Maximilian Kasy Department of Economics, Harvard University 1 / 37 Agenda 6 equivalent representations of the
More informationRidge regression. Patrick Breheny. February 8. Penalized regression Ridge regression Bayesian interpretation
Patrick Breheny February 8 Patrick Breheny High-Dimensional Data Analysis (BIOS 7600) 1/27 Introduction Basic idea Standardization Large-scale testing is, of course, a big area and we could keep talking
More informationRobust MCMC Sampling with Non-Gaussian and Hierarchical Priors
Division of Engineering & Applied Science Robust MCMC Sampling with Non-Gaussian and Hierarchical Priors IPAM, UCLA, November 14, 2017 Matt Dunlop Victor Chen (Caltech) Omiros Papaspiliopoulos (ICREA,
More informationMachine Learning CSE546 Carlos Guestrin University of Washington. September 30, 2013
Bayesian Methods Machine Learning CSE546 Carlos Guestrin University of Washington September 30, 2013 1 What about prior n Billionaire says: Wait, I know that the thumbtack is close to 50-50. What can you
More informationSmall Area Modeling of County Estimates for Corn and Soybean Yields in the US
Small Area Modeling of County Estimates for Corn and Soybean Yields in the US Matt Williams National Agricultural Statistics Service United States Department of Agriculture Matt.Williams@nass.usda.gov
More informationConstrained Gaussian processes: methodology, theory and applications
Constrained Gaussian processes: methodology, theory and applications Hassan Maatouk hassan.maatouk@univ-rennes2.fr Workshop on Gaussian Processes, November 6-7, 2017, St-Etienne (France) Hassan Maatouk
More informationPost-exam 2 practice questions 18.05, Spring 2014
Post-exam 2 practice questions 18.05, Spring 2014 Note: This is a set of practice problems for the material that came after exam 2. In preparing for the final you should use the previous review materials,
More informationQuantile Regression for Extraordinarily Large Data
Quantile Regression for Extraordinarily Large Data Shih-Kang Chao Department of Statistics Purdue University November, 2016 A joint work with Stanislav Volgushev and Guang Cheng Quantile regression Two-step
More informationBayesian Inference. Chapter 9. Linear models and regression
Bayesian Inference Chapter 9. Linear models and regression M. Concepcion Ausin Universidad Carlos III de Madrid Master in Business Administration and Quantitative Methods Master in Mathematical Engineering
More informationPART I INTRODUCTION The meaning of probability Basic definitions for frequentist statistics and Bayesian inference Bayesian inference Combinatorics
Table of Preface page xi PART I INTRODUCTION 1 1 The meaning of probability 3 1.1 Classical definition of probability 3 1.2 Statistical definition of probability 9 1.3 Bayesian understanding of probability
More information