# Parameter Selection Techniques and Surrogate Models

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Parameter Selection Techniques and Surrogate Models Model Reduction: Will discuss two forms Parameter space reduction Surrogate models to reduce model complexity Input Representation Local Sensitivity Analysis Global Sensitivity Analysis Parameter Selection Model Discrepancy Surrogate Models Sparse Grids Model Calibration Stochastic Spectral Methods Uncertainty Propagation Sparse Grids

2 Parameter Selection Techniques and Surrogate Models Parameter Space Reduction: SIR Model ds = N - S - kis, S() =S dt di dt = kis - (r + )I, I() =I dr = ri - R, R() =R dt Susceptible Infectious Recovered Parameters: Response: : Infection coefficient k: Interaction coefficient r: Recovery rate : Birth/death rate y = Z 5 R(t, q)dt Note: Parameters q =[, k, r, ] not uniquely determined by data

3 Parameter Selection Techniques and Surrogate Models First Issue: Parameters often not identifiable in the sense that they are not uniquely determined by the data. Example: Spring model m d 2 z dt 2 Problem: z() =z, + c dz dt + kz = f cos(! F t) dz dt () =z 1 Parameters q =[m, c, k, f ] and q = 1, c m, k m, f m yield same displacements

4 Parameter Selection Techniques and Surrogate Models First Issue: Parameters often not identifiable in the sense that they are not uniquely determined by the data. Example: Spring model m d 2 z dt 2 Problem: z() =z, Solution: Reformulate problem as d 2 z dt 2 + C dz dt + Kz = F cos(! F t) z() =z, + c dz dt + kz = f cos(! F t) dz dt () =z 1 Parameters q =[m, c, k, f ] and q = 1, c m, k m, f m yield same displacements dz dt () =z 1 where C = c m, K = k m and F = f m Techniques for General Models: Linear algebra analysis; e.g., SVD or QR algorithms Identifiability analysis: M. Eisenberg Sensitivity analysis: M. Hyman Active Subspaces: Present discussion Surrogates: Present and later talks

5 Parameter Selection Techniques and Surrogate Models Second Issue: Coupled compartment can have 1 s of nonidentifiable parameters Example: Metabolism Techniques for General Models: Identifiability and sensitivity analysis: M. Eisenberg, M. Hyman Active Subspaces: Present discussion

6 Example: Varies most in [.7,.3] direction No variation in orthogonal direction Active Subspaces Note: Functions may vary significantly in only a few directions Active directions may be linear combination of inputs y = exp(.7q 1 +.3q 2 ) Strategy: Linearly parameterized problems: Employ SVD or QR decomposition. Nonlinear problems: Construct approximate gradient matrix and employ SVD or QR. q2 q 1

7 Parameter Space Reduction Techniques: Linear Problems Second Issue: Models depends on very large number of parameters e.g., millions but only a few are significant. Linear Algebra Techniques: Linearly parameterized problems y = Aq, q 2 R p,y2 R m Singular Value Decomposition (SVD): Rank Revealing QR Decomposition: Problem: Neither is directly applicable when m or p are very large; e.g., millions. Solution: Random range finding algorithms.

8 Random Range Finding Algorithms: Linear Problems Algorithm: Halko, Martinsson and Tropp, SIAM Review, 211 Example: y i = px q k sin(2 kt i ),i=1,,m k=1

9 Random Range Finding Algorithms: Linear Problems Example: m = 11, p = 1: Analytic value for rank is 49 Column Entries of A A(:,1) A(:,11) A(:,21) A(:,91) Time (s) Column Entries of A Aliasing A(:,1) A(:,51) Time (s).8 1 Singular Values Singular Values Deterministic Algorithm Random Algorithm Absolute Difference in Singular Values Absolute Difference Between Singular Values Example: m = 11, p = 1,,: Random algorithm still viable

10 Random Range Finding Algorithms: Linear Problems Example: m = 11, p = 1 Column Entries of A A(:,1) A(:,11) A(:,21) A(:,91) Time (s) Column Entries of A A(:,1) A(:,51) Time (s).8 1 Aliasing Later: Reproduce these results using the code alias.m. You can experiment with the size of the random sample. See how far you can increase the dimension. Singular Values Deterministic Algorithm Random Algorithm Singular Values Absolute Difference in Singular Values Absolute Difference Between Singular Values

11 Example: Varies most in [.7,.3] direction No variation in orthogonal direction Active Subspaces Note: Functions may vary significantly in only a few directions Active directions may be linear combination of inputs y = exp(.7q 1 +.3q 2 ) Strategy: Linearly parameterized problems: Employ SVD or QR decomposition. Nonlinear problems: Construct approximate gradient matrix and employ SVD or QR. q2 q 1

12 Gradient-Based Active Subspace Construction Active Subspace: See [Constantine, SIAM 215]. Consider f = f (q), q 2 Q R p and r q f (q) = Construct outer product C @q p Z (r q f )(r q f ) T dq T (q): Distribution of input parameters q Partition eigenvalues: C = W W T = apple 1 Rotated Coordinates: 2, W =[W 1 W 2 ] y = W T 1 q 2 Rn and z = W T 2 q 2 Rp-n Active Variables Active Subspace: Range of eigenvectors in W 1

13 Gradient-Based Active Subspace Construction Active Subspace: Construction based on random sampling 1. Draw M independent samples {q j } from 2. Evaluate r q f j = r q f (q j ) 3. Approximate outer product C e C = 1 M MX (r q f j )(r q f j ) T j=1 Note: e C = GG T where G = 1 p M [r q f 1,..., r q f M ] 4. Take SVD of G = W p V T Monte Carlo Quadrature: More on Wednesday Active subspace of dimension n is first n columns of W Research: Develop algorithms for codes that do not have derivative capabilities Note: Finite-difference approximations tempting but not very effective Strategy: Algorithms based on linearization and sampling in active directions.

14 Gradient-Based Active Subspace Construction Example: y = exp(.7q 1 +.3q 2 ) Active Subspace: For gradient matrix G, form SVD G = U V T Eigenvalue spectrum indicates 1-D active subspace with basis U(:, 1) = [.91,.39] g(y) Response Surface Testing Points y Exercise: Construct a gradient matrix G and evaluate it at random points in the interval [ 1, 1] [ 1, 1] Now take the SVD to determine a basis for the active subspace. How might you construct a response surface g(y) on the 1-D subspace y?

15 Later Talk: C. Webster Steps in Uncertainty Quantification

16 Surrogate Models: Motivation Sinko-Streifer Model Notation and Assumptions: u (t, x ): Number of fish of size x at time t µ: Death rate Growth rate of same-sized individuals is same and denoted by g (t, x ) dx = g (t, x ) dt Flux Balance: u (t, x ) is a density and rate is (t, x ) = g (t, x )u (t, @ ) + = -µu + = Model: ) + Z g (t, x )u (t, x ) u (, x ) = x =x (x ) = x1 x k (t, )u (t, )d

17 Example: Consider the model Surrogate = f(q) with the response y(q) = Boundary Conditions Initial Conditions Z 1 Z 1 Z 1 Z 1 u(t, x, y, z; q)dxdydzdt t 1 x, y, z Notes: Requires approximation of PDE in 3-D What would be a simple surrogate?

18 Recall: Consider = f(q) with the response y(q) = Boundary Conditions Initial Conditions Z 1 Z 1 Z 1 Z 1 Surrogate Models u(t, x, y, z; q)dxdydzdt t Surrogate: Quadratic 1 x, y, z y s (q) =(q.25) Question: How do you construct a polynomial surrogate? Regression Interpolation

19 Recall: Consider = f(q) with the response y(q) = Boundary Conditions Initial Conditions Z 1 Z 1 Z 1 Z 1 Surrogate Models u(t, x, y, z; q)dxdydzdt t Surrogate: Quadratic 1 x, y, z y s (q) =(q.25) Question: How do you construct a polynomial surrogate? Interpolation Regression Statistical Model: y s (q): Emulator for y(q) y m = y s (q m )+" m,m=1,...,m M=7 k=2

20 Surrogate Models Question: How do we keep from fitting noise? Akaike Information Criterion (AIC) AIC = 2k - 2 log[ (y q)] Bayesian Information Criterion (AIC) BIC = k log(m) - 2 log[ (y q)] Likelihood: 1 2 (y q) = Maximize (2 2 ) M/2 e-ss q/2 MX SS q = [y m - y s (q m )] 2 Minimize m=1 Later Talks: M. Eisenberg, B. Fitzpatrick, R. Smith M=7 k=2

21 Surrogate Models Question: How do we keep from fitting noise? Akaike Information Criterion (AIC) AIC = 2k - 2 log[ (y q)] Bayesian Information Criterion (AIC) Example: Exercise: BIC = k log(m) - 2 log[ (y q)] Likelihood: 1 2 (y q) = (2 2 ) M/2 e-ss q/2 MX SS q = [y m - y s (q m )] 2 m=1 y = exp(.7q 1 +.3q 2 ) Construct a polynomial surrogate using the code response_surface.m. What order seems appropriate? Maximize Minimize q2 q 1 M=7 k=2

22 Example: m = 11, p = 1 y i = Code: Computer Tutorials px q k sin(2 kt i ),i=1,,m k=1 Reproduce these results using the code alias.m. You can experiment with the size of the random sample and tol. See how far you can increase the dimension. Column Entries of A A(:,1) A(:,51) Time (s).8 1 Example: y = exp(.7q 1 +.3q 2 ) Construct a polynomial surrogate using the code response_surface.m. What order seems appropriate. Construct a gradient matrix G and evaluate it at random points in the interval [ 1, 1] [ 1, 1] q2 Take SVD to determine basis for the active subspace. q 1

### Spectral Representation of Random Processes

Spectral Representation of Random Processes Example: Represent u(t,x,q) by! u K (t, x, Q) = u k (t, x) k(q) where k(q) are orthogonal polynomials. Single Random Variable:! Let k (Q) be orthogonal with

### MA 575 Linear Models: Cedric E. Ginestet, Boston University Regularization: Ridge Regression and Lasso Week 14, Lecture 2

MA 575 Linear Models: Cedric E. Ginestet, Boston University Regularization: Ridge Regression and Lasso Week 14, Lecture 2 1 Ridge Regression Ridge regression and the Lasso are two forms of regularized

### DS-GA 1002 Lecture notes 10 November 23, Linear models

DS-GA 2 Lecture notes November 23, 2 Linear functions Linear models A linear model encodes the assumption that two quantities are linearly related. Mathematically, this is characterized using linear functions.

### Background Mathematics (2/2) 1. David Barber

Background Mathematics (2/2) 1 David Barber University College London Modified by Samson Cheung (sccheung@ieee.org) 1 These slides accompany the book Bayesian Reasoning and Machine Learning. The book and

### Properties of Matrices and Operations on Matrices

Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,

### Stochastic Collocation Methods for Polynomial Chaos: Analysis and Applications

Stochastic Collocation Methods for Polynomial Chaos: Analysis and Applications Dongbin Xiu Department of Mathematics, Purdue University Support: AFOSR FA955-8-1-353 (Computational Math) SF CAREER DMS-64535

### Quadratic forms. Here. Thus symmetric matrices are diagonalizable, and the diagonalization can be performed by means of an orthogonal matrix.

Quadratic forms 1. Symmetric matrices An n n matrix (a ij ) n ij=1 with entries on R is called symmetric if A T, that is, if a ij = a ji for all 1 i, j n. We denote by S n (R) the set of all n n symmetric

### Stat 451 Lecture Notes Numerical Integration

Stat 451 Lecture Notes 03 12 Numerical Integration Ryan Martin UIC www.math.uic.edu/~rgmartin 1 Based on Chapter 5 in Givens & Hoeting, and Chapters 4 & 18 of Lange 2 Updated: February 11, 2016 1 / 29

### TENSOR APPROXIMATION TOOLS FREE OF THE CURSE OF DIMENSIONALITY

TENSOR APPROXIMATION TOOLS FREE OF THE CURSE OF DIMENSIONALITY Eugene Tyrtyshnikov Institute of Numerical Mathematics Russian Academy of Sciences (joint work with Ivan Oseledets) WHAT ARE TENSORS? Tensors

### randomized block krylov methods for stronger and faster approximate svd

randomized block krylov methods for stronger and faster approximate svd Cameron Musco and Christopher Musco December 2, 25 Massachusetts Institute of Technology, EECS singular value decomposition n d left

### (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? Solution: dim N(A) 1, since rank(a) 3. Ax =

. (5 points) (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? dim N(A), since rank(a) 3. (b) If we also know that Ax = has no solution, what do we know about the rank of A? C(A)

### LINEAR ALGEBRA (PMTH213) Tutorial Questions

Tutorial Questions The tutorial exercises range from revision and routine practice, through lling in details in the notes, to applications of the theory. While the tutorial problems are not compulsory,

### Linear Algebra Practice Problems

Linear Algebra Practice Problems Math 24 Calculus III Summer 25, Session II. Determine whether the given set is a vector space. If not, give at least one axiom that is not satisfied. Unless otherwise stated,

### Computational Methods. Eigenvalues and Singular Values

Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations

### arxiv: v2 [math.na] 5 Dec 2013

ACTIVE SUBSPACE METHODS IN THEORY AND PRACTICE: APPLICATIONS TO KRIGING SURFACES PAUL G. CONSTANTINE, ERIC DOW, AND QIQI WANG arxiv:1304.2070v2 [math.na 5 Dec 2013 Abstract. Many multivariate functions

### I. Multiple Choice Questions (Answer any eight)

Name of the student : Roll No : CS65: Linear Algebra and Random Processes Exam - Course Instructor : Prashanth L.A. Date : Sep-24, 27 Duration : 5 minutes INSTRUCTIONS: The test will be evaluated ONLY

### Linear Models 1. Isfahan University of Technology Fall Semester, 2014

Linear Models 1 Isfahan University of Technology Fall Semester, 2014 References: [1] G. A. F., Seber and A. J. Lee (2003). Linear Regression Analysis (2nd ed.). Hoboken, NJ: Wiley. [2] A. C. Rencher and

### A Tour of the Lanczos Algorithm and its Convergence Guarantees through the Decades

A Tour of the Lanczos Algorithm and its Convergence Guarantees through the Decades Qiaochu Yuan Department of Mathematics UC Berkeley Joint work with Prof. Ming Gu, Bo Li April 17, 2018 Qiaochu Yuan Tour

### 3. Identify and find the general solution of each of the following first order differential equations.

Final Exam MATH 33, Sample Questions. Fall 6. y = Cx 3 3 is the general solution of a differential equation. Find the equation. Answer: y = 3y + 9 xy. y = C x + C is the general solution of a differential

### DS-GA 1002 Lecture notes 12 Fall Linear regression

DS-GA Lecture notes 1 Fall 16 1 Linear models Linear regression In statistics, regression consists of learning a function relating a certain quantity of interest y, the response or dependent variable,

### Proposition 42. Let M be an m n matrix. Then (32) N (M M)=N (M) (33) R(MM )=R(M)

RODICA D. COSTIN. Singular Value Decomposition.1. Rectangular matrices. For rectangular matrices M the notions of eigenvalue/vector cannot be defined. However, the products MM and/or M M (which are square,

### Preliminary Examination in Numerical Analysis

Department of Applied Mathematics Preliminary Examination in Numerical Analysis August 7, 06, 0 am pm. Submit solutions to four (and no more) of the following six problems. Show all your work, and justify

### 1. Select the unique answer (choice) for each problem. Write only the answer.

MATH 5 Practice Problem Set Spring 7. Select the unique answer (choice) for each problem. Write only the answer. () Determine all the values of a for which the system has infinitely many solutions: x +

### Theorems. Least squares regression

Theorems In this assignment we are trying to classify AML and ALL samples by use of penalized logistic regression. Before we indulge on the adventure of classification we should first explain the most

### Positive Definite Matrix

1/29 Chia-Ping Chen Professor Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra Positive Definite, Negative Definite, Indefinite 2/29 Pure Quadratic Function

### Lecture 4 and 5 Controllability and Observability: Kalman decompositions

1 Lecture 4 and 5 Controllability and Observability: Kalman decompositions Spring 2013 - EE 194, Advanced Control (Prof. Khan) January 30 (Wed.) and Feb. 04 (Mon.), 2013 I. OBSERVABILITY OF DT LTI SYSTEMS

### Lecture 6: Bayesian Inference in SDE Models

Lecture 6: Bayesian Inference in SDE Models Bayesian Filtering and Smoothing Point of View Simo Särkkä Aalto University Simo Särkkä (Aalto) Lecture 6: Bayesian Inference in SDEs 1 / 45 Contents 1 SDEs

### Linear Algebra, part 2 Eigenvalues, eigenvectors and least squares solutions

Linear Algebra, part 2 Eigenvalues, eigenvectors and least squares solutions Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2013 Main problem of linear algebra 2: Given

### CS168: The Modern Algorithmic Toolbox Lecture #8: How PCA Works

CS68: The Modern Algorithmic Toolbox Lecture #8: How PCA Works Tim Roughgarden & Gregory Valiant April 20, 206 Introduction Last lecture introduced the idea of principal components analysis (PCA). The

### Calculating determinants for larger matrices

Day 26 Calculating determinants for larger matrices We now proceed to define det A for n n matrices A As before, we are looking for a function of A that satisfies the product formula det(ab) = det A det

### Final Exam, Machine Learning, Spring 2009

Name: Andrew ID: Final Exam, 10701 Machine Learning, Spring 2009 - The exam is open-book, open-notes, no electronics other than calculators. - The maximum possible score on this exam is 100. You have 3

### 10-725/36-725: Convex Optimization Prerequisite Topics

10-725/36-725: Convex Optimization Prerequisite Topics February 3, 2015 This is meant to be a brief, informal refresher of some topics that will form building blocks in this course. The content of the

### Math 224, Fall 2007 Exam 3 Thursday, December 6, 2007

Math 224, Fall 2007 Exam 3 Thursday, December 6, 2007 You have 1 hour and 20 minutes. No notes, books, or other references. You are permitted to use Maple during this exam, but you must start with a blank

### (a)

Chapter 8 Subspace Methods 8. Introduction Principal Component Analysis (PCA) is applied to the analysis of time series data. In this context we discuss measures of complexity and subspace methods for

### Lecture 2: From Linear Regression to Kalman Filter and Beyond

Lecture 2: From Linear Regression to Kalman Filter and Beyond January 18, 2017 Contents 1 Batch and Recursive Estimation 2 Towards Bayesian Filtering 3 Kalman Filter and Bayesian Filtering and Smoothing

### Numerical Mathematics

Alfio Quarteroni Riccardo Sacco Fausto Saleri Numerical Mathematics Second Edition With 135 Figures and 45 Tables 421 Springer Contents Part I Getting Started 1 Foundations of Matrix Analysis 3 1.1 Vector

### Collaborative Filtering: A Machine Learning Perspective

Collaborative Filtering: A Machine Learning Perspective Chapter 6: Dimensionality Reduction Benjamin Marlin Presenter: Chaitanya Desai Collaborative Filtering: A Machine Learning Perspective p.1/18 Topics

### Matrix Algebra, part 2

Matrix Algebra, part 2 Ming-Ching Luoh 2005.9.12 1 / 38 Diagonalization and Spectral Decomposition of a Matrix Optimization 2 / 38 Diagonalization and Spectral Decomposition of a Matrix Also called Eigenvalues

### FINITE-DIMENSIONAL LINEAR ALGEBRA

DISCRETE MATHEMATICS AND ITS APPLICATIONS Series Editor KENNETH H ROSEN FINITE-DIMENSIONAL LINEAR ALGEBRA Mark S Gockenbach Michigan Technological University Houghton, USA CRC Press Taylor & Francis Croup

### σ(a) = a N (x; 0, 1 2 ) dx. σ(a) = Φ(a) =

Until now we have always worked with likelihoods and prior distributions that were conjugate to each other, allowing the computation of the posterior distribution to be done in closed form. Unfortunately,

### 18.06 Problem Set 8 - Solutions Due Wednesday, 14 November 2007 at 4 pm in

806 Problem Set 8 - Solutions Due Wednesday, 4 November 2007 at 4 pm in 2-06 08 03 Problem : 205+5+5+5 Consider the matrix A 02 07 a Check that A is a positive Markov matrix, and find its steady state

### An Empirical Chaos Expansion Method for Uncertainty Quantification

An Empirical Chaos Expansion Method for Uncertainty Quantification Melvin Leok and Gautam Wilkins Abstract. Uncertainty quantification seeks to provide a quantitative means to understand complex systems

### Modeling Data with Linear Combinations of Basis Functions. Read Chapter 3 in the text by Bishop

Modeling Data with Linear Combinations of Basis Functions Read Chapter 3 in the text by Bishop A Type of Supervised Learning Problem We want to model data (x 1, t 1 ),..., (x N, t N ), where x i is a vector

### CS 231A Section 1: Linear Algebra & Probability Review

CS 231A Section 1: Linear Algebra & Probability Review 1 Topics Support Vector Machines Boosting Viola-Jones face detector Linear Algebra Review Notation Operations & Properties Matrix Calculus Probability

### CS 231A Section 1: Linear Algebra & Probability Review. Kevin Tang

CS 231A Section 1: Linear Algebra & Probability Review Kevin Tang Kevin Tang Section 1-1 9/30/2011 Topics Support Vector Machines Boosting Viola Jones face detector Linear Algebra Review Notation Operations

### High Performance Nonlinear Solvers

What is a nonlinear system? High Performance Nonlinear Solvers Michael McCourt Division Argonne National Laboratory IIT Meshfree Seminar September 19, 2011 Every nonlinear system of equations can be described

### Randomized algorithms for the approximation of matrices

Randomized algorithms for the approximation of matrices Luis Rademacher The Ohio State University Computer Science and Engineering (joint work with Amit Deshpande, Santosh Vempala, Grant Wang) Two topics

### Differential equations

Differential equations Math 7 Spring Practice problems for April Exam Problem Use the method of elimination to find the x-component of the general solution of x y = 6x 9x + y = x 6y 9y Soln: The system

### Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Linear Classifiers. Blaine Nelson, Tobias Scheffer

Universität Potsdam Institut für Informatik Lehrstuhl Linear Classifiers Blaine Nelson, Tobias Scheffer Contents Classification Problem Bayesian Classifier Decision Linear Classifiers, MAP Models Logistic

### Least Squares Optimization

Least Squares Optimization The following is a brief review of least squares optimization and constrained optimization techniques. I assume the reader is familiar with basic linear algebra, including the

### A q x k+q + A q 1 x k+q A 0 x k = 0 (1.1) where k = 0, 1, 2,..., N q, or equivalently. A(σ)x k = 0, k = 0, 1, 2,..., N q (1.

A SPECTRAL CHARACTERIZATION OF THE BEHAVIOR OF DISCRETE TIME AR-REPRESENTATIONS OVER A FINITE TIME INTERVAL E.N.Antoniou, A.I.G.Vardulakis, N.P.Karampetakis Aristotle University of Thessaloniki Faculty

### Fundamentals of Data Assimila1on

014 GSI Community Tutorial NCAR Foothills Campus, Boulder, CO July 14-16, 014 Fundamentals of Data Assimila1on Milija Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University

### AM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition

AM 205: lecture 8 Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition QR Factorization A matrix A R m n, m n, can be factorized

### Numerical Methods I Singular Value Decomposition

Numerical Methods I Singular Value Decomposition Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 9th, 2014 A. Donev (Courant Institute)

### SPRING 2006 PRELIMINARY EXAMINATION SOLUTIONS

SPRING 006 PRELIMINARY EXAMINATION SOLUTIONS 1A. Let G be the subgroup of the free abelian group Z 4 consisting of all integer vectors (x, y, z, w) such that x + 3y + 5z + 7w = 0. (a) Determine a linearly

The Conjugate Gradient Method Classical Iterations We have a problem, We assume that the matrix comes from a discretization of a PDE. The best and most popular model problem is, The matrix will be as large

### Linear Subspace Models

Linear Subspace Models Goal: Explore linear models of a data set. Motivation: A central question in vision concerns how we represent a collection of data vectors. The data vectors may be rasterized images,

### THE GRADIENT PROJECTION ALGORITHM FOR ORTHOGONAL ROTATION. 2 The gradient projection algorithm

THE GRADIENT PROJECTION ALGORITHM FOR ORTHOGONAL ROTATION 1 The problem Let M be the manifold of all k by m column-wise orthonormal matrices and let f be a function defined on arbitrary k by m matrices.

### * Tuesday 17 January :30-16:30 (2 hours) Recored on ESSE3 General introduction to the course.

Name of the course Statistical methods and data analysis Audience The course is intended for students of the first or second year of the Graduate School in Materials Engineering. The aim of the course

### LECTURE 16: PCA AND SVD

Instructor: Sael Lee CS549 Computational Biology LECTURE 16: PCA AND SVD Resource: PCA Slide by Iyad Batal Chapter 12 of PRML Shlens, J. (2003). A tutorial on principal component analysis. CONTENT Principal

### 1 Linearity and Linear Systems

Mathematical Tools for Neuroscience (NEU 34) Princeton University, Spring 26 Jonathan Pillow Lecture 7-8 notes: Linear systems & SVD Linearity and Linear Systems Linear system is a kind of mapping f( x)

### B553 Lecture 5: Matrix Algebra Review

B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations

### Contents as of 12/8/2017. Preface. 1. Overview...1

Contents as of 12/8/2017 Preface 1. Overview...1 1.1 Introduction...1 1.2 Finite element data...1 1.3 Matrix notation...3 1.4 Matrix partitions...8 1.5 Special finite element matrix notations...9 1.6 Finite

### Linearization of Differential Equation Models

Linearization of Differential Equation Models 1 Motivation We cannot solve most nonlinear models, so we often instead try to get an overall feel for the way the model behaves: we sometimes talk about looking

### Full-State Feedback Design for a Multi-Input System

Full-State Feedback Design for a Multi-Input System A. Introduction The open-loop system is described by the following state space model. x(t) = Ax(t)+Bu(t), y(t) =Cx(t)+Du(t) () 4 8.5 A =, B =.5.5, C

### 2. Review of Linear Algebra

2. Review of Linear Algebra ECE 83, Spring 217 In this course we will represent signals as vectors and operators (e.g., filters, transforms, etc) as matrices. This lecture reviews basic concepts from linear

### Preface. Figures Figures appearing in the text were prepared using MATLAB R. For product information, please contact:

Linear algebra forms the basis for much of modern mathematics theoretical, applied, and computational. The purpose of this book is to provide a broad and solid foundation for the study of advanced mathematics.

### Iterative Methods for Solving A x = b

Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http

### STOCHASTIC SAMPLING METHODS

STOCHASTIC SAMPLING METHODS APPROXIMATING QUANTITIES OF INTEREST USING SAMPLING METHODS Recall that quantities of interest often require the evaluation of stochastic integrals of functions of the solutions

### Structure in Data. A major objective in data analysis is to identify interesting features or structure in the data.

Structure in Data A major objective in data analysis is to identify interesting features or structure in the data. The graphical methods are very useful in discovering structure. There are basically two

### The Jordan Canonical Form

The Jordan Canonical Form The Jordan canonical form describes the structure of an arbitrary linear transformation on a finite-dimensional vector space over an algebraically closed field. Here we develop

### Overview. Bayesian assimilation of experimental data into simulation (for Goland wing flutter) Why not uncertainty quantification?

Delft University of Technology Overview Bayesian assimilation of experimental data into simulation (for Goland wing flutter), Simao Marques 1. Why not uncertainty quantification? 2. Why uncertainty quantification?

### The antitriangular factorisation of saddle point matrices

The antitriangular factorisation of saddle point matrices J. Pestana and A. J. Wathen August 29, 2013 Abstract Mastronardi and Van Dooren [this journal, 34 (2013) pp. 173 196] recently introduced the block

### Linear Models. DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis.

Linear Models DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_fall17/index.html Carlos Fernandez-Granda Linear regression Least-squares estimation

### Derivative-Free Trust-Region methods

Derivative-Free Trust-Region methods MTH6418 S. Le Digabel, École Polytechnique de Montréal Fall 2015 (v4) MTH6418: DFTR 1/32 Plan Quadratic models Model Quality Derivative-Free Trust-Region Framework

### Linear Regression (continued)

Linear Regression (continued) Professor Ameet Talwalkar Professor Ameet Talwalkar CS260 Machine Learning Algorithms February 6, 2017 1 / 39 Outline 1 Administration 2 Review of last lecture 3 Linear regression

### Symmetric Matrices and Eigendecomposition

Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions

### Subset Selection. Ilse Ipsen. North Carolina State University, USA

Subset Selection Ilse Ipsen North Carolina State University, USA Subset Selection Given: real or complex matrix A integer k Determine permutation matrix P so that AP = ( A 1 }{{} k A 2 ) Important columns

### Fundamentals of Data Assimila1on

2015 GSI Community Tutorial NCAR Foothills Campus, Boulder, CO August 11-14, 2015 Fundamentals of Data Assimila1on Milija Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University

### Least Squares Optimization

Least Squares Optimization The following is a brief review of least squares optimization and constrained optimization techniques. Broadly, these techniques can be used in data analysis and visualization

### Review of some mathematical tools

MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical

### On the Nature of Random System Matrices in Structural Dynamics

On the Nature of Random System Matrices in Structural Dynamics S. ADHIKARI AND R. S. LANGLEY Cambridge University Engineering Department Cambridge, U.K. Nature of Random System Matrices p.1/20 Outline

### COMP 558 lecture 18 Nov. 15, 2010

Least squares We have seen several least squares problems thus far, and we will see more in the upcoming lectures. For this reason it is good to have a more general picture of these problems and how to

### CME 345: MODEL REDUCTION

CME 345: MODEL REDUCTION Proper Orthogonal Decomposition (POD) Charbel Farhat & David Amsallem Stanford University cfarhat@stanford.edu 1 / 43 Outline 1 Time-continuous Formulation 2 Method of Snapshots

### MA2501 Numerical Methods Spring 2015

Norwegian University of Science and Technology Department of Mathematics MA5 Numerical Methods Spring 5 Solutions to exercise set 9 Find approximate values of the following integrals using the adaptive

### Introduction to Differential Equations

Introduction to Differential Equations J. M. Veal, Ph. D. version 13.08.30 Contents 1 Introduction to Differential Equations 2 1.1 Definitions and Terminology.................... 2 1.2 Initial-Value Problems.......................

### Probabilistic Latent Semantic Analysis

Probabilistic Latent Semantic Analysis Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr

### Graphs, Vectors, and Matrices Daniel A. Spielman Yale University. AMS Josiah Willard Gibbs Lecture January 6, 2016

Graphs, Vectors, and Matrices Daniel A. Spielman Yale University AMS Josiah Willard Gibbs Lecture January 6, 2016 From Applied to Pure Mathematics Algebraic and Spectral Graph Theory Sparsification: approximating

### Data Mining Techniques

Data Mining Techniques CS 6220 - Section 3 - Fall 2016 Lecture 12 Jan-Willem van de Meent (credit: Yijun Zhao, Percy Liang) DIMENSIONALITY REDUCTION Borrowing from: Percy Liang (Stanford) Linear Dimensionality

### Subspace Identification

Chapter 10 Subspace Identification Given observations of m 1 input signals, and p 1 signals resulting from those when fed into a dynamical system under study, can we estimate the internal dynamics regulating

### Linear Algebra- Final Exam Review

Linear Algebra- Final Exam Review. Let A be invertible. Show that, if v, v, v 3 are linearly independent vectors, so are Av, Av, Av 3. NOTE: It should be clear from your answer that you know the definition.

### Lecture 9: SVD, Low Rank Approximation

CSE 521: Design and Analysis of Algorithms I Spring 2016 Lecture 9: SVD, Low Rank Approimation Lecturer: Shayan Oveis Gharan April 25th Scribe: Koosha Khalvati Disclaimer: hese notes have not been subjected

### EE731 Lecture Notes: Matrix Computations for Signal Processing

EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University October 17, 005 Lecture 3 3 he Singular Value Decomposition

### Numerical Methods I: Eigenvalues and eigenvectors

1/25 Numerical Methods I: Eigenvalues and eigenvectors Georg Stadler Courant Institute, NYU stadler@cims.nyu.edu November 2, 2017 Overview 2/25 Conditioning Eigenvalues and eigenvectors How hard are they

### Computational methods for mixed models

Computational methods for mixed models Douglas Bates Department of Statistics University of Wisconsin Madison March 27, 2018 Abstract The lme4 package provides R functions to fit and analyze several different

### ELEC633: Graphical Models

ELEC633: Graphical Models Tahira isa Saleem Scribe from 7 October 2008 References: Casella and George Exploring the Gibbs sampler (1992) Chib and Greenberg Understanding the Metropolis-Hastings algorithm

### Introduction to Matrix Algebra

Introduction to Matrix Algebra August 18, 2010 1 Vectors 1.1 Notations A p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the line. When p

### S. Freitag, B. T. Cao, J. Ninić & G. Meschke

SFB 837 Interaction Modeling Mechanized Tunneling S Freitag, B T Cao, J Ninić & G Meschke Institute for Structural Mechanics Ruhr University Bochum 1 Content 1 Motivation 2 Process-oriented FE model for