Stat. 758: Computation and Programming
|
|
- Andrew Harvey
- 6 years ago
- Views:
Transcription
1 Stat. 758: Computation and Programming Eric B. Laber Department of Statistics, North Carolina State University Lecture 4a Sept. 10, 2015
2 Ambition is like a frog sitting on a Venus flytrap. The flytrap can bite and bite, but it won t bother the frog because it only has little tiny plant teeth. But some other stuff could happen and it could be like ambition. Chiu Chang Suan Shu The word matrix means womb in latin. The title of that stupid Keanu Reeves movie finally makes sense. Terry L. Laber
3 House keeping Beach trip happened! (A thing you should do!) I will give a week s notice for each quiz HW 1 is due September 22 but HW2 will be up before then Python is on the lab machines in SAS hall Work together! But turn in our your own HW and write your own code!
4 Warm-up Explain to your stat buddy How linear systems might arise in statistics What is big-o notation? What is Gaussian elimination? (As a child, I always thought this must be some form of assassination) True or false: The mathematician who coined the term matrix for arrangments of numbers as rectangular arrays fled the U.S. after killing a student with a newspaper stick Research on solving linear least squares problems ceased with the invention of modern computing software An Irish penny has a harp on one side and a chicken on the other (this is why they ask harps or chickens before kickofff in Ireland)
5 Big-O From your mathematics days, f, g : D R f (x) = O {g(x)} as x x 0 lim x x 0 f (x) L g(x), for some fixed constant L True or false f (x) = O {g(x)}: f (x) = 4x 2 10x, g(x) = 10x 2 + x + 2, x f (x) = x log(x 2 ), g(x) = x log(x), x 0 f (n) = log(n!), g(n) = nlog(n), n Z +, n f (x) = x 3 + x 2 log(x), g(x) = 2x 2 log(x) + x, x
6 Little-O Assume f, g : D R f (x) = o {g(x)} as x x 0 lim x x 0 f (x) g(x) = 0 alternatively, for any constant κ > 0 there exists ɛ κ such that f (x) κ g(x) if x x 0 ɛ κ True or false f (x) = o {g(x)} f (x) = x 2 + x, g(x) = x + 1 as x 0 f (x) = x log(x), g(x) = x 2 as x f (x) = x log(x), g(x) = x + x log(x) as x
7 Oh-pee! We often require probabilistic notions of big and little O O p through examples We say r.v. X = OP (1) if lim L P( X L) = 1 Given sequence of r.v. s {(Xn, Y n )} n 1 we say X n = O P (Y n ) if for any ɛ > 0 there exists L ɛ s.t. for all sufficiently large n P ( X n L ɛ Y n ) 1 ɛ Ex. Suppose X1,..., X n are i.i.d. with finite mean µ and variance-covariance Σ show X n = O P (n 1/2 )
8 Oh-pee! cont d More general defn: let X(t), Y(t) be stochastic processes indexed by t T, say X(t) = O P {Y(t)} as t t 0 if for any ɛ there exists L ɛ > 0 and δ ɛ > 0 so that if t t 0 δ ɛ P { X(t) L ɛ Y(t) } 1 ɛ
9 Op-pee! cont d o p through examples We say X n = o P (a n ) to mean P( X n /a n > ɛ) 0 as n for any ɛ > 0 (think delta-method) We say X n = o P (Y n ) if for any κ > 0 and ɛ > 0 P ( X n κ Y n ) 1 ɛ for all sufficiently large n Ex. prove that if Xn = O P (1) and Y n = o P (1) then X n Y n = o P (1)
10 Why do we care? Big O notation used to characterize deterministic algorithm complexity Big O P notation used heavily in asymptotic analyses Stochastic approximation algorithms Bounding Monte Carlo error Dealing with remainder terms in asymptotic expansions
11 Flops We describe algorithms in the number of floating point operations (flops) Addition, subtraction, multiplication, division Built-in functions, e.g,. exp() harder to evaluate How many flops to compute Av where A R n p and v R p? How many flops to compute A A??
12 Linear systems: warm-up Let A R p p and b R p, want soln to Ax = b If A were upper triangular how would you solve for x? Go over linsys.ipynb
13 Linear systems: Gaussian elimination I Triangular systems rare in practice I I Idea! Transform general linear system to triangular system Ax = b BAx = Bb if B, thus sufficient to find B so that BA is triangular invertible
14 Linear systems: Gaussian elimination cont d Primary school example, reduce to triangular system x x x 3 = x
15 Linear systems: Gaussian elimination cont d Algorithm for Gaussian elimination to triangular system A (0) = A, (B (1) ) i,1 = A (0) if j 1 i,1 /A(0) 1,1 if i > 1 and (B(1) ) i,j = 1 i=j Recursively for k = 1,..., p 1 A (k+1) = B (k) A (k 1), and (B (k+1) ) i,k+1 = A (k) i,k+1 /A(k) k+1,k+1 if i > k + 1 and (B (k+1) ) i,j = 1 i=j if j k + 1 (k,k) We assume that A k+1,k+1 0 for all k, which need not hold in general, your will fix this in HW2! Back to linsys.ipynb
16 Iterative methods for large systems Gaussian elimination requires O(p 3 ) operations Manageable for small/moderate-sized problems When p is large iterative methods may be preferable especially if the matrix is sparse Canonical example: Gauss-Seidel iteration for Ax = b Suppose we knew {x j j i} solve for x i via x i = b i k i A i,kx k A i,i Idea! start with initial guess, x 0, then repeatedly update each component of our guess using the above updates
17 Gauss-Seidel pseudo code Input: x (0), A, b Set m = 0 Repeat forever x (m+1) = x (m) For i = 1,..., p x (m+1) i If x (m+1) x (m) ɛ break m = m + 1 = (b i k i A i,kx k )/A i,i With your stat buddy: convert this to python code!
18 Sparse matrices Many applications in statistics involve large sparse matrices Functional data analysis Markov decision processes Matrix completion problems Graphical models... Computational savings obtained by exploiting sparsity Save memory: store only non-zero elements Save flops: matrix ops only with non-zero elements
19 Dictionary of keys Suppose our lin. sys. Ax = b has A = we can store this as the set of triples { (1, 2, 1), (1, 4, 2), (2, 1, 3), (3, 1, 1), (3, 2, 1), (3, 4, 4), (4, 3, 8), (4, 4, 9) } More convenient to store as an associate array where each pair of indices is associated with its respective matrix value, i.e., (1, 2) 1, (1, 4) 2,..., (4, 4) 9
20 Dictionary of keys cont d Dictionary of keys (DOK) storage format is a set of key value pairs Key: indices of non-zero matrix elements Value: non-zero matrix elements Store matrix A as {(i, j) A i,j : A i,j 0} First part of sparsemats.ipynb
21 Compressed row storage DOK is intuitive and useful for constructing sparse matrices Slower than alternatives for numerical operations E.g., matrix-vector mult can be slower than dense case Compressed row storage (CRS) faster for numerical operations Pattern: construct with DOK convert to CRS Suppose that A = CRS stores this as three arrays: Value Col Row
22 Compressed row storage cont d With your stat buddy: Convert to dense format Value Col Row Convert to CSR format A = Back to sparsemats.ipynb
23 Cholesky decomposition If A is symmetric positive definite then A = LL, where L is lower triangular Solve Ax = b by solving triangular systems Ly = b L x = y
24 Cholesky decomposition cont d Algorithm to compute A = LL similar to Gaussian elimination GE and Cholesky both O(p 3 ) but Cholesky better constant Generally Cholesky is more stable Generate Z Normal p (µ, Σ) via 1. Compute Σ = LL 2. Generate W Normal p (0, I p ) 3. Set Z = LW + µ
25 Break: Warm-up quiz II Explain to your stat buddy: What is a random walk? What is a Brownian bridge? What is importance sampling? True of false: Brownian motion was invented by Cavell Brownie The term Monte Carlo was a code-name for stochastic computer experiments related to nuclear research during WWII Hotter than Satan s Toenails is the name of a nail salon in Chattanooga TN
26 Ex. Brownian motion Brownian motion shows up frequently in asymptotic statistics Recall {X (t) : t 0} is a Brownian motion process if: (P1) X (0) = 0; (wp1) (P2) {X (t) : t 0} has ind. increments (what does this mean?) (P3) X (t) Normal(0, c 2 t) for all t 0 (We will assume c = 1 hereafter)
27 Ex. Brownian motion cont d Goal: simulate Brownian motion Problem: computer cannot simulate continuous values Idea: discretize interval [0, T ], 0 = t0 < t 1 < < t n = T and simulate {X (t 1 ),..., X (t n )} Fact: {X (t 1 ),..., X (t n )} is normally distributed with mean 0 and variance-covariance (Σ i,j ) = min(t i, t j ) (see HW2)
28 Generating random functions In some applications necessary to generate random smooth functions over some domain (e.g., time, space, etc.) Grown curves Depression scores Humidity... Basic idea Choose space for domain of random function Choose basis for this space Generate random linear combinations of basis functions
29 Review: basis functions Recall: basis for space of functions, F, is a collection {b j } j 1 in F so that for any f F {λ j } j 1 that satisfy f = j 1 λ j b j Stone-Weierstrass theorem: every continuous function on [0, 1] can be uniformly approximated by a polynomial function. Thus, a basis for the space C[0, 1] is {x j 1 } j 1.
30 Genenerating a random function in F Goal: generate a random element of F Random linear combination of basis functions: Let {b j } j 1 be a basis for F Choose finite truncation J Generate random loadings λ 1,..., λ J, e.g., i.i.d. normal Define f = J j=1 λ jb j, i.e., f (x) = J j=1 λ jb j (x)
31 Ex. Fourier basis A Fourier basis has the form cos ( ) πx 2 b j (x) = } sin { (j+1)πx 2 if j even if j odd Dense in L 2 [0, 1] (square integrable functions on [0, 1]) Go over fourierrando.py
32 In class example Dependent spatial binary data (on board)
Monte Carlo simulation inspired by computational optimization. Colin Fox Al Parker, John Bardsley MCQMC Feb 2012, Sydney
Monte Carlo simulation inspired by computational optimization Colin Fox fox@physics.otago.ac.nz Al Parker, John Bardsley MCQMC Feb 2012, Sydney Sampling from π(x) Consider : x is high-dimensional (10 4
More informationCourse Notes: Week 1
Course Notes: Week 1 Math 270C: Applied Numerical Linear Algebra 1 Lecture 1: Introduction (3/28/11) We will focus on iterative methods for solving linear systems of equations (and some discussion of eigenvalues
More informationScientific Computing
Scientific Computing Direct solution methods Martin van Gijzen Delft University of Technology October 3, 2018 1 Program October 3 Matrix norms LU decomposition Basic algorithm Cost Stability Pivoting Pivoting
More informationNumerical Methods I Non-Square and Sparse Linear Systems
Numerical Methods I Non-Square and Sparse Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 25th, 2014 A. Donev (Courant
More informationIntroduction, basic but important concepts
Introduction, basic but important concepts Felix Kubler 1 1 DBF, University of Zurich and Swiss Finance Institute October 7, 2017 Felix Kubler Comp.Econ. Gerzensee, Ch1 October 7, 2017 1 / 31 Economics
More informationSOLVING LINEAR SYSTEMS
SOLVING LINEAR SYSTEMS We want to solve the linear system a, x + + a,n x n = b a n, x + + a n,n x n = b n This will be done by the method used in beginning algebra, by successively eliminating unknowns
More informationLU Factorization. LU factorization is the most common way of solving linear systems! Ax = b LUx = b
AM 205: lecture 7 Last time: LU factorization Today s lecture: Cholesky factorization, timing, QR factorization Reminder: assignment 1 due at 5 PM on Friday September 22 LU Factorization LU factorization
More information1 Lecture 8: Interpolating polynomials.
1 Lecture 8: Interpolating polynomials. 1.1 Horner s method Before turning to the main idea of this part of the course, we consider how to evaluate a polynomial. Recall that a polynomial is an expression
More informationConsider the following example of a linear system:
LINEAR SYSTEMS Consider the following example of a linear system: Its unique solution is x + 2x 2 + 3x 3 = 5 x + x 3 = 3 3x + x 2 + 3x 3 = 3 x =, x 2 = 0, x 3 = 2 In general we want to solve n equations
More informationPower System Analysis Prof. A. K. Sinha Department of Electrical Engineering Indian Institute of Technology, Kharagpur. Lecture - 21 Power Flow VI
Power System Analysis Prof. A. K. Sinha Department of Electrical Engineering Indian Institute of Technology, Kharagpur Lecture - 21 Power Flow VI (Refer Slide Time: 00:57) Welcome to lesson 21. In this
More informationFast Multipole Methods: Fundamentals & Applications. Ramani Duraiswami Nail A. Gumerov
Fast Multipole Methods: Fundamentals & Applications Ramani Duraiswami Nail A. Gumerov Week 1. Introduction. What are multipole methods and what is this course about. Problems from physics, mathematics,
More informationApplications of Randomized Methods for Decomposing and Simulating from Large Covariance Matrices
Applications of Randomized Methods for Decomposing and Simulating from Large Covariance Matrices Vahid Dehdari and Clayton V. Deutsch Geostatistical modeling involves many variables and many locations.
More informationLINEAR SYSTEMS (11) Intensive Computation
LINEAR SYSTEMS () Intensive Computation 27-8 prof. Annalisa Massini Viviana Arrigoni EXACT METHODS:. GAUSSIAN ELIMINATION. 2. CHOLESKY DECOMPOSITION. ITERATIVE METHODS:. JACOBI. 2. GAUSS-SEIDEL 2 CHOLESKY
More informationLecture for Week 2 (Secs. 1.3 and ) Functions and Limits
Lecture for Week 2 (Secs. 1.3 and 2.2 2.3) Functions and Limits 1 First let s review what a function is. (See Sec. 1 of Review and Preview.) The best way to think of a function is as an imaginary machine,
More informationProcess Model Formulation and Solution, 3E4
Process Model Formulation and Solution, 3E4 Section B: Linear Algebraic Equations Instructor: Kevin Dunn dunnkg@mcmasterca Department of Chemical Engineering Course notes: Dr Benoît Chachuat 06 October
More informationGAUSSIAN ELIMINATION AND LU DECOMPOSITION (SUPPLEMENT FOR MA511)
GAUSSIAN ELIMINATION AND LU DECOMPOSITION (SUPPLEMENT FOR MA511) D. ARAPURA Gaussian elimination is the go to method for all basic linear classes including this one. We go summarize the main ideas. 1.
More informationMatrix decompositions
Matrix decompositions How can we solve Ax = b? 1 Linear algebra Typical linear system of equations : x 1 x +x = x 1 +x +9x = 0 x 1 +x x = The variables x 1, x, and x only appear as linear terms (no powers
More informationMonte-Carlo MMD-MA, Université Paris-Dauphine. Xiaolu Tan
Monte-Carlo MMD-MA, Université Paris-Dauphine Xiaolu Tan tan@ceremade.dauphine.fr Septembre 2015 Contents 1 Introduction 1 1.1 The principle.................................. 1 1.2 The error analysis
More informationLecture 12 (Tue, Mar 5) Gaussian elimination and LU factorization (II)
Math 59 Lecture 2 (Tue Mar 5) Gaussian elimination and LU factorization (II) 2 Gaussian elimination - LU factorization For a general n n matrix A the Gaussian elimination produces an LU factorization if
More informationNumerical Linear Algebra
Numerical Linear Algebra Decompositions, numerical aspects Gerard Sleijpen and Martin van Gijzen September 27, 2017 1 Delft University of Technology Program Lecture 2 LU-decomposition Basic algorithm Cost
More informationLecture 3. Big-O notation, more recurrences!!
Lecture 3 Big-O notation, more recurrences!! Announcements! HW1 is posted! (Due Friday) See Piazza for a list of HW clarifications First recitation section was this morning, there s another tomorrow (same
More informationProgram Lecture 2. Numerical Linear Algebra. Gaussian elimination (2) Gaussian elimination. Decompositions, numerical aspects
Numerical Linear Algebra Decompositions, numerical aspects Program Lecture 2 LU-decomposition Basic algorithm Cost Stability Pivoting Cholesky decomposition Sparse matrices and reorderings Gerard Sleijpen
More informationNext topics: Solving systems of linear equations
Next topics: Solving systems of linear equations 1 Gaussian elimination (today) 2 Gaussian elimination with partial pivoting (Week 9) 3 The method of LU-decomposition (Week 10) 4 Iterative techniques:
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 19: Computing the SVD; Sparse Linear Systems Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationMassachusetts Institute of Technology Department of Electrical Engineering and Computer Science Algorithms For Inference Fall 2014
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 2014 Problem Set 3 Issued: Thursday, September 25, 2014 Due: Thursday,
More informationSolving linear systems (6 lectures)
Chapter 2 Solving linear systems (6 lectures) 2.1 Solving linear systems: LU factorization (1 lectures) Reference: [Trefethen, Bau III] Lecture 20, 21 How do you solve Ax = b? (2.1.1) In numerical linear
More informationLecture 10: Powers of Matrices, Difference Equations
Lecture 10: Powers of Matrices, Difference Equations Difference Equations A difference equation, also sometimes called a recurrence equation is an equation that defines a sequence recursively, i.e. each
More informationNumerical Methods I: Numerical linear algebra
1/3 Numerical Methods I: Numerical linear algebra Georg Stadler Courant Institute, NYU stadler@cimsnyuedu September 1, 017 /3 We study the solution of linear systems of the form Ax = b with A R n n, x,
More informationLinear System of Equations
Linear System of Equations Linear systems are perhaps the most widely applied numerical procedures when real-world situation are to be simulated. Example: computing the forces in a TRUSS. F F 5. 77F F.
More informationCHAPTER 11. A Revision. 1. The Computers and Numbers therein
CHAPTER A Revision. The Computers and Numbers therein Traditional computer science begins with a finite alphabet. By stringing elements of the alphabet one after another, one obtains strings. A set of
More informationCSE 160 Lecture 13. Numerical Linear Algebra
CSE 16 Lecture 13 Numerical Linear Algebra Announcements Section will be held on Friday as announced on Moodle Midterm Return 213 Scott B Baden / CSE 16 / Fall 213 2 Today s lecture Gaussian Elimination
More informationIterative Methods. Splitting Methods
Iterative Methods Splitting Methods 1 Direct Methods Solving Ax = b using direct methods. Gaussian elimination (using LU decomposition) Variants of LU, including Crout and Doolittle Other decomposition
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725 Consider Last time: proximal Newton method min x g(x) + h(x) where g, h convex, g twice differentiable, and h simple. Proximal
More informationLecture Note 2: The Gaussian Elimination and LU Decomposition
MATH 5330: Computational Methods of Linear Algebra Lecture Note 2: The Gaussian Elimination and LU Decomposition The Gaussian elimination Xianyi Zeng Department of Mathematical Sciences, UTEP The method
More informationLU Factorization. Marco Chiarandini. DM559 Linear and Integer Programming. Department of Mathematics & Computer Science University of Southern Denmark
DM559 Linear and Integer Programming LU Factorization Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark [Based on slides by Lieven Vandenberghe, UCLA] Outline
More informationChapter 7. Iterative methods for large sparse linear systems. 7.1 Sparse matrix algebra. Large sparse matrices
Chapter 7 Iterative methods for large sparse linear systems In this chapter we revisit the problem of solving linear systems of equations, but now in the context of large sparse systems. The price to pay
More informationGaussian Elimination and Back Substitution
Jim Lambers MAT 610 Summer Session 2009-10 Lecture 4 Notes These notes correspond to Sections 31 and 32 in the text Gaussian Elimination and Back Substitution The basic idea behind methods for solving
More informationApproximating a single component of the solution to a linear system
Approximating a single component of the solution to a linear system Christina E. Lee, Asuman Ozdaglar, Devavrat Shah celee@mit.edu asuman@mit.edu devavrat@mit.edu MIT LIDS 1 How do I compare to my competitors?
More informationReview of matrices. Let m, n IN. A rectangle of numbers written like A =
Review of matrices Let m, n IN. A rectangle of numbers written like a 11 a 12... a 1n a 21 a 22... a 2n A =...... a m1 a m2... a mn where each a ij IR is called a matrix with m rows and n columns or an
More informationMAA507, Power method, QR-method and sparse matrix representation.
,, and representation. February 11, 2014 Lecture 7: Overview, Today we will look at:.. If time: A look at representation and fill in. Why do we need numerical s? I think everyone have seen how time consuming
More informationCS 4407 Algorithms Lecture 3: Iterative and Divide and Conquer Algorithms
CS 4407 Algorithms Lecture 3: Iterative and Divide and Conquer Algorithms Prof. Gregory Provan Department of Computer Science University College Cork 1 Lecture Outline CS 4407, Algorithms Growth Functions
More informationEAD 115. Numerical Solution of Engineering and Scientific Problems. David M. Rocke Department of Applied Science
EAD 115 Numerical Solution of Engineering and Scientific Problems David M. Rocke Department of Applied Science Taylor s Theorem Can often approximate a function by a polynomial The error in the approximation
More information6. Iterative Methods for Linear Systems. The stepwise approach to the solution...
6 Iterative Methods for Linear Systems The stepwise approach to the solution Miriam Mehl: 6 Iterative Methods for Linear Systems The stepwise approach to the solution, January 18, 2013 1 61 Large Sparse
More information9 - Markov processes and Burt & Allison 1963 AGEC
This document was generated at 8:37 PM on Saturday, March 17, 2018 Copyright 2018 Richard T. Woodward 9 - Markov processes and Burt & Allison 1963 AGEC 642-2018 I. What is a Markov Chain? A Markov chain
More informationJACOBI S ITERATION METHOD
ITERATION METHODS These are methods which compute a sequence of progressively accurate iterates to approximate the solution of Ax = b. We need such methods for solving many large linear systems. Sometimes
More informationNumerical Methods I Solving Square Linear Systems: GEM and LU factorization
Numerical Methods I Solving Square Linear Systems: GEM and LU factorization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 18th,
More informationStat 206: Linear algebra
Stat 206: Linear algebra James Johndrow (adapted from Iain Johnstone s notes) 2016-11-02 Vectors We have already been working with vectors, but let s review a few more concepts. The inner product of two
More information7.2 Linear equation systems. 7.3 Linear least square fit
72 Linear equation systems In the following sections, we will spend some time to solve linear systems of equations This is a tool that will come in handy in many di erent places during this course For
More informationThings we can already do with matrices. Unit II - Matrix arithmetic. Defining the matrix product. Things that fail in matrix arithmetic
Unit II - Matrix arithmetic matrix multiplication matrix inverses elementary matrices finding the inverse of a matrix determinants Unit II - Matrix arithmetic 1 Things we can already do with matrices equality
More informationGaussian Quiz. Preamble to The Humble Gaussian Distribution. David MacKay 1
Preamble to The Humble Gaussian Distribution. David MacKay Gaussian Quiz H y y y 3. Assuming that the variables y, y, y 3 in this belief network have a joint Gaussian distribution, which of the following
More informationST495: Survival Analysis: Hypothesis testing and confidence intervals
ST495: Survival Analysis: Hypothesis testing and confidence intervals Eric B. Laber Department of Statistics, North Carolina State University April 3, 2014 I remember that one fateful day when Coach took
More informationMath 471 (Numerical methods) Chapter 3 (second half). System of equations
Math 47 (Numerical methods) Chapter 3 (second half). System of equations Overlap 3.5 3.8 of Bradie 3.5 LU factorization w/o pivoting. Motivation: ( ) A I Gaussian Elimination (U L ) where U is upper triangular
More informationExample: Current in an Electrical Circuit. Solving Linear Systems:Direct Methods. Linear Systems of Equations. Solving Linear Systems: Direct Methods
Example: Current in an Electrical Circuit Solving Linear Systems:Direct Methods A number of engineering problems or models can be formulated in terms of systems of equations Examples: Electrical Circuit
More informationAn Introduction to NeRDS (Nearly Rank Deficient Systems)
(Nearly Rank Deficient Systems) BY: PAUL W. HANSON Abstract I show that any full rank n n matrix may be decomposento the sum of a diagonal matrix and a matrix of rank m where m < n. This decomposition
More informationIntroduction - Motivation. Many phenomena (physical, chemical, biological, etc.) are model by differential equations. f f(x + h) f(x) (x) = lim
Introduction - Motivation Many phenomena (physical, chemical, biological, etc.) are model by differential equations. Recall the definition of the derivative of f(x) f f(x + h) f(x) (x) = lim. h 0 h Its
More informationLinear Algebra and Eigenproblems
Appendix A A Linear Algebra and Eigenproblems A working knowledge of linear algebra is key to understanding many of the issues raised in this work. In particular, many of the discussions of the details
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 2: Direct Methods PD Dr.
More informationECS289: Scalable Machine Learning
ECS289: Scalable Machine Learning Cho-Jui Hsieh UC Davis Sept 27, 2015 Outline Linear regression Ridge regression and Lasso Time complexity (closed form solution) Iterative Solvers Regression Input: training
More informationChapter 9: Gaussian Elimination
Uchechukwu Ofoegbu Temple University Chapter 9: Gaussian Elimination Graphical Method The solution of a small set of simultaneous equations, can be obtained by graphing them and determining the location
More informationThe Kernel Trick, Gram Matrices, and Feature Extraction. CS6787 Lecture 4 Fall 2017
The Kernel Trick, Gram Matrices, and Feature Extraction CS6787 Lecture 4 Fall 2017 Momentum for Principle Component Analysis CS6787 Lecture 3.1 Fall 2017 Principle Component Analysis Setting: find the
More informationMatrix decompositions
Matrix decompositions How can we solve Ax = b? 1 Linear algebra Typical linear system of equations : x 1 x +x = x 1 +x +9x = 0 x 1 +x x = The variables x 1, x, and x only appear as linear terms (no powers
More informationBasic Linear Algebra. Florida State University. Acknowledgements: Daniele Panozzo. CAP Computer Graphics - Fall 18 Xifeng Gao
Basic Linear Algebra Acknowledgements: Daniele Panozzo Overview We will briefly overview the basic linear algebra concepts that we will need in the class You will not be able to follow the next lectures
More informationSparse Linear Systems. Iterative Methods for Sparse Linear Systems. Motivation for Studying Sparse Linear Systems. Partial Differential Equations
Sparse Linear Systems Iterative Methods for Sparse Linear Systems Matrix Computations and Applications, Lecture C11 Fredrik Bengzon, Robert Söderlund We consider the problem of solving the linear system
More informationP, NP, NP-Complete, and NPhard
P, NP, NP-Complete, and NPhard Problems Zhenjiang Li 21/09/2011 Outline Algorithm time complicity P and NP problems NP-Complete and NP-Hard problems Algorithm time complicity Outline What is this course
More informationMachine Learning and Computational Statistics, Spring 2017 Homework 2: Lasso Regression
Machine Learning and Computational Statistics, Spring 2017 Homework 2: Lasso Regression Due: Monday, February 13, 2017, at 10pm (Submit via Gradescope) Instructions: Your answers to the questions below,
More informationSolving PDEs with CUDA Jonathan Cohen
Solving PDEs with CUDA Jonathan Cohen jocohen@nvidia.com NVIDIA Research PDEs (Partial Differential Equations) Big topic Some common strategies Focus on one type of PDE in this talk Poisson Equation Linear
More informationSimulation. Where real stuff starts
1 Simulation Where real stuff starts ToC 1. What is a simulation? 2. Accuracy of output 3. Random Number Generators 4. How to sample 5. Monte Carlo 6. Bootstrap 2 1. What is a simulation? 3 What is a simulation?
More information, and rewards and transition matrices as shown below:
CSE 50a. Assignment 7 Out: Tue Nov Due: Thu Dec Reading: Sutton & Barto, Chapters -. 7. Policy improvement Consider the Markov decision process (MDP) with two states s {0, }, two actions a {0, }, discount
More informationComputational statistics
Computational statistics Markov Chain Monte Carlo methods Thierry Denœux March 2017 Thierry Denœux Computational statistics March 2017 1 / 71 Contents of this chapter When a target density f can be evaluated
More informationPolynomial accelerated MCMC... and other sampling algorithms inspired by computational optimization
Polynomial accelerated MCMC... and other sampling algorithms inspired by computational optimization Colin Fox fox@physics.otago.ac.nz Al Parker, John Bardsley Fore-words Motivation: an inverse oceanography
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization /36-725
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: proximal gradient descent Consider the problem min g(x) + h(x) with g, h convex, g differentiable, and h simple
More informationOR MSc Maths Revision Course
OR MSc Maths Revision Course Tom Byrne School of Mathematics University of Edinburgh t.m.byrne@sms.ed.ac.uk 15 September 2017 General Information Today JCMB Lecture Theatre A, 09:30-12:30 Mathematics revision
More informationAM 205: lecture 6. Last time: finished the data fitting topic Today s lecture: numerical linear algebra, LU factorization
AM 205: lecture 6 Last time: finished the data fitting topic Today s lecture: numerical linear algebra, LU factorization Unit II: Numerical Linear Algebra Motivation Almost everything in Scientific Computing
More information5.1 Banded Storage. u = temperature. The five-point difference operator. uh (x, y + h) 2u h (x, y)+u h (x, y h) uh (x + h, y) 2u h (x, y)+u h (x h, y)
5.1 Banded Storage u = temperature u= u h temperature at gridpoints u h = 1 u= Laplace s equation u= h u = u h = grid size u=1 The five-point difference operator 1 u h =1 uh (x + h, y) 2u h (x, y)+u h
More information9. Numerical linear algebra background
Convex Optimization Boyd & Vandenberghe 9. Numerical linear algebra background matrix structure and algorithm complexity solving linear equations with factored matrices LU, Cholesky, LDL T factorization
More informationPerform the same three operations as above on the values in the matrix, where some notation is given as a shorthand way to describe each operation:
SECTION 2.1: SOLVING SYSTEMS OF EQUATIONS WITH A UNIQUE SOLUTION In Chapter 1 we took a look at finding the intersection point of two lines on a graph. Chapter 2 begins with a look at a more formal approach
More informationCSE332: Data Structures & Parallelism Lecture 2: Algorithm Analysis. Ruth Anderson Winter 2018
CSE332: Data Structures & Parallelism Lecture 2: Algorithm Analysis Ruth Anderson Winter 2018 Today Algorithm Analysis What do we care about? How to compare two algorithms Analyzing Code Asymptotic Analysis
More informationarxiv: v1 [cs.sc] 17 Apr 2013
EFFICIENT CALCULATION OF DETERMINANTS OF SYMBOLIC MATRICES WITH MANY VARIABLES TANYA KHOVANOVA 1 AND ZIV SCULLY 2 arxiv:1304.4691v1 [cs.sc] 17 Apr 2013 Abstract. Efficient matrix determinant calculations
More informationApproximation. Inderjit S. Dhillon Dept of Computer Science UT Austin. SAMSI Massive Datasets Opening Workshop Raleigh, North Carolina.
Using Quadratic Approximation Inderjit S. Dhillon Dept of Computer Science UT Austin SAMSI Massive Datasets Opening Workshop Raleigh, North Carolina Sept 12, 2012 Joint work with C. Hsieh, M. Sustik and
More informationCSE332: Data Structures & Parallelism Lecture 2: Algorithm Analysis. Ruth Anderson Winter 2018
CSE332: Data Structures & Parallelism Lecture 2: Algorithm Analysis Ruth Anderson Winter 2018 Today Algorithm Analysis What do we care about? How to compare two algorithms Analyzing Code Asymptotic Analysis
More informationMath 304 (Spring 2010) - Lecture 2
Math 304 (Spring 010) - Lecture Emre Mengi Department of Mathematics Koç University emengi@ku.edu.tr Lecture - Floating Point Operation Count p.1/10 Efficiency of an algorithm is determined by the total
More informationQualifying Examination
Summer 24 Day. Monday, September 5, 24 You have three hours to complete this exam. Work all problems. Start each problem on a All problems are 2 points. Please email any electronic files in support of
More informationLecture 5: Linear models for classification. Logistic regression. Gradient Descent. Second-order methods.
Lecture 5: Linear models for classification. Logistic regression. Gradient Descent. Second-order methods. Linear models for classification Logistic regression Gradient descent and second-order methods
More informationReview from Bootcamp: Linear Algebra
Review from Bootcamp: Linear Algebra D. Alex Hughes October 27, 2014 1 Properties of Estimators 2 Linear Algebra Addition and Subtraction Transpose Multiplication Cross Product Trace 3 Special Matrices
More information9.1 Preconditioned Krylov Subspace Methods
Chapter 9 PRECONDITIONING 9.1 Preconditioned Krylov Subspace Methods 9.2 Preconditioned Conjugate Gradient 9.3 Preconditioned Generalized Minimal Residual 9.4 Relaxation Method Preconditioners 9.5 Incomplete
More informationLinear Models for Regression CS534
Linear Models for Regression CS534 Example Regression Problems Predict housing price based on House size, lot size, Location, # of rooms Predict stock price based on Price history of the past month Predict
More informationCS123 INTRODUCTION TO COMPUTER GRAPHICS. Linear Algebra /34
Linear Algebra /34 Vectors A vector is a magnitude and a direction Magnitude = v Direction Also known as norm, length Represented by unit vectors (vectors with a length of 1 that point along distinct axes)
More informationNumerical Analysis: Solutions of System of. Linear Equation. Natasha S. Sharma, PhD
Mathematical Question we are interested in answering numerically How to solve the following linear system for x Ax = b? where A is an n n invertible matrix and b is vector of length n. Notation: x denote
More informationCSE332: Data Structures & Parallelism Lecture 2: Algorithm Analysis. Ruth Anderson Winter 2019
CSE332: Data Structures & Parallelism Lecture 2: Algorithm Analysis Ruth Anderson Winter 2019 Today Algorithm Analysis What do we care about? How to compare two algorithms Analyzing Code Asymptotic Analysis
More informationCS 4407 Algorithms Lecture 2: Iterative and Divide and Conquer Algorithms
CS 4407 Algorithms Lecture 2: Iterative and Divide and Conquer Algorithms Prof. Gregory Provan Department of Computer Science University College Cork 1 Lecture Outline CS 4407, Algorithms Growth Functions
More information(x 1 +x 2 )(x 1 x 2 )+(x 2 +x 3 )(x 2 x 3 )+(x 3 +x 1 )(x 3 x 1 ).
CMPSCI611: Verifying Polynomial Identities Lecture 13 Here is a problem that has a polynomial-time randomized solution, but so far no poly-time deterministic solution. Let F be any field and let Q(x 1,...,
More informationMath Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88
Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant
More informationAM 205: lecture 6. Last time: finished the data fitting topic Today s lecture: numerical linear algebra, LU factorization
AM 205: lecture 6 Last time: finished the data fitting topic Today s lecture: numerical linear algebra, LU factorization Unit II: Numerical Linear Algebra Motivation Almost everything in Scientific Computing
More information(f(x) P 3 (x)) dx. (a) The Lagrange formula for the error is given by
1. QUESTION (a) Given a nth degree Taylor polynomial P n (x) of a function f(x), expanded about x = x 0, write down the Lagrange formula for the truncation error, carefully defining all its elements. How
More informationMathematical Optimisation, Chpt 2: Linear Equations and inequalities
Mathematical Optimisation, Chpt 2: Linear Equations and inequalities Peter J.C. Dickinson p.j.c.dickinson@utwente.nl http://dickinson.website version: 12/02/18 Monday 5th February 2018 Peter J.C. Dickinson
More informationDesigning Information Devices and Systems I Fall 2018 Lecture Notes Note Introduction to Linear Algebra the EECS Way
EECS 16A Designing Information Devices and Systems I Fall 018 Lecture Notes Note 1 1.1 Introduction to Linear Algebra the EECS Way In this note, we will teach the basics of linear algebra and relate it
More informationSparsity-Preserving Difference of Positive Semidefinite Matrix Representation of Indefinite Matrices
Sparsity-Preserving Difference of Positive Semidefinite Matrix Representation of Indefinite Matrices Jaehyun Park June 1 2016 Abstract We consider the problem of writing an arbitrary symmetric matrix as
More information9. Numerical linear algebra background
Convex Optimization Boyd & Vandenberghe 9. Numerical linear algebra background matrix structure and algorithm complexity solving linear equations with factored matrices LU, Cholesky, LDL T factorization
More information1 What is numerical analysis and scientific computing?
Mathematical preliminaries 1 What is numerical analysis and scientific computing? Numerical analysis is the study of algorithms that use numerical approximation (as opposed to general symbolic manipulations)
More informationSTA 294: Stochastic Processes & Bayesian Nonparametrics
MARKOV CHAINS AND CONVERGENCE CONCEPTS Markov chains are among the simplest stochastic processes, just one step beyond iid sequences of random variables. Traditionally they ve been used in modelling a
More information