Recursive Estimation
|
|
- Cody Bailey
- 6 years ago
- Views:
Transcription
1 Recursive Estimation Raffaello D Andrea Spring 07 Problem Set 3: Etracting Estimates from Probability Distributions Last updated: April 5, 07 Notes: Notation: Unlessotherwisenoted,, y,andz denoterandomvariables, f )ortheshort hand f)) denotes the probability density function of, and f y y) or f y)) denotes the conditional probability density function of conditioned on y. The epected value is denoted by E[ ], the variance is denoted by Var[ ], and PrZ) denotes the probability that the event Z occurs. A normally distributed random variable with mean µ and variance σ is denoted by N µ,σ ). Please report any errors found in this problem set to the teaching assistants michaemu@ethz.ch or lhewing@ethz.ch).
2 Problem Set Problem Let and w be scalar CRVs, where,w are independent: f,w) = f)fw), and w [,] is uniformly distributed. Let z = +3w. a) Calculate fz ) using the change of variables. b) Let =. Verify that Prz [,4] = ) = Prw [0,] = ). Problem Let w R and R 4 be CRVs, where and w are independent: f,w) = f)fw). The two elements of the measurement noise vector w are w,w, and the PDF of w is uniform: f w w) = f w,w w,w ) = The measurement model is where z = H+Gw H = /4 for w and w 0 otherwise. [ ] [ ], G = Calculate the conditional joint PDF f z z ) using the change of variables.. Problem 3 Find the maimum likelihood ML) estimate ˆ ML := argmafz ) when z i = +w i, i =, where w i are independent with w i N 0,σ i) and zi,w i, R. Problem 4 Solve Problem 3 if the joint PDF of w is fw) = fw,w ) = /4 for w and w 0 otherwise.
3 Problem 5 Let R n be an unknown, constant parameter. Furthermore, let w R m, n m, represent the measurement noise with its joint PDF defined by the multivariate Gaussian distribution f w w) = π) m / detσ)) ep ) / wt Σ w with mean E[w] = 0 and variance E[ww T ] =: Σ = Σ T R m m. Assume that and w are independent. Let the measurement be z = H+w where the matri H R m n has full column rank. a) Calculate the ML estimate ˆ ML := argmafz ) for a diagonal variance σ 0 0 Σ = 0 σ σm Note that in this case, the elements w i of w are mutually independent. b) Calculate the ML estimate for non-diagonal Σ, which implies that the elements w i of w are correlated. Hint: The following matri calculus is useful for this problem see, for eample, the Matri Cookbook for reference): Recall the definition of a Jacobian matri from the lecture: Let g) : R n R m be a function that maps the vector R n to another vector g) R m. The Jacobian matri is defined as g) := g ) g ). g m) g ) g ).. g m) g ) n g ) n. g m) n Rm n where j is the j-th element of and g i ) the i-th element of g). Derivative of a quadratic form: Chain rule: T A = T A+A T) R n, A R n n. u = g), R n,u R n ut Au = u T A+A T) u, u = g) Rn n. 3
4 Problem 6 Find the maimum a posteriori MAP) estimate ˆ MAP := argmafz )f) when z = +w,,z,w R where is eponentially distributed, 0 for < 0 f) = e for 0 and w N0, ). Problem 7 In the derivation of the recursive least squares algorithm, we used the following result: trace ABA T) = AB if B = B T A where trace ) is the sum of the diagonal elements of a matri. Prove that the above holds for the two-dimensional case, i.e. A,B R. Problem 8 The derivation of the recursive least squares algorithm also used the following result: traceab) A = B T Prove that the above holds for A,B R. Problem 9 Consider the recursive least squares algorithm presented in lecture, where the estimation error at step k was shown to be ) ek) = I Kk)Hk) ek ) Kk)wk). Furthermore, E[wk)] = 0, E[ek)] = 0, and, w),...} are mutually independent. Show that the measurement error variance Pk) = Var[ek)] = E[ek)e T k)] is given by ) T Pk) = I Kk)Hk) Pk ) I Kk)Hk)) +Kk)Rk)K T k) where Rk) = Var[wk)]. Problem 0 Use the results from Problems 7 and 8 to derive the gain matri Kk) for the recursive least squares algorithm that minimizes the mean squared error MMSE) by solving 0 = ) T Kk) trace I Kk)Hk)) Pk ) I Kk)Hk)) +Kk)Rk)K T k). 4
5 Problem Apply the recursive least squares algorithm when where zk) = +wk), k =,,... ) E[] = 0, Var[] =, and E[wk)] = 0, Var[wk)] =. ) The random variables,w),w),...} are mutually independent and zk),wk), R. a) Epress the gain matri Kk) and variance Pk) as eplicit functions of k. b) Simulate the algorithm for normally distributed continuous random variables wk) and with the above properties ). c) Simulate the algorithm for and wk) being discrete random variables that take the values and with equal probability. Note that and wk) satisfy ). Simulate for values of k up to Problem Consider the random variables and measurement equation from Problem c), i.e. zk) = +wk), k =,,... 3) for mutually independent and wk) that take the values and with equal probability. a) WhatistheMMSEestimateofattimek givenallpastmeasurementsz),z),...,zk)? Remember that the MMSE estimate is given by [ ˆ MMSE k) := argmine ˆ ) ] z),z),...,zk). ˆ b) Now consider the MMSE estimate that is restricted to the sample space of, i.e. [ ˆ MMSE k) := argmin E ˆ ) ] z),z),...,zk) ˆ X where X denotes the sample space of. Calculate the estimate of at time k given all past measurements z),z),...,zk). 5
6 Sample solutions Problem a) The change of variables formula for conditional PDFs when z = g,w) is f z z ) = f w hz,) ) g. w,hz,)) where w = hz,) is the unique solution to z = g,w). We solve z = g,w) for w to find hz,) w = z 3 = hz,) and apply the change of variables: f z z ) = f w hz,) ) g w,hz,)) = f w 3 z ) ) 3 = 3 f w ) z ). 3 With the given uniform PDF of w, the PDF of fz ) can then be stated eplicitely: 6 for 3 z +3 f z z ) = 0 otherwise b) We begin by computing Prz [,4] = ): Prz [,4] = ) = 4 The probability Prw [0,] = ) is f z z = )dz = Prw [0,] = ) = Prw [0,]) = which is indeed identical to Prz [,4] = ) dz =. f w w)dw = 0 dw = Problem Analogous to the lecture notes, the multivariate change of variables formula for conditional PDFs is f z z ) = f w hz,) ) detg) where w = hz,) is the unique solution to z = H+Gw. The function hz,) is given by [ ] [ ] ) w = G z H) = 0 z [ ] z = ) 3 z 3 ) 6
7 where we used =,, 3, 4 ) and z = z,z ). Applying the change of variables, it follows that f z z ) = f w hz,) ) detg) = f w G z H) ) 3 = 3 f w Using the PDF f w w), this can then be epressed as f z z ) = z ), ) 3 z 3 ). 6 for z + and 3 +3 z otherwise. Problem 3 For a normally-distributed random variable with w i N 0,σ i), we have fw i ) ep w i ) σi. Using the change of variables, and by the conditional independence of z,z given, we can write )) fz,z ) = fz )fz ) ep z ) σ + z ) σ. Differentiating with respect to, and setting to 0, leads to fz,z ) z ˆ ML) z ˆ ML) = 0 + =ˆ ML σ σ = 0 ˆ ML = z σ +z σ σ +σ which is a weighted sum of the measurements. Note that for σ = 0 : ˆ ML = z, for σ = 0 : ˆ ML = z. Problem 4 Using the total probability theorem, we find that fw i ) = for w i [,] 0 otherwise. and fw,w ) = fw )fw ). Using the change of variables, fz i ) = if z i 0 otherwise. We need to solve for fz,z ) = fz )fz ). Consider the different cases: z z >, for which the likelihoods do not overlap, see the following figure: 7
8 fz ) z fz ) z Since no value for can eplain both, z and z, given the model z i = +w i, we find fz,z ) = 0. z z, where the likelihoods overlap, which is shown in the following figure: fz,z ) fz ) fz ) 4 z z z z We find fz,z ) = 4 for [z,z +] [z,z +] 0 otherwise where A B denotes the intersection of the sets A and B. Therefore the estimate is ˆ ML [z,z +] [z,z +]. All values in this range are equally likely, and any value in the range is therefore a valid maimum likelihood estimate ˆ ML. Problem 5 a) We apply the multivariate change of coordinates: f z z ) = f w hz,) ) detg) = f w z H ) = f w z H) = π) m / detσ)) ep ) / z H)T Σ z H) We proceed similarly as in class, for z,w R m and R n : H = H H. H m, H i = [ ] h i h i... h in, where hij R. Using fz ) ep m )) z i H i ) i= σ i. 8
9 differentiating with respect to j, and setting to zero yields m z i H iˆ ML h ij = 0, i= σ i [ ] h j h mj }} column of H From which we have and H T W z Hˆ ML) = 0 σ ˆ ML = H T WH ) H T Wz. σ... 0 σm j =,,...,n } } W 0 z Hˆ ML ) = 0, j =,...,n. Note that this can also be interpreted as a weighted least squares, w) = z H ˆ = argmin w) T Ww) where W is a weighting of the measurements. The result in both cases is the same! b) We find the maimizing ˆ ML by setting the derivative of fz ) with respect to to zero. With the given facts about matri calculus, we find 0 = fz ) ep ) z H)T Σ z H) where we use the fact that Σ ) T = Σ T ) = Σ. z H)T Σ + Σ ) T ) z H) = z H) T Σ H) Finally, z H) T Σ H = 0 z T Σ H = T H T Σ H and after we transpose the left and right hand sides, we find H T Σ z = H T Σ H. It follows that ˆ ML = H T Σ H ) H T Σ z. 9
10 Problem 6 We calculate the conditional probability density function, fz ) = ep ) z ). π Both the measurement likelihood fz ) and the prior f) are shown in the following figure f) fz ) z We find fz )f) = 0 for < 0 π ep )ep z )) for 0. We calculate the MAP estimate ˆ MAP that maimizes the above function for a given z. We note that fz )f) > 0 for 0, and the maimizing value is therefore ˆ MAP 0. Since the function is discontinuous at = 0, we have to consider two cases:. The maimum is at the boundary, ˆ MAP = 0. It follows that f z ˆ MAP ) ) f MAP ˆ = ep π z ). 4). The maimum is where the derivative of the likelihood fz )f) with respect to is zero, i.e. 0 = fz )f)) from which follows ˆ MAP = z =ˆ MAP = + z ˆ MAP ) at the maimum. Substituting back into the likelihood, we obtain f z ˆ MAP ) ) f MAP ˆ = ep z +)ep ) = ep z + ). 5) π π We note that f z ˆ MAP ) f ˆ MAP therefore 0 for z < ˆ MAP = z for z. ) f z ˆ MAP ) f ˆ MAP ) for z, and the ML estimate is 0
11 Problem 7 We begin by evaluating the individual elements of AB) and ABA T ) with A) ij denoting the i,j)th element of matri A element in ith row and jth column): AB) ij = a i b j +a i b j ABA T ) ij = a ib +a i b )a j +a i b +a i b )a j From this we obtain the trace as trace ABA T) = ABA T) + ABA T) and the derivatives = a b +a b )a +a b +a b )a +a b +a b )a +a b +a b )a trace ABA T) = a b +a b +a a }} b = AB) =b trace ABA T) = a b +a b +a b = AB) a trace ABA T) = a b +a b +a b = AB) a trace ABA T) a = a b +a b +a b = AB) We therefore proved that trace ABA T) = AB A holds for A,B R. Problem 8 We proceed similarly to the previous problem: that is traceab) = AB) +AB) = a b +a b +a b +a b traceab) = b = B) a = B T) traceab) = b = B) a = B T) traceab) = b = B) a = B T) traceab) = b = B) a = B T) traceab) A = B T for A,B R.
12 Problem 9 By definition, Pk) = E[ek)e T k)]. Substituting the estimation error ) ek) = I Kk)Hk) ek ) Kk)wk) in the definition, we obtain ) ) ) ) ] T Pk) =E[ I Kk)Hk) ek ) Kk)wk) I Kk)Hk) ek ) Kk)wk) ) T =E[ I Kk)Hk) ek )e T k ) I Kk)Hk)) +Kk)wk)w T k)k T k) ) ) ] T I Kk)Hk) ek )w T k)k T k) Kk)wk)e T k ) I Kk)Hk) ) T = I Kk)Hk) Pk ) I Kk)Hk)) +Kk)Rk)K T k) ) I Kk)Hk) E [ ek )w T k) ] K T k) Kk)E [ wk)e T k ) ] T. I Kk)Hk)) Becauseek ) andwk)areindependent,e[wk)e T k )] = E[wk)]E[e T k )]. SinceE[wk)] = 0, E[wk)]E[e T k )] = 0, and it follows that ) T Pk) = I Kk)Hk) Pk ) I Kk)Hk)) +Kk)Rk)K T k). Problem 0 First, we epand ) T I Kk)Hk) Pk ) I Kk)Hk)) +Kk)Rk)K T k) ) = Pk ) Kk)Hk)Pk ) Pk )H T k)k T k)+kk) Hk)Pk )H T k)+rk) K T k). We use the results from problems 7 and 8 and evaluate the summands individually, since tracea+b) = tracea)+traceb). We find Kk) trace Pk ) ) = 0 Kk) trace Kk)Hk)Pk ) ) = Pk )H T k), note that Pk ) = P T k ) Kk) trace Pk )H T k)k T k) ) = Pk )H T k), note that tracea) = trace A T) ) ) ) Kk) trace Kk) Hk)Pk )H T k) +Rk) K T k) = Kk) Hk)Pk )H T k) +Rk). Note that R T k) = Rk) and Hk)Pk )H T k) ) T = Hk)Pk )H T k). Finally, ) Kk) Hk)Pk )H T k) +Rk) = Pk )H T k). Kk) = Pk )H T k) Hk)Pk )H T k) +Rk))
13 Problem a) We initialize P0) =, ˆ0) = 0, Hk) = and Rk) = k =,,..., and get: Kk) = Pk ) Pk ) + 6) Pk) = Kk)) Pk ) +Kk) = Pk ) +Pk )) + Pk ) +Pk )) = Pk ) +Pk ). 7) From 7) and P0) =, we have P) =, P) = 3,..., and we can show by induction see below) that Pk) = +k, Kk) = +k from 6). Proof by induction: Assume Pk) = +k. 8) Start with P0) = +0 =, which satisfies 8). Now we show for k+): Pk+) = = = +k+ = k+ = Pk) +Pk) +k + +k +k +) by assumption 8) q.e.d. from 7) b) The Matlab code is available on the class web page. Recursive least squares works, and provides a reasonable estimate after several thousand iterations. The recursive least squares estimator is the optimal estimator for Gaussian noise. c) The Matlab code is available on the class web page. Recursive least squares works, but we can do much better with the proposed nonlinear estimator derived in Problem. 3
14 Problem a) The MMSE estimate is ˆ MMSE k) = argmin ˆ =,) ˆ ) f z),z),...,zk)). 9) Let us first consider the MMSE estimate at time k =. The only possible combinations are: w) z) We can now construct the conditional PDF of when conditioned on z): z) f z)) / 0 / Therefore, if z) = or z) =, we know precisely: = and =, respectively. If z) = 0, it is equally likely that = or =. The MMSE estimate is for z) = : for z) = : for z) = 0: ˆ MMSE ) = since f = z) = ) = 0 ˆ MMSE ) = since f = z) = ) = 0 ˆ MMSE ) = argmin ˆ = argmin ˆ = argmin ˆ ˆ + ) = 0. f = z) = 0)ˆ ) +f = z) = 0)ˆ+) ) ˆ ) + ) ˆ+) Note that the same result could have been obtained by evaluating ˆ MMSE k) = E[ z)] = f z) = 0) = )+ ) ) = 0. =,) At time k =, we could construct a table for the conditional PDF of when conditioned on z) and z) as above. However, note that the MMSE estimate is given for all k if one measures either or at time k =, for z) =, ˆ MMSE k) = k =,,... for z) =, ˆ MMSE k) = k =,,.... 4
15 Ontheotherhand, ameasurementz) = 0doesnotprovideanyinformation, i.e. ) = and ) = stay equally likely. As soon as a subsequent measurement zk), k, has a value of or, is known precisely which is also reflected by the MMSE estimate. As long as we keep measuring 0, we do not gain additional information and ˆ MMSE k) = 0 if zl) = 0 l =,,...,k. Matlab code for this nonlinear MMSE estimator is available on the class webpage. b) The solution is identical to part a), ecept that the estimate is the minimizing value from the sample space of : ˆ MMSE k) = argmin ˆ ) f z),z),...,zk)) ˆ X. 0) =,) The same argument holds that, if one measurement is either - or, we know precisely. However, for z) = 0, the estimate is ˆ MMSE ) = argmin f = z) = 0)ˆ ) +f = z) = 0)ˆ+) ) ˆ X = argmin ˆ X = ±. ˆ ) + ) ˆ+) Note that X =,} and therefore, we cannot differentiate and set to zero theright hand side of Equation 0 to find the minimizing value. This implies that the estimate is not necessarily equal to E[ z], which is the case for measuring z = 0. In this case E[ z = 0] is equal 0, whereas the MMMSE estimator yields ± for ˆ MMSE. The remainder of the solution is identical to the above part a) in that the estimate remains ˆ MMSE = ± until a measurement is either - or. 5
Recursive Estimation
Recursive Estimation Raffaello D Andrea Spring 08 Problem Set 3: Extracting Estimates from Probability Distributions Last updated: April 9, 08 Notes: Notation: Unless otherwise noted, x, y, and z denote
More information6.4 Kalman Filter Equations
6.4 Kalman Filter Equations 6.4.1 Recap: Auxiliary variables Recall the definition of the auxiliary random variables x p k) and x m k): Init: x m 0) := x0) S1: x p k) := Ak 1)x m k 1) +uk 1) +vk 1) S2:
More information6. Vector Random Variables
6. Vector Random Variables In the previous chapter we presented methods for dealing with two random variables. In this chapter we etend these methods to the case of n random variables in the following
More informationLecture 3: Pattern Classification. Pattern classification
EE E68: Speech & Audio Processing & Recognition Lecture 3: Pattern Classification 3 4 5 The problem of classification Linear and nonlinear classifiers Probabilistic classification Gaussians, mitures and
More informationLecture Note 2: Estimation and Information Theory
Univ. of Michigan - NAME 568/EECS 568/ROB 530 Winter 2018 Lecture Note 2: Estimation and Information Theory Lecturer: Maani Ghaffari Jadidi Date: April 6, 2018 2.1 Estimation A static estimation problem
More informationCS 532: 3D Computer Vision 6 th Set of Notes
1 CS 532: 3D Computer Vision 6 th Set of Notes Instructor: Philippos Mordohai Webpage: www.cs.stevens.edu/~mordohai E-mail: Philippos.Mordohai@stevens.edu Office: Lieb 215 Lecture Outline Intro to Covariance
More information2 Statistical Estimation: Basic Concepts
Technion Israel Institute of Technology, Department of Electrical Engineering Estimation and Identification in Dynamical Systems (048825) Lecture Notes, Fall 2009, Prof. N. Shimkin 2 Statistical Estimation:
More informationMath 180A. Lecture 16 Friday May 7 th. Expectation. Recall the three main probability density functions so far (1) Uniform (2) Exponential.
Math 8A Lecture 6 Friday May 7 th Epectation Recall the three main probability density functions so far () Uniform () Eponential (3) Power Law e, ( ), Math 8A Lecture 6 Friday May 7 th Epectation Eample
More informationRecursive Estimation
Recursive Estimation Raffaello D Andrea Spring 08 Problem Set : Bayes Theorem and Bayesian Tracking Last updated: March, 08 Notes: Notation: Unless otherwise noted, x, y, and z denote random variables,
More informationProblem Set 3 Due Oct, 5
EE6: Random Processes in Systems Lecturer: Jean C. Walrand Problem Set 3 Due Oct, 5 Fall 6 GSI: Assane Gueye This problem set essentially reviews detection theory. Not all eercises are to be turned in.
More informationMACHINE LEARNING ADVANCED MACHINE LEARNING
MACHINE LEARNING ADVANCED MACHINE LEARNING Recap of Important Notions on Estimation of Probability Density Functions 2 2 MACHINE LEARNING Overview Definition pdf Definition joint, condition, marginal,
More informationCSE 559A: Computer Vision
CSE 559A: Computer Vision Fall 208: T-R: :30-pm @ Lopata 0 Instructor: Ayan Chakrabarti (ayan@wustl.edu). Course Staff: Zhihao ia, Charlie Wu, Han Liu http://www.cse.wustl.edu/~ayan/courses/cse559a/ Sep
More information3.3.1 Linear functions yet again and dot product In 2D, a homogenous linear scalar function takes the general form:
3.3 Gradient Vector and Jacobian Matri 3 3.3 Gradient Vector and Jacobian Matri Overview: Differentiable functions have a local linear approimation. Near a given point, local changes are determined by
More informationMACHINE LEARNING ADVANCED MACHINE LEARNING
MACHINE LEARNING ADVANCED MACHINE LEARNING Recap of Important Notions on Estimation of Probability Density Functions 22 MACHINE LEARNING Discrete Probabilities Consider two variables and y taking discrete
More informationWhere now? Machine Learning and Bayesian Inference
Machine Learning and Bayesian Inference Dr Sean Holden Computer Laboratory, Room FC6 Telephone etension 67 Email: sbh@clcamacuk wwwclcamacuk/ sbh/ Where now? There are some simple take-home messages from
More informationShort course A vademecum of statistical pattern recognition techniques with applications to image and video analysis. Agenda
Short course A vademecum of statistical pattern recognition techniques with applications to image and video analysis Lecture Recalls of probability theory Massimo Piccardi University of Technology, Sydney,
More informationCSE 559A: Computer Vision Tomorrow Zhihao's Office Hours back in Jolley 309: 10:30am-Noon
CSE 559A: Computer Vision ADMINISTRIVIA Tomorrow Zhihao's Office Hours back in Jolley 309: 0:30am-Noon Fall 08: T-R: :30-pm @ Lopata 0 This Friday: Regular Office Hours Net Friday: Recitation for PSET
More informationEE 302 Division 1. Homework 6 Solutions.
EE 3 Division. Homework 6 Solutions. Problem. A random variable X has probability density { C f X () e λ,,, otherwise, where λ is a positive real number. Find (a) The constant C. Solution. Because of the
More informationLecture 3: Latent Variables Models and Learning with the EM Algorithm. Sam Roweis. Tuesday July25, 2006 Machine Learning Summer School, Taiwan
Lecture 3: Latent Variables Models and Learning with the EM Algorithm Sam Roweis Tuesday July25, 2006 Machine Learning Summer School, Taiwan Latent Variable Models What to do when a variable z is always
More informationEA = I 3 = E = i=1, i k
MTH5 Spring 7 HW Assignment : Sec.., # (a) and (c), 5,, 8; Sec.., #, 5; Sec.., #7 (a), 8; Sec.., # (a), 5 The due date for this assignment is //7. Sec.., # (a) and (c). Use the proof of Theorem. to obtain
More informationBasic concepts in estimation
Basic concepts in estimation Random and nonrandom parameters Definitions of estimates ML Maimum Lielihood MAP Maimum A Posteriori LS Least Squares MMS Minimum Mean square rror Measures of quality of estimates
More informationComputation fundamentals of discrete GMRF representations of continuous domain spatial models
Computation fundamentals of discrete GMRF representations of continuous domain spatial models Finn Lindgren September 23 2015 v0.2.2 Abstract The fundamental formulas and algorithms for Bayesian spatial
More informationKalman Filtering. Namrata Vaswani. March 29, Kalman Filter as a causal MMSE estimator
Kalman Filtering Namrata Vaswani March 29, 2018 Notes are based on Vincent Poor s book. 1 Kalman Filter as a causal MMSE estimator Consider the following state space model (signal and observation model).
More informationAssignment #9: Orthogonal Projections, Gram-Schmidt, and Least Squares. Name:
Assignment 9: Orthogonal Projections, Gram-Schmidt, and Least Squares Due date: Friday, April 0, 08 (:pm) Name: Section Number Assignment 9: Orthogonal Projections, Gram-Schmidt, and Least Squares Due
More informationMAT 2037 LINEAR ALGEBRA I web:
MAT 237 LINEAR ALGEBRA I 2625 Dokuz Eylül University, Faculty of Science, Department of Mathematics web: Instructor: Engin Mermut http://kisideuedutr/enginmermut/ HOMEWORK 2 MATRIX ALGEBRA Textbook: Linear
More informationDecomposable and Directed Graphical Gaussian Models
Decomposable Decomposable and Directed Graphical Gaussian Models Graphical Models and Inference, Lecture 13, Michaelmas Term 2009 November 26, 2009 Decomposable Definition Basic properties Wishart density
More informationINF Introduction to classifiction Anne Solberg Based on Chapter 2 ( ) in Duda and Hart: Pattern Classification
INF 4300 151014 Introduction to classifiction Anne Solberg anne@ifiuiono Based on Chapter 1-6 in Duda and Hart: Pattern Classification 151014 INF 4300 1 Introduction to classification One of the most challenging
More informationNoise-Blind Image Deblurring Supplementary Material
Noise-Blind Image Deblurring Supplementary Material Meiguang Jin University of Bern Switzerland Stefan Roth TU Darmstadt Germany Paolo Favaro University of Bern Switzerland A. Upper and Lower Bounds Our
More informationMachine Learning Basics III
Machine Learning Basics III Benjamin Roth CIS LMU München Benjamin Roth (CIS LMU München) Machine Learning Basics III 1 / 62 Outline 1 Classification Logistic Regression 2 Gradient Based Optimization Gradient
More informationGaussians. Hiroshi Shimodaira. January-March Continuous random variables
Cumulative Distribution Function Gaussians Hiroshi Shimodaira January-March 9 In this chapter we introduce the basics of how to build probabilistic models of continuous-valued data, including the most
More information2.7 The Gaussian Probability Density Function Forms of the Gaussian pdf for Real Variates
.7 The Gaussian Probability Density Function Samples taken from a Gaussian process have a jointly Gaussian pdf (the definition of Gaussian process). Correlator outputs are Gaussian random variables if
More informationPreliminaries. Copyright c 2018 Dan Nettleton (Iowa State University) Statistics / 38
Preliminaries Copyright c 2018 Dan Nettleton (Iowa State University) Statistics 510 1 / 38 Notation for Scalars, Vectors, and Matrices Lowercase letters = scalars: x, c, σ. Boldface, lowercase letters
More informationMathematics 1. Part II: Linear Algebra. Exercises and problems
Bachelor Degree in Informatics Engineering Barcelona School of Informatics Mathematics Part II: Linear Algebra Eercises and problems February 5 Departament de Matemàtica Aplicada Universitat Politècnica
More informationState Space and Hidden Markov Models
State Space and Hidden Markov Models Kunsch H.R. State Space and Hidden Markov Models. ETH- Zurich Zurich; Aliaksandr Hubin Oslo 2014 Contents 1. Introduction 2. Markov Chains 3. Hidden Markov and State
More informationEconomics 205 Exercises
Economics 05 Eercises Prof. Watson, Fall 006 (Includes eaminations through Fall 003) Part 1: Basic Analysis 1. Using ε and δ, write in formal terms the meaning of lim a f() = c, where f : R R.. Write the
More informationMatrices. In this chapter: matrices, determinants. inverse matrix
Matrices In this chapter: matrices, determinants inverse matrix 1 1.1 Matrices A matrix is a retangular array of numbers. Rows: horizontal lines. A = a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33 a 41 a
More informationMachine Learning Lecture 2
Announcements Machine Learning Lecture 2 Eceptional number of lecture participants this year Current count: 449 participants This is very nice, but it stretches our resources to their limits Probability
More informationMachine Learning Lecture 2
Machine Perceptual Learning and Sensory Summer Augmented 6 Computing Announcements Machine Learning Lecture 2 Course webpage http://www.vision.rwth-aachen.de/teaching/ Slides will be made available on
More information3.1: 1, 3, 5, 9, 10, 12, 14, 18
3.:, 3, 5, 9,,, 4, 8 ) We want to solve d d c() d = f() with c() = c = constant and f() = for different boundary conditions to get w() and u(). dw d = dw d d = ( )d w() w() = w() = w() ( ) c d d = u()
More informationINF Introduction to classifiction Anne Solberg
INF 4300 8.09.17 Introduction to classifiction Anne Solberg anne@ifi.uio.no Introduction to classification Based on handout from Pattern Recognition b Theodoridis, available after the lecture INF 4300
More information33A Linear Algebra and Applications: Practice Final Exam - Solutions
33A Linear Algebra and Applications: Practice Final Eam - Solutions Question Consider a plane V in R 3 with a basis given by v = and v =. Suppose, y are both in V. (a) [3 points] If [ ] B =, find. (b)
More informationVariations. ECE 6540, Lecture 02 Multivariate Random Variables & Linear Algebra
Variations ECE 6540, Lecture 02 Multivariate Random Variables & Linear Algebra Last Time Probability Density Functions Normal Distribution Expectation / Expectation of a function Independence Uncorrelated
More informationCopyright, 2008, R.E. Kass, E.N. Brown, and U. Eden REPRODUCTION OR CIRCULATION REQUIRES PERMISSION OF THE AUTHORS
Copright, 8, RE Kass, EN Brown, and U Eden REPRODUCTION OR CIRCULATION REQUIRES PERMISSION OF THE AUTHORS Chapter 6 Random Vectors and Multivariate Distributions 6 Random Vectors In Section?? we etended
More informationNON-NEGATIVE MATRIX FACTORIZATION FOR PARAMETER ESTIMATION IN HIDDEN MARKOV MODELS. Balaji Lakshminarayanan and Raviv Raich
NON-NEGATIVE MATRIX FACTORIZATION FOR PARAMETER ESTIMATION IN HIDDEN MARKOV MODELS Balaji Lakshminarayanan and Raviv Raich School of EECS, Oregon State University, Corvallis, OR 97331-551 {lakshmba,raich@eecs.oregonstate.edu}
More informationLemma 8: Suppose the N by N matrix A has the following block upper triangular form:
17 4 Determinants and the Inverse of a Square Matrix In this section, we are going to use our knowledge of determinants and their properties to derive an explicit formula for the inverse of a square matrix
More informationMatrix Algebra Determinant, Inverse matrix. Matrices. A. Fabretti. Mathematics 2 A.Y. 2015/2016. A. Fabretti Matrices
Matrices A. Fabretti Mathematics 2 A.Y. 2015/2016 Table of contents Matrix Algebra Determinant Inverse Matrix Introduction A matrix is a rectangular array of numbers. The size of a matrix is indicated
More informationFactor Analysis. Qian-Li Xue
Factor Analysis Qian-Li Xue Biostatistics Program Harvard Catalyst The Harvard Clinical & Translational Science Center Short course, October 7, 06 Well-used latent variable models Latent variable scale
More informationLeast Squares with Examples in Signal Processing 1. 2 Overdetermined equations. 1 Notation. The sum of squares of x is denoted by x 2 2, i.e.
Least Squares with Eamples in Signal Processing Ivan Selesnick March 7, 3 NYU-Poly These notes address (approimate) solutions to linear equations by least squares We deal with the easy case wherein the
More information0.1. Linear transformations
Suggestions for midterm review #3 The repetitoria are usually not complete; I am merely bringing up the points that many people didn t now on the recitations Linear transformations The following mostly
More informationConvex relaxation. In example below, we have N = 6, and the cut we are considering
Convex relaxation The art and science of convex relaxation revolves around taking a non-convex problem that you want to solve, and replacing it with a convex problem which you can actually solve the solution
More informationExpression arrays, normalization, and error models
1 Epression arrays, normalization, and error models There are a number of different array technologies available for measuring mrna transcript levels in cell populations, from spotted cdna arrays to in
More informationBayesian Methods / G.D. Hager S. Leonard
Bayesian Methods Recall Robot Localization Given Sensor readings z 1, z 2,, z t = z 1:t Known control inputs u 0, u 1, u t = u 0:t Known model t+1 t, u t ) with initial 1 u 0 ) Known map z t t ) Compute
More informationMultidisciplinary System Design Optimization (MSDO)
oday s opics Multidisciplinary System Design Optimization (MSDO) Approimation Methods Lecture 9 6 April 004 Design variable lining Reduced-Basis Methods Response Surface Approimations Kriging Variable-Fidelity
More informationOrdering the Extraction of Polluting Nonrenewable Resources
Web Appendi for Ordering the Etraction of Polluting Nonrenewable Resources Appendi A: Maimizing umulative Etraction of oal along a otelling Path implies that the Stock of Pollution is Always at the eiling
More informationOptimization Methods: Optimization using Calculus - Equality constraints 1. Module 2 Lecture Notes 4
Optimization Methods: Optimization using Calculus - Equality constraints Module Lecture Notes 4 Optimization of Functions of Multiple Variables subect to Equality Constraints Introduction In the previous
More informationKernel PCA, clustering and canonical correlation analysis
ernel PCA, clustering and canonical correlation analsis Le Song Machine Learning II: Advanced opics CSE 8803ML, Spring 2012 Support Vector Machines (SVM) 1 min w 2 w w + C j ξ j s. t. w j + b j 1 ξ j,
More informationLearning from Data: Regression
November 3, 2005 http://www.anc.ed.ac.uk/ amos/lfd/ Classification or Regression? Classification: want to learn a discrete target variable. Regression: want to learn a continuous target variable. Linear
More informationSpeech and Language Processing
Speech and Language Processing Lecture 5 Neural network based acoustic and language models Information and Communications Engineering Course Takahiro Shinoaki 08//6 Lecture Plan (Shinoaki s part) I gives
More informationFigure 29: AR model fit into speech sample ah (top), the residual, and the random sample of the model (bottom).
Original 0.4 0.0 0.4 ACF 0.5 0.0 0.5 1.0 0 500 1000 1500 2000 0 50 100 150 200 Residual 0.05 0.05 ACF 0 500 1000 1500 2000 0 50 100 150 200 Generated 0.4 0.0 0.4 ACF 0.5 0.0 0.5 1.0 0 500 1000 1500 2000
More informationNow, if you see that x and y are present in both equations, you may write:
Matrices: Suppose you have two simultaneous equations: y y 3 () Now, if you see that and y are present in both equations, you may write: y 3 () You should be able to see where the numbers have come from.
More informationMath for ML: review. Milos Hauskrecht 5329 Sennott Square, x people.cs.pitt.edu/~milos/courses/cs1675/
Math for ML: review Milos Hauskrecht milos@pitt.edu 5 Sennott Square, -5 people.cs.pitt.edu/~milos/courses/cs75/ Administrivia Recitations Held on Wednesdays at :00am and :00pm This week: Matlab tutorial
More information9 - Markov processes and Burt & Allison 1963 AGEC
This document was generated at 8:37 PM on Saturday, March 17, 2018 Copyright 2018 Richard T. Woodward 9 - Markov processes and Burt & Allison 1963 AGEC 642-2018 I. What is a Markov Chain? A Markov chain
More information6.867 Machine learning
6.867 Machine learning Mid-term eam October 8, 6 ( points) Your name and MIT ID: .5.5 y.5 y.5 a).5.5 b).5.5.5.5 y.5 y.5 c).5.5 d).5.5 Figure : Plots of linear regression results with different types of
More informationChapter 6. Nonlinear Equations. 6.1 The Problem of Nonlinear Root-finding. 6.2 Rate of Convergence
Chapter 6 Nonlinear Equations 6. The Problem of Nonlinear Root-finding In this module we consider the problem of using numerical techniques to find the roots of nonlinear equations, f () =. Initially we
More informationRejection sampling - Acceptance probability. Review: How to sample from a multivariate normal in R. Review: Rejection sampling. Weighted resampling
Rejection sampling - Acceptance probability Review: How to sample from a multivariate normal in R Goal: Simulate from N d (µ,σ)? Note: For c to be small, g() must be similar to f(). The art of rejection
More informationProbabilistic Graphical Models
Probabilistic Graphical Models Lecture Notes Fall 2009 November, 2009 Byoung-Ta Zhang School of Computer Science and Engineering & Cognitive Science, Brain Science, and Bioinformatics Seoul National University
More informationUNCONSTRAINED OPTIMIZATION PAUL SCHRIMPF OCTOBER 24, 2013
PAUL SCHRIMPF OCTOBER 24, 213 UNIVERSITY OF BRITISH COLUMBIA ECONOMICS 26 Today s lecture is about unconstrained optimization. If you re following along in the syllabus, you ll notice that we ve skipped
More informationMobile Robot Localization
Mobile Robot Localization 1 The Problem of Robot Localization Given a map of the environment, how can a robot determine its pose (planar coordinates + orientation)? Two sources of uncertainty: - observations
More information1 Cricket chirps: an example
Notes for 2016-09-26 1 Cricket chirps: an example Did you know that you can estimate the temperature by listening to the rate of chirps? The data set in Table 1 1. represents measurements of the number
More informationProbability & Statistics: Infinite Statistics. Robert Leishman Mark Colton ME 363 Spring 2011
Probability & Statistics: Infinite Statistics Robert Leishman Mark Colton ME 363 Spring 0 Large Data Sets What happens to a histogram as N becomes large (N )? Number of bins becomes large (K ) Width of
More informationCorrelation analysis 2: Measures of correlation
Correlation analsis 2: Measures of correlation Ran Tibshirani Data Mining: 36-462/36-662 Februar 19 2013 1 Review: correlation Pearson s correlation is a measure of linear association In the population:
More informationCLASS NOTES Computational Methods for Engineering Applications I Spring 2015
CLASS NOTES Computational Methods for Engineering Applications I Spring 2015 Petros Koumoutsakos Gerardo Tauriello (Last update: July 2, 2015) IMPORTANT DISCLAIMERS 1. REFERENCES: Much of the material
More informationMatrix Algebra. Matrix Algebra. Chapter 8 - S&B
Chapter 8 - S&B Algebraic operations Matrix: The size of a matrix is indicated by the number of its rows and the number of its columns. A matrix with k rows and n columns is called a k n matrix. The number
More informationMatrix Derivatives and Descent Optimization Methods
Matrix Derivatives and Descent Optimization Methods 1 Qiang Ning Department of Electrical and Computer Engineering Beckman Institute for Advanced Science and Techonology University of Illinois at Urbana-Champaign
More informationExpectation Maximization Deconvolution Algorithm
Epectation Maimization Deconvolution Algorithm Miaomiao ZHANG March 30, 2011 Abstract In this paper, we use a general mathematical and eperimental methodology to analyze image deconvolution. The main procedure
More informationParticle Methods as Message Passing
Particle Methods as Message Passing Justin Dauwels RIKEN Brain Science Institute Hirosawa,2-1,Wako-shi,Saitama,Japan Email: justin@dauwels.com Sascha Korl Phonak AG CH-8712 Staefa, Switzerland Email: sascha.korl@phonak.ch
More informationA6523 Modeling, Inference, and Mining Jim Cordes, Cornell University
A6523 Modeling, Inference, and Mining Jim Cordes, Cornell University Lecture 19 Modeling Topics plan: Modeling (linear/non- linear least squares) Bayesian inference Bayesian approaches to spectral esbmabon;
More informationManaging Uncertainty
Managing Uncertainty Bayesian Linear Regression and Kalman Filter December 4, 2017 Objectives The goal of this lab is multiple: 1. First it is a reminder of some central elementary notions of Bayesian
More informationECE Lecture #9 Part 2 Overview
ECE 450 - Lecture #9 Part Overview Bivariate Moments Mean or Expected Value of Z = g(x, Y) Correlation and Covariance of RV s Functions of RV s: Z = g(x, Y); finding f Z (z) Method : First find F(z), by
More informationECON 5350 Class Notes Review of Probability and Distribution Theory
ECON 535 Class Notes Review of Probability and Distribution Theory 1 Random Variables Definition. Let c represent an element of the sample space C of a random eperiment, c C. A random variable is a one-to-one
More information14.30 Introduction to Statistical Methods in Economics Spring 2009
MIT OpenCourseWare http://ocw.mit.edu 4.0 Introduction to Statistical Methods in Economics Spring 009 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.
More informationINF Anne Solberg One of the most challenging topics in image analysis is recognizing a specific object in an image.
INF 4300 700 Introduction to classifiction Anne Solberg anne@ifiuiono Based on Chapter -6 6inDuda and Hart: attern Classification 303 INF 4300 Introduction to classification One of the most challenging
More informationLecture 3: September 10
CS294 Markov Chain Monte Carlo: Foundations & Applications Fall 2009 Lecture 3: September 10 Lecturer: Prof. Alistair Sinclair Scribes: Andrew H. Chan, Piyush Srivastava Disclaimer: These notes have not
More informationProblem Set 2. MAS 622J/1.126J: Pattern Recognition and Analysis. Due: 5:00 p.m. on September 30
Problem Set 2 MAS 622J/1.126J: Pattern Recognition and Analysis Due: 5:00 p.m. on September 30 [Note: All instructions to plot data or write a program should be carried out using Matlab. In order to maintain
More informationProbabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Gaussian graphical models and Ising models: modeling networks Eric Xing Lecture 0, February 5, 06 Reading: See class website Eric Xing @ CMU, 005-06
More informationM.S. Project Report. Efficient Failure Rate Prediction for SRAM Cells via Gibbs Sampling. Yamei Feng 12/15/2011
.S. Project Report Efficient Failure Rate Prediction for SRA Cells via Gibbs Sampling Yamei Feng /5/ Committee embers: Prof. Xin Li Prof. Ken ai Table of Contents CHAPTER INTRODUCTION...3 CHAPTER BACKGROUND...5
More informationMachine Learning Lecture 3
Announcements Machine Learning Lecture 3 Eam dates We re in the process of fiing the first eam date Probability Density Estimation II 9.0.207 Eercises The first eercise sheet is available on L2P now First
More informationANOVA: Analysis of Variance - Part I
ANOVA: Analysis of Variance - Part I The purpose of these notes is to discuss the theory behind the analysis of variance. It is a summary of the definitions and results presented in class with a few exercises.
More informationKalman Filter. Predict: Update: x k k 1 = F k x k 1 k 1 + B k u k P k k 1 = F k P k 1 k 1 F T k + Q
Kalman Filter Kalman Filter Predict: x k k 1 = F k x k 1 k 1 + B k u k P k k 1 = F k P k 1 k 1 F T k + Q Update: K = P k k 1 Hk T (H k P k k 1 Hk T + R) 1 x k k = x k k 1 + K(z k H k x k k 1 ) P k k =(I
More informationLocal Probability Models
Readings: K&F 3.4, 5.~5.5 Local Probability Models Lecture 3 pr 4, 2 SE 55, Statistical Methods, Spring 2 Instructor: Su-In Lee University of Washington, Seattle Outline Last time onditional parameterization
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationInfinite Horizon LQ. Given continuous-time state equation. Find the control function u(t) to minimize
Infinite Horizon LQ Given continuous-time state equation x = Ax + Bu Find the control function ut) to minimize J = 1 " # [ x T t)qxt) + u T t)rut)] dt 2 0 Q $ 0, R > 0 and symmetric Solution is obtained
More informationLinear Equations and Matrix
1/60 Chia-Ping Chen Professor Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra Gaussian Elimination 2/60 Alpha Go Linear algebra begins with a system of linear
More informationPre-Calculus Summer Packet
Pre-Calculus Summer Packet Name ALLEN PARK HIGH SCHOOL Summer Assessment Pre-Calculus Summer Packet For Students Entering Pre-Calculus Summer 05 This summer packet is intended to be completed by the FIRST
More informationAP Calculus BC Summer Packet 2017
AP Calculus BC Summer Packet 7 o The attached packet is required for all FHS students who took AP Calculus AB in 6-7 and will be continuing on to AP Calculus BC in 7-8. o It is to be turned in to your
More informationELEG 5633 Detection and Estimation Minimum Variance Unbiased Estimators (MVUE)
1 ELEG 5633 Detection and Estimation Minimum Variance Unbiased Estimators (MVUE) Jingxian Wu Department of Electrical Engineering University of Arkansas Outline Minimum Variance Unbiased Estimators (MVUE)
More informationPre-Calculus and Trigonometry Capacity Matrix
Pre-Calculus and Capacity Matri Review Polynomials A1.1.4 A1.2.5 Add, subtract, multiply and simplify polynomials and rational epressions Solve polynomial equations and equations involving rational epressions
More informationA Robust State Estimator Based on Maximum Exponential Square (MES)
11 th Int. Workshop on EPCC, May -5, 011, Altea, Spain A Robust State Estimator Based on Maimum Eponential Square (MES) Wenchuan Wu Ye Guo Boming Zhang, et al Dept. of Electrical Eng. Tsinghua University
More informationThe Kalman filter is arguably one of the most notable algorithms
LECTURE E NOTES «Kalman Filtering with Newton s Method JEFFREY HUMPHERYS and JEREMY WEST The Kalman filter is arguably one of the most notable algorithms of the 0th century [1]. In this article, we derive
More information