Linear Prediction Theory
|
|
- Martha Hodges
- 5 years ago
- Views:
Transcription
1 Linear Prediction Theory Joseph A. O Sullivan ESE 524 Spring 29 March 3, 29 Overview The problem of estimating a value of a random process given other values of the random process is pervasive. Many problems in forecasting fall into this category. The Kalman filter applies if there is a nown underlying state space model for the system. If no such state space model exists, then some approximation must be used. If the covariance function for a zero mean random process is nown, then the use of a linear predictor is often a good choice. In linear prediction, the next value is estimated as a linear combination of past values, the parameters being chosen to minimize the mean square error between the next value and its estimate. If the true distribution is Gaussian, then the linear estimator of a given order minimizes the mean square error over all estimators of that order. This ability to use the algorithms independent of the true distribution maes them universal in the sense that they achieve performance for any true distributions. When the true distribution is an autoregressive model of order, then the th order linear filter is optimal. In that case, the previous inputs can be considered the state, and the Kalman filter applied. The steady state version of the Kalman filter is equivalent to the th order linear filters here. The choice of the order of a linear estimator (that is, the number of past values used to predict the current value) is important. The mean square error monotonically decreases with increasing order of the estimator. Including too many coefficients increases complexity unnecessarily. In addition, if there is some uncertainty in the covariance function, then increasing the model order beyond some critical value may lead to overfitting the data, increasing the mean square error. When the model order increases, the coefficients used may be computed recursively. These recursive computations are classical, being based on efficient inversion due to the Toeplitz structure of the data covariance matrix. For the theoretical analysis, we assume Gaussian statistics and derive the optimal estimators. The linear predictors defined here form the basis for much of modern adaptive signal processing, including the least mean square (LMS) and recursive least squares (RLS) algorithms and their many variants. The algorithms are also instructive for many other estimation problems including array signal processing. The filter structures that result include the transversal and lattice filters. 2 Summary of Recursive Estimation Equations Let r n be a stationary, zero mean Gaussian random process with covariance function c l Er n r n l. ()
2 The problem of interest is estimating (predicting) one value of the random process given previous values, Er n r n,r n 2,...,r n. Linearity and Gaussian statistics yield ˆr n w r n + w 2 r n w r n (2) w T r (n ). (3) The coefficients do not depend on time due to stationarity. In this equation, The orthogonality principle states that and thus that r (n ) r n r n +... r n 2 r n T (4) w w w... w 2 w T. (5) E(r n w T r (n ))r (n ) T (6) γ T w T Γ (7) w Γ γ. (8) Here, Γ is a array with (i, j) elementequaltoc i j. The vector γ is a vector γ c c... c 2 c T. (9) Note the following recursive structures: γ + c+ γ () and Γ + γ T c c (Jγ ) T Γ γ Jγ Γ, () (2) where J is called an exchange matrix and has ones along its antidiagonal and zeros elsewhere; the matrix J has the property that J times a vector equals that vector with its entries reordered from bottom to top. These two decompositions of Γ + yield two different, but closely related ways of recursively computing its inverse. These two ways rely on defining the ( +) vector a w Using this vector, the forward estimation (prediction) error may be written F (n) r n. (3) w l r n l (4) l a T r +(n). (5) 2
3 The inverse of Γ + using () may be written as where Γ + Γ + p a a T, (6) p c γ T Γ γ (7) E(r n w T r (n )) 2 (8) EF (n) 2 (9) is the th order forward prediction error variance. Note that γ Γ Γ + a w c γ T w c γ T Γ p γ (2) (2) (22) This result actually verifies the inversion formula above because ( Γ Γ + I γ T Γ I γ T Γ + p a a T ) (23) + p p a T (24) + γ T Γ (25) I. (26) To write the inverse related to the second decomposition of Γ + in a similar form, we define a new vector b to have the elements of a in the opposite order. Then we have the following equations: The vector b determines the bacward prediction error G (n), JΓ + J Γ + (27) Ja b (28) JJ I (29) G (n) r n Er n r n +,...,r n,r n (3) r n (Jw) T r (n) (3) r T + b. (32) This vector satisfies b Jw (33) 3
4 and JΓ + a JΓ + JJa (34) JΓ + Jb (35) Γ + b. (36) We also have so JΓ + a J p p Γ + b p (37) (38) (39) where The inverse of Γ + using (2) may be written as Γ + Γ The recursive structure is further clarified through + p b b T. (4) w + Γ + γ + (4) c+ Γ + b γ b T p γ + (42) + Δ b, (43) w p Δ b T γ + (44) a T Jγ + (45) Plugging this last form for w + into the definition of a + yields w+ a + (46) w Δ b (47) p Δ b. (48) a p Similarly, plugging into the definition of b + yields b + Jw + (49) 4
5 Jw b Δ Δ p p a Jb (5). (5) 2. Recursive Transversal Filter Coefficient Computation Inputs: c,c,c 2,... Outputs: prediction error filters in transversal and lattice forms.. Initialization step: p c ; a b ;γ c ;Γ c ;. 2. Reflection coefficient and prediction variance computation: Δ a T Jγ + (52) p a T γ (53) c 3. Update forward and bacward prediction error filters: a + a b b + Δ p Δ p b a (54). (55) 4. Recursion step: +; γ + c+ γ ; (56) return to reflection coefficient computation. The computational complexity of this algorithm is determined by the computations for the reflection coefficient and the filter update steps. In the reflection coefficient and prediction variance computation, there are 2 multiplies and 2 additions. In the two filter update equations, the multiplies are all the same and the resulting values are just reordered versions of each other (reordered using J). Thus, there is one division and there are multiplies and additions. The number of computations is thus 3 multiplications, division, and 3 additions per stage. The total number of computations from stage through stage is 3( + )/2 multiplies, divisions, and 3( + )/2 additions. Some divisions can be avoided by using as input the sequence of correlation coefficients,c /c,c 2 /c,... instead of c,c,c 2,..., the sequence of correlations. 2.2 Lattice Filter Structure Two equivalent representations of the filters described above are the transversal and lattice filter representations. The transversal filters are described in terms of the coefficients a, a 2,...,a and b, b 2,...,b.The lattice filters are described in terms of the coefficients Δ p, Δ p,..., Δ p. 5
6 The update equations (54) and (55) may be used to describe the lattice structure in terms of the forward prediction errors F (n) and the bacward prediction errors G (n): F + (n) a T + r +2(n) (57) T r a +2(n) Δ T b R p +2(n) (58) a T r + (n) Δ b T r + (n ) p (59) F (n) Δ G (n ), p (6) where the ey step is recognizing that zeros in a T and b T correspond to reducing the length of r +2 (n) from +2to +, and shifting one time unit in the latter case. Similarly, G + (n) b T + r +2(n) (6) T b R +2(n) Δ T R p a +2(n) (62) b T r +(n ) Δ a T p R +(n) (63) G (n ) Δ p F (n). (64) In matrix form, one stage of the lattice filter has the form F+ (n) Δ p G + (n) Δ p F (n) G (n ). (65) If the lattice filter structure is used in the implementation, then the multiplications needed to update a and b are not needed. The computational complexity of the algorithm may be reduced even further as described below. 2.3 Faster Computations In the recursive algorithm, the computations associated with the reflection coefficient and the prediction variance can essentially be eliminated. To see this, consider the forward and bacward filters used with inputs equal to the correlations c n. At time (noting that c n c n ), the output of the forward prediction error filter of order equals c w T γ which equals p.attime, the output of the bacward prediction error filter of order equals c (Jw ) T γ b T γ + (66) Δ. (67) In this interpretation of the computations, the filters up through order may be used to compute the quantities needed for the next update by simply running the correlation coefficients through the filters. This saves computations by using the lattice filter structure: p Δ a T b T γ+ c, (68) 6
7 where the zeros in the matrix on the right side are scalars; that is, the matrix is 2 ( + 2) and has a zero in the upper left and bottom right corners. Δ p Δ p F (n) G (n ). (69) 2.4 Key Properties 2.5 Autoregressive Gaussian Processes A zero mean, stationary, Gaussian random process r,r 2,... is an mth order autoregressive process if r n a r n a 2 r n 2... a m r n m + w n, (7) for all n, wherew n are independent and identically distributed Gaussian random variables with zero mean and variance σ 2. An mth order autoregressive process is mth order Marov in the sense that the probability density function of r n given r n,r n 2,...,r equals the probability density function of r n given r n,r n 2,...,r n m. Defining the vector a m a m a m... a 2 a T, (7) may be rewritten as Let the covariance function for the random process be C,so a T m r m+(n) w n. (7) C E{r n r n }. (72) Comment: In order for this equation to model a stationary random process and to be viewed as a generative model for the data, the corresponding discrete time system must be stable. That is, if one were to compute the transfer function in the Z-transform domain, then all of the poles of the transfer function must be inside of the unit dis in the complex plane. These poles are obviously the roots of the characteristic equation with coefficients a j. a. Using the autoregressive model in equation (??), show that the covariance function satisfies the equations C + a C + a 2 C a m C m σ 2 (73) C + a C + a 2 C a m C m, (74) where the second equation holds for all >. Hint: Multiply both sides of (??) by a value of the random sequence and tae expected values. Use the symmetry property of covariance functions for the first equality. 2.6 Bacground and Understanding of Autoregressive Models Suppose that r,r 2,...is a stationary sequence of Gaussian random variables with zero mean. The covariance function is determined by an autoregressive model which the random variables satisfy. The autoregressive model is an mth order Marov model, meaning that More specifically, suppose that b. Derive a recursive structure for computing the logarithm of the probability density function of r n,r n 2,...,r. More specifically, let v n lnp(r,r 2,...,r n ). (75) 7
8 Derive an expression for v n in terms of v n and an update. Focus on the case where n>m. Hint: This is a ey part of the problem, so mae sure you do it correctly. It obviously relates to the Marov property expressed through the autoregressive model in (??). c. Consider the special case of m. Suppose that C. Find a relationship between a and σ 2 (essentially you must solve (74) in this general case). Comment: Note that the stability requirement implies that a <. 2.7 Recursive Detection for Autoregressive Models Suppose that one has to decide whether data arise from an autoregressive model or from white noise. In this problem, the log-lielihood ratio is computed recursively. Under hypothesis H, the data arise from the autoregressive model (??). Under hypothesis H, the data R n are i.i.d. Gaussian with zero mean and variance C. That is, under either hypothesis the marginal distribution on any sample R n is the same. The only difference between the two models is in the covariance structure. a. Find the log-lielihood ratio for n samples. Call this log-lielihood ratio l n. Derive a recursive expression for l n in terms of l n and an update. Focus on the case n>m. b. Consider the special case of m. Write down the recursive structure for this case. c. The performance increases as n grows. This can be quantified in various ways. One way is to compute the information rate functions for each n. In this problem, you will compute a special case. Consider again m. Find the log-moment generating function for the difference between l n and l n conditioned on each hypothesis, and conditioned on previous measurements; call these two log-moment generating functions m (s) andm (s): m (s) lne{e s(ln ln ) H,r,r 2,...,r n }. (76) Compute and plot the information rate functions I (x) andi (x) for these two log-moment generating functions. Comment: These two functions quantify the increase in information for detection provided by the new measurement. 2.8 Recursive Estimation for Autoregressive Models In this problem, you will estimate the parameters in an autoregressive model given observations of the data r n,r n,...,r. a. First, assume that the maximum lielihood estimate for the parameters given data r n,r n 2,...,r satisfies B n â n d n, (77) where the vector â n is the maximum lielihood estimate of the parameter vector a a a 2... a m T. (78) Find the update equations for B n and d n. These may be obtained by writing down the lielihood equation using the recursive update for the log-lielihood function, and taing the derivative with respect to the parameter vector. b. The computation for â n may also be written in recursive form. This is accomplished using the matrix inversion lemma. The matrix inversion lemma states that a ran one update to a matrix yields a ran one 8
9 update to its inverse. More specifically, if A is an m m symmetric, invertible matrix and f is an m vector, then (A + ff T ) A A f +f T A f f T A. (79) Use this equation to derive an equation for the estimate â n in terms of â n. Hint: The final form should loo lie â n â n + g n r n + â T n (r n r n 2...r n m ) T, (8) where an auxiliary equation defines the vector g n in terms of B n and the appropriate definition of f. 2.9 Recursive Detection: Order Versus Order 2 Autoregressive Model A decision must be made between two models for a sequence of Gaussian distributed random variables. Each model is an autoregressive model. The first model is autoregressive of order one, while the second is autoregressive of order two. There are two goals here as outlined below. First, the optimal test statistic for a Neyman-Pearson test must be computed for a fixed number N of consecutive samples of a realization. Second, an efficient update of this test statistic to the case with N + samples must be derived. Consider the following two hypotheses. Under H, the model for the measurements is y i.75y i + w i, (8) where w i are independent and identically distributed Gaussian random variables with zero mean and variance equal to 7/4.75; w i are independent of y for all i; andy is Gaussian distributed with zero mean and variance 4. Under H 2, the model for the measurements is y i.75y i +.2y i 2 + w i, (82) where w i are independent and identically distributed Gaussian random variables with zero mean and variance equal to.75; w i are independent of y for all i; andy is Gaussian distributed with zero mean and variance 4. y.75y + w where w is a zero mean Gaussian random variable with zero mean and variance.75. a. Given y,y,...,y N, find the optimal test statistic for a Neyman-Pearson test. Simplify the expression as much as possible. Interpret your answer. b. Denote the test statistic computed in part a by l N. The optimal test statistic for N + measurements is l N+. Find an efficient update rule for computing l N+ from l N. 9
ESE 524 Detection and Estimation Theory
ESE 524 Detection and Estimation heory Joseh A. O Sullivan Samuel C. Sachs Professor Electronic Systems and Signals Research Laboratory Electrical and Systems Engineering Washington University 2 Urbauer
More informationFor final project discussion every afternoon Mark and I will be available
Worshop report 1. Daniels report is on website 2. Don t expect to write it based on listening to one project (we had 6 only 2 was sufficient quality) 3. I suggest writing it on one presentation. 4. Include
More informationTime Series Analysis. James D. Hamilton PRINCETON UNIVERSITY PRESS PRINCETON, NEW JERSEY
Time Series Analysis James D. Hamilton PRINCETON UNIVERSITY PRESS PRINCETON, NEW JERSEY & Contents PREFACE xiii 1 1.1. 1.2. Difference Equations First-Order Difference Equations 1 /?th-order Difference
More informationLeast Squares. Ken Kreutz-Delgado (Nuno Vasconcelos) ECE 175A Winter UCSD
Least Squares Ken Kreutz-Delgado (Nuno Vasconcelos) ECE 75A Winter 0 - UCSD (Unweighted) Least Squares Assume linearity in the unnown, deterministic model parameters Scalar, additive noise model: y f (
More informationStatistical and Adaptive Signal Processing
r Statistical and Adaptive Signal Processing Spectral Estimation, Signal Modeling, Adaptive Filtering and Array Processing Dimitris G. Manolakis Massachusetts Institute of Technology Lincoln Laboratory
More informationTime Series Analysis. James D. Hamilton PRINCETON UNIVERSITY PRESS PRINCETON, NEW JERSEY
Time Series Analysis James D. Hamilton PRINCETON UNIVERSITY PRESS PRINCETON, NEW JERSEY PREFACE xiii 1 Difference Equations 1.1. First-Order Difference Equations 1 1.2. pth-order Difference Equations 7
More informationSome Time-Series Models
Some Time-Series Models Outline 1. Stochastic processes and their properties 2. Stationary processes 3. Some properties of the autocorrelation function 4. Some useful models Purely random processes, random
More informationADAPTIVE FILTER THEORY
ADAPTIVE FILTER THEORY Fourth Edition Simon Haykin Communications Research Laboratory McMaster University Hamilton, Ontario, Canada Front ice Hall PRENTICE HALL Upper Saddle River, New Jersey 07458 Preface
More informationCramér-Rao Bounds for Estimation of Linear System Noise Covariances
Journal of Mechanical Engineering and Automation (): 6- DOI: 593/jjmea Cramér-Rao Bounds for Estimation of Linear System oise Covariances Peter Matiso * Vladimír Havlena Czech echnical University in Prague
More information12.4 Known Channel (Water-Filling Solution)
ECEn 665: Antennas and Propagation for Wireless Communications 54 2.4 Known Channel (Water-Filling Solution) The channel scenarios we have looed at above represent special cases for which the capacity
More informationECONOMETRIC METHODS II: TIME SERIES LECTURE NOTES ON THE KALMAN FILTER. The Kalman Filter. We will be concerned with state space systems of the form
ECONOMETRIC METHODS II: TIME SERIES LECTURE NOTES ON THE KALMAN FILTER KRISTOFFER P. NIMARK The Kalman Filter We will be concerned with state space systems of the form X t = A t X t 1 + C t u t 0.1 Z t
More informationDetection and Estimation Theory
ESE 524 Detetion and Estimation heory Joseh A. O Sullivan Samuel C. Sahs Professor Eletroni Systems and Signals Researh Laboratory Eletrial and Systems Engineering Washington University 2 Urbauer Hall
More informationProperties of Zero-Free Spectral Matrices Brian D. O. Anderson, Life Fellow, IEEE, and Manfred Deistler, Fellow, IEEE
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL 54, NO 10, OCTOBER 2009 2365 Properties of Zero-Free Spectral Matrices Brian D O Anderson, Life Fellow, IEEE, and Manfred Deistler, Fellow, IEEE Abstract In
More informationUTILIZING PRIOR KNOWLEDGE IN ROBUST OPTIMAL EXPERIMENT DESIGN. EE & CS, The University of Newcastle, Australia EE, Technion, Israel.
UTILIZING PRIOR KNOWLEDGE IN ROBUST OPTIMAL EXPERIMENT DESIGN Graham C. Goodwin James S. Welsh Arie Feuer Milan Depich EE & CS, The University of Newcastle, Australia 38. EE, Technion, Israel. Abstract:
More information2 Introduction of Discrete-Time Systems
2 Introduction of Discrete-Time Systems This chapter concerns an important subclass of discrete-time systems, which are the linear and time-invariant systems excited by Gaussian distributed stochastic
More informationParametric Signal Modeling and Linear Prediction Theory 4. The Levinson-Durbin Recursion
Parametric Signal Modeling and Linear Prediction Theory 4. The Levinson-Durbin Recursion Electrical & Computer Engineering North Carolina State University Acknowledgment: ECE792-41 slides were adapted
More informationConvergence of Square Root Ensemble Kalman Filters in the Large Ensemble Limit
Convergence of Square Root Ensemble Kalman Filters in the Large Ensemble Limit Evan Kwiatkowski, Jan Mandel University of Colorado Denver December 11, 2014 OUTLINE 2 Data Assimilation Bayesian Estimation
More informationLecture 4: Types of errors. Bayesian regression models. Logistic regression
Lecture 4: Types of errors. Bayesian regression models. Logistic regression A Bayesian interpretation of regularization Bayesian vs maximum likelihood fitting more generally COMP-652 and ECSE-68, Lecture
More informationSIMON FRASER UNIVERSITY School of Engineering Science
SIMON FRASER UNIVERSITY School of Engineering Science Course Outline ENSC 810-3 Digital Signal Processing Calendar Description This course covers advanced digital signal processing techniques. The main
More informationADAPTIVE FILTER THEORY
ADAPTIVE FILTER THEORY Fifth Edition Simon Haykin Communications Research Laboratory McMaster University Hamilton, Ontario, Canada International Edition contributions by Telagarapu Prabhakar Department
More informationTime-Varying Systems and Computations Lecture 3
Time-Varying Systems and Computations Lecture 3 Klaus Diepold November 202 Linear Time-Varying Systems State-Space System Model We aim to derive the matrix containing the time-varying impulse responses
More informationLecture 5: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics. 1 Executive summary
ECE 830 Spring 207 Instructor: R. Willett Lecture 5: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics Executive summary In the last lecture we saw that the likelihood
More informationNotes on Time Series Modeling
Notes on Time Series Modeling Garey Ramey University of California, San Diego January 17 1 Stationary processes De nition A stochastic process is any set of random variables y t indexed by t T : fy t g
More informationX t = a t + r t, (7.1)
Chapter 7 State Space Models 71 Introduction State Space models, developed over the past 10 20 years, are alternative models for time series They include both the ARIMA models of Chapters 3 6 and the Classical
More informationPreviously on TT, Target Tracking: Lecture 2 Single Target Tracking Issues. Lecture-2 Outline. Basic ideas on track life
REGLERTEKNIK Previously on TT, AUTOMATIC CONTROL Target Tracing: Lecture 2 Single Target Tracing Issues Emre Özan emre@isy.liu.se Division of Automatic Control Department of Electrical Engineering Linöping
More information14.30 Introduction to Statistical Methods in Economics Spring 2009
MIT OpenCourseWare http://ocw.mit.edu.30 Introduction to Statistical Methods in Economics Spring 009 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms. .30
More informationLecture 3: Autoregressive Moving Average (ARMA) Models and their Practical Applications
Lecture 3: Autoregressive Moving Average (ARMA) Models and their Practical Applications Prof. Massimo Guidolin 20192 Financial Econometrics Winter/Spring 2018 Overview Moving average processes Autoregressive
More informationFINANCIAL ECONOMETRICS AND EMPIRICAL FINANCE -MODULE2 Midterm Exam Solutions - March 2015
FINANCIAL ECONOMETRICS AND EMPIRICAL FINANCE -MODULE2 Midterm Exam Solutions - March 205 Time Allowed: 60 minutes Family Name (Surname) First Name Student Number (Matr.) Please answer all questions by
More informationRECURSIVE ESTIMATION AND KALMAN FILTERING
Chapter 3 RECURSIVE ESTIMATION AND KALMAN FILTERING 3. The Discrete Time Kalman Filter Consider the following estimation problem. Given the stochastic system with x k+ = Ax k + Gw k (3.) y k = Cx k + Hv
More informationTitle. Description. var intro Introduction to vector autoregressive models
Title var intro Introduction to vector autoregressive models Description Stata has a suite of commands for fitting, forecasting, interpreting, and performing inference on vector autoregressive (VAR) models
More informationLecture 8: Bayesian Estimation of Parameters in State Space Models
in State Space Models March 30, 2016 Contents 1 Bayesian estimation of parameters in state space models 2 Computational methods for parameter estimation 3 Practical parameter estimation in state space
More informationcovariance function, 174 probability structure of; Yule-Walker equations, 174 Moving average process, fluctuations, 5-6, 175 probability structure of
Index* The Statistical Analysis of Time Series by T. W. Anderson Copyright 1971 John Wiley & Sons, Inc. Aliasing, 387-388 Autoregressive {continued) Amplitude, 4, 94 case of first-order, 174 Associated
More informationF denotes cumulative density. denotes probability density function; (.)
BAYESIAN ANALYSIS: FOREWORDS Notation. System means the real thing and a model is an assumed mathematical form for the system.. he probability model class M contains the set of the all admissible models
More informationDerivation of the Kalman Filter
Derivation of the Kalman Filter Kai Borre Danish GPS Center, Denmark Block Matrix Identities The key formulas give the inverse of a 2 by 2 block matrix, assuming T is invertible: T U 1 L M. (1) V W N P
More informationLecture 2: Univariate Time Series
Lecture 2: Univariate Time Series Analysis: Conditional and Unconditional Densities, Stationarity, ARMA Processes Prof. Massimo Guidolin 20192 Financial Econometrics Spring/Winter 2017 Overview Motivation:
More informationLecture 3: Autoregressive Moving Average (ARMA) Models and their Practical Applications
Lecture 3: Autoregressive Moving Average (ARMA) Models and their Practical Applications Prof. Massimo Guidolin 20192 Financial Econometrics Winter/Spring 2018 Overview Moving average processes Autoregressive
More informationStatistical Signal Processing Detection, Estimation, and Time Series Analysis
Statistical Signal Processing Detection, Estimation, and Time Series Analysis Louis L. Scharf University of Colorado at Boulder with Cedric Demeure collaborating on Chapters 10 and 11 A TT ADDISON-WESLEY
More information{ ϕ(f )Xt, 1 t n p, X t, n p + 1 t n, { 0, t=1, W t = E(W t W s, 1 s t 1), 2 t n.
Innovations, Sufficient Statistics, And Maximum Lielihood In ARM A Models Georgi N. Boshnaov Institute of Mathematics, Bulgarian Academy of Sciences 1. Introduction. Let X t, t Z} be a zero mean ARMA(p,
More informationLecture 7: Linear Prediction
1 Lecture 7: Linear Prediction Overview Dealing with three notions: PREDICTION, PREDICTOR, PREDICTION ERROR; FORWARD versus BACKWARD: Predicting the future versus (improper terminology) predicting the
More informationMachine Learning Lecture Notes
Machine Learning Lecture Notes Predrag Radivojac January 25, 205 Basic Principles of Parameter Estimation In probabilistic modeling, we are typically presented with a set of observations and the objective
More informationTime Series Examples Sheet
Lent Term 2001 Richard Weber Time Series Examples Sheet This is the examples sheet for the M. Phil. course in Time Series. A copy can be found at: http://www.statslab.cam.ac.uk/~rrw1/timeseries/ Throughout,
More informationNew Introduction to Multiple Time Series Analysis
Helmut Lütkepohl New Introduction to Multiple Time Series Analysis With 49 Figures and 36 Tables Springer Contents 1 Introduction 1 1.1 Objectives of Analyzing Multiple Time Series 1 1.2 Some Basics 2
More informationARMA Estimation Recipes
Econ. 1B D. McFadden, Fall 000 1. Preliminaries ARMA Estimation Recipes hese notes summarize procedures for estimating the lag coefficients in the stationary ARMA(p,q) model (1) y t = µ +a 1 (y t-1 -µ)
More information10-701/15-781, Machine Learning: Homework 4
10-701/15-781, Machine Learning: Homewor 4 Aarti Singh Carnegie Mellon University ˆ The assignment is due at 10:30 am beginning of class on Mon, Nov 15, 2010. ˆ Separate you answers into five parts, one
More information(Extended) Kalman Filter
(Extended) Kalman Filter Brian Hunt 7 June 2013 Goals of Data Assimilation (DA) Estimate the state of a system based on both current and all past observations of the system, using a model for the system
More informationLatent Variable Models and EM algorithm
Latent Variable Models and EM algorithm SC4/SM4 Data Mining and Machine Learning, Hilary Term 2017 Dino Sejdinovic 3.1 Clustering and Mixture Modelling K-means and hierarchical clustering are non-probabilistic
More informationSequential Procedure for Testing Hypothesis about Mean of Latent Gaussian Process
Applied Mathematical Sciences, Vol. 4, 2010, no. 62, 3083-3093 Sequential Procedure for Testing Hypothesis about Mean of Latent Gaussian Process Julia Bondarenko Helmut-Schmidt University Hamburg University
More informationENGR352 Problem Set 02
engr352/engr352p02 September 13, 2018) ENGR352 Problem Set 02 Transfer function of an estimator 1. Using Eq. (1.1.4-27) from the text, find the correct value of r ss (the result given in the text is incorrect).
More information8 Basics of Hypothesis Testing
8 Basics of Hypothesis Testing 4 Problems Problem : The stochastic signal S is either 0 or E with equal probability, for a known value E > 0. Consider an observation X = x of the stochastic variable X
More informationTime Series: Theory and Methods
Peter J. Brockwell Richard A. Davis Time Series: Theory and Methods Second Edition With 124 Illustrations Springer Contents Preface to the Second Edition Preface to the First Edition vn ix CHAPTER 1 Stationary
More informationThe Unscented Particle Filter
The Unscented Particle Filter Rudolph van der Merwe (OGI) Nando de Freitas (UC Bereley) Arnaud Doucet (Cambridge University) Eric Wan (OGI) Outline Optimal Estimation & Filtering Optimal Recursive Bayesian
More informationIntroduction. Chapter 1
Chapter 1 Introduction In this book we will be concerned with supervised learning, which is the problem of learning input-output mappings from empirical data (the training dataset). Depending on the characteristics
More informationEECS564 Estimation, Filtering, and Detection Exam 2 Week of April 20, 2015
EECS564 Estimation, Filtering, and Detection Exam Week of April 0, 015 This is an open book takehome exam. You have 48 hours to complete the exam. All work on the exam should be your own. problems have
More informationFE570 Financial Markets and Trading. Stevens Institute of Technology
FE570 Financial Markets and Trading Lecture 5. Linear Time Series Analysis and Its Applications (Ref. Joel Hasbrouck - Empirical Market Microstructure ) Steve Yang Stevens Institute of Technology 9/25/2012
More informationStabilization with Disturbance Attenuation over a Gaussian Channel
Proceedings of the 46th IEEE Conference on Decision and Control New Orleans, LA, USA, Dec. 1-14, 007 Stabilization with Disturbance Attenuation over a Gaussian Channel J. S. Freudenberg, R. H. Middleton,
More informationHypothesis testing (cont d)
Hypothesis testing (cont d) Ulrich Heintz Brown University 4/12/2016 Ulrich Heintz - PHYS 1560 Lecture 11 1 Hypothesis testing Is our hypothesis about the fundamental physics correct? We will not be able
More informationTime Series Models and Inference. James L. Powell Department of Economics University of California, Berkeley
Time Series Models and Inference James L. Powell Department of Economics University of California, Berkeley Overview In contrast to the classical linear regression model, in which the components of the
More informationInstitute of Actuaries of India
Institute of Actuaries of India Subject CT3 Probability & Mathematical Statistics May 2011 Examinations INDICATIVE SOLUTION Introduction The indicative solution has been written by the Examiners with the
More informationEUSIPCO
EUSIPCO 3 569736677 FULLY ISTRIBUTE SIGNAL ETECTION: APPLICATION TO COGNITIVE RAIO Franc Iutzeler Philippe Ciblat Telecom ParisTech, 46 rue Barrault 753 Paris, France email: firstnamelastname@telecom-paristechfr
More informationDATA ASSIMILATION FOR FLOOD FORECASTING
DATA ASSIMILATION FOR FLOOD FORECASTING Arnold Heemin Delft University of Technology 09/16/14 1 Data assimilation is the incorporation of measurement into a numerical model to improve the model results
More informationIntroduction: The Perceptron
Introduction: The Perceptron Haim Sompolinsy, MIT October 4, 203 Perceptron Architecture The simplest type of perceptron has a single layer of weights connecting the inputs and output. Formally, the perceptron
More informationSTAT 443 Final Exam Review. 1 Basic Definitions. 2 Statistical Tests. L A TEXer: W. Kong
STAT 443 Final Exam Review L A TEXer: W Kong 1 Basic Definitions Definition 11 The time series {X t } with E[X 2 t ] < is said to be weakly stationary if: 1 µ X (t) = E[X t ] is independent of t 2 γ X
More informationModel-based Correlation Measure for Gain and Offset Nonuniformity in Infrared Focal-Plane-Array Sensors
Model-based Correlation Measure for Gain and Offset Nonuniformity in Infrared Focal-Plane-Array Sensors César San Martin Sergio Torres Abstract In this paper, a model-based correlation measure between
More information2. What are the tradeoffs among different measures of error (e.g. probability of false alarm, probability of miss, etc.)?
ECE 830 / CS 76 Spring 06 Instructors: R. Willett & R. Nowak Lecture 3: Likelihood ratio tests, Neyman-Pearson detectors, ROC curves, and sufficient statistics Executive summary In the last lecture we
More informationReview: General Approach to Hypothesis Testing. 1. Define the research question and formulate the appropriate null and alternative hypotheses.
1 Review: Let X 1, X,..., X n denote n independent random variables sampled from some distribution might not be normal!) with mean µ) and standard deviation σ). Then X µ σ n In other words, X is approximately
More informationCIS 390 Fall 2016 Robotics: Planning and Perception Final Review Questions
CIS 390 Fall 2016 Robotics: Planning and Perception Final Review Questions December 14, 2016 Questions Throughout the following questions we will assume that x t is the state vector at time t, z t is the
More informationProblem Set 1 Solution Sketches Time Series Analysis Spring 2010
Problem Set 1 Solution Sketches Time Series Analysis Spring 2010 1. Construct a martingale difference process that is not weakly stationary. Simplest e.g.: Let Y t be a sequence of independent, non-identically
More informationFrequentist-Bayesian Model Comparisons: A Simple Example
Frequentist-Bayesian Model Comparisons: A Simple Example Consider data that consist of a signal y with additive noise: Data vector (N elements): D = y + n The additive noise n has zero mean and diagonal
More informationStochastic Processes. Theory for Applications. Robert G. Gallager CAMBRIDGE UNIVERSITY PRESS
Stochastic Processes Theory for Applications Robert G. Gallager CAMBRIDGE UNIVERSITY PRESS Contents Preface page xv Swgg&sfzoMj ybr zmjfr%cforj owf fmdy xix Acknowledgements xxi 1 Introduction and review
More informationMathematical Formulation of Our Example
Mathematical Formulation of Our Example We define two binary random variables: open and, where is light on or light off. Our question is: What is? Computer Vision 1 Combining Evidence Suppose our robot
More informationElements of Multivariate Time Series Analysis
Gregory C. Reinsel Elements of Multivariate Time Series Analysis Second Edition With 14 Figures Springer Contents Preface to the Second Edition Preface to the First Edition vii ix 1. Vector Time Series
More informationRobust Backtesting Tests for Value-at-Risk Models
Robust Backtesting Tests for Value-at-Risk Models Jose Olmo City University London (joint work with Juan Carlos Escanciano, Indiana University) Far East and South Asia Meeting of the Econometric Society
More information14 - Gaussian Stochastic Processes
14-1 Gaussian Stochastic Processes S. Lall, Stanford 211.2.24.1 14 - Gaussian Stochastic Processes Linear systems driven by IID noise Evolution of mean and covariance Example: mass-spring system Steady-state
More informationWeek 5 Quantitative Analysis of Financial Markets Characterizing Cycles
Week 5 Quantitative Analysis of Financial Markets Characterizing Cycles Christopher Ting http://www.mysmu.edu/faculty/christophert/ Christopher Ting : christopherting@smu.edu.sg : 6828 0364 : LKCSB 5036
More informationGaussian Message Passing on Linear Models: An Update
Int. Symp. on Turbo Codes & Related Topics, pril 2006 Gaussian Message Passing on Linear Models: n Update Hans-ndrea Loeliger 1, Junli Hu 1, Sascha Korl 2, Qinghua Guo 3, and Li Ping 3 1 Dept. of Information
More informationA HOMOTOPY CLASS OF SEMI-RECURSIVE CHAIN LADDER MODELS
A HOMOTOPY CLASS OF SEMI-RECURSIVE CHAIN LADDER MODELS Greg Taylor Taylor Fry Consulting Actuaries Level, 55 Clarence Street Sydney NSW 2000 Australia Professorial Associate Centre for Actuarial Studies
More informationIf we want to analyze experimental or simulated data we might encounter the following tasks:
Chapter 1 Introduction If we want to analyze experimental or simulated data we might encounter the following tasks: Characterization of the source of the signal and diagnosis Studying dependencies Prediction
More informationMCMC analysis of classical time series algorithms.
MCMC analysis of classical time series algorithms. mbalawata@yahoo.com Lappeenranta University of Technology Lappeenranta, 19.03.2009 Outline Introduction 1 Introduction 2 3 Series generation Box-Jenkins
More informationQ-Learning and Stochastic Approximation
MS&E338 Reinforcement Learning Lecture 4-04.11.018 Q-Learning and Stochastic Approximation Lecturer: Ben Van Roy Scribe: Christopher Lazarus Javier Sagastuy In this lecture we study the convergence of
More informationLinear Models for Regression CS534
Linear Models for Regression CS534 Prediction Problems Predict housing price based on House size, lot size, Location, # of rooms Predict stock price based on Price history of the past month Predict the
More informationDecomposition. bq (m n) R b (n n) r 11 r 1n
The QR Decomposition Lab Objective: The QR decomposition is a fundamentally important matrix factorization. It is straightforward to implement, is numerically stable, and provides the basis of several
More informationVAR Model. (k-variate) VAR(p) model (in the Reduced Form): Y t-2. Y t-1 = A + B 1. Y t + B 2. Y t-p. + ε t. + + B p. where:
VAR Model (k-variate VAR(p model (in the Reduced Form: where: Y t = A + B 1 Y t-1 + B 2 Y t-2 + + B p Y t-p + ε t Y t = (y 1t, y 2t,, y kt : a (k x 1 vector of time series variables A: a (k x 1 vector
More informationExercises - Time series analysis
Descriptive analysis of a time series (1) Estimate the trend of the series of gasoline consumption in Spain using a straight line in the period from 1945 to 1995 and generate forecasts for 24 months. Compare
More informationThe Kalman Filter. Data Assimilation & Inverse Problems from Weather Forecasting to Neuroscience. Sarah Dance
The Kalman Filter Data Assimilation & Inverse Problems from Weather Forecasting to Neuroscience Sarah Dance School of Mathematical and Physical Sciences, University of Reading s.l.dance@reading.ac.uk July
More informationDETECTION theory deals primarily with techniques for
ADVANCED SIGNAL PROCESSING SE Optimum Detection of Deterministic and Random Signals Stefan Tertinek Graz University of Technology turtle@sbox.tugraz.at Abstract This paper introduces various methods for
More informationMaximum Likelihood Diffusive Source Localization Based on Binary Observations
Maximum Lielihood Diffusive Source Localization Based on Binary Observations Yoav Levinboo and an F. Wong Wireless Information Networing Group, University of Florida Gainesville, Florida 32611-6130, USA
More informationInformation Formulation of the UDU Kalman Filter
Information Formulation of the UDU Kalman Filter Christopher D Souza and Renato Zanetti 1 Abstract A new information formulation of the Kalman filter is presented where the information matrix is parameterized
More informationLinear Processes in Function Spaces
D. Bosq Linear Processes in Function Spaces Theory and Applications Springer Preface Notation vi xi Synopsis 1 1. The object of study 1 2. Finite-dimensional linear processes 3 3. Random variables in function
More informationEconomics 573 Problem Set 5 Fall 2002 Due: 4 October b. The sample mean converges in probability to the population mean.
Economics 573 Problem Set 5 Fall 00 Due: 4 October 00 1. In random sampling from any population with E(X) = and Var(X) =, show (using Chebyshev's inequality) that sample mean converges in probability to..
More informationSTATIC AND DYNAMIC RECURSIVE LEAST SQUARES
STATC AND DYNAMC RECURSVE LEAST SQUARES 3rd February 2006 1 Problem #1: additional information Problem At step we want to solve by least squares A 1 b 1 A 1 A 2 b 2 A 2 A x b, A := A, b := b 1 b 2 b with
More informationAnalysis of Redundant-Wavelet Multihypothesis for Motion Compensation
Analysis of Redundant-Wavelet Multihypothesis for Motion Compensation James E. Fowler Department of Electrical and Computer Engineering GeoResources Institute GRI Mississippi State University, Starville,
More informationMultiple realizations: Model variance and data uncertainty
Stanford Exploration Project, Report 108, April 29, 2001, pages 1?? Multiple realizations: Model variance and data uncertainty Robert G. Clapp 1 ABSTRACT Geophysicists typically produce a single model,
More informationState Observers and the Kalman filter
Modelling and Control of Dynamic Systems State Observers and the Kalman filter Prof. Oreste S. Bursi University of Trento Page 1 Feedback System State variable feedback system: Control feedback law:u =
More informationParameter Estimation, Sampling Distributions & Hypothesis Testing
Parameter Estimation, Sampling Distributions & Hypothesis Testing Parameter Estimation & Hypothesis Testing In doing research, we are usually interested in some feature of a population distribution (which
More informationARIMA Modelling and Forecasting
ARIMA Modelling and Forecasting Economic time series often appear nonstationary, because of trends, seasonal patterns, cycles, etc. However, the differences may appear stationary. Δx t x t x t 1 (first
More informationDependence and independence
Roberto s Notes on Linear Algebra Chapter 7: Subspaces Section 1 Dependence and independence What you need to now already: Basic facts and operations involving Euclidean vectors. Matrices determinants
More informationThe Viterbi Algorithm EECS 869: Error Control Coding Fall 2009
1 Bacground Material 1.1 Organization of the Trellis The Viterbi Algorithm EECS 869: Error Control Coding Fall 2009 The Viterbi algorithm (VA) processes the (noisy) output sequence from a state machine
More informationProblem Set 2 Solution Sketches Time Series Analysis Spring 2010
Problem Set 2 Solution Sketches Time Series Analysis Spring 2010 Forecasting 1. Let X and Y be two random variables such that E(X 2 ) < and E(Y 2 )
More informationDetection and Estimation Theory
ESE 524 Detection and Estimation Theory Joseph A. O Sullivan Samuel C. Sachs Professor Electronic Systems and Signals Research Laboratory Electrical and Systems Engineering Washington University 2 Urbauer
More informationGradient-Adaptive Algorithms for Minimum Phase - All Pass Decomposition of an FIR System
1 Gradient-Adaptive Algorithms for Minimum Phase - All Pass Decomposition of an FIR System Mar F. Flanagan, Member, IEEE, Michael McLaughlin, and Anthony D. Fagan, Member, IEEE Abstract Adaptive algorithms
More information