Least Squares with Examples in Signal Processing 1. 2 Overdetermined equations. 1 Notation. The sum of squares of x is denoted by x 2 2, i.e.
|
|
- Brice Matthew Bridges
- 6 years ago
- Views:
Transcription
1 Least Squares with Eamples in Signal Processing Ivan Selesnick March 7, 3 NYU-Poly These notes address (approimate) solutions to linear equations by least squares We deal with the easy case wherein the system matri is full rank If the system matri is rank deficient, then other methods are needed, eg, QR decomposition, singular value decomposition, or the pseudo-inverse, [, 3] In these notes, least squares is illustrated by applying it to several basic problems in signal processing: Linear prediction Smoothing 3 Deconvolution 4 System identification 5 Estimating missing data For the use of least squares in filter design, see [] Notation We denote vectors in lower-case bold, ie, = () N We denote matrices in upper-case bold The transpose of a vector or matri in indicated by a superscript T, ie, T is the transpose of The notation refers to the Euclidian length of the vector, ie, = N () For feedback/corrections, selesi@polyedu Matlab software to reproduce the eamples in these notes is available on the web or from the author Last edit: November 6, 3 The sum of squares of is denoted by, ie, = n (n) = T (3) The energy of a vector refers to In these notes, it is assumed that all vectors and matrices are real-valued In the comple-valued case, the conjugate transpose should be used in place of the transpose, etc Overdetered equations Consider the system of linear equations y = H If there is no solution to this system of equations, then the system is overdetered This frequently happens when H is a tall matri (more rows than columns) with linearly independent columns In this case, it is common to seek a solution imizing the energy of the error: J() = y H Epanding J() gives J() = (y H) T (y H) (4) = y T y y T H T H T y + T H T H (5) = y T y y T H + T H T H (6) Note that each of the four terms in (5) are scalars Note also that the scalar T H T y is the transpose of the scalar y T H, and hence T H T y = y T H Taking the derivative (see Appendi A), gives J() = HT y + H T H Setting the derivative to zero, J() = = HT H = H T y
2 Let us assume that H T H is invertible Then the solution is given by = (H T H) H T y This is the least squares solution y H = = (H T H) H T y (7) In some situations, it is desirable to imize the weighted square error, ie, n w n r n where r is the residual, or error, r = y H, and w n are positive weights This corresponds to imizing W / (y H) where W is the diagonal matri, [W] n,n = w n Using (7) gives W / (y H) = = (H T WH) H T Wy (8) where we have used the fact that W is symmetric 3 Underdetered equations Consider the system of linear equations y = H Set the derivatives to zero to get: = HT µ () y = H () Plugging () into () gives y = HHT µ Let us assume HH T is invertible Then µ = (HH T ) y (3) Plugging (3) into () gives the least squares solution: = H T (HH T ) y We can verify that in this formula does in fact satisfy y = H by plugging in: If there are many solutions, then the system is underdetered This frequently happens when H is a wide matri (more columns than rows) with linearly independent rows In this case, it is common to seek a solution with imum norm That is, we would like to solve the optimization problem So, H = H [ H T (HH T ) y ] = (HH T )(HH T ) y = y st y = H = = H T (HH T ) y (4) (9) such that y = H () Minimization with constraints can be done with Lagrange multipliers So, define the Lagrangian: L(, µ) = + µ T (y H) Take the derivatives of the Lagrangian: L() = HT µ L() = y H µ In some situations, it is desirable to imize the weighted energy, ie, n w n n, where w n are positive weights This corresponds to imizing W / where W is the diagonal matri, [W] n,n = w n The derivation of the solution is similar, and gives W / st y = H = = W H T ( HW H T ) y This solution is also derived below, see (5) 4 Regularization (5) In the overdetered case, we imized y H In the underdetered case, we imized Another approach is to imize the
3 weighted sum: c y H + c The solution depends on the ratio c /c, not on c and c individually A common approach to obtain an ineact solution to a linear system is to imize the objective function: J() = y H + λ (6) Setting the derivative to zero, J() = = HT H + λa T A = H T y So the solution is given by = (H T H + λa T A) = H T y where λ > Taking the derivative, we get J() = HT (H y) + λ So, = (H T H + λa T A) H T y Setting the derivative to zero, J() = = HT H + λ = H T y So the solution is given by So, = (H T H + λi) H T y = (H T H + λi) = H T y y H + λ = = (H T H + λi) H T y (7) This is referred to as diagonal loading because a constant, λ, is added to the diagonal elements of H T H The approach also avoids the problem of rank deficiency because H T H + λi is invertible even if H T H is not In addition, the solution (7) can be used in both cases: when H is tall and when H is wide 5 Weighted regularization A more general form of the regularized objective function (6) is: J() = y H + λ A where λ > Taking the derivative, we get J() = HT (H y) + λa T A y H + λ A = = (H T H + λa T A) H T y Note that if A is the identity matri, then equation (8) becomes (7) 6 Constrained least squares (8) Constrained least squares refers to the problem of finding a least squares solution that eactly satisfies additional constraints If the additional constraints are a set of linear equations, then the solution is obtained as follows The constrained least squares problem is of the form: y H (9) such that C = b () Define the Lagrangian, L(, µ) = y H + µ T (C b) The derivatives are: L() = HT (H y) + C T µ L() = C b µ 3
4 Setting the derivatives to zero, L() = = = (HT H) (H T y 5C T µ) () L() = = C = b () µ Multiplying () on the left by C gives C, which from () is b, so we have C(H T H) (H T y 5C T µ) = b or, epanding, C(H T H) H T y 5C(H T H) C T µ = b Solving for µ gives µ = ( C(H T H) C T ) ( C(H T H) H T y b ) Plugging µ into () gives = (H T H) ( H T y C T ( C(H T H) C T ) ( C(H T H) H T y b )) Let us verify that in this formula does in fact satisfy C = b, So, C = C(H T H) ( H T y C T ( C(H T H) C T ) ( C(H T H) H T y b )) = C(H T H) H T y C(H T H) C T ( C(H T H) C T ) ( C(H T H) H T y b ) = C(H T H) H T y ( C(H T H) H T y b ) = b y H st C = b = = (H T H) ( H T y C T ( C(H T H) C T ) ( C(H T H) H T y b )) (3) 6 Special cases Simpler forms of (3) are frequently useful For eample, if H = I and b = in (3), then we get y st C = = = y C T ( CC T ) Cy (4) If y = in (3), then we get H st C = b = = (H T H) C T ( C(H T H) C T ) b (5) If y = and H = I in (3), then we get st C = b = = C T ( CC T ) b (6) which is the same as (4) 7 Note The epressions above involve matri inverses For eample, (7) involves (H T H) However, it must be emphasized that finding the least square solution does not require computing the inverse of H T H even though the inverse appears in the formula Instead, in (7) should be obtained, in practice, by solving the system A = b where A = H T H and b = H T y The most direct way to solve a linear system of equations is by Gaussian eliation Gaussian eliation is much faster than computing the inverse of the matri A 8 Eamples 8 Polynomial approimation An important eample of least squares is fitting a low-order polynomial to data Suppose the N-point data is of the form (t i, y i ) for i N The goal is to find a polynomial that approimates the data by imizing the energy of the residual: E = i (y i p(t i )) 4
5 where p(t) is a polynomial, eg, p(t) = a + a t + a t Data 5 5 Polynomial approimation (degree = ) 5 5 Polynomial approimation (degree = 4) 5 5 Figure : Least squares polynomial approimation The problem can be viewed as solving the overdetered system of equations, y t t y t t a a, a y N t N t N which we denote as y Ha The energy of the residual, E, is written as E = y Ha From (7), the least squares solution is given by a = (H T H) H T y An eample is illustrated in Fig 8 Linear prediction One approach to predict future values of a time-series is based on linear prediction, eg, y(n) a y(n ) + a y(n ) + a 3 y(n 3) (7) If past data y(n) is available, then the problem of finding a i can be solved using least squares Finding a = (a, a, a ) T can be viewed as one of solving an overdetered system of equations For eample, if y(n) is available for n N, and we seek a third order linear predictor, then the overdetered system of equations are given by y(3) y() y() y() y(4) y(3) y() y() a a, a 3 y(n ) y(n ) y(n 3) y(n 4) which we can write as ȳ = Ha where H is a matri of size (N 3) 3 From (7), the least squares solution is given by a = (H T H) H T ȳ Note that H T H is small, of size 3 3 only Hence, a is obtained by solving a small linear system of equations 5
6 Given data to be predicted Once the coefficients a i are found, then y(n) for n > N can be estimated using the recursive difference equation (7) An eample is illustrated in Fig One hundred samples of data are available, ie, y(n) for n 99 From these samples, a p-order linear predictor is obtained by least squares, and the subsequent samples are predicted 5 5 Predicted samples (deg = 3) p = Predicted samples (deg = 4) p = Predicted samples (deg = 6) p = Figure : Least squares linear prediction 83 Smoothing One approach to smooth a noisy signal is based on least squares weighted regularization The idea is to obtain a signal similar to the noisy one, but smoother The smoothness of a signal can be measured by the energy of its derivative (or second-order derivative) The smoother a signal is, the smaller the energy of its derivative is Define the matri D as D = (8) Then D is the second-order difference (a discrete form of the second-order derivative) of the signal (n) See Appendi B If is smooth, then D is small in value If y(n) is a noisy signal, then a smooth signal (n), that approimates y(n), can be obtained as the solution to the problem: y + λ D (9) where λ > is a parameter to be specified Minimizing y forces to be similar to the noisy signal y Minimizing D forces to be smooth Minimizing the sum in (9) forces to be both similar to y and smooth (as far as possible, and depending on λ) If λ =, then the solution will be the noisy data, ie, = y, because this solution makes (9) equal to zero In this case, no smoothing is achieved On the other hand, the greater λ is, the smoother will be Using (8), the signal imizing (9) is given by = (I + λd T D) y (3) 6
7 5 5 Noisy data Least squares smoothing Figure 3: Least squares smoothing Note that the matri I + λd T D is banded (The only non-zero values are near the main diagonal) Therefore, the solution can be obtained using fast solvers for banded systems [5, Sect 4] An eample of least squares smoothing is illustrated in Fig 3 A noisy ECG signal is smoothed using (3) We have used the ECG waveform generator ECGSYN [4] 84 Deconvolution Deconvolution refers to the problem of finding the input to an LTI system when the output signal is known Here, we assume the impulse response of the system is known The output, y(n), is given by y(n) = h() (n) + h() (n ) + + h(n) (n N) (3) where (n) is the input signal and h(n) is the impulse response Equation (3) can be written as y = H where H is a matri of the form h() h() h() H = h() h() h() This matri is constant-valued along its diagonals Such matrices are called Toeplitz matrices It may be epected that can be obtained from y by solving the linear system y = H In some situations, this is possible However, the matri H is often singular or almost singular In this case, Gaussian eliation encounters division by zeros For eample, Fig 4 illustrates an input signal, (n), an impulse response, h(n), and the output signal, y(n) When we attempt to obtain by solving y = H in Matlab, we receive the warning message: Matri is singular to working precision and we obtain a vector of all NaN (not a number) Due to H being singular, we regularize the problem Note that the input signal in Fig 4 is mostly zero, hence, it is reasonable to seek a solution with small energy The signal we seek should also satisfy y = H, at least approimately To obtain such a signal,, we solve the problem: y H + λ (3) where λ > is a parameter to be specified Minimizing y H forces to be consistent with the output signal y Minimizing forces to have low energy Minimizing the sum in (3) forces to be consistent with y and to have low energy (as far as possible, and depending on λ) Using (8), the signal imizing (3) is given by = (H T H + λi) H T y (33) This technique is called diagonal loading because λ is added the diagonal of H T H A small value of λ is sufficient to make the matri invertible The solution, illustrated in Fig 4, is very similar to the original input signal, shown in the figure In practice, the available data is also noisy In this case, the data y is given by y = H + w where w is the noise The noise is often modeled as an additive white Gaussian random signal In this case, diagonal loading with a small λ will generally produce a noisy estimate of the input signal In Fig 4, we used λ = When the same value is used with the noisy data, a noisy result is obtained, as illustrated in Fig 5 A larger λ is needed so as to attenuate the noise But if λ is too large, then the estimate of the input signal is distorted Notice that with λ =, the noise is reduced 7
8 3 Input signal Impulse response Output signal (noise free) Deconvolution (diagonal loading) λ = Figure 4: Deconvolution of noise-free data by diagonal loading but the height of the the pulses present in the original signal are somewhat attenuated With λ = 5, the noise is reduced slightly more, but the pulses are substantially more attenuated To improve the deconvolution result in the presence of noise, we can imize the energy of the derivative (or second-order derivative) of instead As in the smoothing eample above, imizing the energy of the second-order derivative forces to be smooth In order that is consistent with the data y and is also smooth, we solve the problem: y H + λ D (34) where D is the second-order difference matri (8) Using (8), the signal imizing (34) is given by = (H T H + λd T D) H T y (35) The solution obtained using (35) is illustrated in Fig 6 Compared to the solutions obtained by diagonal loading, illustrated in Fig 5, this solution is less noisy and less distorted This eample illustrates the need for regularization even when the data is noise-free (an unrealistic ideal case) It also illustrates the choice of regularizer (ie,, D, or other) affects the quality of the result 85 System identification System identification refers to the problem of estimating an unknown system In its simplest form, the system is LTI and input-output data is available Here, we assume that the output signal is noisy We also assume that the impulse response is relatively short The output, y(n), of the system can be written as y(n) = h (n) + h (n ) + h (n ) + w(n) (36) where (n) is the input signal and w(n) is the noise Here, we have assumed the impulse response h n is of length 3 We can write this in matri form as y y y y 3 3 h h h 8
9 Output signal (noisy) Output signal (noisy) Deconvolution (derivative regularization) 3 Deconvolution (diagonal loading) λ = λ = Figure 6: Deconvolution of noisy data by derivative regularization 3 Deconvolution (diagonal loading) which we denote as y Xh If y is much longer than the length of the impulse response h, then X is a tall matri and y Xh is an overdetered system of equations In this case, h can be estimated from (7) as h = (X T X) X T y (37) λ = Deconvolution (diagonal loading) λ = Figure 5: Deconvolution of noisy data by diagonal loading An eample is illustrated in Fig 7 A binary input signal and noisy output signal are shown When it is assumed that h is of length, then we obtain the impulse response shown The residual, ie, r = y Xh, is also shown in the figure It is informative to plot the root-mean-square-error (RMSE), ie, r, as a function of the length of the impulse response This is a decreasing function If the data really is the input-output data of an LTI system with a finite impulse response, and if the noise is not too severe, then the RMSE tends to flatten out at the correct impulse response length This provides an indication of the length of the unknown impulse response 86 Missing sample estimation Due to transmission errors, transient interference, or impulsive noise, some samples of a signal may be lost or so badly corrupted as to be unusable In 9
10 Input signal 5 5 Signal with missing samples Output signal (noisy) 5 5 Estimated samples Estimated impulse response (length ) Final output RMSE Residual (RMSE = 83) RMSE vs impulse response length 5 5 Length of impulse response Figure 7: Least squares system identification Figure 8: Least squares estimation of missing data this case, the missing samples should be estimated based on the available uncorrupted data To complicate the problem, the missing samples may be randomly distributed through out the signal Filling in missing values in order to conceal errors is called error concealment [6] This eample shows how the missing samples can be estimated by least squares As an eample, Fig 8 shows a -point ECG signal wherein samples are missing The problem is to fill in the missing samples To formulate the problem as a least squares problem, we introduce some notation Let be a signal of length N Suppose K samples of are known, where K < N The K-point known signal, y, can be written as y = S (38) where S is a selection (or sampling ) matri of size K N For eample,
11 if only the first, second and last elements of a 5-point signal are observed, then the matri S is given by S = (39) The matri S is the identity matri with rows removed, corresponding to the missing samples Note that S removes two samples from the signal, () () () S = () = () = y (4) (3) (4) (4) The vector y consists of the known samples of So, the vector y is shorter than (K < N) The problem can be stated as: Given the signal, y, and the matri, S, find such that y = S Of course, there are infinitely many solutions Below, it is shown how to obtain a smooth solution by least squares Note that S T y has the effect of setting the missing samples to zero For eample, with S in (39) we have y() y() y() S T y = y() = (4) y() y() Let us define S c as the complement of S The matri S c consists of the rows of the identity matri not appearing in S Continuing the 5-point eample, S c = [ Now, an estimate ˆ can be represented as ] (4) ˆ = S T y + S T c v (43) where y is the available data and v consists of the samples to be detered For eample, y() y() [ ] y() S T y + S T c v = y() + v() = v() v() (44) y() v() y() The problem is to estimate the vector v, which is of length N K Let us assume that the original signal,, is smooth Then it is reasonable to find v to optimize the smoothness of ˆ, ie, to imize the energy of the second-order derivative of ˆ Therefore, v can be obtained by imizing Dˆ where D is the second-order difference matri (8) Using (43), we find v by solving the problem ie, v D(S T y + S T c v), (45) v DS T y + DS T c v (46) From (7), the solution is given by v = (S c D T D S T c ) S c D T D S T y (47) Once v is obtained, the estimate ˆ in (43) can be constructed simply by inserting entries v(i) into y An eample of least square estimation of missing data using (47) is illustrated in Fig 8 The result is a smoothly interpolated signal We make several remarks All matrices in (47) are banded, so the computation of v can be implemented with very high efficiency using a fast solver for banded systems [5, Sect 4] The banded property of the matri G = S c D T D S T c (48) arising in (47) is illustrated in Fig 9
12 nz = 8 Figure 9: Visualization of the banded matri G in (48) All the non-zero values lie near the main diagonal The method does not require the pattern of missing samples to have any particular structure The missing samples can be distributed quite randomly 3 This method (47) does not require any regularization parameter λ be specified However, this derivation does assume the available data, y, is noise free If y is noisy, then simultaneous smoothing and missing sample estimation is required (see the Eercises) Speech de-clipping: In audio recording, if the amplitude of the audio source is too high, then the recorded waveform may suffer from clipping (ie, saturation) Figure shows a speech waveform that is clipped All values greater than in absolute value are lost due to clipping To estimate the missing data, we can use the least squares approach given by (47) That is, we fill in the missing data so as to imize the energy of the derivative of the total signal In this eample, we imize the energy of the the third derivative This encourages the filled in data to have the form of a parabola (second order polynomial), because the third derivative of a parabola is zero In order to use (47) for this problem, we Clipped speech with 39 missing samples Estimated samples Final output Figure : Estimation of speech waveform samples lost due to clipping The lost samples are estimated by least squares only need to change the matri D to the following one If we define the matri D as D = , (49) then D is an approimation of the third-order derivative of the signal Using (47) with D defined in (49), we obtain the signal shown in the Fig The samples lost due to clipping have been smoothly filled in Figure shows both, the clipped signal and the estimated samples, on the same ais
13 Filled in (red) Clipped data (blue) eperiments and compare results of -nd and 3-rd order difference matrices Figure : The available clipped speech waveform is shown in blue The filled in signal, estimated by least squares, is shown in red 9 Eercises Find the solution to the least squares problem: y A + λ b Show that the solution to the least squares problem is λ b A + λ b A + λ 3 b 3 A 3 = ( λ A T A + λ A T A + λ 3 A T 3 A 3 ) ( λ A T b + λ A T b + λ 3 A T 3 b 3 ) (5) 3 In reference to (7), why is H T H + λi with λ > invertible even if H T H is not? 4 Show (56) 5 Smoothing Demonstrate least square smoothing of noisy data Use various values of λ What behavior do you observe when λ is very high? 6 The second-order difference matri (8) was used in the eamples for smoothing, deconvolution, and estimating missing samples Discuss the use of the third-order difference instead Perform numerical A 7 System identification Perform a system identification eperiment with varying variance of additive Gaussian noise Plot the RMSE versus impulse response length How does the plot of RMSE change with respect to the variance of the noise? 8 Speech de-clipping Record your own speech and use it to artificially create a clipped signal Perform numerical eperiments to test the least square estimation of the lost samples 9 Suppose the available data is both noisy and that some samples are missing Formulate a suitable least squares optimization problem to smooth the data and recover the missing samples Illustrate the effectiveness by a numerical demonstration (eg using Matlab) Vector derivatives If f() is a function of,, N, then the derivative of f() with respect to is the vector of derivatives, f() f() f() = (5) f() N This is the gradient of f, denoted f By direct calculation, we have and T b = b (5) bt = b (53) Suppose that A is a symmetric real matri, A T = A Then, by direct calculation, we also have T A = A (54) 3
14 Also, and (y )T A(y ) = A( y), (55) A b = A T (A b) (56) We illustrate (54) by an eample Set A as the matri, [ ] 3 A = (57) 5 Then, by direct calculation [ ] [ ] [ T 3 A = 5 so and ] = (58) ( T A ) = (59) ( T A ) = 4 + (6) Let us verify that the right-hand side of (54) gives the same: [ ] [ ] [ ] A = = (6) B The Kth-order difference The first order difference of a discrete-time signal (n), n Z, is defined as y(n) = (n) (n ) (6) This is represented as a system with input and output y, The second order difference is obtained by taking the first order difference twice: D D y which give the difference equation y(n) = (n) (n ) (n ) (63) The third order difference is obtained by taking the first order difference three times: D D D y which give the difference equation y(n) = (n) 3 (n ) + 3 (n ) (n 3) (64) In terms of discrete-time linear time-invariant systems (LTI), the first order difference is an LTI system with transfer function D(z) = z The second order difference has the transfer function D (z) = ( z ) = z + z The third order difference has the transfer function D 3 (z) = ( z ) 3 = 3z + 3z z 3 Note that the coefficient come from Pascal s triangle: D y 4
15 C Additional Eercises for Signal Processing Students For smoothing a noisy signal using least squares, we obtained (3), = (I + λd T D) y, λ > The matri G = (I + λd T D) can be understood as a low-pass filter Using Matlab, compute and plot the output, y(n), when the input is an impulse, (n) = δ(n n o ) This is an impulse located at inde n o Try placing the impulse at various points in the signal For eample, put the impulse around the middle of the signal What happens when the impulse is located near the ends of the signal (n o = or n o = N )? [5] W H Press, S A Teukolsky, W T Vetterling, and B P Flannery Numerical recipes in C: the art of scientific computing (nd ed) Cambridge University Press, 99 [6] Y Wang and Q Zhu Error control and concealment for video communication: A review Proc IEEE, 86(5): , May 998 For smoothing a noisy signal using least squares, we obtained (3), = (I + λd T D) y, λ >, where D represents the K-th order derivative Assume D is the first-order difference and that the matrices I and D are infinite is size Then I + λd T D is a convolution matri and represents an LTI system (a) Find and sketch the frequency response of the system I + λd T D (b) Based on (a), find the frequency response of G = (I + λd T D) Sketch the frequency response of G What kind of filter is it (low-pass, high-pass, band-pass, etc)? What is its dc gain? References [] C Burrus Least squared error design of FIR filters Conneions Web site, [] C Burrus General solutions of simultaneous equations Conneions Web site, 3 [3] C L Lawson and R J Hanson Solving Least Squares Problems SIAM, 987 [4] P E McSharry, G D Clifford, L Tarassenko, and L A Smith A dynamical model for generating synthetic electrocardiogram signals Trans on Biomed Eng, 5(3):89 94, March 3 5
LECTURE 7. Least Squares and Variants. Optimization Models EE 127 / EE 227AT. Outline. Least Squares. Notes. Notes. Notes. Notes.
Optimization Models EE 127 / EE 227AT Laurent El Ghaoui EECS department UC Berkeley Spring 2015 Sp 15 1 / 23 LECTURE 7 Least Squares and Variants If others would but reflect on mathematical truths as deeply
More informationGeometric Modeling Summer Semester 2010 Mathematical Tools (1)
Geometric Modeling Summer Semester 2010 Mathematical Tools (1) Recap: Linear Algebra Today... Topics: Mathematical Background Linear algebra Analysis & differential geometry Numerical techniques Geometric
More informationStatistical Geometry Processing Winter Semester 2011/2012
Statistical Geometry Processing Winter Semester 2011/2012 Linear Algebra, Function Spaces & Inverse Problems Vector and Function Spaces 3 Vectors vectors are arrows in space classically: 2 or 3 dim. Euclidian
More informationSPARSITY-ASSISTED SIGNAL SMOOTHING (REVISITED) Ivan Selesnick
SPARSITY-ASSISTED SIGNAL SMOOTHING (REVISITED) Ivan Selesnick Electrical and Computer Engineering Tandon School of Engineering, New York University Brooklyn, New York, USA ABSTRACT This paper proposes
More informationIntroduction to Sparsity in Signal Processing
1 Introduction to Sparsity in Signal Processing Ivan Selesnick Polytechnic Institute of New York University Brooklyn, New York selesi@poly.edu 212 2 Under-determined linear equations Consider a system
More informationFitting functions to data
1 Fitting functions to data 1.1 Exact fitting 1.1.1 Introduction Suppose we have a set of real-number data pairs x i, y i, i = 1, 2,, N. These can be considered to be a set of points in the xy-plane. They
More informationLesson 1. Optimal signalbehandling LTH. September Statistical Digital Signal Processing and Modeling, Hayes, M:
Lesson 1 Optimal Signal Processing Optimal signalbehandling LTH September 2013 Statistical Digital Signal Processing and Modeling, Hayes, M: John Wiley & Sons, 1996. ISBN 0471594318 Nedelko Grbic Mtrl
More informationOn the Frequency-Domain Properties of Savitzky-Golay Filters
On the Frequency-Domain Properties of Savitzky-Golay Filters Ronald W Schafer HP Laboratories HPL-2-9 Keyword(s): Savitzky-Golay filter, least-squares polynomial approximation, smoothing Abstract: This
More informationIntroduction to Sparsity in Signal Processing
1 Introduction to Sparsity in Signal Processing Ivan Selesnick Polytechnic Institute of New York University Brooklyn, New York selesi@poly.edu 212 2 Under-determined linear equations Consider a system
More informationStatistical and Adaptive Signal Processing
r Statistical and Adaptive Signal Processing Spectral Estimation, Signal Modeling, Adaptive Filtering and Array Processing Dimitris G. Manolakis Massachusetts Institute of Technology Lincoln Laboratory
More informationSPARSE SIGNAL RESTORATION. 1. Introduction
SPARSE SIGNAL RESTORATION IVAN W. SELESNICK 1. Introduction These notes describe an approach for the restoration of degraded signals using sparsity. This approach, which has become quite popular, is useful
More informationAssignment #9: Orthogonal Projections, Gram-Schmidt, and Least Squares. Name:
Assignment 9: Orthogonal Projections, Gram-Schmidt, and Least Squares Due date: Friday, April 0, 08 (:pm) Name: Section Number Assignment 9: Orthogonal Projections, Gram-Schmidt, and Least Squares Due
More informationInverses. Stephen Boyd. EE103 Stanford University. October 28, 2017
Inverses Stephen Boyd EE103 Stanford University October 28, 2017 Outline Left and right inverses Inverse Solving linear equations Examples Pseudo-inverse Left and right inverses 2 Left inverses a number
More informationLecture 11 FIR Filters
Lecture 11 FIR Filters Fundamentals of Digital Signal Processing Spring, 2012 Wei-Ta Chu 2012/4/12 1 The Unit Impulse Sequence Any sequence can be represented in this way. The equation is true if k ranges
More information0.1. Linear transformations
Suggestions for midterm review #3 The repetitoria are usually not complete; I am merely bringing up the points that many people didn t now on the recitations Linear transformations The following mostly
More informationChap 3. Linear Algebra
Chap 3. Linear Algebra Outlines 1. Introduction 2. Basis, Representation, and Orthonormalization 3. Linear Algebraic Equations 4. Similarity Transformation 5. Diagonal Form and Jordan Form 6. Functions
More informationIndex. book 2009/5/27 page 121. (Page numbers set in bold type indicate the definition of an entry.)
page 121 Index (Page numbers set in bold type indicate the definition of an entry.) A absolute error...26 componentwise...31 in subtraction...27 normwise...31 angle in least squares problem...98,99 approximation
More informationPOLYNOMIAL SINGULAR VALUES FOR NUMBER OF WIDEBAND SOURCES ESTIMATION AND PRINCIPAL COMPONENT ANALYSIS
POLYNOMIAL SINGULAR VALUES FOR NUMBER OF WIDEBAND SOURCES ESTIMATION AND PRINCIPAL COMPONENT ANALYSIS Russell H. Lambert RF and Advanced Mixed Signal Unit Broadcom Pasadena, CA USA russ@broadcom.com Marcel
More informationBasic Concepts in Linear Algebra
Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39 Numerical Linear Algebra Linear
More informationDiscrete-Time Systems
FIR Filters With this chapter we turn to systems as opposed to signals. The systems discussed in this chapter are finite impulse response (FIR) digital filters. The term digital filter arises because these
More informationIntroduction to Sparsity in Signal Processing 1
Introduction to Sparsity in Signal Processing Ivan Selesnick June, NYU-Poly Introduction These notes describe how sparsity can be used in several signal processing problems. A common theme throughout these
More informationLecture 6. Numerical methods. Approximation of functions
Lecture 6 Numerical methods Approximation of functions Lecture 6 OUTLINE 1. Approximation and interpolation 2. Least-square method basis functions design matrix residual weighted least squares normal equation
More informationReview of Basic Concepts in Linear Algebra
Review of Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University September 7, 2017 Math 565 Linear Algebra Review September 7, 2017 1 / 40 Numerical Linear Algebra
More informationGeorge Mason University ECE 201: Introduction to Signal Analysis Spring 2017
Assigned: March 20, 2017 Due Date: Week of April 03, 2017 George Mason University ECE 201: Introduction to Signal Analysis Spring 2017 Laboratory Project #6 Due Date Your lab report must be submitted on
More informationLecture 7: Weak Duality
EE 227A: Conve Optimization and Applications February 7, 2012 Lecture 7: Weak Duality Lecturer: Laurent El Ghaoui 7.1 Lagrange Dual problem 7.1.1 Primal problem In this section, we consider a possibly
More informationTrace balancing with PEF plane annihilators
Stanford Exploration Project, Report 84, May 9, 2001, pages 1 333 Short Note Trace balancing with PEF plane annihilators Sean Crawley 1 INTRODUCTION Land seismic data often shows large variations in energy
More information4. Matrix inverses. left and right inverse. linear independence. nonsingular matrices. matrices with linearly independent columns
L. Vandenberghe ECE133A (Winter 2018) 4. Matrix inverses left and right inverse linear independence nonsingular matrices matrices with linearly independent columns matrices with linearly independent rows
More informationExpanding brackets and factorising
CHAPTER 8 Epanding brackets and factorising 8 CHAPTER Epanding brackets and factorising 8.1 Epanding brackets There are three rows. Each row has n students. The number of students is 3 n 3n. Two students
More informationCOUNCIL ROCK HIGH SCHOOL MATHEMATICS. A Note Guideline of Algebraic Concepts. Designed to assist students in A Summer Review of Algebra
COUNCIL ROCK HIGH SCHOOL MATHEMATICS A Note Guideline of Algebraic Concepts Designed to assist students in A Summer Review of Algebra [A teacher prepared compilation of the 7 Algebraic concepts deemed
More informationPROBLEMS In each of Problems 1 through 12:
6.5 Impulse Functions 33 which is the formal solution of the given problem. It is also possible to write y in the form 0, t < 5, y = 5 e (t 5/ sin 5 (t 5, t 5. ( The graph of Eq. ( is shown in Figure 6.5.3.
More informationAdvanced Digital Signal Processing -Introduction
Advanced Digital Signal Processing -Introduction LECTURE-2 1 AP9211- ADVANCED DIGITAL SIGNAL PROCESSING UNIT I DISCRETE RANDOM SIGNAL PROCESSING Discrete Random Processes- Ensemble Averages, Stationary
More informationLU Factorization. A m x n matrix A admits an LU factorization if it can be written in the form of A = LU
LU Factorization A m n matri A admits an LU factorization if it can be written in the form of Where, A = LU L : is a m m lower triangular matri with s on the diagonal. The matri L is invertible and is
More informationBasic Linear Inverse Method Theory - DRAFT NOTES
Basic Linear Inverse Method Theory - DRAFT NOTES Peter P. Jones 1 1 Centre for Compleity Science, University of Warwick, Coventry CV4 7AL, UK (Dated: 21st June 2012) BASIC LINEAR INVERSE METHOD THEORY
More informationVector Spaces, Orthogonality, and Linear Least Squares
Week Vector Spaces, Orthogonality, and Linear Least Squares. Opening Remarks.. Visualizing Planes, Lines, and Solutions Consider the following system of linear equations from the opener for Week 9: χ χ
More informationLinear Models in Machine Learning
CS540 Intro to AI Linear Models in Machine Learning Lecturer: Xiaojin Zhu jerryzhu@cs.wisc.edu We briefly go over two linear models frequently used in machine learning: linear regression for, well, regression,
More informationLinear Least-Squares Data Fitting
CHAPTER 6 Linear Least-Squares Data Fitting 61 Introduction Recall that in chapter 3 we were discussing linear systems of equations, written in shorthand in the form Ax = b In chapter 3, we just considered
More informationPolynomial Functions of Higher Degree
SAMPLE CHAPTER. NOT FOR DISTRIBUTION. 4 Polynomial Functions of Higher Degree Polynomial functions of degree greater than 2 can be used to model data such as the annual temperature fluctuations in Daytona
More informationMobile Robotics 1. A Compact Course on Linear Algebra. Giorgio Grisetti
Mobile Robotics 1 A Compact Course on Linear Algebra Giorgio Grisetti SA-1 Vectors Arrays of numbers They represent a point in a n dimensional space 2 Vectors: Scalar Product Scalar-Vector Product Changes
More informationExamples. 2-input, 1-output discrete-time systems: 1-input, 1-output discrete-time systems:
Discrete-Time s - I Time-Domain Representation CHAPTER 4 These lecture slides are based on "Digital Signal Processing: A Computer-Based Approach, 4th ed." textbook by S.K. Mitra and its instructor materials.
More informationELECTRONOTES APPLICATION NOTE NO Hanshaw Road Ithaca, NY Mar 6, 2015
ELECTRONOTES APPLICATION NOTE NO. 422 1016 Hanshaw Road Ithaca, NY 14850 Mar 6, 2015 NOTCH FILTER AS A WASHED-OUT COMB INTRODUCTION: We recently reviewed notch filters [1] and thought of them as a class
More informationLecture Notes 5: Multiresolution Analysis
Optimization-based data analysis Fall 2017 Lecture Notes 5: Multiresolution Analysis 1 Frames A frame is a generalization of an orthonormal basis. The inner products between the vectors in a frame and
More informationMidterm Solutions. EE127A L. El Ghaoui 3/19/11
EE27A L. El Ghaoui 3/9/ Midterm Solutions. (6 points Find the projection z of the vector = (2, on the line that passes through 0 = (, 2 and with direction given by the vector u = (,. Solution: The line
More informationMultistat2. The course is focused on the parts in bold; a short description will be presented for the other parts
1 2 The course is focused on the parts in bold; a short description will be presented for the other parts 3 4 5 What I cannot teach you: how to look for relevant literature, how to use statistical software
More informationNumerical Integration (Quadrature) Another application for our interpolation tools!
Numerical Integration (Quadrature) Another application for our interpolation tools! Integration: Area under a curve Curve = data or function Integrating data Finite number of data points spacing specified
More informationLinear Algebra. and
Instructions Please answer the six problems on your own paper. These are essay questions: you should write in complete sentences. 1. Are the two matrices 1 2 2 1 3 5 2 7 and 1 1 1 4 4 2 5 5 2 row equivalent?
More informationA Padé approximation to the scalar wavefield extrapolator for inhomogeneous media
A Padé approimation A Padé approimation to the scalar wavefield etrapolator for inhomogeneous media Yanpeng Mi, Zhengsheng Yao, and Gary F. Margrave ABSTRACT A seismic wavefield at depth z can be obtained
More informationSignal Processing COS 323
Signal Processing COS 323 Digital Signals D: functions of space or time e.g., sound 2D: often functions of 2 spatial dimensions e.g. images 3D: functions of 3 spatial dimensions CAT, MRI scans or 2 space,
More informationNOISE ROBUST RELATIVE TRANSFER FUNCTION ESTIMATION. M. Schwab, P. Noll, and T. Sikora. Technical University Berlin, Germany Communication System Group
NOISE ROBUST RELATIVE TRANSFER FUNCTION ESTIMATION M. Schwab, P. Noll, and T. Sikora Technical University Berlin, Germany Communication System Group Einsteinufer 17, 1557 Berlin (Germany) {schwab noll
More informationA structured low-rank approximation approach to system identification. Ivan Markovsky
1 / 35 A structured low-rank approximation approach to system identification Ivan Markovsky Main message: system identification SLRA minimize over B dist(a,b) subject to rank(b) r and B structured (SLRA)
More informationNotes on Eigenvalues, Singular Values and QR
Notes on Eigenvalues, Singular Values and QR Michael Overton, Numerical Computing, Spring 2017 March 30, 2017 1 Eigenvalues Everyone who has studied linear algebra knows the definition: given a square
More informationMatrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =
30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can
More informationIntroduction to Data Mining
Introduction to Data Mining Lecture #21: Dimensionality Reduction Seoul National University 1 In This Lecture Understand the motivation and applications of dimensionality reduction Learn the definition
More informationY m = y n e 2πi(m 1)(n 1)/N (1) Y m e 2πi(m 1)(n 1)/N (2) m=1
The Discrete Fourier Transform (Bretherton notes): 1 Definition Let y n, n = 1,..., N be a sequence of N possibly comple values. The discrete Fourier transform () of this sequence is the sequence Y m,
More informationCSCI5654 (Linear Programming, Fall 2013) Lectures Lectures 10,11 Slide# 1
CSCI5654 (Linear Programming, Fall 2013) Lectures 10-12 Lectures 10,11 Slide# 1 Today s Lecture 1. Introduction to norms: L 1,L 2,L. 2. Casting absolute value and max operators. 3. Norm minimization problems.
More informationLecture 19 IIR Filters
Lecture 19 IIR Filters Fundamentals of Digital Signal Processing Spring, 2012 Wei-Ta Chu 2012/5/10 1 General IIR Difference Equation IIR system: infinite-impulse response system The most general class
More informationReview of some mathematical tools
MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical
More informationRobustness of Principal Components
PCA for Clustering An objective of principal components analysis is to identify linear combinations of the original variables that are useful in accounting for the variation in those original variables.
More informationSection 3.3 Graphs of Polynomial Functions
3.3 Graphs of Polynomial Functions 179 Section 3.3 Graphs of Polynomial Functions In the previous section we eplored the short run behavior of quadratics, a special case of polynomials. In this section
More informationIntroduction to Mobile Robotics Compact Course on Linear Algebra. Wolfram Burgard, Bastian Steder
Introduction to Mobile Robotics Compact Course on Linear Algebra Wolfram Burgard, Bastian Steder Reference Book Thrun, Burgard, and Fox: Probabilistic Robotics Vectors Arrays of numbers Vectors represent
More informationpset3-sol September 7, 2017
pset3-sol September 7, 2017 1 18.06 pset 3 Solutions 1.1 Problem 1 Suppose that you solve AX = B with and find that X is 1 1 1 1 B = 0 2 2 2 1 1 0 1 1 1 0 1 X = 1 0 1 3 1 0 2 1 1.1.1 (a) What is A 1? (You
More informationDesigning Information Devices and Systems I Fall 2018 Homework 5
Last Updated: 08-09-9 0:6 EECS 6A Designing Information Devices and Systems I Fall 08 Homework 5 This homework is due September 8, 08, at 3:59. Self-grades are due October, 08, at 3:59. Submission Format
More informationCS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares
CS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares Robert Bridson October 29, 2008 1 Hessian Problems in Newton Last time we fixed one of plain Newton s problems by introducing line search
More informationGeneralized Pencil Of Function Method
Generalized Pencil Of Function Method In recent times, the methodology of approximating a function by a sum of complex exponentials has found applications in many areas of electromagnetics For example
More informationCh. 7: Z-transform Reading
c J. Fessler, June 9, 3, 6:3 (student version) 7. Ch. 7: Z-transform Definition Properties linearity / superposition time shift convolution: y[n] =h[n] x[n] Y (z) =H(z) X(z) Inverse z-transform by coefficient
More information4. Linear transformations as a vector space 17
4 Linear transformations as a vector space 17 d) 1 2 0 0 1 2 0 0 1 0 0 0 1 2 3 4 32 Let a linear transformation in R 2 be the reflection in the line = x 2 Find its matrix 33 For each linear transformation
More informationMathematical Methods in Engineering and Science Prof. Bhaskar Dasgupta Department of Mechanical Engineering Indian Institute of Technology, Kanpur
Mathematical Methods in Engineering and Science Prof. Bhaskar Dasgupta Department of Mechanical Engineering Indian Institute of Technology, Kanpur Module - I Solution of Linear Systems Lecture - 05 Ill-Conditioned
More informationComputational Methods. Least Squares Approximation/Optimization
Computational Methods Least Squares Approximation/Optimization Manfred Huber 2011 1 Least Squares Least squares methods are aimed at finding approximate solutions when no precise solution exists Find the
More informationMath Analysis Chapter 2 Notes: Polynomial and Rational Functions
Math Analysis Chapter Notes: Polynomial and Rational Functions Day 13: Section -1 Comple Numbers; Sections - Quadratic Functions -1: Comple Numbers After completing section -1 you should be able to do
More informationLinear Regression (9/11/13)
STA561: Probabilistic machine learning Linear Regression (9/11/13) Lecturer: Barbara Engelhardt Scribes: Zachary Abzug, Mike Gloudemans, Zhuosheng Gu, Zhao Song 1 Why use linear regression? Figure 1: Scatter
More informationJeffrey D. Ullman Stanford University
Jeffrey D. Ullman Stanford University 2 Often, our data can be represented by an m-by-n matrix. And this matrix can be closely approximated by the product of two matrices that share a small common dimension
More informationDS-GA 1002 Lecture notes 10 November 23, Linear models
DS-GA 2 Lecture notes November 23, 2 Linear functions Linear models A linear model encodes the assumption that two quantities are linearly related. Mathematically, this is characterized using linear functions.
More informationData-driven signal processing
1 / 35 Data-driven signal processing Ivan Markovsky 2 / 35 Modern signal processing is model-based 1. system identification prior information model structure 2. model-based design identification data parameter
More informationNumerical Analysis: Solving Systems of Linear Equations
Numerical Analysis: Solving Systems of Linear Equations Mirko Navara http://cmpfelkcvutcz/ navara/ Center for Machine Perception, Department of Cybernetics, FEE, CTU Karlovo náměstí, building G, office
More informationMultivariate Statistical Analysis
Multivariate Statistical Analysis Fall 2011 C. L. Williams, Ph.D. Lecture 4 for Applied Multivariate Analysis Outline 1 Eigen values and eigen vectors Characteristic equation Some properties of eigendecompositions
More informationIdentification of MIMO linear models: introduction to subspace methods
Identification of MIMO linear models: introduction to subspace methods Marco Lovera Dipartimento di Scienze e Tecnologie Aerospaziali Politecnico di Milano marco.lovera@polimi.it State space identification
More information6.02 Fall 2012 Lecture #10
6.02 Fall 2012 Lecture #10 Linear time-invariant (LTI) models Convolution 6.02 Fall 2012 Lecture 10, Slide #1 Modeling Channel Behavior codeword bits in generate x[n] 1001110101 digitized modulate DAC
More informationLecture Note 2: Estimation and Information Theory
Univ. of Michigan - NAME 568/EECS 568/ROB 530 Winter 2018 Lecture Note 2: Estimation and Information Theory Lecturer: Maani Ghaffari Jadidi Date: April 6, 2018 2.1 Estimation A static estimation problem
More informationMath 307 Learning Goals. March 23, 2010
Math 307 Learning Goals March 23, 2010 Course Description The course presents core concepts of linear algebra by focusing on applications in Science and Engineering. Examples of applications from recent
More informationTheory and Problems of Signals and Systems
SCHAUM'S OUTLINES OF Theory and Problems of Signals and Systems HWEI P. HSU is Professor of Electrical Engineering at Fairleigh Dickinson University. He received his B.S. from National Taiwan University
More informationPolynomials and Polynomial Functions
Unit 5: Polynomials and Polynomial Functions Evaluating Polynomial Functions Objectives: SWBAT identify polynomial functions SWBAT evaluate polynomial functions. SWBAT find the end behaviors of polynomial
More informationand Rational Functions
chapter This detail from The School of Athens (painted by Raphael around 1510) depicts Euclid eplaining geometry. Linear, Quadratic, Polynomial, and Rational Functions In this chapter we focus on four
More informationSparsity. The implication is that we would like to find ways to increase efficiency of LU decomposition.
Sparsity. Introduction We saw in previous notes that the very common problem, to solve for the n vector in A b ( when n is very large, is done without inverting the n n matri A, using LU decomposition.
More informationIntroduction to Mobile Robotics Compact Course on Linear Algebra. Wolfram Burgard, Cyrill Stachniss, Kai Arras, Maren Bennewitz
Introduction to Mobile Robotics Compact Course on Linear Algebra Wolfram Burgard, Cyrill Stachniss, Kai Arras, Maren Bennewitz Vectors Arrays of numbers Vectors represent a point in a n dimensional space
More information3.3.1 Linear functions yet again and dot product In 2D, a homogenous linear scalar function takes the general form:
3.3 Gradient Vector and Jacobian Matri 3 3.3 Gradient Vector and Jacobian Matri Overview: Differentiable functions have a local linear approimation. Near a given point, local changes are determined by
More informationAdaptive Noise Cancellation
Adaptive Noise Cancellation P. Comon and V. Zarzoso January 5, 2010 1 Introduction In numerous application areas, including biomedical engineering, radar, sonar and digital communications, the goal is
More informationSparsity in system identification and data-driven control
1 / 40 Sparsity in system identification and data-driven control Ivan Markovsky This signal is not sparse in the "time domain" 2 / 40 But it is sparse in the "frequency domain" (it is weighted sum of six
More informationMultiplying matrices by diagonal matrices is faster than usual matrix multiplication.
7-6 Multiplying matrices by diagonal matrices is faster than usual matrix multiplication. The following equations generalize to matrices of any size. Multiplying a matrix from the left by a diagonal matrix
More informationName: INSERT YOUR NAME HERE. Due to dropbox by 6pm PDT, Wednesday, December 14, 2011
AMath 584 Name: INSERT YOUR NAME HERE Take-home Final UWNetID: INSERT YOUR NETID Due to dropbox by 6pm PDT, Wednesday, December 14, 2011 The main part of the assignment (Problems 1 3) is worth 80 points.
More informationCLASS NOTES Computational Methods for Engineering Applications I Spring 2015
CLASS NOTES Computational Methods for Engineering Applications I Spring 2015 Petros Koumoutsakos Gerardo Tauriello (Last update: July 27, 2015) IMPORTANT DISCLAIMERS 1. REFERENCES: Much of the material
More informationELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications
ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications Professor M. Chiang Electrical Engineering Department, Princeton University March
More informationVery useful for designing and analyzing signal processing systems
z-transform z-transform The z-transform generalizes the Discrete-Time Fourier Transform (DTFT) for analyzing infinite-length signals and systems Very useful for designing and analyzing signal processing
More informationBlind Identification of FIR Systems and Deconvolution of White Input Sequences
Blind Identification of FIR Systems and Deconvolution of White Input Sequences U. SOVERINI, P. CASTALDI, R. DIVERSI and R. GUIDORZI Dipartimento di Elettronica, Informatica e Sistemistica Università di
More informationA6523 Modeling, Inference, and Mining Jim Cordes, Cornell University
A6523 Modeling, Inference, and Mining Jim Cordes, Cornell University Lecture 19 Modeling Topics plan: Modeling (linear/non- linear least squares) Bayesian inference Bayesian approaches to spectral esbmabon;
More informationMachine Learning. A Bayesian and Optimization Perspective. Academic Press, Sergios Theodoridis 1. of Athens, Athens, Greece.
Machine Learning A Bayesian and Optimization Perspective Academic Press, 2015 Sergios Theodoridis 1 1 Dept. of Informatics and Telecommunications, National and Kapodistrian University of Athens, Athens,
More informationWe use the overhead arrow to denote a column vector, i.e., a number with a direction. For example, in three-space, we write
1 MATH FACTS 11 Vectors 111 Definition We use the overhead arrow to denote a column vector, ie, a number with a direction For example, in three-space, we write The elements of a vector have a graphical
More informationScientific Computing: Dense Linear Systems
Scientific Computing: Dense Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 February 9th, 2012 A. Donev (Courant Institute)
More informationRECURSIVE ESTIMATION AND KALMAN FILTERING
Chapter 3 RECURSIVE ESTIMATION AND KALMAN FILTERING 3. The Discrete Time Kalman Filter Consider the following estimation problem. Given the stochastic system with x k+ = Ax k + Gw k (3.) y k = Cx k + Hv
More informationA Multi-Affine Model for Tensor Decomposition
Yiqing Yang UW Madison breakds@cs.wisc.edu A Multi-Affine Model for Tensor Decomposition Hongrui Jiang UW Madison hongrui@engr.wisc.edu Li Zhang UW Madison lizhang@cs.wisc.edu Chris J. Murphy UC Davis
More informationAdaptive Systems Homework Assignment 1
Signal Processing and Speech Communication Lab. Graz University of Technology Adaptive Systems Homework Assignment 1 Name(s) Matr.No(s). The analytical part of your homework (your calculation sheets) as
More information