Lecture Notes 4 Vector Detection and Estimation. Vector Detection Reconstruction Problem Detection for Vector AGN Channel
|
|
- Rosalyn Stevenson
- 6 years ago
- Views:
Transcription
1 Lecture Notes 4 Vector Detection and Estimation Vector Detection Reconstruction Problem Detection for Vector AGN Channel Vector Linear Estimation Linear Innovation Sequence Kalman Filter EE 278B: Random Vectors 4 1
2 Vector Detection Let the signal Θ = θ 0 with probability p 0 and Θ = θ 1 with probability p 1 = 1 p 0 We observe the RV Y, where Y {Θ = θ 0 } f Y Θ (y θ 0 ) and Y {Θ = θ 1 } f Y Θ (y θ 1 ) We wish to find the estimate ˆΘ(Y) that minimizes the probability of detection error P{ˆΘ Θ} The optimal estimate is obtained using the MAP decoder θ 0 if p Θ Y (θ 0 y) p ˆΘ(y) = Θ Y (θ 1 y) > 1 θ 1 otherwise When p 0 = p 1 = 1/2, the MAP decoder reduces to the ML decoder θ 0 if f Y Θ (y θ 0) f ˆΘ(y) = Y Θ (y θ 1 ) > 1 θ 1 otherwise EE 278B: Random Vectors 4 2
3 Reconstruction on the Tree Consider a complete binary reconstruction tree of finite depth k X 1 Z 1 Z 2 X 2 X 3 Z 3 Z 4 Z 5 Z 6 X 4 X 5 X 6 X 7 The root node is assigned a r.v. X 1 Bern(1/2) (the signal) Denote the two children of each non-leaf node i as l i and r i (e.g., for i = 1, l 1 = 2 and r 1 = 3) EE 278B: Random Vectors 4 3
4 A random variable is assigned to each non-root node as follows X li = X i Z li, X ri = X i Z ri, where Z 1,Z 2,... are i.i.d. Bern(ǫ), ǫ 1/2, r.v.s independent of X 1 That is, the r.v. assigned to a node is the output of a binary symmetric channel (BSC) whose input is the r.v. of its parent Denote the set of leaf r.v.s that are descendants of node i as X i (e.g., for i = 1, X 1 = (X 4,X 5,X 6,X 7 ), and for i = 4, X 4 = X 4 in figure) We observe the leaf node r.v.s X 1 and wish to find the estimate ˆX 1 (X 1 ) that minimizes the probability of error P e = P{ ˆX 1 X 1 } This problem is a simple example of the reconstruction on the tree problem, which arises in computational evolutionary biology (phylogenetic reconstruction), statistical physics, and theoretical computer science. A question of interest in these fields is under what condition on the channel noise can X 1 be reconstructed with P e < 1/2 as the tree depth k The reconstruction problem itself is an example of graphical models in which random variables dependencies are specified by a graph (STAT 375, CS 228) EE 278B: Random Vectors 4 4
5 Since X 1 Bern(1/2), the optimal estimate is obtained using the ML decoder 0 if p X 1 X (x 1 1 0) p ˆX 1 (X 1 ) = X1 X (x 1 1 1) > 1 1 otherwise Because of the special structure of the observation r.v.s, the optimal estimate can be computed using a fast iterative message passing algorithm Define L i,0 = p Xi X i (x i 0) L i,1 = p Xi X i (x i 1) Then the ML estimate can be written as ˆX 1 (X 1 ) = { 0 if L1,0 > L 1,1 1 otherwise We now show that L 1,0,L 1,1 can be computed (in order of the number of nodes in the tree) by iteratively computing the intermediate likelihoods L i,0,l i,1 beginning with the leaf nodes for which L i,0 = 1 x i L i,1 = x i EE 278B: Random Vectors 4 5
6 By the law of total probability, for a non-leaf node i, we can write L i,0 = p Xi X i (x i 0) = p Xli,X ri X i (0,0 0) p Xi X i,x li,x ri (x i 0,0,0) +p Xli,X ri X i (0,1 0) p Xi X i,x li,x ri (x i 0,0,1) +p Xli,X ri X i (1,0 0) p Xi X i,x li,x ri (x i 0,1,0) +p Xli,X ri X i (1,1 0) p Xi X i,x li,x ri (x i 0,1,1) = p Xli X i (0 0)p Xri X i (0 0) p Xi X i,x li,x ri (x i 0,0,0) +p Xli X i (0 0) p Xri X i (1 0)p Xi X i,x li,x ri (x i 0,0,1) +p Xli X i (1 0) p Xri X i (0 0)p Xi X i,x li,x ri (x i 0,1,0) +p Xli X i (1 0)p Xri X i (1 0)p Xi X i,x li,x ri (x i 0,1,1) conditional independence = ǫ 2 p Xi X i,x li,x ri (x i 0,0,0)+ ǫǫ p Xi X i,x li,x ri (x i 0,0,1) +ǫ ǫ p Xi X i,x li,x ri (x i 0,1,0)+ǫ 2 p Xi X i,x li,x ri (x i 0,1,1) L i,1 can be expressed similarly EE 278B: Random Vectors 4 6
7 Now, since X i = (X li,x ri ), by conditional independence, p Xi X i,x li,x ri (x i x i,x li,x ri ) = p Xli,X ri X i,x li,x ri (x li,x ri x i,x li,x ri ) Hence we obtain the following iteratively equations where, at the leaf nodes = p Xli X li (x li x li )p Xri X ri (x ri x ri ) L i,0 = ( ǫl li,0+ǫl li,1)( ǫl ri,0+ǫl ri,1), L i,1 = (ǫl li,0+ ǫl li,1)(ǫl ri,0+ ǫl ri,1), L i,0 = p Xi X i (x i 0) = 1 x i L i,1 = p Xi X i (x i 1) = x i Hence to compute L 1,0 and L 1,1, we start with the likelihoods at each leaf node, then compute the likelihoods for the nodes at level k 1, and so on until we arrive at node 1 EE 278B: Random Vectors 4 7
8 Detection for Vector Additive Gaussian Noise Channel Consider the vector additive Gaussian noise (AGN) channel Y = Θ+Z, where the signal Θ = θ 0, an n-dimensional real vector, with probability 1/2 and Θ = θ 1 with probability 1/2, and the noise Z N(0,Σ Z ) are independent We observe y and wish to find the estimate ˆΘ(Y) that minimizes the probability of decoding error P{ˆΘ Θ} First assume that Σ Z = NI, i.e., additive white Gaussian noise channel The optimal decoding rule is the ML decoder. Define the log likelihood ratio Then, the ML decoder is ˆΘ(y) = Λ(y) = ln f(y θ 0) f(y θ 1 ) { θ0 if Λ(y) > 0 θ 1 otherwise EE 278B: Random Vectors 4 8
9 Now, Λ(y) = 1 2N [ (y θ1 ) T (y θ 1 ) (y θ 0 ) T (y θ 0 ) ] Hence, the ML decoder reduces to the minimum distance decoder { θ0 if y θ 0 < y θ 1 ˆΘ(y) = θ 1 otherwise We can simplify this further to { θ0 if y ˆΘ(y) T (θ 1 θ 0 ) < 1 = 2 (θt 1θ 1 θ T 0θ 0 ) θ 1 otherwise Hence, the decision depends only on the value of a scalar r.v. W = Y T (θ 1 θ 0 ). Such r.v. is referred to as a sufficient statistic for the optimal decoder. Further, W {Θ = θ 0 } N(θ T 0(θ 1 θ 0 ),N(θ 1 θ 0 ) T (θ 1 θ 0 )), W {Θ = θ 1 } N(θ T 1(θ 1 θ 0 ),N(θ 1 θ 0 ) T (θ 1 θ 0 )) EE 278B: Random Vectors 4 9
10 Assuming that the signals have the same power, i.e., θ T 0θ 0 = θ T 1θ 1 = P, the optimal decoding rule reduces to the matched filter decoder (receiver) { θ0 if y ˆΘ(y) T (θ 1 θ 0 ) < 0 = θ 1 otherwise, that is, ˆΘ(y) = { θ0 if w < 0 θ 1 if w 0 This is the same as the optimal rule for the scalar case discussed in Lecture notes 1! The minimum probability of error is P e = Q 0θ 1 2N(P θ T 0θ 1 ) = Q P θ T 0θ 1 2N EE 278B: Random Vectors 4 10
11 This is minimized by using antipodal signals θ 0 = θ 1, which yields ( ) P P e = Q N Exactly the same as scalar antipodal signals Now suppose that the noise is not white, i.e., Σ Z NI. Then the ML decoder reduces to { θ0 if (y θ 0 ) ˆΘ(y) T Σ 1 Z = (y θ 0) < (y θ 1 ) T Σ 1 Z (y θ 1) θ 1 otherwise Now, let y = Σ 1/2 Z y and θ i = Σ 1/2 Z θ i for i = 0,1, then the rule becomes the same as that for the white noise case { θ0 if y ˆΘ(y) θ 0 < y θ 1 = θ 1 otherwise and can be simplified to the scalar case as before Thus, the optimal decoder is to first multiply Y by Σ 1/2 Z to obtain Y and then to apply the optimal rule for the white noise case with the transformed signals θ i = Σ 1/2 Z θ i, i = 0,1 EE 278B: Random Vectors 4 11
12 Vector Linear Estimation Let X f X (x) be a r.v. representing the signal and let Y be an n-dimensional RV representing the observations The minimum MSE estimate of X given Y is the conditional expectation E(X Y). This is often not practical to compute either because the conditional pdf of X given Y is not known or because of high computational cost The MMSE linear (or affine) estimate is easier to find since it depends only on the means, variances, and covariances of the r.v.s involved To find the MMSE linear estimate, first assume that E(X) = 0 and E(Y) = 0. The problem reduces to finding a real n-vector h such that n ˆX = h T Y = h i Y i minimizes the MSE = E [ (X ˆX) 2] i=1 EE 278B: Random Vectors 4 12
13 MMSE Linear Estimate via Orthogonality Principle To find ˆX we use the orthogonality principle: we view the r.v.s X,Y 1,Y 2,...,Y n as vectors in the inner product space consisting of all zero mean r.v.s defined over the underlying probability space The linear estimation problem reduces to a geometry problem: find the vector ˆX that is closest to X (in norm of error X ˆX) X signal ˆX error vector X ˆX subspace spanned by Y 1,Y 2,...,Y n EE 278B: Random Vectors 4 13
14 To minimize MSE = X ˆX 2, we choose ˆX so that the error vector X ˆX is orthogonal to the subspace spanned by the observations Y 1,Y 2,...,Y n, i.e., hence E [ (X ˆX)Y i ] = 0, i = 1,2,...,n, n E(Y i X) = E(Y i ˆX) = h j E(Y i Y j ), i = 1,2,...,n j=1 Define the cross covariance of Y and X as the n-vector Σ YX = E [ (Y E(Y))(X E(X)) ] = For n = 1 this is simply the covariance σ Y1 X σ Y2 X. σ Yn X The above equations can be written in vector form as Σ Y h = Σ YX If Σ Y is nonsingular, we can solve the equations to obtain h = Σ 1 Y Σ YX EE 278B: Random Vectors 4 14
15 Thus, if Σ Y is nonsingular then the best linear MSE estimate is: ˆX = h T Y = Σ T YX Σ 1 Y Y Compare this to the scalar case, where ˆX = Cov(X,Y) Y σy 2 Now to find the minimum MSE, consider MSE = E [ (X ˆX) 2] = E [ (X ˆX)X ] E [ ] (X ˆX) ˆX = E [ (X ˆX)X ], since by orthogonality (X ˆX) ˆX = E(X 2 ) E( ˆXX) = Var(X) E ( Σ T YXΣ 1 Y YX) = Var(X) Σ T YXΣ 1 Y Σ YX Compare this to the scalar case, where minimum MSE is Var(X) Cov(X,Y )2 If X or Y have nonzero mean, the MMSE affine estimate ˆX = h 0 +h T Y is determined by first finding the MMSE linear estimate of X E(X) given Y E(Y) (minimum MSE for ˆX and ˆX are the same), which is ˆX = Σ T YX Σ 1 Y (Y E(Y)), and then setting ˆX = ˆX +E(X) (since E( ˆX) = E(X) is necessary) σ 2 Y EE 278B: Random Vectors 4 15
16 Example Let X be the r.v. representing a signal with mean µ and variance P. The observations are Y i = X +Z i, for i = 1,2,...,n, where the Z i are zero mean uncorrelated noise with variance N, and X and Z i are also uncorrelated Find the MMSE linear estimate of X given Y and its MSE For n = 1, we already know that ˆX 1 = P P +N Y 1 + N P +N µ To find the MMSE linear estimate for general n, first let X = X µ and Y i = Y i µ. Thus X and Y are zero mean The MMSE linear estimate of X given Y is given by ˆX n = h T Y, where Σ Y h = Σ YX, thus P +N P P h 1 P P P +N P h = P. P P P +N P h n EE 278B: Random Vectors 4 16
17 By symmetry, h 1 = h 2 = = h n = Therefore ˆX n = ˆX n = ( P n ) (Y i µ) np +N i=1 The mean square error of the estimate: P np + N. Thus P np +N +µ = n i=1 Y i ( P n Y i )+ np +N i=1 MSE n = P E( ˆX nx ) = PN np +N N np +N µ Thus as n, MSE n 0, i.e., the linear estimate becomes perfect (even though we don t know the complete statistics of X and Y ) EE 278B: Random Vectors 4 17
18 Linear Innovation Sequence Let X be the signal and Y be the observation vector (all zero mean) Suppose the Y i s are orthogonal, i.e., E(Y i Y j ) = 0 for all i j, and let ˆX(Y) be the best linear MSE estimate of X given Y and ˆX(Y i ) be the best linear MSE estimate of X given only Y i for i = 1,...,n, then we can write n ˆX(Y) = ˆX(Y i ), i=1 MSE = Var(X) n i=1 Cov 2 (X,Y i ) Var(Y i ) Hence the computation of the best linear MSE estimate and its MSE are very simple In fact, we can compute the estimates and the MSE causally (recursively) ˆX(Y i+1 ) = ˆX(Y i )+ ˆX(Y i+1 ) MSE i+1 = MSE i Cov2 (X,Y i+1 ) Var(Y i+1 ) EE 278B: Random Vectors 4 18
19 This can be proved by direct evaluation of MMSE linear estimate or using orthogonality: X ˆX(Y 1 ) Y 1 ˆX(Y 2 ) ˆX(Y 2 ) Y 2 EE 278B: Random Vectors 4 19
20 Now suppose the Y i s are not orthogonal. We can still express the estimate and its MSE as sums We first whiten Y to obtain Z. The best linear MSE estimate of X given Y is exactly the same as that given Z (why?) The estimate and its MSE can then be computed as n ˆX(Y) = ˆX(Z i ) i=1 MSE = Var(X) n Cov 2 (X,Z i ) i=1 We can compute an orthogonal observation sequence Ỹ from Y causally: Given Y i, we compute the error of the best linear MSE estimate of Y i+1, Ỹ i+1 (Y i ) = Y i+1 Ŷ i+1 (Y i ) Clearly, Ỹi+1 (Ỹ1,Ỹ2,...,Ỹi), hence we can write i Ŷ i+1 (Y i ) = Ŷ i+1 (Ỹj) j=1 EE 278B: Random Vectors 4 20
21 Interpretation: Ŷ i+1 is the part of Y i+1 predictable by Y i, hence carries no useful new information for estimating X beyond Y i Ỹ i+1 by comparison is the unpredictable part, hence carries new information As such, Ỹ is called the linear innovation sequence of Y Remark: If we normalize Ỹ (by dividing each Ỹ i by its standard deviation), we obtain the same sequence as using the Cholesky decomposition in Lecture notes 3 Example: Let the observation sequence be Y i = X +Z i for i = 1,2,...,n, where X, Z 1,..., Z n are zero mean, uncorrelated r.v.s with E(X 2 ) = P and E(Zi 2 ) = N for i = 1,2,...,n. Find the linear innovation sequence of Y Using the innovation sequence, the MMSE linear estimate of X given Ỹ i+1 and its MSE can be computed causally ˆX(Ỹ i+1 ) = ˆX(Ỹ i )+ ˆX(Ỹi+1), MSE i+1 = MSE i Cov2 (X,Ỹ i+1 ) Var(Ỹ i+1 ) The innovation sequence will prove useful in deriving the Kalman filter EE 278B: Random Vectors 4 21
22 Kalman Filter The Kalman filter is an efficient, recursive algorithm for computing the MMSE linear estimate and its MSE when the signal X and observations Y evolve according to a state-space model Consider a linear dynamical system described by the state-space model: with noisy observations (output) X i+1 = A i X i +U i, i = 0,1,...,n Y i = X i +V i, i = 0,1,...,n, where X 0, U 0,U 1,...,U n, V 0,V 1,...,V n are zero mean, uncorrelated RVs with Σ X0 = P 0, Σ Ui = Q i, Σ Vi = N i ; A i is a known sequence of matrices V i U i X i+1 X i Delay Y i A i EE 278B: Random Vectors 4 22
23 This state space model is used in many applications: Navigation, e.g., of a car: State: is location, speed, heading, acceleration, tilt, steering wheel position of vehicle Observations: inertial (accelerometer, gyroscopes), electronic compass, GPS Phase locked loop: State: phase and frequency offsets Observations: noisy observation of phase Computer vision, e.g., face tracking: State: Pose, motion, shape (size, articulation), appearance (light, color) Observations: video frame sequence Economics... EE 278B: Random Vectors 4 23
24 The goal is to compute the MMSE linear estimate of the state from causal observations: Prediction: Find the estimate ˆX i+1 i of X i+1 from Y i and its MSE Σ i+1 i Filtering: Find the estimate ˆX i i of X i from Y i and its MSE Σ i i The Kalman filter provides clever recursive equations for computing these estimates and their error covariance matrices EE 278B: Random Vectors 4 24
25 Scalar Kalman Filter Consider the scalar state space system: X i+1 = a i X i +U i, i = 0,1,...,n with noisy observations Y i = X i +V i, i = 0,1,...,n, where X 0, U 0,U 1,...,U n, V 0,V 1,...,V n are zero mean, uncorrelated r.v.s with Var(X 0 ) = P 0, Var(U i ) = Q i, Var(V i ) = N i, and a i is a known sequence V i U i X i+1 X i Delay Y i a i EE 278B: Random Vectors 4 25
26 Kalman filter (prediction): Initialization: ˆX 0 1 = 0, σ = P 0 Update equations: For i = 0,1,2,...,n, the estimate is where the filter gain is The MSE of ˆX i+1 i is ˆX i+1 i = a i ˆXi i 1 +k i (Y i ˆX i i 1 ), k i = a iσ 2 i i 1 σ 2 i i 1 +N i σ 2 i+1 i = a i(a i k i )σ 2 i i 1 +Q i EE 278B: Random Vectors 4 26
27 Example: Let a i = 1, Q i = 0, N i = N, and P 0 = P (so X 0 = X 1 = X 2 = = X), and Y i = X +V i (this is the same as the earlier estimation example) Kalman filter: Initialization: ˆX 0 1 = 0 and σ = P The update in each step is ˆX i+1 i = (1 k i ) ˆX i i 1 +k i Y i with and the MSE is k i = σ2 i i 1 σ 2 i i 1 +N, σ 2 i+1 i = (1 k i)σ 2 i i 1 EE 278B: Random Vectors 4 27
28 We can solve for σi+1 i 2 explicitly ( ) σi+1 i 2 = 1 σ2 i i 1 σi i 1 2 +N σi i 1 2 = Nσ2 i i 1 σi i 1 2 +N The gain is 1 σ 2 i+1 i The recursive estimate is = 1 N + 1 σ 2 i i 1 σ 2 i+1 i = 1 i/n +1/P = NP ip +N ˆX i+1 i = k i = (i 1)P +N ip +N P ip +N ˆX i i 1 + We thus obtain the previous result in a recursive form P ip +N Y i EE 278B: Random Vectors 4 28
29 Example: Let n = 200, P 0 = 1, N i = 1 For i = 1 to 100: a i = α 2, Q i = P 0 (1 α 2 ) with α = 0.95 (memory factor) For i = 100 to 200: a i = 1, Q i = 0 (i.e., state remains constant) 2 1 Xi Yi ˆXi+1 i i EE 278B: Random Vectors 4 29
30 Xi+1 ˆX i+1 i Yi Xi σ 2 i+1 i i EE 278B: Random Vectors 4 30
31 Derivation of the Kalman Filter We use innovations. Let Ỹi be the innovation r.v. for Y i, then we can write ˆX i+1 i = ˆX i+1 i 1 +k i Ỹ i, σ i+1 i = σ i+1 i 1 +k i Cov(X i+1,ỹi) where ˆX i+1 i 1 and σ i+1 i 1 are the MMSE linear estimate of X given Y i 1 and its MSE, and k i = Cov(X i+1,ỹi) Var(Ỹi) Now, since X i+1 = a i X i +U i, by linearity of MMSE linear estimate, we have and ˆX i+1 i 1 = a i ˆXi i 1 σ 2 i+1 i 1 = a2 iσ 2 i i 1 +Q i EE 278B: Random Vectors 4 31
32 Now, the innovation r.v. for Y i is Ỹ i = Y i Ŷ i (Y i 1 ) Since Y i = X i +V i and V i is uncorrelated with Y j, j = 1,2,...,i 1, Ŷ i (Y i 1 ) = ˆX i i 1 Hence, This yields Ỹ i = Y i ˆX i i 1 ˆX i+1 i = a i ˆXi i 1 +k i Ỹ i = a i ˆXi i 1 +k i (Y i ˆX i i 1 ) Now, consider σ 2 i+1 i = σ2 i+1 i 1 k icov(x i+1,ỹ i ), k i = Cov(X i+1,ỹi) Var(Ỹi) = Cov(a ix i +U i,x i ˆX i i 1 +V i ) Var(X i ˆX i i 1 +V i ) = Cov(a ix i,x i ˆX i i 1 ) Var(X i ˆX i i 1 +V i ) EE 278B: Random Vectors 4 32
33 = a icov(x i,x i ˆX i i 1 ) Var(X i ˆX i i 1 +V i ) = a icov(x i ˆX i i 1,X i ˆX i i 1 ) Var(X i ˆX i i 1 +V i ) since (X i ˆX i i 1 ) ˆX i i 1 The MSE is = a ivar(x i ˆX i i 1 ) Var(X i ˆX i i 1 )+N i = a iσ 2 i i 1 σ 2 i i 1 +N i σ 2 i+1 i = σ2 i+1 i 1 k icov(a i X i +U i,x i ˆX i i 1 +V i ) = σ 2 i+1 i 1 k ia i σ 2 i i 1 = a i (a i k i )σ 2 i i 1 +Q i This completes the derivation of the scalar Kalman filter EE 278B: Random Vectors 4 33
34 Vector Kalman Filter The above scalar Kalman filter can be extended to the vector state space model: Initialization: ˆX0 1 = 0, Σ 0 1 = P 0 Update equations: For i = 0,1,2,...,n, the estimate is where the filter gain matrix The covariance of the error is ˆX i+1 i = A iˆxi i 1 +K i (Y i ˆX i i 1 ), K i = A i Σ i i 1 (Σ i i 1 +N i ) 1 Σ i+1 i = A i Σ i i 1 A T i K i Σ i i 1 A T i +Q i Remark: If X 0, U 0,U 1,...,U n and V 0,V 1,...,V n are Gaussian (zero mean, uncorrelated), then the Kalman filter yields the best MSE estimate of X i, i = 0,...,n EE 278B: Random Vectors 4 34
35 Filtering Now assume the goal is to compute the MMSE linear estimate of X i given Y i, i.e., instead of predicting the next state, we are interested in estimating the current state We denote this estimate by ˆX i i and its MSE by σ 2 i i The Kalman filter can be adapted to this case as follows: Initialization: ˆX 0 0 = P 0 P 0 +N 0 Y 0 σ = P 0N 0 P 0 +N 0 Update equations: For i = 1,2,...,n, the estimate is ˆX i i = a i 1 (1 k i ) ˆX i 1 i 1 +k i Y i EE 278B: Random Vectors 4 35
36 with filter gain and MSE recursion k i = σ 2 i i = (1 k i) a2 i 1 σ2 i 1 i 1 +Q i 1 a 2 i 1 σ2 i 1 i 1 +Q i 1+N i ( ) a 2 i 1σi 1 i 1 2 +Q i 1 Vector case Initialization: ˆX 0 0 = P 0 (P 0 +N 0 ) 1 Y 0 Σ 0 0 = P 0 (I (P 0 +N 0 ) 1 P 0 ) Update equations: For i = 1,2,...,n, the estimate is ˆX i i = (I K i )A i 1ˆXi 1 i 1 +K i Y i with filter gain K i = (A i 1 Σ i 1 i 1 A T i 1+Q i 1 ) ( A i 1 Σ i 1 i 1 A T i 1+Q i 1 +N i ) 1 and MSE recursion Σ i i = (A i 1 Σ i 1 i 1 A T i 1+Q i 1 )(I K T i ) EE 278B: Random Vectors 4 36
ESTIMATION THEORY. Chapter Estimation of Random Variables
Chapter ESTIMATION THEORY. Estimation of Random Variables Suppose X,Y,Y 2,...,Y n are random variables defined on the same probability space (Ω, S,P). We consider Y,...,Y n to be the observed random variables
More informationLeast Squares and Kalman Filtering Questions: me,
Least Squares and Kalman Filtering Questions: Email me, namrata@ece.gatech.edu Least Squares and Kalman Filtering 1 Recall: Weighted Least Squares y = Hx + e Minimize Solution: J(x) = (y Hx) T W (y Hx)
More informationLecture 4: Least Squares (LS) Estimation
ME 233, UC Berkeley, Spring 2014 Xu Chen Lecture 4: Least Squares (LS) Estimation Background and general solution Solution in the Gaussian case Properties Example Big picture general least squares estimation:
More informationStatistics 910, #15 1. Kalman Filter
Statistics 910, #15 1 Overview 1. Summary of Kalman filter 2. Derivations 3. ARMA likelihoods 4. Recursions for the variance Kalman Filter Summary of Kalman filter Simplifications To make the derivations
More information4 Derivations of the Discrete-Time Kalman Filter
Technion Israel Institute of Technology, Department of Electrical Engineering Estimation and Identification in Dynamical Systems (048825) Lecture Notes, Fall 2009, Prof N Shimkin 4 Derivations of the Discrete-Time
More informationLecture Notes 5 Convergence and Limit Theorems. Convergence with Probability 1. Convergence in Mean Square. Convergence in Probability, WLLN
Lecture Notes 5 Convergence and Limit Theorems Motivation Convergence with Probability Convergence in Mean Square Convergence in Probability, WLLN Convergence in Distribution, CLT EE 278: Convergence and
More informationKalman Filtering. Namrata Vaswani. March 29, Kalman Filter as a causal MMSE estimator
Kalman Filtering Namrata Vaswani March 29, 2018 Notes are based on Vincent Poor s book. 1 Kalman Filter as a causal MMSE estimator Consider the following state space model (signal and observation model).
More informationEstimation theory. Parametric estimation. Properties of estimators. Minimum variance estimator. Cramer-Rao bound. Maximum likelihood estimators
Estimation theory Parametric estimation Properties of estimators Minimum variance estimator Cramer-Rao bound Maximum likelihood estimators Confidence intervals Bayesian estimation 1 Random Variables Let
More informationAppendix A : Introduction to Probability and stochastic processes
A-1 Mathematical methods in communication July 5th, 2009 Appendix A : Introduction to Probability and stochastic processes Lecturer: Haim Permuter Scribe: Shai Shapira and Uri Livnat The probability of
More informationUCSD ECE153 Handout #34 Prof. Young-Han Kim Tuesday, May 27, Solutions to Homework Set #6 (Prepared by TA Fatemeh Arbabjolfaei)
UCSD ECE53 Handout #34 Prof Young-Han Kim Tuesday, May 7, 04 Solutions to Homework Set #6 (Prepared by TA Fatemeh Arbabjolfaei) Linear estimator Consider a channel with the observation Y XZ, where the
More informationLeast Squares Estimation Namrata Vaswani,
Least Squares Estimation Namrata Vaswani, namrata@iastate.edu Least Squares Estimation 1 Recall: Geometric Intuition for Least Squares Minimize J(x) = y Hx 2 Solution satisfies: H T H ˆx = H T y, i.e.
More informationX t = a t + r t, (7.1)
Chapter 7 State Space Models 71 Introduction State Space models, developed over the past 10 20 years, are alternative models for time series They include both the ARIMA models of Chapters 3 6 and the Classical
More informationA Theoretical Overview on Kalman Filtering
A Theoretical Overview on Kalman Filtering Constantinos Mavroeidis Vanier College Presented to professors: IVANOV T. IVAN STAHN CHRISTIAN Email: cmavroeidis@gmail.com June 6, 208 Abstract Kalman filtering
More informationUCSD ECE153 Handout #30 Prof. Young-Han Kim Thursday, May 15, Homework Set #6 Due: Thursday, May 22, 2011
UCSD ECE153 Handout #30 Prof. Young-Han Kim Thursday, May 15, 2014 Homework Set #6 Due: Thursday, May 22, 2011 1. Linear estimator. Consider a channel with the observation Y = XZ, where the signal X and
More informationSolutions to Homework Set #5 (Prepared by Lele Wang) MSE = E [ (sgn(x) g(y)) 2],, where f X (x) = 1 2 2π e. e (x y)2 2 dx 2π
Solutions to Homework Set #5 (Prepared by Lele Wang). Neural net. Let Y X + Z, where the signal X U[,] and noise Z N(,) are independent. (a) Find the function g(y) that minimizes MSE E [ (sgn(x) g(y))
More informationVariance reduction. Michel Bierlaire. Transport and Mobility Laboratory. Variance reduction p. 1/18
Variance reduction p. 1/18 Variance reduction Michel Bierlaire michel.bierlaire@epfl.ch Transport and Mobility Laboratory Variance reduction p. 2/18 Example Use simulation to compute I = 1 0 e x dx We
More informationShannon meets Wiener II: On MMSE estimation in successive decoding schemes
Shannon meets Wiener II: On MMSE estimation in successive decoding schemes G. David Forney, Jr. MIT Cambridge, MA 0239 USA forneyd@comcast.net Abstract We continue to discuss why MMSE estimation arises
More informationECE531 Lecture 11: Dynamic Parameter Estimation: Kalman-Bucy Filter
ECE531 Lecture 11: Dynamic Parameter Estimation: Kalman-Bucy Filter D. Richard Brown III Worcester Polytechnic Institute 09-Apr-2009 Worcester Polytechnic Institute D. Richard Brown III 09-Apr-2009 1 /
More informationCh. 12 Linear Bayesian Estimators
Ch. 1 Linear Bayesian Estimators 1 In chapter 11 we saw: the MMSE estimator takes a simple form when and are jointly Gaussian it is linear and used only the 1 st and nd order moments (means and covariances).
More informationLecture Notes 1 Probability and Random Variables. Conditional Probability and Independence. Functions of a Random Variable
Lecture Notes 1 Probability and Random Variables Probability Spaces Conditional Probability and Independence Random Variables Functions of a Random Variable Generation of a Random Variable Jointly Distributed
More information1 Kalman Filter Introduction
1 Kalman Filter Introduction You should first read Chapter 1 of Stochastic models, estimation, and control: Volume 1 by Peter S. Maybec (available here). 1.1 Explanation of Equations (1-3) and (1-4) Equation
More informationProblem Solving. Correlation and Covariance. Yi Lu. Problem Solving. Yi Lu ECE 313 2/51
Yi Lu Correlation and Covariance Yi Lu ECE 313 2/51 Definition Let X and Y be random variables with finite second moments. the correlation: E[XY ] Yi Lu ECE 313 3/51 Definition Let X and Y be random variables
More informationECE 541 Stochastic Signals and Systems Problem Set 9 Solutions
ECE 541 Stochastic Signals and Systems Problem Set 9 Solutions Problem Solutions : Yates and Goodman, 9.5.3 9.1.4 9.2.2 9.2.6 9.3.2 9.4.2 9.4.6 9.4.7 and Problem 9.1.4 Solution The joint PDF of X and Y
More informationProblem Set 3 Due Oct, 5
EE6: Random Processes in Systems Lecturer: Jean C. Walrand Problem Set 3 Due Oct, 5 Fall 6 GSI: Assane Gueye This problem set essentially reviews detection theory. Not all eercises are to be turned in.
More informationLinear Dynamical Systems
Linear Dynamical Systems Sargur N. srihari@cedar.buffalo.edu Machine Learning Course: http://www.cedar.buffalo.edu/~srihari/cse574/index.html Two Models Described by Same Graph Latent variables Observations
More informationSimple Linear Regression for the MPG Data
Simple Linear Regression for the MPG Data 2000 2500 3000 3500 15 20 25 30 35 40 45 Wgt MPG What do we do with the data? y i = MPG of i th car x i = Weight of i th car i =1,...,n n = Sample Size Exploratory
More informationParameter Estimation
1 / 44 Parameter Estimation Saravanan Vijayakumaran sarva@ee.iitb.ac.in Department of Electrical Engineering Indian Institute of Technology Bombay October 25, 2012 Motivation System Model used to Derive
More informationECE531 Lecture 8: Non-Random Parameter Estimation
ECE531 Lecture 8: Non-Random Parameter Estimation D. Richard Brown III Worcester Polytechnic Institute 19-March-2009 Worcester Polytechnic Institute D. Richard Brown III 19-March-2009 1 / 25 Introduction
More informationLecture Notes 1 Probability and Random Variables. Conditional Probability and Independence. Functions of a Random Variable
Lecture Notes 1 Probability and Random Variables Probability Spaces Conditional Probability and Independence Random Variables Functions of a Random Variable Generation of a Random Variable Jointly Distributed
More information9 Forward-backward algorithm, sum-product on factor graphs
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 2014 9 Forward-backward algorithm, sum-product on factor graphs The previous
More informationFactor Analysis and Kalman Filtering (11/2/04)
CS281A/Stat241A: Statistical Learning Theory Factor Analysis and Kalman Filtering (11/2/04) Lecturer: Michael I. Jordan Scribes: Byung-Gon Chun and Sunghoon Kim 1 Factor Analysis Factor analysis is used
More informationA Probability Review
A Probability Review Outline: A probability review Shorthand notation: RV stands for random variable EE 527, Detection and Estimation Theory, # 0b 1 A Probability Review Reading: Go over handouts 2 5 in
More informationECONOMETRIC METHODS II: TIME SERIES LECTURE NOTES ON THE KALMAN FILTER. The Kalman Filter. We will be concerned with state space systems of the form
ECONOMETRIC METHODS II: TIME SERIES LECTURE NOTES ON THE KALMAN FILTER KRISTOFFER P. NIMARK The Kalman Filter We will be concerned with state space systems of the form X t = A t X t 1 + C t u t 0.1 Z t
More informationStatistics Homework #4
Statistics 910 1 Homework #4 Chapter 6, Shumway and Stoffer These are outlines of the solutions. If you would like to fill in other details, please come see me during office hours. 6.1 State-space representation
More informationUCSD ECE153 Handout #27 Prof. Young-Han Kim Tuesday, May 6, Solutions to Homework Set #5 (Prepared by TA Fatemeh Arbabjolfaei)
UCSD ECE53 Handout #7 Prof. Young-Han Kim Tuesday, May 6, 4 Solutions to Homework Set #5 (Prepared by TA Fatemeh Arbabjolfaei). Neural net. Let Y = X + Z, where the signal X U[,] and noise Z N(,) are independent.
More information5. Random Vectors. probabilities. characteristic function. cross correlation, cross covariance. Gaussian random vectors. functions of random vectors
EE401 (Semester 1) 5. Random Vectors Jitkomut Songsiri probabilities characteristic function cross correlation, cross covariance Gaussian random vectors functions of random vectors 5-1 Random vectors we
More informationEstimation techniques
Estimation techniques March 2, 2006 Contents 1 Problem Statement 2 2 Bayesian Estimation Techniques 2 2.1 Minimum Mean Squared Error (MMSE) estimation........................ 2 2.1.1 General formulation......................................
More informationSTAT 100C: Linear models
STAT 100C: Linear models Arash A. Amini April 27, 2018 1 / 1 Table of Contents 2 / 1 Linear Algebra Review Read 3.1 and 3.2 from text. 1. Fundamental subspace (rank-nullity, etc.) Im(X ) = ker(x T ) R
More informationENGR352 Problem Set 02
engr352/engr352p02 September 13, 2018) ENGR352 Problem Set 02 Transfer function of an estimator 1. Using Eq. (1.1.4-27) from the text, find the correct value of r ss (the result given in the text is incorrect).
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 11 Project
More informationCS281A/Stat241A Lecture 17
CS281A/Stat241A Lecture 17 p. 1/4 CS281A/Stat241A Lecture 17 Factor Analysis and State Space Models Peter Bartlett CS281A/Stat241A Lecture 17 p. 2/4 Key ideas of this lecture Factor Analysis. Recall: Gaussian
More information9 Multi-Model State Estimation
Technion Israel Institute of Technology, Department of Electrical Engineering Estimation and Identification in Dynamical Systems (048825) Lecture Notes, Fall 2009, Prof. N. Shimkin 9 Multi-Model State
More informationLecture Note 12: Kalman Filter
ECE 645: Estimation Theory Spring 2015 Instructor: Prof. Stanley H. Chan Lecture Note 12: Kalman Filter LaTeX prepared by Stylianos Chatzidakis) May 4, 2015 This lecture note is based on ECE 645Spring
More informationJoint Distributions. (a) Scalar multiplication: k = c d. (b) Product of two matrices: c d. (c) The transpose of a matrix:
Joint Distributions Joint Distributions A bivariate normal distribution generalizes the concept of normal distribution to bivariate random variables It requires a matrix formulation of quadratic forms,
More informationKalman Filter Computer Vision (Kris Kitani) Carnegie Mellon University
Kalman Filter 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University Examples up to now have been discrete (binary) random variables Kalman filtering can be seen as a special case of a temporal
More informationState-space Model. Eduardo Rossi University of Pavia. November Rossi State-space Model Financial Econometrics / 49
State-space Model Eduardo Rossi University of Pavia November 2013 Rossi State-space Model Financial Econometrics - 2013 1 / 49 Outline 1 Introduction 2 The Kalman filter 3 Forecast errors 4 State smoothing
More information11 - The linear model
11-1 The linear model S. Lall, Stanford 2011.02.15.01 11 - The linear model The linear model The joint pdf and covariance Example: uniform pdfs The importance of the prior Linear measurements with Gaussian
More informationf X, Y (x, y)dx (x), where f(x,y) is the joint pdf of X and Y. (x) dx
INDEPENDENCE, COVARIANCE AND CORRELATION Independence: Intuitive idea of "Y is independent of X": The distribution of Y doesn't depend on the value of X. In terms of the conditional pdf's: "f(y x doesn't
More informationLevinson Durbin Recursions: I
Levinson Durbin Recursions: I note: B&D and S&S say Durbin Levinson but Levinson Durbin is more commonly used (Levinson, 1947, and Durbin, 1960, are source articles sometimes just Levinson is used) recursions
More informationKalman Filter. Man-Wai MAK
Kalman Filter Man-Wai MAK Dept. of Electronic and Information Engineering, The Hong Kong Polytechnic University enmwmak@polyu.edu.hk http://www.eie.polyu.edu.hk/ mwmak References: S. Gannot and A. Yeredor,
More informationLecture Outline. Target Tracking: Lecture 3 Maneuvering Target Tracking Issues. Maneuver Illustration. Maneuver Illustration. Maneuver Detection
REGLERTEKNIK Lecture Outline AUTOMATIC CONTROL Target Tracking: Lecture 3 Maneuvering Target Tracking Issues Maneuver Detection Emre Özkan emre@isy.liu.se Division of Automatic Control Department of Electrical
More information401 Review. 6. Power analysis for one/two-sample hypothesis tests and for correlation analysis.
401 Review Major topics of the course 1. Univariate analysis 2. Bivariate analysis 3. Simple linear regression 4. Linear algebra 5. Multiple regression analysis Major analysis methods 1. Graphical analysis
More informationA measurement error model approach to small area estimation
A measurement error model approach to small area estimation Jae-kwang Kim 1 Spring, 2015 1 Joint work with Seunghwan Park and Seoyoung Kim Ouline Introduction Basic Theory Application to Korean LFS Discussion
More informationState-space Model. Eduardo Rossi University of Pavia. November Rossi State-space Model Fin. Econometrics / 53
State-space Model Eduardo Rossi University of Pavia November 2014 Rossi State-space Model Fin. Econometrics - 2014 1 / 53 Outline 1 Motivation 2 Introduction 3 The Kalman filter 4 Forecast errors 5 State
More information2 Statistical Estimation: Basic Concepts
Technion Israel Institute of Technology, Department of Electrical Engineering Estimation and Identification in Dynamical Systems (048825) Lecture Notes, Fall 2009, Prof. N. Shimkin 2 Statistical Estimation:
More informationENGG2430A-Homework 2
ENGG3A-Homework Due on Feb 9th,. Independence vs correlation a For each of the following cases, compute the marginal pmfs from the joint pmfs. Explain whether the random variables X and Y are independent,
More informationFinal Exam. Economics 835: Econometrics. Fall 2010
Final Exam Economics 835: Econometrics Fall 2010 Please answer the question I ask - no more and no less - and remember that the correct answer is often short and simple. 1 Some short questions a) For each
More informationSample Problems. Note: If you find the following statements true, you should briefly prove them. If you find them false, you should correct them.
Sample Problems 1. True or False Note: If you find the following statements true, you should briefly prove them. If you find them false, you should correct them. (a) The sample average of estimated residuals
More informationL06. LINEAR KALMAN FILTERS. NA568 Mobile Robotics: Methods & Algorithms
L06. LINEAR KALMAN FILTERS NA568 Mobile Robotics: Methods & Algorithms 2 PS2 is out! Landmark-based Localization: EKF, UKF, PF Today s Lecture Minimum Mean Square Error (MMSE) Linear Kalman Filter Gaussian
More information5. Density evolution. Density evolution 5-1
5. Density evolution Density evolution 5-1 Probabilistic analysis of message passing algorithms variable nodes factor nodes x1 a x i x2 a(x i ; x j ; x k ) x3 b x4 consider factor graph model G = (V ;
More informationLevinson Durbin Recursions: I
Levinson Durbin Recursions: I note: B&D and S&S say Durbin Levinson but Levinson Durbin is more commonly used (Levinson, 1947, and Durbin, 1960, are source articles sometimes just Levinson is used) recursions
More informationLecture Note 1: Probability Theory and Statistics
Univ. of Michigan - NAME 568/EECS 568/ROB 530 Winter 2018 Lecture Note 1: Probability Theory and Statistics Lecturer: Maani Ghaffari Jadidi Date: April 6, 2018 For this and all future notes, if you would
More informationTSRT14: Sensor Fusion Lecture 8
TSRT14: Sensor Fusion Lecture 8 Particle filter theory Marginalized particle filter Gustaf Hendeby gustaf.hendeby@liu.se TSRT14 Lecture 8 Gustaf Hendeby Spring 2018 1 / 25 Le 8: particle filter theory,
More informationMachine Learning 4771
Machine Learning 4771 Instructor: ony Jebara Kalman Filtering Linear Dynamical Systems and Kalman Filtering Structure from Motion Linear Dynamical Systems Audio: x=pitch y=acoustic waveform Vision: x=object
More information[POLS 8500] Review of Linear Algebra, Probability and Information Theory
[POLS 8500] Review of Linear Algebra, Probability and Information Theory Professor Jason Anastasopoulos ljanastas@uga.edu January 12, 2017 For today... Basic linear algebra. Basic probability. Programming
More informationBayesian Networks: Construction, Inference, Learning and Causal Interpretation. Volker Tresp Summer 2014
Bayesian Networks: Construction, Inference, Learning and Causal Interpretation Volker Tresp Summer 2014 1 Introduction So far we were mostly concerned with supervised learning: we predicted one or several
More informationSolutions to Homework Set #6 (Prepared by Lele Wang)
Solutions to Homework Set #6 (Prepared by Lele Wang) Gaussian random vector Given a Gaussian random vector X N (µ, Σ), where µ ( 5 ) T and 0 Σ 4 0 0 0 9 (a) Find the pdfs of i X, ii X + X 3, iii X + X
More informationThe Hilbert Space of Random Variables
The Hilbert Space of Random Variables Electrical Engineering 126 (UC Berkeley) Spring 2018 1 Outline Fix a probability space and consider the set H := {X : X is a real-valued random variable with E[X 2
More informationThe Kalman Filter. An Algorithm for Dealing with Uncertainty. Steven Janke. May Steven Janke (Seminar) The Kalman Filter May / 29
The Kalman Filter An Algorithm for Dealing with Uncertainty Steven Janke May 2011 Steven Janke (Seminar) The Kalman Filter May 2011 1 / 29 Autonomous Robots Steven Janke (Seminar) The Kalman Filter May
More informationMiscellaneous. Regarding reading materials. Again, ask questions (if you have) and ask them earlier
Miscellaneous Regarding reading materials Reading materials will be provided as needed If no assigned reading, it means I think the material from class is sufficient Should be enough for you to do your
More informationFinal Examination Solutions (Total: 100 points)
Final Examination Solutions (Total: points) There are 4 problems, each problem with multiple parts, each worth 5 points. Make sure you answer all questions. Your answer should be as clear and readable
More information18 Bivariate normal distribution I
8 Bivariate normal distribution I 8 Example Imagine firing arrows at a target Hopefully they will fall close to the target centre As we fire more arrows we find a high density near the centre and fewer
More informationFor more information about how to cite these materials visit
Author(s): Kerby Shedden, Ph.D., 2010 License: Unless otherwise noted, this material is made available under the terms of the Creative Commons Attribution Share Alike 3.0 License: http://creativecommons.org/licenses/by-sa/3.0/
More informationInformation Theoretic Imaging
Information Theoretic Imaging WU Faculty: J. A. O Sullivan WU Doctoral Student: Naveen Singla Boeing Engineer: James Meany First Year Focus: Imaging for Data Storage Image Reconstruction Data Retrieval
More informationTiming Recovery at Low SNR Cramer-Rao bound, and outperforming the PLL
T F T I G E O R G A I N S T I T U T E O H E O F E A L P R O G R ESS S A N D 1 8 8 5 S E R V L O G Y I C E E C H N O Timing Recovery at Low SNR Cramer-Rao bound, and outperforming the PLL Aravind R. Nayak
More informationA Short Course in Basic Statistics
A Short Course in Basic Statistics Ian Schindler November 5, 2017 Creative commons license share and share alike BY: C 1 Descriptive Statistics 1.1 Presenting statistical data Definition 1 A statistical
More informationECE531 Lecture 10b: Dynamic Parameter Estimation: System Model
ECE531 Lecture 10b: Dynamic Parameter Estimation: System Model D. Richard Brown III Worcester Polytechnic Institute 02-Apr-2009 Worcester Polytechnic Institute D. Richard Brown III 02-Apr-2009 1 / 14 Introduction
More informationLecture 13: Simple Linear Regression in Matrix Format. 1 Expectations and Variances with Vectors and Matrices
Lecture 3: Simple Linear Regression in Matrix Format To move beyond simple regression we need to use matrix algebra We ll start by re-expressing simple linear regression in matrix form Linear algebra is
More informationCS 532: 3D Computer Vision 6 th Set of Notes
1 CS 532: 3D Computer Vision 6 th Set of Notes Instructor: Philippos Mordohai Webpage: www.cs.stevens.edu/~mordohai E-mail: Philippos.Mordohai@stevens.edu Office: Lieb 215 Lecture Outline Intro to Covariance
More informationBayesian Networks: Construction, Inference, Learning and Causal Interpretation. Volker Tresp Summer 2016
Bayesian Networks: Construction, Inference, Learning and Causal Interpretation Volker Tresp Summer 2016 1 Introduction So far we were mostly concerned with supervised learning: we predicted one or several
More informationME 597: AUTONOMOUS MOBILE ROBOTICS SECTION 2 PROBABILITY. Prof. Steven Waslander
ME 597: AUTONOMOUS MOBILE ROBOTICS SECTION 2 Prof. Steven Waslander p(a): Probability that A is true 0 pa ( ) 1 p( True) 1, p( False) 0 p( A B) p( A) p( B) p( A B) A A B B 2 Discrete Random Variable X
More informationLINEAR MMSE ESTIMATION
LINEAR MMSE ESTIMATION TERM PAPER FOR EE 602 STATISTICAL SIGNAL PROCESSING By, DHEERAJ KUMAR VARUN KHAITAN 1 Introduction Linear MMSE estimators are chosen in practice because they are simpler than the
More informationMS&E 226: Small Data. Lecture 11: Maximum likelihood (v2) Ramesh Johari
MS&E 226: Small Data Lecture 11: Maximum likelihood (v2) Ramesh Johari ramesh.johari@stanford.edu 1 / 18 The likelihood function 2 / 18 Estimating the parameter This lecture develops the methodology behind
More informationChapter 4. Data Transmission and Channel Capacity. Po-Ning Chen, Professor. Department of Communications Engineering. National Chiao Tung University
Chapter 4 Data Transmission and Channel Capacity Po-Ning Chen, Professor Department of Communications Engineering National Chiao Tung University Hsin Chu, Taiwan 30050, R.O.C. Principle of Data Transmission
More informationFirst Year Examination Department of Statistics, University of Florida
First Year Examination Department of Statistics, University of Florida August 19, 010, 8:00 am - 1:00 noon Instructions: 1. You have four hours to answer questions in this examination.. You must show your
More informationMidterm Exam 1 Solution
EECS 126 Probability and Random Processes University of California, Berkeley: Fall 2015 Kannan Ramchandran September 22, 2015 Midterm Exam 1 Solution Last name First name SID Name of student on your left:
More informationProblem Set 7 Due March, 22
EE16: Probability and Random Processes SP 07 Problem Set 7 Due March, Lecturer: Jean C. Walrand GSI: Daniel Preda, Assane Gueye Problem 7.1. Let u and v be independent, standard normal random variables
More informationIntroduction to Simple Linear Regression
Introduction to Simple Linear Regression Yang Feng http://www.stat.columbia.edu/~yangfeng Yang Feng (Columbia University) Introduction to Simple Linear Regression 1 / 68 About me Faculty in the Department
More informationDiscrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Note 26. Estimation: Regression and Least Squares
CS 70 Discrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Note 26 Estimation: Regression and Least Squares This note explains how to use observations to estimate unobserved random variables.
More informationUCSD ECE250 Handout #20 Prof. Young-Han Kim Monday, February 26, Solutions to Exercise Set #7
UCSD ECE50 Handout #0 Prof. Young-Han Kim Monday, February 6, 07 Solutions to Exercise Set #7. Minimum waiting time. Let X,X,... be i.i.d. exponentially distributed random variables with parameter λ, i.e.,
More informationLow-Density Parity-Check Codes
Department of Computer Sciences Applied Algorithms Lab. July 24, 2011 Outline 1 Introduction 2 Algorithms for LDPC 3 Properties 4 Iterative Learning in Crowds 5 Algorithm 6 Results 7 Conclusion PART I
More informationCS181 Midterm 2 Practice Solutions
CS181 Midterm 2 Practice Solutions 1. Convergence of -Means Consider Lloyd s algorithm for finding a -Means clustering of N data, i.e., minimizing the distortion measure objective function J({r n } N n=1,
More informationPrincipal Components Theory Notes
Principal Components Theory Notes Charles J. Geyer August 29, 2007 1 Introduction These are class notes for Stat 5601 (nonparametrics) taught at the University of Minnesota, Spring 2006. This not a theory
More informationChapter 3: Maximum Likelihood Theory
Chapter 3: Maximum Likelihood Theory Florian Pelgrin HEC September-December, 2010 Florian Pelgrin (HEC) Maximum Likelihood Theory September-December, 2010 1 / 40 1 Introduction Example 2 Maximum likelihood
More informationVectors and Matrices Statistics with Vectors and Matrices
Vectors and Matrices Statistics with Vectors and Matrices Lecture 3 September 7, 005 Analysis Lecture #3-9/7/005 Slide 1 of 55 Today s Lecture Vectors and Matrices (Supplement A - augmented with SAS proc
More informationDetection and Estimation Theory
Detection and Estimation Theory Instructor: Prof. Namrata Vaswani Dept. of Electrical and Computer Engineering Iowa State University http://www.ece.iastate.edu/ namrata Slide 1 What is Estimation and Detection
More informationConvergence of Square Root Ensemble Kalman Filters in the Large Ensemble Limit
Convergence of Square Root Ensemble Kalman Filters in the Large Ensemble Limit Evan Kwiatkowski, Jan Mandel University of Colorado Denver December 11, 2014 OUTLINE 2 Data Assimilation Bayesian Estimation
More informationStatistics II. Management Degree Management Statistics IIDegree. Statistics II. 2 nd Sem. 2013/2014. Management Degree. Simple Linear Regression
Model 1 2 Ordinary Least Squares 3 4 Non-linearities 5 of the coefficients and their to the model We saw that econometrics studies E (Y x). More generally, we shall study regression analysis. : The regression
More informationTutorial on Principal Component Analysis
Tutorial on Principal Component Analysis Copyright c 1997, 2003 Javier R. Movellan. This is an open source document. Permission is granted to copy, distribute and/or modify this document under the terms
More informationFundamentals of Statistical Signal Processing Volume II Detection Theory
Fundamentals of Statistical Signal Processing Volume II Detection Theory Steven M. Kay University of Rhode Island PH PTR Prentice Hall PTR Upper Saddle River, New Jersey 07458 http://www.phptr.com Contents
More information