Chapter 6: Nonparametric Time- and Frequency-Domain Methods. Problems presented by Uwe

System Identification written by L. Ljung, Prentice Hall PTR, 1999 Chapter 6: Nonparametric Time- and Frequency-Domain Methods Problems presented by Uwe System Identification Problems Chapter 6 p. 1/33

Outline Introduction Problem 6G.3 Problem 6E.1 Problem 6E.3 System Identification Problems Chapter 6 p. 2/33

Transient-Response Analysis Impulse-Response: If a strictly stable LTI system is subjected to a pulse input { α, t = 0 y(t) = G 0 (q)u(t) + v(t), u(t) = 0, t 0 System Identification Problems Chapter 6 p. 3/33

Transient-Response Analysis Impulse-Response: If a strictly stable LTI system is subjected to a pulse input { α, t = 0 y(t) = G 0 (q)u(t) + v(t), u(t) = 0, t 0 the output will be y(t) = k=1 g(k)q k u(t) + v(t) = k=1 g(k)u(t k) + v(t) = αg 0 (t) + v(t) System Identification Problems Chapter 6 p. 3/33

Transient-Response Analysis Impulse-Response: If a strictly stable LTI system is subjected to a pulse input { α, t = 0 y(t) = G 0 (q)u(t) + v(t), u(t) = 0, t 0 the output will be y(t) = αg 0 (t) + v(t) If the noise level is low, estimates for the impulse-response coefficients can be determined by ĝ 0 (t) = y(t) α with the errors ε(t) = v(t) α System Identification Problems Chapter 6 p. 3/33

Transient-Response Analysis Step-Response: If a strictly stable LTI system is subjected to a step input { α, t 0 y(t) = G 0 (q)u(t) + v(t), u(t) = 0, t < 0 System Identification Problems Chapter 6 p. 4/33

Transient-Response Analysis Step-Response: If a strictly stable LTI system is subjected to a step input { α, t 0 y(t) = G 0 (q)u(t) + v(t), u(t) = 0, t < 0 the output will be y(t) = k=1 g(k)q k u(t) + v(t) = k=1 g(k)u(t k) + v(t) = α t k=1 g 0(k) + v(t) System Identification Problems Chapter 6 p. 4/33

Transient-Response Analysis Step-Response: If a strictly stable LTI system is subjected to a step input { α, t 0 y(t) = G 0 (q)u(t) + v(t), u(t) = 0, t < 0 the output will be y(t) = α t k=1 g 0(k) + v(t) Since the step response is the integral of the impulseresponse, the estimates for g 0 (k) can be determined as y(t) y(t 1) v(t) v(t 1) ĝ 0 (t) = with ε(t) = 1 α 1 α System Identification Problems Chapter 6 p. 4/33

Constraints in Transient-Response Analysis Many physical processes do not allow pulse / step inputs of such an amplitude that the error is insignificant compared to the impulse-response coefficients. However, the step-response is suitable to determine some model characteristics, such as delay time, static gain, and dominating time constants. System Identification Problems Chapter 6 p. 5/33

Correlation Analysis Considering a strictly stable LTI system, where the input is a quasi-stationary sequence with R u (τ) = E {u(t)u(t τ)} R uv (τ) = E {u(t)v(t τ)} 0 (open-loop!) System Identification Problems Chapter 6 p. 6/33

Correlation Analysis Considering a strictly stable LTI system, where the input is a quasi-stationary sequence with R u (τ) = E {u(t)u(t τ)} R uv (τ) = E {u(t)v(t τ)} 0 (open-loop!) The input-output covariance function will be E {y(t)u(t τ)} = G 0 (q)e {u(t)u(t τ)} +E {v(t)u(t τ)} R yu (τ) = G 0 (q)r u (τ) = k=1 g 0(k)R u (k τ) (time domain) System Identification Problems Chapter 6 p. 6/33

Correlation Analysis (cont d) The input-output covariance function will be R yu (τ) = k=1 g 0(k)R u (k τ) If u(t) is chosen as white noise, it follows for the estimate R yu (τ) = λ g 0 (τ) = ĝ 0 (τ) = ˆR N yu (τ)/λ where ˆR N yu (τ) = 1 N N t=τ y(t)u(t τ). If the input is not white noise, we may estimate ˆR N u (τ) = 1 N N t=τ and solve u(t)u(t τ) ˆR N yu (τ) = M k=1 ĝ0(k) ˆR N u (k τ) = ĝ 0(k) System Identification Problems Chapter 6 p. 7/33

Frequency-Response Analysis Applying an input sinusoid to the strictly stable LTI system allows to determine y(t) = G 0 (q)u(t) + v(t) the amplitude α ĜN(e jω ) and phase shift ˆϕ N = argĝ N (e jω ) of the resulting output sinusoid for a number of frequencies. The influence of the noise component v(t) can be eliminated by the correlation method or filtering. System Identification Problems Chapter 6 p. 8/33

Empirical Transfer-Function Estimate (ETFE) In the frequency domain, the transfer-function estimate is Ĝ N (e jω ) = Y N(ω) U N (ω) based on data over the interval 1 t N, where Y N (ω) = 1 N N t=1 U N (ω) = 1 N N t=1 y(t)e jωt u(t)e jωt System Identification Problems Chapter 6 p. 9/33

Properties of the ETFE Given the strictly stable LTI system y(t) = G 0 (q)u(t) + v(t) it follows for the data interval 1 t N Y N (ω) = G 0 (e jω )U N (ω) + ρ 1 (N) + V N (ω), System Identification Problems Chapter 6 p. 10/33

Properties of the ETFE Given the strictly stable LTI system y(t) = G 0 (q)u(t) + v(t) it follows for the data interval 1 t N Y N (ω) = G 0 (e jω )U N (ω) + ρ 1 (N) + V N (ω), where ρ 1 (N) shows up as rest term due to the finite summation ρ 1 (N) C 1 N. If the input is periodic with period N, ρ 1 (N) is zero. System Identification Problems Chapter 6 p. 10/33

Properties of the ETFE (cont d) Ĝ N (e jω ) = Y N(ω) U N (ω) = G 0(e jω ) + ρ 1(N) U N (ω) + V N(ω) U N (ω) System Identification Problems Chapter 6 p. 11/33

Properties of the ETFE (cont d) Ĝ N (e jω ) = Y N(ω) U N (ω) = G 0(e jω ) + ρ 1(N) U N (ω) + V N(ω) U N (ω) Mean value (most expected value): Since the disturbance v(t) is assumed to have zero mean, E {V N (ω)} = 0, ω where ρ 1 (N) C 1. N } E {Ĝ N (e jω ) = G 0 (e jω ) + ρ 1(N) U N (ω) That means, the ETFE is asymptotically unbiased. System Identification Problems Chapter 6 p. 11/33

Properties of the ETFE (cont d) The covariance function of the disturbance can be computed as R v (τ) = E {v(t)v(t τ)} = E {V N (ω)v N ( ξ)} System Identification Problems Chapter 6 p. 12/33

Properties of the ETFE (cont d) The covariance function of the disturbance can be computed as R v (τ) = E {v(t)v(t τ)} = E {V N (ω)v N ( ξ)} E {V N (ω)v N ( ξ)} = Φ v (ω) + ρ 2 (N), = ρ 2 (N), if ξ = ω if ξ ω = 2πk N k = 1, 2,..., N 1 where ρ 2 (N) C 2. N System Identification Problems Chapter 6 p. 12/33

Properties of the ETFE (cont d) Covariance of the estimation error: ]} E {[Ĝ N (e jω ) G 0 (e )] [Ĝ jω N (e jξ ) G 0 (e jξ ) 1 [Φ U N (ω) 2 v (ω) + ρ 2 (N)], if ξ = ω = 1 U N (ω)u N ( ξ) ρ 2(N), where ρ 2 (N) C 2. N = if ξ ω = 2πk N k = 1, 2,..., N 1 That means, the ETFE has a variance of Φ v (ω)/ U N (ω) 2 at each frequency, but the estimates at different frequencies are asymptotically uncorrelated. System Identification Problems Chapter 6 p. 13/33

Problems with the ETFE The variance of the ETFE does not decay with increasing N. We determine as many independent estimates as we have data points. Linearity of the true system is the only assumption we made, i.e. the systems properties at different frequencies may be totally unrelated. Assuming that the values of the true transfer function at different frequencies are related will improve the poor variance properties. The result will be data and information compression. System Identification Problems Chapter 6 p. 14/33

Spectral Analysis Smoothing the ETFE The ETFE Ĝ N (e jω ) represents unbiased and uncorrelated estimates of G 0 (e jω ) for ω = 2πk/N with the variance Φ v (ω)/ U N (ω) 2. System Identification Problems Chapter 6 p. 15/33

Spectral Analysis Smoothing the ETFE The ETFE Ĝ N (e jω ) represents unbiased and uncorrelated estimates of G 0 (e jω ) for ω = 2πk/N with the variance Φ v (ω)/ U N (ω) 2. Provided that the frequency distance 2π/N is small compared to how quickly G 0 (e jω ) changes, we can assume G 0 (e jω ) to be constant over the interval 2πk 1 N = ω 0 ω < ω < ω 0 + ω = 2πk 2 N Forming a weighted average of the "measurements" for this interval, will estimate the constant G 0 (e jω 0 ). System Identification Problems Chapter 6 p. 15/33

Spectral Analysis Smoothing the ETFE (cont d) Weighting according to the inverse variance gives Ĝ N (e jω 0 ) = ω0 + ω ω 0 ω α(ξ)ĝ N (e jξ )dξ ω0 + ω ω 0 ω α(ξ)dξ, α(ξ) = U N (ξ) 2 Φ v (ξ) System Identification Problems Chapter 6 p. 16/33

Spectral Analysis Smoothing the ETFE (cont d) Weighting according to the inverse variance gives Ĝ N (e jω 0 ) = ω0 + ω ω 0 ω α(ξ)ĝ N (e jξ )dξ ω0 + ω ω 0 ω α(ξ)dξ, α(ξ) = U N (ξ) 2 Φ v (ξ) If the transfer function G 0 (e jω ) is not constant over the interval, additional weighting close to ω 0 is required. Ĝ N (e jω 0 ) = π π W γ(ξ ω 0 ) U N (ξ) 2 Ĝ N (e jξ )dξ π π W γ(ξ ω 0 ) U N (ξ) 2 dξ W γ (ξ) is a frequency window centered in ξ = 0, with γ as shape parameter. Φ v (ξ) is assumed constant around ω 0. System Identification Problems Chapter 6 p. 16/33

Spectral Analysis Smoothing the ETFE Example y(t) 1.5y(t 1)+0.7y(t 2) = u(t 1)+0.5u(t 2)+e(t) System Identification Problems Chapter 6 p. 17/33

Spectral Analysis Smoothing the ETFE Example System Identification Problems Chapter 6 p. 18/33

Relation of the ETFE to the Spectral Estimates Provided that u(t) and v(t) are independent, the cross spectrum between input and output is defined as Φ yu (ω) = G(e jω )Φ u (ω) This formula can also be applied to the smoothed ETFE Ĝ N (e jω 0 ) = ˆΦ N yu (ω 0) ˆΦ N u (ω 0) with the corresponding spectral estimates ˆΦ N yu (ω 0) = π π W γ(ξ ω 0 ) U N (ξ) 2 Ĝ N (e jξ )dξ ˆΦ N u (ω 0) = π π W γ(ξ ω 0 ) U N (ξ) 2 dξ System Identification Problems Chapter 6 p. 19/33

Estimating the Disturbance Spectrum Analogous to the input spectrum, we can write for the disturbance spectrum ˆΦ N v (ω) = π π W γ (ξ ω) V N (ξ) 2 dξ If v(t) is not measurable, an estimate can be used by ˆΦ N v (ω) = π π ˆv(t) = y(t) Ĝ N (q)u(t) W γ (ξ ω) Y N (ξ) Ĝ N (e jξ )U N (ξ) 2 dξ System Identification Problems Chapter 6 p. 20/33

Properties of the estimates Ĝ N (e jω 0 ) and Φ N v (ω) The properties are dependent on the shaping parameter γ: large γ generates a narrow window with small bias but high variance small γ generates a wide window with high bias but small variance System Identification Problems Chapter 6 p. 21/33

Problem 6G.3 The amplitude of the ETFE appears to be systematically larger than the true amplitude, despite the fact that the ETFE is unbiased (cf. slide 11). However, Ĝ being an unbiased estimate of G 0 does not imply that Ĝ is an unbiased estimate of G 0. In fact, prove that { E Ĝ N (e jω ) 2} = G0 (e jω ) 2 + Φ v(ω) asymptotically for large N. U N (ω) 2 System Identification Problems Chapter 6 p. 22/33

Problem 6G.3 The ETFE is defined as (cf. slide 11) Ĝ N (e jω ) = Y N(ω) U N (ω) = G 0(e jω ) + ρ 1(N) U N (ω) + V N(ω) U N (ω) The magnitude of the ETFE can be determined by Ĝ N (e jω ) 2 = Ĝ N (e jω )Ĝ N (e jω ) ( ) = G 0 (e jω ) + ρ 1(N) U N (ω) + V N(ω) U N ( (ω) G 0 (e jω ) + ρ 1(N) = G0 (e jω ) 2 ( ) + G 0 (e jω ) ρ1 (N) U N ( ω) + V N( ω) U N ( ω) U N ( ω) + V N( ω) U N ( ω) + ρ 1(N) U N (ω) G 0(e jω ) + ρ2 1(N) + ρ 1(N)V N ( ω) U N (ω) 2 U N (ω) 2 + V N(ω) U N (ω) G 0(e jω ) + ρ 1(N)V N (ω) + V N(ω)V N ( ω) U N (ω) 2 U N (ω) 2 ) System Identification Problems Chapter 6 p. 23/33

Problem 6G.3 Taking expectation with respect to v(t) (cf. slide 12): { E Ĝ N (e jω ) 2} = G0 (e jω ) 2 + G 0(e jω )ρ 1 (N) + G 0(e jω )ρ 1 (N) U N (ω) + Φ v(ω)+ρ 2 (N) U N (ω) 2 U N ( ω) + ρ2 1(N) U N (ω) 2 System Identification Problems Chapter 6 p. 24/33

Problem 6G.3 Taking expectation with respect to v(t) (cf. slide 12): { E Ĝ N (e jω ) 2} = G0 (e jω ) 2 + G 0(e jω )ρ 1 (N) + G 0(e jω )ρ 1 (N) U N (ω) + Φ v(ω)+ρ 2 (N) U N (ω) 2 U N ( ω) + ρ2 1(N) U N (ω) 2 For large N, the function becomes asymptotically to { E Ĝ N (e jω ) 2} = G0 (e jω ) 2 + Φ v(ω) U N (ω) 2 It means that the magnitude of the ETFE Ĝ N (e jω ) is an biased estimate of G0 (e jω ), where Φ v (ω)/ U N (ω) 2 represents the variance. System Identification Problems Chapter 6 p. 24/33

Problem 6E.1 Determine an estimate for G 0 (e jω ) based on the impulse-response estimates ĝ 0 (t) (cf. slide 3). Show that the estimate coincides with the ETFE. System Identification Problems Chapter 6 p. 25/33

Problem 6E.1 Determine an estimate for G 0 (e jω ) based on the impulse-response estimates ĝ 0 (t) (cf. slide 3). Show that the estimate coincides with the ETFE. Estimates for the impulse-response coefficients are ĝ 0 (t) = y(t) α The transfer function G 0 (q) is defined as G(q) = g(k)q k = G(e jω ) = k=1 k=1 g(k)e jωk which corresponds to G(e jω ) in the frequency domain. System Identification Problems Chapter 6 p. 25/33

Problem 6E.1 The transfer function estimate for the data interval [1, N] will be N Ĝ N (e jω ) = g(k)e jωk k=1 Using the impulse-response estimates, it follows N N Ĝ N (e jω ) = ĝ(k)e jωk y(k) = α e jωk k=1 = 1 α N k=1 y(k)e jωk k=1 System Identification Problems Chapter 6 p. 26/33

Problem 6E.1 Since the pulse input is defined as { α, t = 0 u(t) = 0, t 0 the ETFE will be Ĝ N (e jω ) = Y N(ω) U N (ω) = = N 1 k=0 y(k)e jωk α 1 N 1 N k=0 y(k)e jωk 1 N 1 N k=0 u(k)e jωk which is the same as the transfer function estimate based on the estimated impulse-response coefficients. System Identification Problems Chapter 6 p. 27/33

Problem 6E.3 Let ω k, k = 1,..., M, be independent random variables, all with mean values 1 and variances E { (ω k 1) 2} = λ k. Consider the weighted sum ω = M k=1 α k ω k Determine α k, k = 1,..., M, so that (a) E {ω} = 1. (b) E { (ω 1) 2} is minimized. System Identification Problems Chapter 6 p. 28/33

Problem 6E.3 The mean values of ω k are E {ω k } = 1 According to (a), it follows for the expectation of ω M M E {ω} = α k E {ω k } = α k = 1 k=1 k=1 System Identification Problems Chapter 6 p. 29/33

Problem 6E.3 The mean values of ω k are E {ω k } = 1 According to (a), it follows for the expectation of ω M M E {ω} = α k E {ω k } = α k = 1 k=1 k=1 The variance of the weighted sum ω equals to E { { (ω 1) 2} [ M = E k=1 α kω k M { [ M ] } 2 = E k=1 α k (ω k 1) k=1 α k ] 2 } System Identification Problems Chapter 6 p. 29/33

Problem 6E.3 Opening the bracket, it follows E { (ω 1) 2} { M = E k=1 α2 k (ω k 1) 2 +2 M 1 k=1 α k(ω k 1) } M i=k+1 α i(ω i 1) System Identification Problems Chapter 6 p. 30/33

Problem 6E.3 Opening the bracket, it follows E { (ω 1) 2} { M = E k=1 α2 k (ω k 1) 2 +2 M 1 k=1 α k(ω k 1) } M i=k+1 α i(ω i 1) Applying the variances E { (ω k 1) 2} = λ k and the covariances E {(ω k 1)(ω i 1)} = 0, for k i provided that ω k are independent, we obtain E { (ω 1) 2} M = α 2 k λ k k=1 System Identification Problems Chapter 6 p. 30/33

Problem 6E.3 The minimization problem, regarding the variance for the weighted sum ω, will be E { (ω 1) 2} M = α 2 k λ k min k=1 where the following condition has to be fulfilled: M E {ω} = α k = 1 k=1 System Identification Problems Chapter 6 p. 31/33

Problem 6E.3 The Lagrangian L leads to the system of equations L α k = 2α k λ k + γ = 0 L γ = M k=1 α k 1 = 0 System Identification Problems Chapter 6 p. 32/33

Problem 6E.3 The Lagrangian L leads to the system of equations L α k = 2α k λ k + γ = 0 L γ = M k=1 α k 1 = 0 where the solutions can be computed by α k = γ 1 2 λ k solving the following equation for γ M α k = 1 = γ 2 k=1 M k=1 1 λ k System Identification Problems Chapter 6 p. 32/33

Problem 6E.3 For example if M = 3, the coefficients α k are computed as γ = α 1 = α 2 = α 3 = 1 1 2 λ 1 + 1 λ 2 + 1 λ 3 1 λ 1 λ 1 + 1 λ 2 + 1 λ 3 1 1 λ 2 λ 1 + 1 λ 2 + 1 λ 3 1 λ 3 1 λ 1 + 1 λ 2 + 1 λ 3 System Identification Problems Chapter 6 p. 33/33