System Identification & Parameter Estimation Wb3: SIPE lecture Correlation functions in time & frequency domain Alfred C. Schouten, Dept. of Biomechanical Engineering (BMechE), Fac. 3mE // Delft University of Technology
Contents Correlation functions in time domain Autocorrelations (Lecture ) Cross-correlations General properties of estimators Bias / variance / consistency Estimators for correlation functions Fourier transform Correlation functions in frequency domain Auto-spectral density Cross-spectral density Estimators for spectral densities
Lecture : (auto)correlation functions Autocorrelation function [ τ ] Φ ( τ) = E ( t ) ( t) Autocovariance function ( )( ) C ( τ ) = ( τ ) μ ( ) μ =Φ ( τ ) E t t μ Autocorrelation coefficient r ( τ ) t ( τ ) μ μ t ( ) Φ( τ) μ C( τ) = E = = σ σ σ C( ) 3
Lecture : autocovariance function Autocovariance function: ( )( ) [ ( τ ) ( )] [ ( τ) μ ] [ μ ( )] μ C ( τ) = E t ( τ) μ t ( ) μ = E t t E t + E t E =Φ ( τ) μ μ + μ =Φ ( τ) μ If μ =, then autocovariance and autocorrelation functions are identical At zero lag: ( ) C( ) = E ( t) μ = σ 4
white Eample autocovariances signal 5 Matlab demo: Lec_eample_Cuu.m C uu low-pass sinusoid sin. + noise -5 3 4 5-3 4 5-3 4 5 5-5 3 4 5 time [s] - - -.5 -.5 - -.5 -.5 - - - - - tau [s] 5
D Probability density function f y { < ( t; ζ ) + d y < y( t + τ ; y dy} (, y; τ ) ddy = Pr ζ ) + 6
D Probability density function Probability for certain values of y(t) given certain values of (t) Co-variance of y(t) with (t): y(t) is related with (t) o co-variance between y(t) and (t): D probability density function is circular Covariance between y(t) and (t): D probability density function is ellipsoidal o co-variance between y(t) and (t): y(t) and (t) are independent no relation eist transfer function is zero! 7
Cross-correlation functions Measure for common structure in two signals: Cross-correlation [ τ ] Φ y ( τ) = E ( t ) y( t) Cross-covariance ( )( ) C ( τ ) = τ μ μ ( ) ( ) = Φ ( τ ) y E t y t y y μμy Cross-correlation coefficient r y ( τ ) t ( τ) μ yt () μ y C ( τ) y = E = σ σ y C( ) Cyy ( ) 8
Eample: Estimation of a Time Delay Consider the system: yt () = α t ( τ ) + vt () Additive noise v(t) is stochastic and uncorrelated to (t), Cross-covariance of the measured input and output in case: input (t) is white noise (stochastic) 9
Eample: Estimation of a Time Delay 5 Matlab demo: Lec_eample_Cuy_delay.m u(t) u -5 5 5 5 3 35 4 45 5 time [s] u 5 y(t) -5 5 5 5 3 35 4 45 5 time [s] C uy (τ) - - -.5 - -.5.5.5 τ [s]
Effects of oise on Autocorrelations Often, correlation functions must be estimated from measurements. Consider (t ) and y (t ), corrupted with noise n (t ) and v (t ) resp.: wt () = t () + nt () zt () = yt () + vt () oises have zero mean and are independent of the signals and of each other. Autocorrelation: ( )( ) Φ zz( τ) = E y( t τ) + v( t τ) y() t + v() t = Φ ( τ ) +Φ ( τ) +Φ ( τ) +Φ ( τ) yy yv vy vv = Φ ( τ ) +Φ ( τ ) yy vv Additive noise will bias autocorrelation functions!
Effects of oise on Cross-correlations Cross-correlation: ( )( ) Φ wz( τ) = E ( t τ) + n( t τ) y( t) + v( t) =Φ ( τ ) +Φ ( τ) +Φ ( τ) +Φ ( τ) =Φ y v ny nv ( τ ) Additive noise will not bias cross-correlation functions! y
oise Reduction Longer recordings More information, same amount of noise Repeat the eperiment oise cancels out by averaging from eactly the same inputs Improve Signal-to-oise Ratio Concentrate power at specified frequencies, assuming the noise power remains the same 3
Properties of Estimators Formula given (autocovariances / spectral densities) are estimators for the true relations. What are the properties of these estimators? bias / variance / consistency Bias: structural error Variance: random error Consistent: A consistent estimator is an estimator that converges, in probability, to the quantity being estimated as the sample size grows. 4
Eample: estimator for mean value and variance of a signal Signal k, with k= Estimator for the signal mean: Estimator for the signal variance: ˆ μ = ˆ σ = k= ( ˆ k μ ) k= k What are the epected values of both estimators? Epectation operator: E{.} 5
6 Estimator for the mean value { } { } { } { } { } k k k k k k k k E E E E E E μ μ μ μ μ μ = = = = = = = = = = = ˆ ˆ ˆ
Estimator for the variance ˆ σ ( ˆ = k μ) = k l k= k= l= = + k kl lk k= l= l= k= E E E E k= l= l= k= { ˆ σ } = { k } { k l} + { l k} E{ ˆ σ } = K kl K kl σ + μ σ μ + σ + μ k= l= l= k = E{ ˆ σ} = σ + = σ = σ Estimator is biased and consistent 7
8 Estimator for the variance biased estimator: unbiased estimator: (default!) ( ) { } ˆ ˆ ˆ k k E σ σ σ μ σ = = = = ( ) { } ˆ ˆ ˆ k k E σ σ μ σ = = =
Estimates of Correlation Functions Cross-correlation: Let (t ) and y(t ) are finite time realizations of ergodic processes. Then: T Practically, signals are finite time and sampled every Δt. Signal (t) is sampled at t=, Δt,, (-)Δt giving (i) with i=,,,. Then: with i =,,, Φ ˆ ( ) lim ( ) ( ) y τ = t τ ytdt T T T Φ ˆ ( τ ) = τ ( ) ( ) y i yi τ i = τ [ τ ] Φ y ( τ) = E ( t ) y( t) 9
Estimates of Correlation Functions Unbiased estimator: Φ ˆ ( τ ) = τ ( ) ( ) y i yi τ i = τ Variance of the estimator increases with lag! To avoid this, divide by (biased estimator): Use large to minimize bias! I.e. ˆ Φ y ( τ) = i ( τ) yi ( ) Similar estimators can be derived for the covariance and correlation coefficient functions i = τ τ
Additional demo s Calculation of cross-covariance: Lec_calculation_Cuy.m ˆ Φ y ( τ) = i ( τ) yi ( ) i = τ
Cross-covariance of some basic systems Eample systems (Matlab Demo: Lec_eample_systems.m): Time delay st order nd order Input signals: White noise (W) Low pass filtered noise (LP: /(.5s+) )
time delay. st order system.5 nd order system C uy (W).5. -.5-5 5. -. -5 5.4 -.5-5 5. C uy (LP). -. -5 5 -. -5 5 -. -5 5 impulse.5 5-5 5 time [s] -5 5 time [s] -5-5 5 time [s] 3
Intermezzo: Fourier Transform Fourier Transform: Mapping between time-domain and frequency-domain One-to-one mapping Unique: inverse Fourier Transform eists Linear technique Inverse Fourier Transform: () ( ) j () π ft X( f) =I t = t e dt () j πtf t X f X f e df ( ( )) ( ) =I = 5
Fourier transformation y(t) is an arbitrary signal jπ ft Y( f) y( t) e dt = Euler formula: symmetric part: cos(z) anti-symmetric part: sin(z) ( ) sin ( ) jz e = cos z + j z jπ ft jπ ft jπ ft e = re e + im e = π ft j π ft ( ) ( ) cos( ) *sin( ) 6
Eample Fourier Transformation jπ ft Y ( f) = y( te ) dt y( t) = sin + 5.5 (.5Hz) sin( Hz) Y(.5Hz): 5 Hz signal will be averaged out.5 -.5 - -.5.5.5.5 3 3.5 4 4.5 5 7
Eample Fourier Transformation.5 y(t) = sin(.5t) 5 Y(ω) = δ(ω-.5).5 4.5 4 3.5 3.5 -.5 -.5.5 -.5.5.5.5 3 3.5 4 4.5 5 -.5.5.5 8
Frequency Domain Epressions Discrete Fourier Transform: =I ( ) = t = j Uf () ut () ute () π ft where f takes values,,, - multiples of Δ f = Δ t Inverse Fourier Transform: ut () =I ( Uf ()) = Ufe () f = ft j π 9
Fourier transform of signals Discrete Fourier Transform (DFT) u ( k) ; k [,,, ] U( r) = DFT U( r); k= { u( k) } = u( k) r e jπrk / [,,, ] DFT maps real values in time domain to comple values in frequency domain Double information? DFT is symmetric U(-r)=U(r)* Information for / comple values r,,, 3
Power spectrum or auto-spectral density auto-spectral density S is Fourier Transform of autocorrelation ( ) jωτ S ( ω) =I Φ ( τ) = Φ ( τ) e dτ ( ) π jfτ S ( f) =I Φ ( τ ) = Φ ( τ) e dτ properties: Real values (no imaginary part) Symmetry: S (f)=s (-f) => only positive frequencies are analyzed C () = S (f) df = σ Area under S (f) is equal to variance of signal (Parseval s theorem) 3
Cross-spectrum or Cross-spectral density cross-spectral density S y : ( ) j π fτ S ( f ) =I Φ ( τ ) = Φ ( τ) e dτ y y y properties: Comple values S y* (f)=s y (-f) => only positive frequencies are analyzed S y (f) describes the interdependency of signals (t) and y(t) in frequency domain (gain and phase) if Θ y (τ) =, then S y (f) = for all frequencies 3
Estimators for the power spectrum Fourier transform of the autocorrelation function, the power spectrum: using the time average estimate: and multiplication with gives: e Sˆ () f = Φˆ () τ e uu ( i π i) f j τ = ˆ Suu() f = u( i τ ) e u() i e τ= i = τ uu fτ j π ˆ Φ uu( τ) = ui ( τ) ui ( ) = e = i = τ ( i τ ) f if jπ jπ Indirect approach and direct approach give equal result! = U * ( nδf) U( nδf) * : comple conjugate: U * ( nδ f) = U( nδf) 33
Estimator for the cross-spectrum Fourier transform of the cross-correlation function is called the cross-spectrum: Sˆ () = Φ ˆ uy f uy () τ e τ = fτ j π * Sˆ uy ( f) = U ( nδf) Y( nδf) 34
Time-domain vs. Frequency-domain Time Domain Fourier Transformation Frequency Domain input, output (t), y(t) X(f), Y(f) input, output cross-correlation function Φ y (τ) S y (f) cross-spectral density cross-covariance function C y (τ) correlation coefficient r y (τ) Γ y (f) coherence 35
Discrete and Continuous Signals Reconstruction DA conversion Distortion: Hold Circuits Sampling AD conversion Distortion: "Dirac Comb" Continuous system: Plant Digital Controller: Plant performance enhancement 36
Sampling, Dirac s Comb 37
Models of Linear Systems System: u(t) yt () = ut ( ()) y(t) Linear systems obey both scaling and superposition property! ky() t = ku ( ()) t ky() t = ku ( ()) t ky() t + ky () t = ku ( () t + ku()) t 38
Time domain models Assume input is a pulse with: Δt for t < d( t, Δ t) = Δt otherwise Response of linear system to pulse: ( (, Δ )) = (, Δ ) d t t h t t Assume u(t) is weighted sum of pulses: = k Δ Δ k = () (, ) u t u d t k t t 39
Time domain models The output is: y() t = ( u()) t k = k = ( (, )) = u d t kδtδt k (, ) = uht kδtδt k Limit case: unit-area pulse d(t,δt) is an impulse: Δ t ( Δ ) = δ ( ) lim d t, t t Then h(t) is the impulse response function (IRF) and the output the convolution integral ( ) ( ) y() t = h τ u t τ dτ 4
Time domain models Causal system: h(t)= for t< Finite memory: h(t)= for t>t T ( ) ( ) y() t = h τ u t τ dτ o Discrete T τ = ( ) ( ) yt () = hτ ut τ Δτ 4
Frequency domain models Time-domain: convolution integral Fourier transform: Yf () =I ( yt ()) =I h( τ ) ut ( τ) dτ = ( ) ( ) jπft h τ u t τ dτ e dt πft τ τ τ ( ) ( ) = h u t e dtd πτ f Uf h e d Yf () = HfUf () () ( τ) ( ) ( ) yt () = hτ ut τ dτ Convolution in time = domain () is multiplication τ in frequency domain (and vice-versa) 4
Estimators in time domain Cross-covariance When analyzing the dynamics of a dynamical system interested in variations of the signals around it mean (and not average value). Assuming signals with zero-mean yt () = ( h u)() t = ht (') ut ( t') dt' C ( ) { ( ) ( )} uy τ = E u t τ y t = E h( t') u( t τ) u( t t') dt' 43
Basic identification with crosscovariance ( ) = ( ) + ( ') ( ) y t n t h t u t-t' dt' ( ) () ( ) () ( ) ( ) multiply with ut ( τ): u t τ y t = u t τ n t + h t' u( t τ) u t-t' dt' ( ) = ( τ) + ( ) ( τ ') C τ C h t' C t dt' uy un uu ( ) uu ( ) ( ) = ( τ) + ( τ) white noise: C τ = for τ ; C = uu uy n(t)? u(t) y(t) un C τ C h Other 'tricks' needed when u(t) is not white 44
Basic identification with spectral densities ( ) = ( ) + ( ') ( ) yt nt ht ut-t'dt' ( ) ( ) ( ) ( ) ( ) ( ) multiply with ut ( τ): u t τ y t = u t τ n t + h t' u( t τ) u t-t' dt' Fourier transform: n(t)? u(t) y(t) ( ) = ( τ) + ( ) ( τ ') ( ) = ( ) + ( ) ( ) C τ C h t' C t dt' uy un uu S f S f H f S f uy un uu ( ) ( ) if S f = : H f = un S S uy uu ( f ) ( f ) 45
Identification: time-domain vs. frequency-domain n(t) u(t)? y(t) Time domain Unknown system: impulse response function (IRF) of h(t ) over limited time Mostly: direct model parameterization (fit of physics model) Frequency domain Unknown system: frequency response function (FRF) H(f) for number of frequencies 46
Eample IRF & FRF of typical systems Time delay First order system Second order system Unstable system (try Matlab!) 47
Book: Westwick & Kearney Lecture : signals Chapter, all Chapter, sec...3. Lecture : Correlation functions in time & frequency domain Chapter, sec..3..3.4 Chapter 3, sec. 3. 3. Lecture 3: Estimators for impulse & frequency response functions Chapter 5, sec. 5. 5.3 48
et week: lecture 3 Lecture 3: Estimation of IRF and FRF Estimator for coherence Open loop and closed loop system identification 49