EXACT QUANTIFICATION OF VARIANCE ERROR

Transcription

1 Copyright 00 IFAC 5th Triennial World Congress, Barcelona, Spain EXACT QUATIFICATIO OF VARIACE ERROR Brett inness, Håkan Hjalmarsson Dept. of Elec. & Comp. Eng, Uni. ewcastle, Australia. FAX: Department of Sensors, Signals and Systems Automatic Control), The Royal Institute of Technology, S Stockholm, Sweden, FAX: Abstract: This paper establishes a method for quantifying variance error in cases where the input spectral density has a rational factorisation. Compared to previous work which has involved asymptotic-in-model-order techniques and yielded expressions which are only approximate for finite orders, the quantifications provided here are exact for finite model order, although they still only apply asymptotically in the observed data length. The key principle employed here is the use of a reproducing kernel in order to characterise the model class, and this implies a certain geometric-type approach to the problem. Keywords: Parameter Estimation, Maximum Likelihood Estimators.. ITRODUCTIO There has been much work over the last several years directed at quantifying the noise induced error variance error) in frequency response estimates L.Ljung and Z.D.Yuan 985, L.Ljung 985, P.M.J. Van den Hof et al. 995, Wahlberg 99, Wahlberg 994, inness et al. 999). In these contributions, in the interests of tractable analysis, a common theme has been to consider the limiting value of the variance error as the model order grows, and then assume that this same value can be approximately used for finite model order. This approach leaves open the question of the accuracy of the ensuing approximate quantification, and indeed recent works P.M.J. Van den Hof et al. 995, Wahlberg 99, Wahlberg 994, inness et al. 999) have specifically addressed the issue by using certain orthonormal basis techniques designed to maximise the finite model order accuracy. The paper here takes a different approach, and shows how in some circumstances it is possible to quantify the variance error in a manner that is exact even for finite model order, although it is still asymptotic in the data length. The key technique used to achieve this is a geometric one, in which recognition of certain subspace invariants reproducing kernels) provides a means for the exact quantification. The importance of this issue of non-asymptotic in model order variance quantification has been recognised by other authors Xie and Ljung 000), who have employed an This work was supported by the Australian Research Council and the Centre for Integrated Dynamics and Control CIDAC). It was completed while the first author was visiting the Department of Sensors, Signals and Systems Automatic Control), The Royal Institute of Technology, Stockholm, Sweden, analysis technique that is quite different to that of this work. The penultimate section of this paper profiles the results of this approach versus the one taken here.. PROBLEM SETTIG The estimation problem considered here is one in which observed input-output data y t, u t is generated according to y t = Gq)u t + e t. Here e t is a zero-mean white noise sequence that satisfies Ee t = σ. As well, Gq) is assumed to be an n th order rational transfer function with poles at ξ 0,, ξ n, and u t is taken to be a second order stationary process with associated spectral density 0 < Φ u ω) <. For the purposes of estimating the dynamics Gq), it is supposed that the following parameterised model structure is used where y t = Gq, θ)u t + e t ) n Gq, θ) = A q) θ k q k, Aq) = n q ξ k ). Here, the fixed poles ξ 0,, ξ m are meant to be chosen via a-priori knowledge Wahlberg 99, Wahlberg 994, Heuberger et al. 995, P.M.J. Van den Hof et al. 995), and the special case of ξ k = 0 renders ) as the common FIR model structure. With this in mind, an estimate θ of θ [θ 0,, θ n ] T based on observations of y t and u t may be found via a least squares approach:

2 θ = arg min V θ) ) θ V θ) y t Gq, θ)u t ). t= Furthermore, since Gq) is assumed to be n th order, then there exists a true parameter vector θ such that Gq, θ ) = Gq) and hence with ψ t = A q)λq)u t, Λq) [q n,, q, ] T 3) it is clear that the expression θ θ = φ t φ T t t= ) φ t e t 4) t= is a quantification of the parameter space estimation error. Furthermore, since the estimated frequency response Ge jω, θ ) depends linearly on the parameter estimate θ according to Ge jω, θ ) = A e jω )Λ T e jω ) θ then the frequency response estimation error may be quantified from 4) as Ge jω, θ ) Ge jω )=A e jω )Λ T e jω ) ) φ t φ T t φ t e t 5) t= t= Of course, this expression 5) depends on the exact noise realisation e t which cannot be known, so in order to quantify the estimation error Ge jω, θ ) Ge jω ) it is usual to consider its average over the ensemble of possible noise realisations. That is, consideration is given to ), which by 5) and assuming openloop data collection, may be computed as where ) = σ R Λ e jω )R Λejω ) Ae jω ) 6) φ t φ T t t= and represents the operation of conjugate transpose. ow, while 6) gives an exact quantification of the noise induced estimation error, it does not expose very much about how experiment design choices or other factors) might affect it. In consideration of this, the approach pioneered in works such as Whittle 953, Hannan and icholls 977, L.Ljung and Z.D.Yuan 985) is to notice that according to the assumptions made on the input R lim R Φu = T n A 7) Λe jλ )Λ e jλ ) Φ uλ) Ae jλ dλ. 8) ) Here, the notation T n Φ u ) is chosen to denote a Toeplitz matrix, of size n n, that is completely characterised by the spectral density function Φ u. In fact, a well known feature of such matrices is that for large n, then ) A T n T n Φu A Φ u where the approximation sign means that the Hilbert Schmidt matrix norm Golub and Loan 989) between the two quantities can be made arbitrarily small for arbitrarily large model roder n Grenander and Szegö 958). In this case, for the same large n it is inviting to approximate ) from 6) as Finally, note that ) σ Ae jω ) A Λ e jω )T n Λe jω ). 9) nπ n Λ e jω )T n Φ u A Φ u ) Λe jω ) = Λ e jω )Λe jλ ) Aejλ ) Φ u λ) τ= dλ = τ ) c τ e jωτ 0) where the c τ are the Fourier co-efficients of A /Φ u : c τ Ae jλ ) Φ u λ) e jλτ dλ. ) Since the Fourier series in 0) can be expected to converge to the function A /Φ u determining its coefficients via ), it is arguable via 9), 0) that for large model order n ) σ n Ae jω ) Aejω ) Φ u ω) = n σ Φ u ω). ) In contrast to 6), this approximate expression ) very clearly exposes the key factors that contribute to the noise induced estimation error. As such, its use has become a fundamental part of the science of System Identification Ljung 999) since it was first presented for the FIR case ξ k = 0) in L.Ljung and Z.D.Yuan 985) and then extended to much more general model structures in L.Ljung 985, P.M.J. Van den Hof et al. 995, Wahlberg 99, Wahlberg 994). However, as argued in inness et al. 999, inness and Hjalmarsson 00a, inness and Hjalmarsson 00b), this reliance can be misplaced. For example, the Achillee s Heel of ) is that it clearly depends on the model order

3 n being large in order for the approximating steps to be accurate. In recognition of this, the work inness et al. 999) has suggested that a quantification that can be more reliable for low model order n is ) σ n Φ u ω) ξ k e jω ξ k 3) ote that as pointed out in inness et al. 999, Xie and Ljung 000), the greater the proportion of the poles ξ k located at the origin, the closer the approximation 3) is to the seminal one ), and when all the poles are at the origin, then ) becomes a special case of 3). evertheless, the approximation 3) still depends on high model order n in order to obtain high accuracy. In recognition of this last point, the purpose of this paper is to derive an expression for lim ) that is exact for finite model order n. 3. MATHEMATICAL BACKGROUD The key tool used in this paper is what might be called a geometric one, and it depends on the idea of a Reproducing Kernel. To explain this, note that any frequency response of interest can be assumed to lie in a certain Hilbert Space H, where any geometric ideas such as orthogonality are obtained from the inner product between two functions f, g H which is defined as f, g µ fλ)gλ) µλ) dλ 4) with µλ) being some positive definite weighting function. ow, suppose that f is an element of an n-dimensional subspace X n defined by certain orthonormal basis elements ϕ 0,, ϕ n as f X n Span ϕ 0,, ϕ n. Then it is a consequence of the Riesz Representation TheoremRiesz and Sz.-agy 955, Rudin 966) that for any fixed ω, a further element K n λ, ω) X n exists such that fω) = fλ), K n λ, ω) µ f X n. 5) This reproducing kernel will be vital to the arguments of this paper, so it is important to present its fundamental features. Firstly it is Hermitian Symmetric in that, since for any fixed λ the kernel K n ω, λ) X n, then integration in the inner product is with respect to the common variable) K n ω, λ) = K n σ, λ), K m σ, ω) µ = K n σ, ω), K n σ, λ) µ = K n λ, ω). This then implies that K n ω, λ) is the unique element in X n that has the property 5), since if another function H n ω, σ) also satisfied 5), then it would hold that H n ω, λ) = H n ω, λ) = H n σ, λ), K n σ, ω) µ = K n σ, ω), H n σ, λ) µ = K n λ, ω). Finally, although the reproducing kernel is unique, there may of course) be many different ways of expressing it. One obvious one uses the orthonormal basis ϕ k for X n to express the quantity as n K n λ, ω) = ϕ k λ)ϕ k ω). 6) This formulation can be verified by noting that if X n f = τ c τϕ τ for some constants c τ, then n n c τ ϕ τ λ), ϕ k λ)ϕ k ω) = τ=0 n n c τ τ=0 ϕ k ω) ϕ τ λ), ϕ k λ) µ = n c τ ϕ τ ω) = fω). 7) τ=0 4. MAI RESULT With these preliminary ideas in mind, then as just argued via 6) and 9) lim ) = σ Ae jω ) Λ e jω )Tn Φu A Λe jω ). 8) The challenge then is to quantify this quadratic form in an exact manner, as opposed to the previously profiled approximation schemes based on asymptotic analysis. This ambition may be achieved by the following Theorem, which is the main result of the paper. Theorem 4.. Suppose that Φ u ω) has a finite dimensional rational spectral factorisation Φ u ω) = κ Ψe jω )Ψe jω ), Then Ψz) ν t=0 µ z βt z α t Ae jω ) Λ e jω )Tn Φu A Λe jω ) ). = n κ ϕ k e jω ) 9) where ϕ k z) is such that Ψz)ϕ 0 z),, Ψz)ϕ n z) is an orthonormal set of functions satisfying

4 X n Span Ψz)ϕ 0 z),, Ψz)ϕ n z) = Span Ψz) zn Az),, Ψz). 0) Az) Proof: ote that as a simple consequence of the definition 8) Ψe jω ) Ae jω ) Λ ne jω )Tn Φu A Λ n e jλ ) κ Ψejλ )Ψe jλ ) Ae jλ )Ae jλ ) Λ ne jλ ) dλ = Ψejω ) Ae jω ) Λ ne jω ). Therefore, the reproducing kernel K n λ, ω) for the space X n defined in 0) is given by the following quadratic form: K n λ, ω) = κ Ψejλ )Ψe jω ) Ae jλ )Ae jω ) Λ e jω )Tn Φu A Λe jλ ). ) However, as established in 7), if the set Ψz)ϕ 0 z),, Ψz)ϕ n z) is an orthonormal basis for X n then n K n λ, ω)= Ψe jλ )ϕ k e jλ )Ψe jω )ϕ k e jω ) n =Ψe jλ )Ψe jω ) ϕ k e jλ )ϕ k e jω ). ) Equating ) with ) when λ = ω then gives 9). On account of 8), this result immediately gives the following exact in n) formulation for variance error lim ) = σ n κ ϕ k e jω ) 3) which implies the following error quantification, which is not asymptotic in model order n: ) σ n κ ϕ k e jω ). 4) Of course, for this to produce an explicit quantification it is necessary to compute the basis Ψz)ϕ k z) for the space X n defined in 0). The following examples will demonstrate how this may be achieved analytically. 5. FIRST ORDER EXAMPLE In order to illustrate the preceding ideas, consider the simplest and lowest order example possible in which the observed data is obtained from the first order system ξ0 y t = u t + e t q ξ 0 where the spectral factor Ψz) of Φ u is of the form Ψz) = z β z α. Then an elementary calculation shows that Ψz) z ξ 0 ) = + αξ 0) + β ) βα + ξ 0 ) α ) αξ 0 ) ξ 0 ) and hence the unit norm version Ψz)ϕ 0 z) of Ψz)z ξ 0 ) is [ α ] ) αξ 0 ) Ψz)ϕ 0 z) = + αξ 0 ) + β ) βα + ξ 0 ) Therefore, according to 3) Ψz) z ξ 0 ). lim VarGejω, θ n ) = σ κ α ) ξ 0 ) αξ 0 ) + αξ 0 ) + β ) βα + ξ 0 ) e jω ξ 0. 5) For the case of α = 0.5, β = 0, ξ 0 = 0.9, κ =, σ = e 4 the results of using the ensuing quantification VarGe jω, θ n ) σ α ) ξ 0 ) αξ 0 ) κ + αξ 0 ) + β ) βα + ξ 0 ) e jω ξ 0. 6) are shown in figure. ote that the expression 6) is es Estimate variability vs exact quantification and asymptotic approximation Sample Variability Exact Quantification Asymptotic Approximation Fig.. Variability of Fixed Denominator Estimate - First Order Example. True variability vs. theoretically derived approximations. Solid line is Monte Carlo estimate of true variability, dashed line is quantification 6), dash-dot line is asymptotic-based approximation ). sentially) exact, which provides one example validating the claim of this paper that it is possible to derive accurate variance error quantifications that apply for arbitrary low model order. Here, and in what follows, the true variability is estimated via the sample average over 0000 different realisations of input and measurement noise, and is illustrated as a solid line. The exact quantification, such as given by 6) is shown as a dashed line, and the well known approximation ) is shown as a dash-dot line.

5 6. SECOD ORDER EXAMPLE Suppose we increase the model complexity in the previous example to that of Gq) = ξ 0) ξ ) q ξ 0 )q ξ ). In order to compute the asymptotic variance associated with this model structure, it is necessary to compute the orthonormal basis Ψϕ 0, Ψϕ that spans the same space as Ψz)z ξ 0 ), Ψz)z ξ ). In fact, this is quite difficult, primarily because of the restriction that all elements in this space have the same zeroes as Ψz). evertheless it can be achieved via the well known Gramm Schmidt procedure. Specifically, we have already established that [ Ψz)ϕ 0 z) = α ) αξ 0 ) + αξ 0 ) + β ) βα + ξ 0 ) Ψz) z ξ 0 ) ] is of unit norm, and hence a basis element fz) that is orthonormal to Ψz)ϕ 0 z) and also such that SpanΨz)ϕ 0, f = SpanΨz)z ξ 0 ), Ψz ξ ) is given by fz)=ψz) z ξ ) Ψz) z ξ ), ϕ 0Ψ Ψz)ϕ 0 z). While this looks innocent enough, it evaluates to where fz) = z β)γz + δ) z ξ 0 )z ξ )z α) ξ 0 )β α)αβ ) γ [ + αξ 0 ) + β ) βα + ξ 0 )]αξ ) ξ 0, δ + and hence, f = ξ 0 )β α)αβ )ξ [ + αξ 0 ) + β ) βα + ξ 0 )]αξ ) 7) ξ 0 β)ξ 0 γ + δ) βξ 0 )γ + δξ 0 ) ξ 0 ξ )ξ 0 α) ξ0 ) ξ 0ξ ) αξ 0 ) + ξ β)ξ γ + δ) βξ )γ + δξ ) ξ ξ 0 )ξ α) ξ ) ξ 0ξ ) αξ ) + α β)αγ + δ) βα)γ + δα) α ξ 0 )α ξ ) ξ 0 α) ξ α) α ). This is surprisingly complicated! Furthermore, it only applies for the case when all of ξ 0, ξ and α are real valued and distinct. evertheless, with the definitions K 0 α ) αξ 0 ) ξ 0 ) + αξ 0 ) + β ) βα + ξ 0 ), K f 8) it permits the computation in this situation for the second order case of VarGe jω, θ n ) σ [ κ K 0 e jω ξ 0 + K γe jω + δ ] e jω ξ 0 e jω ξ 9) The performance of this quantification, for the same experiment conditions as for the first order case and with ξ = 0.8, is shown in figure, and again it is seen to be essentially exact Estimate variability vs exact quantification and asymptotic approximation Sample Variability Exact Quantification Asymptotic Approximation Fig.. Variability of Fixed Denominator Estimate: True variability vs. theoretically derived approximations. Solid line is Monte Carlo estimate of true variability, dashed line is approximation 9), dash-dot line is asymptotic-based approximation ) 7. COMPARISO WITH OTHER APPROACHES The recent paper Xie and Ljung 000) has also tackled this problem of providing variance error quantification that is exact for finite model order. In that work, a completely different analysis approach is taken. Furthermore, some specific assumptions are made there that are different to those of this paper. Specifically, in Xie and Ljung 000) it is required that ) The spectral factor Ψz) is strictly all-pole ie. u t is a strictly auto-regressive time series); ) The model order chosen must be greater than or equal to the order n of the underlying dynamics plus the order ν of the spectral factor Ψz); 3) In the model structure ), all fixed poles ξ k for n < k n + ν are set to zero. With these conditions in mind, the quantification proposed in Xie and Ljung 000) is lim ) = σ n ) ξ k ν Φ u ω) e jω ξ k + α k e jω α k. 30) Clearly, it is important to reconcile this expression with that derived here. For this purpose, consider the first order example of 5 in which case, since Ψz) is also first order in that example, to satisfy the above requirements of Xie and Ljung 000) a second order model of the form Gq, θ) = θ 0 + θ q qq ξ 0 )

6 needs to be employed. In this case, substituting ξ = 0, β = 0 into the analysis of 6 leads to a quantification of ) σ κ K0 + K γe jω δ e jω ξ 0 3) with γ, δ, K 0 and K given by 7), 8). On the other hand, according to 30), the analysis of Xie and Ljung 000) gives a quantification of: ) σ ξ 0 ) e jω α + α ) e jω ξ 0 κ e jω ξ 0 3) The performance of both these quantifications for the case of ξ 0 = 0.9, α = 0.5 is shown in figure 3. Clearly, the quantification 3) seems to be essentially exact, and is also identical to that of 3) derived in Xie and Ljung 000), so that for the scenario where 3) applies, the results of this paper are consistent with it Estimate variability vs exact quantification and asymptotic approximation Sample Variability Exact Quantification Xie Ljung Quantification Fig. 3. Variability of Fixed Denominator Estimate - Second Order Case. True variability vs. theoretically derived approximations. Solid line is Monte Carlo estimate of true variability, dashed line is approximation 3), dash-dot line is Xie Ljung quantification 3) 8. COCLUSIOS The good news presented here is that in cases of rational input spectral density, it is possible to obtain variance quantifications that apply for arbitrarily low model order. The bad news is that the analytical manipulations required to compute the necessary orthonormal basis lead to very complicated expressions; this happens because some zeroes as well as all the poles are constrained in the required orthonormal spanning elements. This implies that some sort of numerical procedure may need to be developed to compute these quantities. This work is related to work by other authors, but there are important differences. In particular, the results here are derived via new reproducing kernel techniques that avoid the need for the detailed residue-based computations employed in Xie and Ljung 000). This approach allows for the inclusion of more general classes of input spectra and also avoids the requirement of model order being larger than the underlying true dynamics. However, because of this generality the quantifications here are more complicated than those of Xie and Ljung 000). 9. REFERECES Golub, G. and Charles Van Loan 989). Matrix Computations. Johns Hopkins University Press. Grenander, U. and G. Szegö 958). Toeplitz Forms and their Applications. University of California Press, Berkeley. Hannan, E. and D.F. icholls 977). The estimation of the prediction error variance. Journal of the American Statistical Association 7360, part ), Heuberger, P., P.M.J. Van den Hof and O.H. Bosgra 995). A generalized orthonormal basis for linear dynamical systems. IEEE Transactions on Automatic Control AC-403), Ljung, L. 999). System Identification: Theory for the User, nd edition). Prentice-Hall, Inc.. ew Jersey. L.Ljung 985). Asymptotic variance expressions for identified black-box transfer function models. IEEE Transactions on Automatic Control AC-309), L.Ljung and Z.D.Yuan 985). Asymptotic properties of black-box identification of transfer functions. IEEE Transactions on Automatic Control 306), inness, B. and Hakan Hjalmarsson 00a). The quantification of variance error part I: Fundamental principles. Submitted to IEEE Transactions on Automatic Control. inness, B. and Hakan Hjalmarsson 00b). The quantification of variance error part II: General models and applications. Submitted to IEEE Transactions on Automatic Control. inness, B., Håkan Hjalmarsson and Fredrik Gustafsson 999). The fundamental role of general orthonormal bases in system identification. IEEE Transactions on Automatic Control 447), P.M.J. Van den Hof, P.S.C. Heuberger and J. Bokor 995). System identification with generalized orthonormal basis functions. Automatica 3), Riesz, F. and B. Sz.-agy 955). Functional Analysis. Fredrick Ungar. ew York. Rudin, W. 966). Real and Complex Analysis. McGraw Hill. ew York. Wahlberg, B. 99). System identification using Laguerre models. IEEE Transactions on Automatic Control AC-365), Wahlberg, B. 994). System identification using Kautz models. IEEE Transactions on Automatic Control AC-396), Whittle, P. 953). Estimation and information in stationary time series. Arkiv För Matematik 3), Xie, L.-L. and Lennart Ljung 000). Asymptotic variance expressions for estimated frequency functions. Technical Report LiTH-ISY-R-77. Department of Electrical Engineering, Linköping University,To appear, IEEE Trans. Automatic Control, 00.