NONLINEAR STRUCTURE IDENTIFICATION WITH LINEAR LEAST SQUARES AND ANOVA. Ingela Lind

Transcription

1 OLIEAR STRUCTURE IDETIFICATIO WITH LIEAR LEAST SQUARES AD AOVA Ingela Lind Division of Automatic Control, Department of Electrical Engineering, Linköping University, SE Linköping, Sweden Abstract: The objective of this paper is to find the structure of a nonlinear system from measurement data, as a prior step to model estimation Applying AOVA directly on a dataset is compared to applying AOVA on residuals from a linear model The distributions of the involved test variables are computed used to show that AOVA is effective in finding which regressors give linear effects what regressors produce nonlinear effects The ability to find nonlinear substructures depending on only subsets of regressors is an AOVA feature which is shown not to be affected by subtracting a linear modelcopyright c 2005 IFAC Keywords: System identification, onlinear systems, Structural properties, Analysis of variance, Linear estimation ITRODUCTIO The aim of this paper is to quantify the difference between two different ways to use Analysis of Variance (AOVA) as a tool for nonlinear system identification In general, system identification is data centred Simple things are tried first; Is a linear model sufficient to describe the data? To invalidate a linear model, the residuals are examined with whiteness tests the fit of the model on validation data is used to form an opinion of how good the model is Thus, a linear model is often available, or easily computed AOVA (Miller, 997; Montgomery, 99) can be used for finding proper regressors model structure for a nonlinear model by fitting a locally constant model to the response surface of the data (Lind, 2000; Lind Ljung, 2005, 2003) A clever parameterisation of a locally constant model makes it possible to perform hypothesis tests in a balanced computationally very effective way Let y(t) = g(u(t), u(t T ),, u(t kt )) + e(t) = θ T ϕ (t) + g 2 (ϕ 2 (t)) + e(t) be a general nonlinear finite impulse response model with input u(t) output y(t), sampled with sampling time T Let ϕ (t) be a vector containing the regressors that affect the output linearly (with parameters θ ) ϕ 2 (t) the regressors that affect y(t) nonlinearly through the function g 2 ( ) Three main questions can be answered by both running AOVA directly on identification data running AOVA on the residuals from a linear model: Should the regressor u(t k i T ) be included in the model at all, should it be included in ϕ (t) or ϕ 2 (t)? What interaction pattern is present? Can g( ) be divided into additive parts containing only subsets of the regressors? What subsets? Are there nonlinear effects in the residuals from a linear model?

2 There are much to be gained by the division into a linear a nonlinear subset of the regressors (Ljung et al, 2004) instead of assuming a full nonlinear model The complexity of any black-box type of model depends heavily on the size of ϕ 2 (t) An idealised case is examined to quantify the difference between running AOVA directly on identification data first estimate an affine (linear with constant offset) model then running AOVA on its residuals The input signal is chosen to keep computations simple, while being sufficiently exciting to make a nonlinear black-box identification possible The structure of the paper is as follows: First, the true data model is stated In section 3, the linear model is estimated the residuals are formed In section 4, AOVA is run directly on the estimation data, in section 5 AOVA is run on the residuals from the linear model Section 6 explores the differences between the two approaches give examples The conclusions are made in section 7 2 TRUE DATA MODEL The system is a finite impulse response model: y(t) = g(u(t), u(t T )) + e(t), () where e(t) (0, σ 2 ) is independent identically distributed Gaussian noise with mean zero variance σ 2 The input signal u(t) is a pseudorom multi-level signal with mean zero, in which each level combination of u(t) u(t T ) occur equally many times The last condition defines a balanced dataset will give independence between sums of squares This type of signal can be given a nearly white spectra, see Godfrey (993) The number of levels the signal assumes is m the levels are denoted u i, where i =,, m An integer number, n, of periods from the input/output data are collected denoted by Z ( = np, where p is the length of the period) g( ) is a function of two variables The results extend to more regressors, but since the equations do not fit in this short format, only two regressors are considered below 3 ESTIMATIO OF THE AFFIE MODEL A linear FIR model with an extra parameter for the mean level of the signal, ŷ(t) = âu(t) + ˆbu(t T ) + ĉ [ = â ˆb ĉ] u(t) u(t T ) = ˆθ T ϕ(t), is estimated using linear least squares (Ljung, 999, page 203) The loss function V (θ, Z ) = t= is minimised by the estimate ˆθ LS = 2 (y(t) ŷ(t))2, [ â ˆb ĉ] T = arg min V (θ, Z ) = [ ϕ(t)ϕ T (t)] t= ϕ(t)y(t) t= For the pseudo-rom multi-level input signal we have that: u(t) = 0, u(t T ) = 0, (2) t= t= u(t)u(t T ) = 0, (3) t= u 2 (t) = m t= u 2 i = R u i= If assumption (3) is not valid, change variables to ũ(t T ) = u(t T ) αu(t), such that / t= u(t)ũ(t T ) = 0 ow, [ ] ϕ(t)ϕ T (t) = R t= u 0 0 R u ϕ(t)y(t) t= = u(t) u(t T ) (g(u(t), u(t T )) + e(t)) t= = m u i g(u i, u j ) u m 2 j g(u i, u j ) i= j= g(u i, u j ) + u i e ijk nm 2, i= j= k= u j e ijk e ijk where e ijk is the value of e(t) when u(t) = u i u(t T ) = u j for the k:th time, that is, the k:th measurement in cell ij With vector notation; f = h = u = f f m h h m u u m = m = m, =, g(u, u j ) j= g(u u j ) g(u i, u ) i= g(u i, u m ) = G = GT

3 v = w = v v m w w m = mn = mn j= k= i= k= e jk e mjk e ik e imk = E = ET where G ij = g(u i, u j ) E ij = /n n k= e ijk is the noise average in cell ij, the parameter estimates can be written as: ˆθ LS = u T (f + v) u T (h + w) m 2 T (G + E) (4) The residuals from the affine model are denoted ɛ(t) = g(u(t), u(t T )) + e(t) (ˆθ LS ) T ϕ(t) 4 AOVA APPLIED TO THE DATA DIRECTLY The statistical analysis method AOVA (Miller (997); Montgomery (99), among many others) is a widely spread tool for finding out which factors contribute to given measurements It has been used discussed since the 930 s is a common tool in, eg, medicine quality control applications The method is based on hypothesis tests with F-distributed test variables computed from the residual quadratic sum There are several slightly different variants (Miller, 997) Here the fixed effects model with two factors will be used Assume that the collected measurement data can be described by a linear statistical model, y ijk = µ + τ i + β j + (τβ) ij + w ijk, (5) where the w ijk are independent Gaussian distributed variables with zero mean constant variance σ 2 The parameter µ is the overall mean For each level i =,, m of the first regressor (u(t)) there is a corresponding effect τ i, for each level j =,, m of the second regressor(u(t T )) the corresponding effect is β j The interaction between the regressors is described by the parameters (τβ) ij The sum of a batch of indexed parameters over any of its index is zero For a linear (y(t) = au(t) + bu(t T ) + e(t)) or a non-linear additive system (y(t) = g (u(t)) + g 2 (u(t T )) + e(t)), the interaction parameters (τβ) ij are zero These are needed when the nonlinearities have a non-additive nature, ie, y(t) = g(u(t), u(t T )) + e(t) Since the regressors are quantised, it is a very simple procedure to estimate the model parameters by the computation of means For example, the constant µ would correspond to y, while the effects from the first regressor are computed as τ i = y i y In each cell ij, for the model (), y ijk = g(u i, u j ) + e ijk This gives the total mean over all cells (all data) ȳ = m 2 T (G + E) The m different row means ȳ i, column means ȳ j, are given by ȳ i = f i + v i = m it (G + E) ȳ j = h j + w j = m T (G + E)j respectively Here i j are vectors with one nonzero element in row i j respectively i = j = The m 2 different cell means are given by ȳ ij = i T (G + E)j, AOVA is used for testing which of the parameters that significantly differ from zero for estimating the values of the parameters with stard errors, which makes it a tool for exploratory data analysis The residual quadratic sum, SS T, is used to design test variables for the different batches (eg, the τ i :s) of parameters The total residual sum of squares is divided into the four parts SSA d (ȳ i ȳ ) 2 i= τ 2 i i= (i T (f + v) m T (f + v)) 2 i= = (f + v) T A(f + v), SSB d βj 2 (ȳ j ȳ ) 2 j= j= (j T (h + w) m T (h + w)) 2 j= = (h + w) T A(h + w), SSAB d = n (τβ) 2 ij = n = n i= j= i= j= i= j= (ȳ ij ȳ i ȳ j + ȳ ) 2 ((i m )T Y(j m ))2 = m trace(yt AY) m 2 T Y T AY, SSE d = wijk 2 = (y ijk ȳ ij ) 2 = i= j= k= i= j= k= i= j= k= (e ijk ē ij ) 2,

4 with A (I m T ), Y = G + E where d sts for sums of squares computed directly from the dataset Z The index for each part is related to one batch of parameters in (5) If all the parameters in the batch are zero, the corresponding quadratic sum is χ 2 -distributed if divided by the true variance σ 2 (see, eg, Montgomery (99, page 59)) Since the true variance is not available, the estimate ˆσ 2 = SSE m 2 (n ) is used to form F - distributed test variables, eg, for τ i ; va d = SSd A /(m ) SSE d /(m2 (n )), H0 A,d : τ i = 0 i If all the τ i :s are zero, v A belongs to an F - distribution with m m 2 (n ) degrees of freedom If any τ i is nonzero it will give a large value of va d, compared to an F -table This is, of course, a test of the null hypothesis that all the τ i :s are zero, which correspond to the case that the regressor u(t) does not have any main effect on the measurements y(t) 5 AOVA APPLIED TO THE RESIDUALS FROM THE AFFIE MODEL In each cell ij we have the residuals ɛ ijk = g(u i, u j ) + e ijk âu i ˆbu j ĉ, where the parameters â, ˆb ĉ are computed according to (4) The total mean is now given by ɛ = nm 2 ɛ ijk i= j= k= = m 2 T (G + E) â m ut ˆb m ut ĉ = 0, where the last equality is due to (2) (4) The row means changes to ɛ i = ɛ ijk = i T (f + v âu) ĉ, nm j= k= since the sum over u j is zero The column means are given by ɛ j = ɛ ijk = j T (h + w nm ˆbu) ĉ, i= k=, finally, the cell means are given by ɛ ij = ɛ ijk = i T (G + E)j âi T u n ˆbu T j ĉ k= The sums of squares SS A SS B are changed to SSA r ( ɛ i ɛ ) 2 i= i= ((i T m T it u u T )(f + v)) 2 = (f + v) T A (f + v), with A (I m T uu T ), SSB r ( ɛ j ɛ ) 2 j= ((j T m T jt u u T )(h + w)) 2 j= = (h + w) T A (h + w) It is easy to verify that SSAB r = n ( ɛ ij ɛ i ɛ j + ɛ ) 2 = SSAB d that SS r E = i= j= = i= j= k= i= j= k= (ɛ ijk ɛ ij ) 2 (e ijk ē ij ) 2 = SSE, d (6) where r sts for sums of squares computed from the residuals from the affine model 6 DIFFERECES AD DISTRIBUTIOS The sum of squares corresponding to the regressor u(t), SS A, changes with the following amount when an affine model is extracted from the data: SS d A SS r A = (f + v) T A 2 (f + v), with A 2 = A A = n R u uu T The change in sums of squares corresponding to the regressor u(t T ) is SS d B SS r B = (h + w) T A 2 (h + w) 6 Distributions From, eg, Miller (997), the following is known: SS d AB SS d A σ 2 χ 2 (m, f T Af σ 2 ), SS d B σ 2 χ 2 (m, ht Ah σ 2 ), σ 2 χ 2 ((m ) 2, mtrace(gt AG) T G T AG m 2 σ 2 ), SS 2 E σ 2 χ 2 (m 2 (n )), where χ 2 (d, ω) means distributed as a noncentral Chi-square distribution with d degrees of freedom non-centrality parameter ω The sums of squares SSA d, SSd B, SSd AB SSd E are independently distributed if the dataset is balanced, ie, if all combinations of u(t) = u i, u(t T ) = u j are present equally many times in the input

5 To find the distributions of SSA r, SSr B, SSd A SSA r SSd B SSr B the following theorems, numbered as in Khatri (980), can be applied Let v (µ, σvv) 2 q = v T Av + 2l T v + c The notation in Theorem 2 is changed from matrix valued to vector valued v Theorem 2 q λσvχ 2 2 Ω (d, λ 2 σ ) iff (i) λ is the v 2 nonzero eigenvalue of VA (or AV) repeated d times, (ii) (l T +µ T A)V = k T VAV for some vector k (iii) Ω = (l T + µ T A)V(l T + µ T A) T µ T Aµ+2l T µ+c = (l T + µ T A)V(l T + µ T A) T /λ q λσvχ 2 2 (d) iff (i) VAVAV = λvav (ii) (l T + µ T A)V = 0 = µ T Aµ + 2l T µ + c Theorem 4 Let q i = v T A i v + 2l T i v + c i, i =, 2, where A A 2 are symmetric matrices Then q q 2 are independently distributed iff (i) VA VA 2 V = 0, (ii) VA 2 V(A µ + l ) = VA V(A 2 µ + l 2 ) = 0 (iii) (l + A µ) T V(l 2 + A 2 µ) = 0 ow set q = SS r A q 2 = SS d A SS r A Let l = A f, c = f T A f, l 2 = A 2 f, v (0, σ2 nm I) Then independence is shown by; (i) VA VA 2 V = A A 2 c 2 = f T A 2 f = n2 m R u (I m T uu T )uu T = 0, since u T u = T u = 0 (ii) VA 2 V(A µ + l ) = A 2 A f = n2 m uu T (I R u m T uu T ) = 0, VA V(A 2 µ + l 2 ) = A A 2 f = 0, (iii) (l + A µ) T V(l 2 + A 2 µ) = f T A A 2 f = 0 for the same reasons as in (i) Since conditions (i),(ii) (iii) in Theorem 4 are fulfilled, q q 2 (that is, SSA r SSd A SSA r ) are independently distributed The same argument is valid for the independence of SSB r SSB d SSr B if all occurrences of f are replaced with h v with w If assumption (3) is not valid, the independence is lost To compute the distributions of q q 2 the conditions in Theorem 2 are checked: (i) λ = eig(va ) with d = m 2 λ 2 = eig(va 2 ) with d 2 = (ii) (l + µa )V = f T A = k T VA V for k = f (l 2 + µa 2 )V = f T A 2 = k T VA 2 V for k = f (iii) Ω = (l + µa )V(l + µa ) T = f T A A f = λ f T A f µ T A µ + 2l T µ + c = c = f T A f = Ω /λ Ω 2 = (l 2 + µa 2 )V(l 2 + µa 2 ) T = f T A 2 A 2 f = λ 2 f T A 2 f µ T A 2 µ + 2l T 2 µ + c 2 = c 2 = f T A 2 f = Ω 2 /λ 2 By Theorem 2, SS r A = q σ 2 χ 2 (m 2, f T A f σ 2 ), SS d A SS r A = q 2 σ 2 χ 2 (, f T A 2 f σ 2 ) As before, all SS A can be replaced by SS B if f is replaced by h 62 Interpretation There are five test variables of interest: v AB = SS AB/(m ) 2 SS E /m 2 (n ), H0 AB : (τβ) ij = 0 i, va d = SSd A /(m ) SS E /m 2 (n ), H0 A,d : τ i = 0 i, vb d = SSd B /(m ) SS E /m 2 (n ), H0 B,d : β j = 0 j, va r = SSr A /(m 2) SS E /m 2 (n ), H0 A,r : τ i = 0 i, vb r = SSr B /(m 2) SS E /m 2 (n ), H0 B,r : β j = 0 j All of these belong to F-distributions if the corresponding null hypotheses are true, that is, large values of the test variables (compared to an F (d, d 2 )-table) are interpreted as that there are effects from the corresponding regressor If v AB is large an interaction effect between u(t) u(t T ) is assumed This means that the system can not be decomposed into additive subsystems If v AB is small, the null hypothesis can not be rejected, so it is assumed that the system can be decomposed into additive subsystems For both va d vr A large, the interpretation is that the effect from u(t) is nonlinear If va d is large, but va r is small, the effect from u(t) can be described by the linear model If both va d vr A are small, u(t) can not be shown to affect the output of the system The same reasoning built on vb d vr B is valid for the effects from u(t T ) Since SS AB is not changed when the linear model is extracted from the data, see (6), we can make the conclusion that all information about the interactions in the system is left in the residuals The interaction information is not destroyed by subtracting a linear model in a balanced dataset

6 63 Linear example Let the true system be given by y(t) = au(t) + bu(t T ) + e(t) Then f = au h = bu This gives SS AB σ 2 χ 2 ((m ) 2, 0), SS d A σ 2 χ 2 (m, nm 2 a 2 R u /σ 2 ), (7) SS d B σ 2 χ 2 (m, nm 2 b 2 R u /σ 2 ), (8) SS r A, SS r B σ 2 χ 2 (m 2, 0) (9) The size of the non-centrality parameters in (7) (8) depends on how many data are collected, the size of the true linear effects the variance of the input the noise This is what effects the power of the F-tests In the sums of squares computed from the residuals from the linear model all dependence on the true model is removed (9) Thus the conclusion can be made; that if SSA r or SSB r are found to be large by the F-tests, then the data are probably not collected from a true linear system 64 Quadratic example Let y(t) = u 2 (t) + e(t) Then f = [u 2,, u 2 m] T, h = 0, with SS AB σ 2 χ 2 ((m ) 2, 0), SS d A σ 2 χ 2 (m, nc d ), SS d B σ 2 χ 2 (m, 0), SS r A σ 2 χ 2 (m 2, nc r ), SS r B σ 2 χ 2 (m 2, 0) nc d = n m σ 2 (m u 4 i Ru), 2 i= nc r = n σ 2 (m m i= u 4 i R 2 u R u ( u 3 i ) 2 ) i= Here, it is clear that it matters how the levels of the input signal are chosen m i= u3 i can vary considerably while (2) is valid, since there are no constraints on the level distribution Also â is proportional to m i= u3 i, so a large difference between AOVA directly AOVA applied to the residuals means that the nonlinear effect have been picked up by the affine model, due to irregular sampling of the function If u(t) is symmetric around its mean, it is clear that m i= u3 i = 0, so nc d = nc r 7 COCLUSIOS a linear model estimated with linear least squares The distributions for the sums of squares needed for the AOVA analysis in the latter case were computed These distributions were used to show that by combining the two approaches AOVA is effective in finding what regressors give linear effects what regressors give nonlinear effects In section 62 it was shown how to divide the regressors into a linear a nonlinear subset, depending on the outcome of the AOVA tests The ability to structure the proposed nonlinear function into additive parts depending on only subsets of regressors is an AOVA feature which is not affected by subtraction of a linear model The results in the paper extend to more regressors, but an important limitation is that the dataset should be balanced, see section 2 This work has been supported by the Swedish Research Council(VR) which is gratefully acknowledged REFERECES K Godfrey Perturbation Signals for System Identification Prentice Hall, ew York, 993 C G Khatri Quadratic forms in normal variables In PR Krishnaiah, editor, Hbook of Statistics, volume, pages , Amsterdam, 980 orth-holl I Lind Model order selection of -FIR models by the analysis of variance method In Proc IFAC Symposium SYSID 2000, pages , Santa Barbara, Jun 2000 I Lind L Ljung Structure selection with AOVA: Local linear models In P van der Hof, B Wahlberg, S Weil, editors, Proc 3th IFAC Symposium on System Identification, pages 5 56, Rotterdam, the etherls, aug 2003 I Lind L Ljung Regressor selection with the analysis of variance method Automatica, 4(4): , Apr 2005 L Ljung System Identification, Theory for the User Prentice Hall, ew Jersey, 2nd edition, 999 L Ljung, Q Zhang, P Lindskog, A Juditsky Modeling a non-linear electric circuit with black box grey box models In Proc OLCOS IFAC Symposium on onlinear Control Systems, pages , Stuttgart, Germany, Sep 2004 RG Miller, Jr Beyond AOVA Chapman Hall, London, 997 DC Montgomery Design Analysis of Experiments John Wiley & Sons, ew York, 3rd edition, 99 Applying AOVA directly on a dataset was compared to applying AOVA on the residuals from