Application-Oriented Estimator Selection

Transcription

1 IEEE SIGNAL PROCESSING LETTERS, VOL. 22, NO. 4, APRIL Application-Oriented Estimator Selection Dimitrios Katselis, Member, IEEE, and Cristian R. Rojas, Member, IEEE Abstract Designing the optimal experiment for the recovery of an unknown system with respect to the end performance metric of interest is a recently established practice in the system identification literature. This practice leads to superior end performance to designing the experiment with respect to some generic metric quantifying the distance of the estimated model from the true one. This is usually done by choosing and fixing the estimation method to either a standard maximum likelihood (ML) or a Bayesian estimator. In this paper, we pose the intuitive question: Can we design better estimators than the usual ones with respect to an end performance metric of interest? Based on a simple linear regression exampleweaffirmatively answer this question. Index Terms End performance metric, estimation, experiment design, training. I. INTRODUCTION ABASIC subproblem in the context of system identification is that of experiment design. Overviews of this topic over the last decade can be found in [4], [5], [6], [15]. Contributions include convexification [9], robust design [12], [16], least-costly design [3], and closed vs open loop experiments [1]. In the context of application-oriented experiment design, the training is designed to optimize a performance metric associated with the particular application where the estimated model will be used [10], [11]. A conceptual framework for application-oriented experiment design was outlined in [6]. The least-costly experiment in this framework is given as follows: Experimental effort (1) where is the identified model and is the set of admissible models. Here, is the end-performance metric of interest dependent on the true model and is a predefined accuracy parameter. For the experimental effort, different measures commonly used are input or output power, and experimental length. For, standard maximum likelihood (ML) and Bayesian estimation methods are usually employed, e.g., minimum mean square error (MMSE) estimation. Fixing the system estimators to usual methods possessing some optimality attributes, application-oriented training designs have Manuscript received August 09, 2014; revised October 07, 2014; accepted October 09, Date of publication October 16, 2014; date of current version October 22, The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Phillip Ainsleigh. D. Katselis is with the Industrial and Enterprise Systems Engineering Department, University of Illinois at Urbana-Champaign, Urbana, IL USA ( katselis@illinois.edu). C. R. Rojas is with ACCESS Linnaeus Center, KTH Royal Institute of Technology, Stockholm S , Sweden ( cristian.rojas@ee.kth.se). Color versions of one or more of the figures in this paper are available online at Digital Object Identifier /LSP been shown to outperform classical training designs based on usual quadratic loss functions [10], [11]. By the previous line of thinking a dual question is: Can we design better estimators than the usual ones with respect to an end performance metric of interest when considering finite length training sequences? Note that under the small sample size assumption, the asymptotic efficiency of the ML estimator together with its invariance property are irrelevant. Optimizing the training and optimally choosing the system estimator are two problems that should ultimately be tackled in a joint context. Nevertheless, both in the framework of classical and application-oriented training designs, a separation strategy is applied: initially, we select and fix the system estimator to a choice that is known to possess some optimality aspects, e.g., the ML or MMSE estimators, and then we optimize the training. This implies that in the case of the dual question one should choose and fix the training to something that possesses some optimality attributes for any system estimator or to something that is independent of the system estimator so that the isolated optimal selection of the system estimator is meaningful. In this paper, we affirmatively answer the aforementioned dual question based on a simple linear Gaussian regression example that is used here as the simplest possible paradigm. Finally, we numerically demonstrate the validity of the claims verifying the purchased analysis. It is worth mentioning that the approach taken in this paper is closely related to the so-called Stein phenomenon [8], [17], according to which there are biased estimators that outperform the standard least-squares (LS) and ML estimators for specific performance metrics. These biased estimators have been studied extensively in statistics, and have shown great potential in signal processing applications [2], [14]. Notation: Vectors are denoted by bold letters. Superscripts, and stand for transposition, Hermitian transposition and conjugation, respectively. is the complex modulus. For a vector, denotes its -th entry. The expectation operator is denoted by. Finally, denotes the complex Gaussian distribution with mean and variance. II. PROBLEM STATEMENT Consider the scalar linear Gaussian model where is the observed signal at time instant, is the unknown system parameter assumed to be complex-valued, istheinputatthesametimeinstantand is complex, circularly symmetric, Gaussian noise with zero mean and variance. We further assume that and. (2) IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See for more information.

2 490 IEEE SIGNAL PROCESSING LETTERS, VOL. 22, NO. 4, APRIL 2015 In addition, and are independent random sequences, while is a white random sequence. Assume that the training length is limited to time slots and that the maximum allowed training energy for estimation purposes is. We can collect the received samples corresponding to the training in a vector: (3) where is the vector of received samples corresponding to the training, is the vector of training symbols and is the vector of noise samples. Considering the class of linear parameter estimators, the system is estimated as follows: (4) where is a estimating filter. To create a paradigm suitable for answering the aforementioned dual question, we consider as performance metric the mean square error (MSE) of a linear input estimator that uses the system knowledge and delivers an estimate of the input variable: Two standard input estimators are the minimum MSE (MMSE) estimator, which minimizes when the input is treated as a random variable, and the zero forcing (ZF) estimator, which coincides with the LS input estimator when the input is deterministically treated. If the signal-to-noise ratio (SNR) is high, i.e., is small, both estimators lead to very similar results. In the sequel, we will focus on the ZF estimator, as it leads to simpler derivations and it does not depend on the knowledge of the noise variance ; however, due to the connection between and,itisexpected that the results in the paper can be extended to the MMSE estimator using perturbation analysis (starting from the design for the ZF estimator). Given the input estimator, we define the considered end performance metric based on the input estimator that only knows a system estimate as follows: Our goal will be to determine the optimal parameter estimators for fixed experiments of finite length so that is minimized. To this end, the following section presents some useful ideas. III. RUDIMENTS Consider the ML estimator. For the linear Gaussian regression this estimator coincides with the minimum variance unbiased (MVU) estimator. We therefore replace our references to the ML estimator by references to the MVU estimator from now on. Since the MVU is an unbiased estimator, it satisfies.by[13],. If we assume that is a random variable and that its prior distribution is known, then instead of the MVU one could use the MMSE parameter estimator. With our assumptions and the extra assumption that (5) (6), one obtains. Assuming that is a deterministic but unknown variable, it turns out that (c.f. (6)). Here, the superscript d stands for deterministic. If is assumed to be a random variable, then the corresponding end-performance metric is obtained by averaging the last expression over. Depending on the probability distributions of and and mayfailtoexistduetoinfinite moment problems. In order to obtain well-behaved estimators that will be used in conjunction with the actual performance metrics, some sort of regularization is needed. Some ideas for appropriate regularization techniques may be obtained by modifying robust estimators (against heavy-tailed distributions), e.g., by trimming a standard estimator, if it gives a value very close to zero [7]. An example of such a trimmed estimator is given as follows: where can be any estimator and a regularization parameter 1. Clearly, the reader may observe that the definition of the trimmed preserves the continuity at. Additionally, the event has zero probability since the distribution of is continuous. Therefore, in this case can be arbitrarily defined, e.g.,. We focus now on the.assumeafixed. In the appendix, we show that for a sufficiently small and a sufficiently high SNR during training, minimizing is equivalent to minimizing the following approximation Following similar steps and using some minor additional technicalities, we can work with (7) (8) (9) (10) instead of. We will call the last approximations zeroth order metrics. The following analysis and results will be based on the zeroth order metrics and they will reveal the dependency of the parameter estimator s selection on the considered (any) end performance metric. Remark: In (9), one can observe that after approximating the mean value of the ratio by the ratio of the mean values the infinite moment problem is eliminated. In the following, all zeroth order metrics will be defined based on the non-trimmed to ease the derivations. This treatment is approximately valid when is sufficiently small as it is actually shown in eq. (19) of the appendix. 1 This parameter can be tuned via cross-validation or any other technique, although in the simulation section we empirically select it for simplicity purposes.

3 KATSELIS AND ROJAS: APPLICATION-ORIENTED ESTIMATOR SELECTION 491 IV. MINIMIZING THE ZEROTH ORDER METRICS In this section, we assume that the training sequence is fixed and we examine the estimator selection problem for and. A. Deterministic The expectation operators in Eq. (9) are with respect to, and. We have: (11) Taking the gradient in (11) with respect to, discarding the outer and setting, we obtain an expression with the following numerator: (12) Note that no choice of will zero this expression for any. Therefore, the MVU is not an optimal system estimator in this case. We can state this result more formally: Proposition 1: The MVU estimator is not an optimal system estimator for the task of minimizing,when is considered deterministic but otherwise an unknown quantity. The question that arises in this case is how to find the optimal system estimator in this setup or generally how to determine a uniformly better estimator for minimizing. Equating the numerator of the gradient of w.r.t. to and taking the inner product of both sides with, we obtain the following optimal estimating filter (13) The problem with (13) is that the optimal solution depends on the unknown system. To deal with this dependence we may assume a noninformative prior distribution for. If the real and imaginary parts of are considered bounded in the intervals and, 2 then one can treat them as independent random variables uniformly distributed on and, respectively. is now replaced by,where denotes the expectation with respect to the joint (uniform) distribution of the real and imaginary parts of. Furthermore, this approach leads to the substitution of by in (13). B. Random In this case, the actual prior statistics of are known. Easily, the optimal,say, is given by (13) but with replaced by, while it can be easily shown that or do not zero the corresponding gradient. This implies the following result: Proposition 2: The MVU and MMSE estimators are not optimal system estimators for the task of minimizing,whenthe prior distribution is known. Remarks: 1) The claimed optimality of the derived estimators in this section is with respect to the zeroth order performance metrics. These estimators also turn out to be uniformly better 2 This assumption is usually reasonable in practice. Fig. 1. with SNR during training equal to 0 db, and. Moreover,. The error bars correspond to one standard deviation confidence bounds around the mean values. than the MVU and MMSE estimators with respect to the true end performance metric as we demonstrate in the simulation section. 2) An alternative way to express eq. (13) is,where is the inverse SNR. Depending on how we implement the last estimator in practice, turns to a tuning parameter controlling the introduction of bias in the MVU estimator. C. Discussion on the Optimal Training To optimize the training vector, one should first fixtheparameter estimator. This is a complementary problem with respect to the approach that we have followed so far. Suppose that we use either the MVU or the MMSE estimators. One can observe that by fixing for example the problem of selecting optimally the training vector is meaningless. To see this consider the case of. We then have: which only depends on. Furthermore, setting, it follows that at sufficiently high SNR, i.e., is minimized when,whichis intuitively appealing. Therefore, any with energy equal to is an equally good training vector for the MVU estimator. Thus, for the same,theestimator will be better than the MVU. Similar conclusions can be reached for the MMSE estimator, as well. V. SIMULATIONS In this section we present numerical results to verify our analysis. In the figure,. The SNR during training highlights how good the system estimate is. Due to space limitations, we only examine the performance of the derived estimators with respect to the true performance metric. The system estimator is implemented based on (8) to combat the infinite moment problems. In Fig. 1, is presented for and SNR during training equal to 0 db. The derived

4 492 IEEE SIGNAL PROCESSING LETTERS, VOL. 22, NO. 4, APRIL 2015 optimal 3 estimators in this paper are better than the MVU and MMSE estimators. We can see that the zeroth order approximation in this case is satisfactory even for low SNR during training, in the sense that the corresponding optimal estimators outperform the MVU and MMSE estimators for the true performance metric. Additionally, the MVU estimator appears to be better than the MMSE estimator for this performance metric. This is a snapshot contradicting what one would naturally expect and verifying the motivation of this paper. Fig.1alsoshowsconfidence bounds of one standard deviation for the MSE plots. These bounds confirm that the gains of the optimal estimators do not hold only in mean sense, but also with high probability, in particular for low SNR. VI. CONCLUSIONS In this paper, application-oriented estimator selection has been compared with common estimators such as the MVU and MMSE estimators. We have shown that the application-oriented selection is the right way to choose estimators in practice. We have verified this observation based on a simple linear Gaussian regression example. Furthermore, if the SNR during training is sufficiently high and the probability mass of is concentrated around,then it can be shown that (15) The same holds even if is a biased estimator of at high training SNR and tends to concentrate around a value bounded away from (and of course from 0). To show the last claim, we set and.since, it also holds that.furthermore, it can be seen that (16) At high training SNR, and in the mean square sense and therefore it can be easily shown that the right hand side of (16) converges to 0. To see this, notice that the Cauchy-Schwarz inequality yields APPENDIX I This section proposes a simplification of the metric for the estimator given in (8) with a fixed. Due to the Gaussianity of, for any (infinite moment problem). Using (8), the corresponding metric becomes: (17) Since and in the mean square sense,, and. For the last case, notice that (14) where denotes conditioning and reg signifies the use of the regularized system estimator in (8). To simplify this expression, we observe that, since by the mean value theorem this probability is equal to the area of the region, which is of order, multiplied by some value of the probability density function of in that region, which is of order. In addition, where the last inequality follows again from the Cauchy- Schwarz inequality. By the mean square convergence of to and to the right hand side of the last inequality tends to 0. Therefore, the right hand side of (17) tends to 0. Moreover, under the high SNR assumption the conditional expectations can be approximated by their unconditional ones, since for a sufficiently small their difference is due to an event of probability. Therefore, (18) Combining all the above results yields (19) 3 The term optimal is used in this case to refer to uniformly better estimators than the MVU/MMSE estimators and not to actually optimal estimators in the strict sense. The estimators are optimal only with respect to the zeroth order metrics. The term is not negligible but for sufficiently small its dependence on is insignificant. Hence, for a sufficiently small and a sufficiently high SNR during training, minimizing is equivalent to minimizing (9).

5 KATSELIS AND ROJAS: APPLICATION-ORIENTED ESTIMATOR SELECTION 493 REFERENCES [1] J. C. Agüero and G. C. Goodwin, Choosing between open and closed-loop experiments in linear system identification, IEEE Trans. Automat. Contr., vol. 52, no. 8, pp , Aug [2] Z. Ben-Haim and Y. C. Eldar, Blind minimax estimation, IEEE Trans. Inf. Theory, vol. 53, no. 9, pp , [3] X. Bombois, G. Scorletti, M. Gevers, P. Van den Hof, and R. Hildebrand, Least costly identification experiment for control, Automatica, vol. 42, no. 10, pp , Oct [4] M. Gevers, Identification for control: From the early achievements to the revival of experiment design, Eur. J. Contr., vol. 11, pp. 1 18, [5] H. Hjalmarsson, From experiment design to closed-loop control, Automatica, vol. 41, no. 3, pp , Mar [6] H. Hjalmarsson, System identification of complex and structured systems, in Plenary Address Eur. Control Conf./Eur. J. Cont., 2009, vol. 15, no. 4, pp [7] P.J.Huber, Robust Statistics. : John Wiley & Sons, [8] W. James and C. Stein, Estimation with quadratic loss, in Proc. Fourth Berkeley Symp. Mathematical Statistics and Probability, 1961, vol. 1, no. 1961, pp [9] H. Jansson and H. Hjalmarsson, Input design via LMIs admitting frequency-wise model specifications in confidence regions, IEEE Trans. Automat. Contr., vol. 50, no. 10, pp , Oct [10] D. Katselis, C. R. Rojas, H. Hjalmarsson, and M. Bengtsson, Application-oriented finite sample experiment design: A semidefinite relaxation approach, in Proc. SYSID 2012, Brussels,Belgium,Jul [11] D. Katselis, C. R. Rojas, M. Bengtsson, E. Bjrnson, X. Bombois, N. Shariati, M. Jansson, and H. Hjalmarsson, Training sequence design for MIMO channels: An application-oriented approach, EURASIP J. Wireless Commun. Netw., p. 245, [12] D.Katselis,C.R.Rojas,J.S.Welsh,andH.Hjalmarsson, Robust Experiment Design for System Identification via Semi-Infinite Programming Techniques, in Proc. SYSID 2012, Brussels,Belgium,Jul [13] S. M. Kay, Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory. Englewood Cliffs, NJ, USA: Prentice-Hall, [14] S. Kay and Y. C. Eldar, Rethinking biased estimatio, IEEE Signal Process. Mag., vol. 25, no. 3, pp , [15] L. Pronzato, Optimal experimental design and some related control problems, Automatica, vol. 44, no. 2, pp , Feb [16] C. R. Rojas, J. S. Welsh, G. C. Goodwin, and A. Feuer, Robust optimal experiment design for system identification, Automatica, vol. 43, no. 6, pp , [17] C. Stein, Inadmissibility of the usual estimator for the mean of a multivariate normal distribution, in Proc. Third Berkeley Symp. Mathematical Statistics and Probability, 1956, vol. 1, no. 399, pp