Fit assessment for nonlinear model. Application to astrometric binary orbits DMS-DP-02 D. Pourbaix March 29, 2005, Revision : Abstract 1 Introduction Today much more than in the past (e.g., Dommanget 1995), assessing the likelihood of the orbit of an astrometric binary star is more difficult than deriving it in the first place. Whereas Hipparcos (ESA 1997; Lindegren et al. 1997) could take advantage of already published orbits, Gaia will not. It is therefore required to both derive the full orbit from scratch and to give a fair assessment of its likelihood. Pourbaix & Arenou (2001) have shown that the latter is not easy even when some parameters are assumed. This paper first gives a brief recap of some definitions overlooked too often (Sect. 2). In Section 3, the effects of the nonlinearity of the model are then established while an efficient way of smoothening them is presented in section 4. Finally, it is shown that seeking too short orbital period can yield to false solutions which cannot be discarded with standard statistical tools (Sect. 5). 2 Observations, least squares, and chi square If ɛ i are N independent stochastic variables that follow a N(0, 1) distribution, the sum of their squares follows a χ 2 distribution with N degrees of freedom. This definition can be generalized to the case of multivariate variables: if ɛ i are Gaussian and V is their covariance matrix, then χ 2 = ɛ t V 1 ɛ (1) also follows a χ 2 distribution with N degrees of freedom (Bulmer 1979). Let y = f(a, x) be the theoretical model that gives y as a function of x and a, the p parameters of the model. Owing to measurement error (ɛ i ), find a unique a such that y i = f(a, x i ) for every single observation (x i, y i ) (i = 1... N) is very unlikely. It turns out that the least-square fitting, i.e. finding a such that (y i f(a, x i )) 2 min! i is a maximum likelihood estimation of the fitted parameter a. Again, that result can be generalized to the case of Gaussian correlated observations. If y i = f(a, x i ) + ɛ i where {ɛ i } are normally distributed with covariance matrix V, Ξ 2 = (y f(a, x)) t V 1 (y f(a, x)) min! (2) 1
is also a maximum likelihood estimation. There are numerous methods available to minimize Ξ 2 (e.g. Dennis Jr. & Schnabel 1995; Horst et al. 1995; Björck 1996). Although Eqs. 1 and 2 are very similar, one additional key assumption is required in order to equal the two. Ξ 2 = (y f(a, x)) t V 1 (y f(a, x)) χ 2 N p only holds if the model f is a linear combination of the components of a (Lupton 1993). Therefore, the assessment of the fit based on the comparison between the minimum of the quadratic form and the tabulated χ 2 N p value is only legitimate in the case of a linear model. With a nonlinear model, the troublesome quantity is the degree of freedom, not the objective function per say. N p overestimates the number of independent added Gaussians. The quality of the fit is therefore subsequently overestimated too. If the fit already looks bad, it is indeed so. However, one can wonder whether a good fit looks so by mistake only. Overestimating the number of degrees of freedom also affects all the quantities that do require a genuine chi square. For instance, the goodness of fit (Kovalevsky & Seidelmann 2004) defined as 9ν χ F 2 = 2 ( 2 3 ν + 2 1) (3) 9ν where ν is the number of degrees of freedom follows a N(0, 1) (Kendall & Stuart 1978). This quantity makes it possible to directly compare fits based upon observation samples of different sizes without requiring the computation of additional probabilities. However, F 2 explicitly requires that the objective function behaves like a chi square. Another quantity often used to assess the relative merit of two fits is the ratio of the two chi squares (chi 2 N p and chi 2 N q, p > q). That ratio follows F-distribution with p q, N p degrees of freedom. It is for instance used to assess the benefit of one additional parameter (p = q +1) as described by Bevington & Robinson (1992). Here again, the F-like behavior only holds if the objective functions are genuine chi squares. 3 From almost linear to strongly nonlinear There is a tendency to forget about that constraint upon the chi square and therefore upon the linearity of the model. For instance, Lucy & Sweeney (1971) based their test for the assessment of the ellipticity of an orbit upon an F-test (the additional parameter is the eccentricity). The authors do not even bother mentioning their non-circular orbit was described by a non-linear model. They claim that earlier assessment methods overestimated the number of non-circular orbits but, clearly, their F-test does too. However, the Lucy-Sweeney test was designed to deal with small eccentricities and, therefore, the lack of linearity does not severely change the number of degrees of freedom. Pourbaix (2001) made a similar mistake when he used the F-test of a Campbell s elements based orbital solution against the single star solution to assess the detection of extrasolar planets in the Hipparcos astrometric data. The model with the Campbell elements is no linear and the probability is therefore underestimated. However, since all these probabilities where already above the adopted threshold, this mistake does not change the result of the paper. It is worth noting that the Thiele-Innes model (Pourbaix & Arenou 2001) is linear and, therefore, the F-test used in that second paper is legitimate. The situation can be way worse if one wants to know whether an eccentric orbit is worth keeping or not. In the Hipparcos catalogue (ESA 1997), one hundred thousand stars were processed as single stars (linear model). Instead of having a mean at 0 as expected, the distribution of F2 peaks near 0.25, thus suggesting that some more elaborate model might be worth trying. A full orbital model was applied to the 100k stars, i.e. seven parameters were added to the five of the single star model. The F-test was then applied to assess the benefit of the orbital model, i.e. the improvement of the fit caused by these seven additional parameters, snobing the 2
Figure 1: Probability of the F-test (orbital vs single star model). The thick line is based on the original data where the thin line is the synthetic genuine single stars observations. constraint of linearity. The probability of the F-test is thus the probability of rejecting the null hypothesis that the orbital model does improve the fit. The probability of the F-test is plotted as a thin line in Fig. 1. Although one cannot rule out the possibility that a few of these single stars are binaries, the cumulative frequency distribution is too steep at low probabilities. Indeed, since all these stars were already classified as single (actually, their observations as being satisfactorily fitted with a single star model), one does not expect any binary to pop up. At the 99% level (i.e. a false detection rate of 1% of the sample is tolerated), 21.5% of the stars are binaries. In order to assess this result, artificial observations of these 100k stars were generated using the single star model. Here again, the distribution of the probability of the F-test (tick line Fig. 1) is positively skewed. At the 99% confidence level, 12.6% of the genuine single stars turn out to be flagged as binaries. Because of the very nature of the observations (one dimensional data), nothing prevents the minimization algorithm from getting very eccentric orbits which do improve the fit (despite being physically meaningless (Pourbaix 2002)) as illustrated in Fig. 2. The distributions of the eccentricities are essentially the same whether the original or synthetic data are used. The problem is not that we find orbital solutions which improve the fit but rather that a lot of them are assigned a high confidence. The Keplerian model used to fit the data is only linear if the orbit is circular. And so are the chi square behavior of the least square fit and, subsequently, its assessment through an F-test. 3
Figure 2: Distribution of the eccentricities based on the original data (thick line) and the synthetic single star observations (thin line). The distribution of the eccentricities of solutions based on the original data and for which the probability of the F-test is below 1% is plotted in dashed line. Clearly, one pays cash snobing the linear assumption. 4 Orbit assessment The only reason of the hypothesis about the linearity of the model is to make it straightforward to obtain the number of degrees of freedom. What we need to be back on business is a way to guess that number in case of a nonlinear model or the smallest linear model (in terms of number of parameters) that yields the same level of Ξ 2 as the nonlinear model. Among the linearization methods, Fourier expansion is probably the most convenient to handle periodic functions. Monet (1979) used the Fourier transform to guess the orbital parameters of binaries by exactly solving for them from the minimum number of expansion coefficients. Such a minimalist approach behaves well for small eccentricities only as illustrated in Fig. 3. However, we are not looking for the orbital solution, so there is no need to limit the expansion to low harmonic ranks. Jarnadin (1965) showed that the coefficient of each additional harmonic is dominated by a higher power of the eccentricity. Since the power of the eccentricity vanishes more rapidly as the exponent increases, the 4
Figure 3: Successive approximations of the unit ellipse with increasing number of Fourier coefficients (1,2,3,4 in the top panels, 5, 10, 15, and 20 in th bottom panels) for different values of the eccentricity (with expansions up to the 20th power of e). number of harmonics required for a convenient linearization increases with the eccentricity. Moreover, the value of the linearized Ξ 2 (Eq. 2) also decreases with the number of harmonics. Let Ξ 2 N be the linearized Ξ2 with N coefficients. We define the number of degrees of freedom of the nonlinear model as the smallest N such that Ξ 2 N < Ξ2, supplemented by three to account for e, P, and the periastron time. One needs to be a bit careful about what is meant by linearized model and Fourier transform, especially in the context of astrometric binaries. The whole picture is a dual-period model: a one-year period describing the absolute motion of the center of mass (position, parallax and proper motion) and the actual orbital model. The former is already linear so only the latter needs to be linearized through the Fourier expansion. The two components (in x and y-direction) corresponding to the time independent part of that expansion are embedded in the position of the center of mass so the expansion actually begins at the fundamental frequency. As foreseen, the dimension of the linearized model increases with e. The trend is clear (Fig. 4) but this is not a one-to-one relation. There is therefore no way to directly guess N from e without the linearization stage. Owing to the speed of the step, this is a very minor drawback. The reduction of the number of accepted (α = 1%) orbital solutions for genuine single stars is substantial, 5
Figure 4: Dimension of the linearized model versus the eccentricity from 12.6% to 1.4% (Fig. 5). This is however still slightly in excess with respect to the theoretical value (only 1% of false detection is expected). Once applied to the original observation, the same procedure yields 2.3% of potential binaries. Only 48 objects are flagged as binaries with both data sets, thus suggesting that the distribution of the observing time might be responsible for the result. 5 Shortest period Any evenly sampled time series can be modeled with a periodic function of an arbitrary high frequency. The Nyquist frequency is defined as the highest frequency constrained by such time series. Eyer & Bartholdi (1999) extended that result to unevenly sampled data and successfully identified a period as short as 0.085 day in the Hipparcos photometry. Regardless of the physical soundness of an astrometric orbit as short as 0.1 day, the shortest period accessible for an astrometric orbit reconstruction is likely longer than the photometric one. In the Hipparcos catalogue, 235 entries were fitted with an orbital model (DMSA/O). Some orbital parameters were sometime adopted from a published solution (spectroscopic or interferometric orbit). Our period with no assumed parameter essentially matches the DMSA/O one when ours is longer than 50 days. In other words, some of the assumed periods have an alias below 50 days for which the F-test at 1% is also satisfied. Fifty days being close to the precession 6
Figure 5: Probability of the F-test (orbital vs single star model) when the number of degrees of freedom is derived from the linearized model. The thick (reps. thin) line is based on the original (resp. synthetic genuine single) observation. period of Hipparcos, one can foresee similar problem with Gaia below 70 days. The problem of aliasing at short periods is clearly present in synthetic data of single stars (Fig. 7). 82.3% of the stars fulfilling the F-test at 1% have a period below 50 days. This has nothing to do with the nonlinearity of the model and will be investigated as a special issue. 6 Conclusion Unlike Hipparcos, Gaia will not benefit from a catalogue of published orbits from which some parameters can be assumed for a Gaia-based astrometric solution. Though the shell task of astrometric orbit fitting will only be called if the single star model does not yield a satisfactory fit, the shell module should also return an assessment of its own solution. We have shown that the F-test only makes sense if the number of degrees of freedom is adjusted to account for the nonlinearity of the model. Although that correction substantially improves the soundness of the assessment, it does not prevent too short orbits from being falsely accepted. 7
Figure 6: Orbital periods derived from scratch (horizontal axis) versus the period listed in the Hipparcos Double and Multiple Star Annex/Orbital solution. Squares stand for systems which do not pass our 1% F-test and triangles for those fulfilling that condition. Filled (open) symbols are for systems whose DMSA/O period was assumed (fitted). Crosses designate systems for which a circular orbit was assumed in the DMSA/O. References Bevington, P. R. & Robinson, D. K. 1992, Data reduction and error analysis for the physical sciences, 2nd edn. (McGraw-Hill) Björck, A. 1996, Numerical Methods for Least Squares Problems (SIAM) Bulmer, M. G. 1979, Principles of Statistics (Dover) Dennis Jr., J. E. & Schnabel, R. B. 1995, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, 2nd edn. (SIAM) Dommanget, J. 1995, A&A, 301, 919 ESA. 1997, The Hipparcos and Tycho Catalogues (ESA SP-1200) Eyer, L. & Bartholdi, P. 1999, A&AS, 135, 1 8
Figure 7: Distribution of the orbital period of accepted (F-test at 1%) binary solution based on single star synthetic data. The dashed line is after the correction for the nonlinearity of the model Horst, R. H., Pardalos, P. M., & Thoai, N. V. 1995, Introduction to Global Optimization (Kluwer Academic Publishers) Jarnadin, Jr., M. P. 1965, Astronomical Papers, 18, 1 Kendall, M. G. & Stuart, A. 1978, The Advanced Theory of Statistics (Griffin) Kovalevsky, J. & Seidelmann, P. K. 2004, Fundamentals of Astrometry (Cambridge University Press) Lindegren, L., Mignard, F., Söderhjelm, S., et al. 1997, A&A, 323, L53 Lucy, L. B. & Sweeney, M. A. 1971, AJ, 76, 544 Lupton, R. 1993, Statistics in theory and practice (Princeton University Press) Monet, D. G. 1979, ApJ, 234, 275 Pourbaix, D. 2001, A&A, 369, L22 Pourbaix, D. 2002, A&A, 385, 686 9
Pourbaix, D. & Arenou, F. 2001, A&A, 372, 935 10