IACHEC 13, Apr , La Tenuta dei Ciclamini. CalStat WG Report. Vinay Kashyap (CXC/CfA)

IACHEC 13, Apr 9-12 2018, La Tenuta dei Ciclamini CalStat WG Report Vinay Kashyap (CXC/CfA)

CalStats Working Group A forum for the discussion of statistical, methodological, and algorithmic issues that affect the calibration of astronomical instruments and how they are used in data analysis and interpretation of analysis results.

CalStats Working Group @IACHEC13 III WG Session 1, Tutorial on Gaussian Processes, David Jones VII (plenary) Updates to Concordance, Herman Marshall IX (plenary) Optimal binning and fit statistics for X-ray spectra, Jelle Kaastra XI WG Session 2, Binning and cstat (contd.) If you are not on the CalStat WG mailing list (iachec-calstat@cfa.harvard.edu),

I would have definitely attended your session but I was too busy having fun.

iachec-calstat@cfa.harvard.edu vkashyap@cfa.harvard.edu I. Tutorial on Gaussian Process New methodologies are coming up rapidly in astronomical analysis: Hierarchical Bayes, Nested Sampling, Deep Learning, Gaussian Processes, etc. Gaussian Process is probably the most immediately useful technique for calibration problems, as it allows a semiparametric, data-driven accounting of fudge factors that are impossible to get away from in real world calibration A tutorial was conducted on Apr 9 by statistician David Jones (Duke/SAMSI and TAMU)

David Jones Mention of Gaussian Process'' in SAO/NASA ADS Abstract 70 60 50 40 30 20 10 0 01/82 12/86 01/87 12/91 01/92 12/96 01/97 12/01 01/02 12/06 01/07 12/11 01/12 12/16 I

David Jones 2 y = f(x) 0 2 10 5 0 5 10 x

~y ~y David Jones 2 ~y ~y N ~ ~ =,! ~y ~y N ~y, y = f(x) 0 2 10 5 0 5 10 x

10 5 0 5 10 x David Jones 2 y 0 2

10 5 0 5 10 x I y,...,y m x,...,x m ~y N ~, + t I m David Jones 2 I ij = R (x i,x j ) ~y N ~f,t Im ~ f N ~, y 0 ~ f =(f (x ),...,f (xm )) T 2

iachec-calstat@cfa.harvard.edu vkashyap@cfa.harvard.edu I. Tutorial on Gaussian Process Some resources for further learning Gaussian Processes for Machine Learning, Rasmussen and Williams 2006, MIT Press http://www.gaussianprocess.org/gpml/chapters/ The Kernel Cookbook, David Duvenaud 2014 http://www.cs.toronto.edu/~duvenaud/cookbook/index.html Beyond Chi-Squared: An Introduction to Correlated Noise, Ben Montet 2014 https://astrobites.org/2014/07/01/beyond-chi-squared-an-introduction-to-correlated-noise/ Fitting Gaussian Process Models in Python, Chris Fonnesbeck 2017 https://blog.dominodatalab.com/fitting-gaussian-process-models-python/ Contact David Jones if you want assistance with a project

iachec-calstat@cfa.harvard.edu vkashyap@cfa.harvard.edu II. Update on Concordance On Apr 10, Herman Marshall (MIT) discussed the status of the Concordance project and discussed the process from an IACHEC perspective A paper describing the full statistical foundation of the method has been submitted to a statistics journal and is on the verge of resubmission after dealing with the referees reports Calibration Concordance for Astronomical Instruments via Multiplicative Shrinkage, Chen, Meng, Wang, van Dyk, Marshall, & Kashyap 2018, submitted to Journal of Applied Statistical Methods https://arxiv.org/abs/1711.09429 A trial run describing astronomically relevant analysis procedures will be presented at SPIE 2018 (Astronomical Instrumentation and Operations), with follow-on submission to JATIS An astronomy oriented paper which includes variable prior uncertainty factors and applications to other datasets (e.g., strong lines from Capella in Chandra gratings) is in preparation (Marshall et al. 201X)

iachec-calstat@cfa.harvard.edu vkashyap@cfa.harvard.edu II. Update on Concordance To complete analysis, we need: τ a measure of the prior uncertainty for detectors over different energy bands DONE datasets compilations of homogeneously estimated fluxes for simultaneously (or in some cases contemporaneously) observed sources. Perhaps easier to send Herman spectra and corresponding ARFs and RMFs (via Heritage WG).

iachec-calstat@cfa.harvard.edu vkashyap@cfa.harvard.edu III. A Practical Guide On Apr 11, Jelle Kaastra (SRON) gave a plenary talk on recent advances he has made 1.on optimal binning of RMFs a prescription for how to determine the most informative binning scheme given instrumental features (LSF, effective area) and counts in the spectrum, see Kaastra & Bleeker 2016 (A&A 587, A151) 2.on computing goodness of fit measures when using the cstat fit statistic calculating the expected contribution of a nominal model intensity in a bin to cstat (and its variance), averaged over all possible counts, and presenting it as a lookup table for ease of computation, see Kaastra 2017 (A&A 605, A51) and followed it up with a detailed discussion during the afternoon WG session. Keith Arnaud (GSFC) reports from afar: the optimal binning scheme is implemented in the new tool ftgrouppha (there s a bug for square RMFs) and an unreleased tool that bins the RMF along the energy-axis.

Monte Carlo results Jelle Kaastra (Data binning) Order 2 Order 1 Order 0 ( SPEX recommended ) ( classical approach)

Jelle Kaastra

Jelle Kaastra Estimating the contribution to cstat from each bin, given the expected model intensity in the bin; tabulated as a look-up table and interpolated therefrom 41

iachec-calstat@cfa.harvard.edu vkashyap@cfa.harvard.edu III. A Practical Guide A recommendation was made to write a practical guide to best practices in X-ray analysis [I will set up an overleaf document and ask CalStat WG members to contribute sections.] Wstat is an alternative statistic in XSPEC that computes maximum likelihood source and background intensities assuming Poisson regime and using observed counts in source and background. This can lead to bad fits when background counts fluctuate and/or model intensities are unstable. Jelle recommends that spectral fitting always be done by simultaneous fitting of background and source models, or computing a smooth background and adding it to the source. Apropos to that, would be a good idea for all missions to describe how to set up a model background spectrum Asked of Jelle that power of cstat parameterization be computed (how often is a bad fit accepted as good?); Monte Carlo uncertainties for cstat goodness-of-fit be calculated to show fluctuations in the asymptotic limits are as expected; simulations be carried out to see how a continuumdominated vs line-dominated spectrum affect goodness of fit estimates. Some questions require consultation with statisticians. E.g., what effect does determining binning from the data have on the process of the fit? Can we compute χ² and determine goodness from that? How robust is the K-S test (or equivalent) to measure goodness?

Next year s theme High resolution spectra