Rehabilitating Research Correlations, Matters Avoiding Inversion, and Extracting Roots

Transcription

1 Rehabilitating Research Correlations, Matters Avoiding Inversion, and Extracting Roots February 25, 2009 Nick Nick Higham Higham Director School of of Research Mathematics The University of Manchester School of nickhigham.wordpress.com The Actuarial Profession March 20, 2013, London 1 / 6

2 Accuracy Estimates Guarantees Precision Single Double Quad Variable Efficiency Speed Parallelism Energy efficiency

3 Outline 1 Rehabilitating Correlations 2 Matrix Inversion 3 Matrix Roots University of Manchester Nick Higham Actuarial Matrix Computations 3 / 37

4 Correlation Matrix An n n symmetric matrix A is a correlation matrix if It has ones on the diagonal. All its eigenvalues are nonnegative. University of Manchester Nick Higham Actuarial Matrix Computations 4 / 37

5 Correlation Matrix An n n symmetric matrix A is a correlation matrix if It has ones on the diagonal. All its eigenvalues are nonnegative. Is this a correlation matrix? University of Manchester Nick Higham Actuarial Matrix Computations 4 / 37

6 Correlation Matrix An n n symmetric matrix A is a correlation matrix if It has ones on the diagonal. All its eigenvalues are nonnegative. Is this a correlation matrix? Spectrum: , , University of Manchester Nick Higham Actuarial Matrix Computations 4 / 37

7 Question from London Fund Management Company (2000) Given a real symmetric matrix A which is almost a correlation matrix... What is the best approximating (in Frobenius norm?) correlation matrix? Is it unique? Can we compute it? Typically we are working with at the moment, but this will probably grow to

8 How to Proceed Make ad hoc modifications to matrix: e.g., shift negative e vals up to zero then diagonally scale. Plug the gaps in the missing data, then compute an exact correlation matrix. Compute the nearest correlation matrix in the weighted Frobenius norm ( A 2 = i,j w iw j a 2 ij ). Given approx correlation matrix A find correlation matrix C to minimize A C. Constraint set is a closed, convex set, so unique minimizer. University of Manchester Nick Higham Actuarial Matrix Computations 6 / 37

9 Development Derived theory and algorithm: N. J. Higham, Computing the Nearest Correlation Matrix A Problem from Finance, IMA J. Numer. Anal. 22, , Extensions in: Craig Lucas, Computing Nearest Covariance and Correlation Matrices, M.Sc. Thesis, University of Manchester, University of Manchester Nick Higham Actuarial Matrix Computations 7 / 37

10 Alternating Projections Algorithm von Neumann (1933): for subspaces. Dykstra (1983): corrections for closed convex sets. S 1 S 2 Easy to implement. Guaranteed convergence, at a linear rate. Can add further constraints/projections. University of Manchester Nick Higham Actuarial Matrix Computations 8 / 37

11 Unexpected Applications Some recent papers: Applying stochastic small-scale damage functions to German winter storms (2012) Estimating variance components and predicting breeding values for eventing disciplines and grades in sport horses (2012) Characterisation of tool marks on cartridge cases by combining multiple images (2012) Experiments in reconstructing twentieth-century sea levels (2011) University of Manchester Nick Higham Actuarial Matrix Computations 9 / 37

12

13 Newton Method Qi & Sun (2006): Newton method based on theory of strongly semismooth matrix functions. Applies Newton to dual (unconstrained) of min 1 2 A X 2 F problem. Globally and quadratically convergent. H & Borsdorf (2010) improve efficiency and reliability: use minres for Newton equation, Jacobi preconditioner, reliability improved by line search tweaks, extra scaling step to ensure unit diagonal. University of Manchester Nick Higham Actuarial Matrix Computations 11 / 37

14

15

16 Alternating Projections vs Newton Matrix tol Code Time (s) Iters 1. Random (100) 1e-10 g02aa nearcorr Random (500) 1e-10 g02aa nearcorr Real-life (1399) 1e-4 g02aa nearcorr University of Manchester Nick Higham Actuarial Matrix Computations 14 / 37

17 Performance of NAG Codes University of Manchester Nick Higham Actuarial Matrix Computations 15 / 37

18 Factor Model (1) ξ = }{{} X η }{{} n k k 1 + F }{{} n n }{{} ε n 1 where var(ξ i ) 1, F = diag(f ii ). Implies k j=1, η i, ε i N(0, 1), x 2 ij 1, i = 1: n. Multifactor normal copula model. Collateralized debt obligations (CDOs). Multivariate time series. University of Manchester Nick Higham Actuarial Matrix Computations 16 / 37

19 Factor Model (2) Yields correlation matrix of form C(X) = D + XX T = D + k j=1 x j x T j, D = diag(i XX T ), X = [x 1,..., x k ]. C(X) has k factor correlation matrix structure. 1 y1 T y 2... y1 T y n y C(X) = 1 T y y T n 1 y n, y i R k. y1 T y n... yn 1 T y n 1 University of Manchester Nick Higham Actuarial Matrix Computations 17 / 37

20 Factor Structure Nearest correlation matrix with factor structure. Principal factors method (Andersen et al., 2003) has no convergence theory and can converge to an incorrect answer. University of Manchester Nick Higham Actuarial Matrix Computations 18 / 37

21 Factor Structure Nearest correlation matrix with factor structure. Principal factors method (Andersen et al., 2003) has no convergence theory and can converge to an incorrect answer. Algorithm based on spectral projected gradient method (Borsdorf, H & Raydan, 2010). Respects the constraints, exploits their convexity, and converges to a feasible stationary point. NAG routine g02aef (Mark 23, 2012). University of Manchester Nick Higham Actuarial Matrix Computations 18 / 37

22 Variations Specific rank or bound on correlations. Block structure. Bounds on individual correlations. Other requirements? University of Manchester Nick Higham Actuarial Matrix Computations 19 / 37

23 Outline 1 Rehabilitating Correlations 2 Matrix Inversion 3 Matrix Roots University of Manchester Nick Higham Actuarial Matrix Computations 20 / 37

24 Avoiding Inversion Fundamental Tenet of Numerical Analysis Don t invert matrices. University of Manchester Nick Higham Actuarial Matrix Computations 21 / 37

25 Avoiding Inversion Fundamental Tenet of Numerical Analysis Don t invert matrices. Ax = b : use Gaussian elimination (with pivoting), not x = A 1 b. x T A 1 y = x T (A 1 y). (A 1 ) ii = e T i A 1 e i. trace(a 1 ) m 1 m k=1 v T k A 1 v k, v k uniform{ 1, 1}. (Bekas et al., 2007) University of Manchester Nick Higham Actuarial Matrix Computations 21 / 37

26 How to Invert (1) Use a condition estimator to warn when A is nearly singular. >> A = hilb(16); >> X = inv(a); Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND = e-19. University of Manchester Nick Higham Actuarial Matrix Computations 22 / 37

27 How to Invert (2) To compute variance covariance matrix (X T X) 1 of least squares estimator, use QR factorization X = QR ((X T X) 1 = R 1 R T ) and do not explicitly form X T X. Quality of computed inverse Ŷ A 1 measured by Ŷ A I Ŷ A or AŶ I Ŷ A but not both. University of Manchester Nick Higham Actuarial Matrix Computations 23 / 37

28 Computing the Sample Variance Sample variance of x 1,..., x n : s 2 n = 1 n 1 n (x i x) 2, where x = 1 n i=1 n x i. (1) Can compute using one-pass formula: sn 2 = 1 ( n xi 2 1 ( n ) 2 x i ). (2) n 1 n i=1 i=1 i=1 University of Manchester Nick Higham Actuarial Matrix Computations 24 / 37

29 Computing the Sample Variance Sample variance of x 1,..., x n : s 2 n = 1 n 1 n (x i x) 2, where x = 1 n i=1 n x i. (1) Can compute using one-pass formula: sn 2 = 1 ( n xi 2 1 ( n ) 2 x i ). (2) n 1 n i=1 For x = (10000, 10001, 10002) using 8-digit arithmetic, (1) gives: 1.0, (2) gives 0.0. i=1 i=1 (2) can even give negative results in floating point arithmetic! University of Manchester Nick Higham Actuarial Matrix Computations 24 / 37

30 Casio fx-992vb University of Manchester Nick Higham Actuarial Matrix Computations 25 / 37

31 Spreadsheets in the Cloud Standard deviation of x = [n, n + 1, n + 2] T. n Exact Google Sheet University of Manchester Nick Higham Actuarial Matrix Computations 26 / 37

32 Spreadsheets in the Cloud Standard deviation of x = [n, n + 1, n + 2] T. n Exact Google Sheet University of Manchester Nick Higham Actuarial Matrix Computations 26 / 37

33 Spreadsheets in the Cloud Standard deviation of x = [n, n + 1, n + 2] T. n Exact Google Sheet McCullough & Yalta (2013) University of Manchester Nick Higham Actuarial Matrix Computations 26 / 37

34 Outline 1 Rehabilitating Correlations 2 Matrix Inversion 3 Matrix Roots University of Manchester Nick Higham Actuarial Matrix Computations 27 / 37

35 Cayley and Sylvester Term matrix coined in 1850 by James Joseph Sylvester, FRS ( ). Matrix algebra developed by Arthur Cayley, FRS ( ). Memoir on the Theory of Matrices (1858). University of Manchester Nick Higham Actuarial Matrix Computations 28 / 37

36 Cayley and Sylvester on Matrix Functions Cayley considered matrix square roots in his 1858 memoir. Tony Crilly, Arthur Cayley: Mathematician Laureate of the Victorian Age, Sylvester (1883) gave first definition of f (A) for general f. Karen Hunger Parshall, James Joseph Sylvester. Jewish Mathematician in a Victorian World, University of Manchester Nick Higham Actuarial Matrix Computations 29 / 37

37 Matrix Roots in Markov Models Let vectors v 2011, v 2010 represent risks, credit ratings or stock prices in 2011 and Assume a Markov model v 2011 = P v 2010, where P is a transition probability matrix. P 1/2 enables predictions to be made at 6-monthly intervals. University of Manchester Nick Higham Actuarial Matrix Computations 30 / 37

38 Matrix Roots in Markov Models Let vectors v 2011, v 2010 represent risks, credit ratings or stock prices in 2011 and Assume a Markov model v 2011 = P v 2010, where P is a transition probability matrix. P 1/2 enables predictions to be made at 6-monthly intervals. P 1/2 is matrix X such that X 2 = P. What are P 2/3, P 0.9? P s = exp(s log P). Problem: log P, P 1/k may have wrong sign patterns regularize. University of Manchester Nick Higham Actuarial Matrix Computations 30 / 37

39 Chronic Disease Example Estimated 6-month transition matrix. Four AIDS-free states and 1 AIDS state observations (Charitos et al., 2008) P = Want to estimate the 1-month transition matrix. Λ(P) = {1, , , , }. H & Lin (2011). Lin (2011, for survey of regularization methods. University of Manchester Nick Higham Actuarial Matrix Computations 31 / 37

40 MATLAB: Arbitrary Powers >> A = [1 1e-8; 0 1] A = e e e+000 >> A^0.1 ans = >> expm(0.1*logm(a)) ans = e e e+000 University of Manchester Nick Higham Actuarial Matrix Computations 32 / 37

41 MATLAB Arbitrary Power New Schur algorithm (H & Lin, 2011, 2013) reliably computes A p for any real p. New backward-error based inverse scaling and squaring alg for matrix logarithm (Al-Mohy, H & Relton, 2012) faster and more accurate. Alternative Newton-based algorithms available for A 1/q with q an integer, e.g., for X k+1 = 1 q [ (q + 1)Xk X q+1 k A ], X 0 = A, X k A 1/q. University of Manchester Nick Higham Actuarial Matrix Computations 33 / 37

42 Knowledge Transfer Partnership University of Manchester and NAG ( ) funded by EPSRC, NAG and TSB. Developing suite of NAG Library codes for matrix functions. Extensive set of new codes included in Mark 23 (2012), Mark 24 (2013). Improvements to existing state of the art: faster and more accurate. My work also supported by 2M ERC Advanced Grant. University of Manchester Nick Higham Actuarial Matrix Computations 34 / 37

43

44

45 Final Remarks Fast moving developments in numerical linear algebra algorithms. Numerical reliability is essential. Partnership with NAG enables rapid inclusion of our algorithms in the NAG Library. Keen to hear about your matrix problems. University of Manchester Nick Higham Actuarial Matrix Computations 37 / 37

46 References I A. H. Al-Mohy and N. J. Higham. Improved inverse scaling and squaring algorithms for the matrix logarithm. SIAM J. Sci. Comput., 34(4):C153 C169, A. H. Al-Mohy, N. J. Higham, and S. D. Relton. Computing the Fréchet derivative of the matrix logarithm and estimating the condition number. MIMS EPrint , Manchester Institute for Mathematical Sciences, The University of Manchester, UK, July pp. Revised December University of Manchester Nick Higham Actuarial Matrix Computations 1 / 8

47 References II L. Anderson, J. Sidenius, and S. Basu. All your hedges in one basket. Risk, pages 67 72, Nov C. Bekas, E. Kokiopoulou, and Y. Saad. An estimator for the diagonal of a matrix. Appl. Numer. Math., 57: , R. Borsdorf, N. J. Higham, and M. Raydan. Computing a nearest correlation matrix with factor structure. SIAM J. Matrix Anal. Appl., 31(5): , University of Manchester Nick Higham Actuarial Matrix Computations 2 / 8

48 References III T. Charitos, P. R. de Waal, and L. C. van der Gaag. Computing short-interval transition matrices of a discrete-time Markov chain from partially observed data. Statistics in Medicine, 27: , T. Crilly. Arthur Cayley: Mathematician Laureate of the Victorian Age. Johns Hopkins University Press, Baltimore, MD, USA, ISBN xxi+610 pp. University of Manchester Nick Higham Actuarial Matrix Computations 3 / 8

49 References IV P. Glasserman and S. Suchintabandid. Correlation expansions for CDO pricing. Journal of Banking & Finance, 31: , C.-H. Guo and N. J. Higham. A Schur Newton method for the matrix pth root and its inverse. SIAM J. Matrix Anal. Appl., 28(3): , University of Manchester Nick Higham Actuarial Matrix Computations 4 / 8

50 References V N. J. Higham. Accuracy and Stability of Numerical Algorithms. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, second edition, ISBN xxx+680 pp. N. J. Higham. Computing the nearest correlation matrix A problem from finance. IMA J. Numer. Anal., 22(3): , University of Manchester Nick Higham Actuarial Matrix Computations 5 / 8

51 References VI N. J. Higham and L. Lin. On pth roots of stochastic matrices. Linear Algebra Appl., 435(3): , N. J. Higham and L. Lin. An improved Schur Padé algorithm for fractional powers of a matrix and their Fréchet derivatives. MIMS EPrint , Manchester Institute for Mathematical Sciences, The University of Manchester, UK, Jan pp. University of Manchester Nick Higham Actuarial Matrix Computations 6 / 8

52 References VII L. Lin. Roots of Stochastic Matrices and Fractional Matrix Powers. PhD thesis, The University of Manchester, Manchester, UK, pp. MIMS EPrint , Manchester Institute for Mathematical Sciences. B. D. McCullough and A. T. Yalta. Spreadsheets in the cloud not ready yet. Journal of Statistical Software, 52:1 14, University of Manchester Nick Higham Actuarial Matrix Computations 7 / 8

53 References VIII K. H. Parshall. James Joseph Sylvester. Jewish Mathematician in a Victorian World. Johns Hopkins University Press, Baltimore, MD, USA, ISBN xiii+461 pp. H.-D. Qi and D. Sun. A quadratically convergent Newton method for computing the nearest correlation matrix. SIAM J. Matrix Anal. Appl., 28(2): , University of Manchester Nick Higham Actuarial Matrix Computations 8 / 8