Covariance Estimation for High Dimensional Data Vectors

Transcription

1 Covariance Estimation for High Dimensional Data Vectors Charles A. Bouman Purdue University School of Electrical and Computer Engineering Co-authored with: Guangzhi Cao and Leonardo R Bachega

2 Covariance Estimation for High Dimensional Data Vectors Let Y =!" y 1, y 2,!, y n # $ where y! N(0, R) is a p! dimensional random vector i Objective: Estimate the eigenvalues and eigenvectors of R R = E!E t Problem: This a classically difficult problem when n<p Curse of dimensionality Proposed Solution: Model based estimation Does not depend on ordering of vector or stationarity assumption

3 Data Model Notation: Y =!" y 1, y 2,!, y n # " $ $ # $$$ % $ Observed Data S & 1 YY t n " $ # %$ Sample Covariance R = E!" S " $ # $ % # $ True Covariance Likelihood of Y given R: p R (Y ) = 1 (2! ) R " n 2 exp # $ " 1 np/ 2 2 tr{y t R "1 Y} & ' % ( ML estimate of eigenvectors and eigenvalues is given by Ê = arg min E!Prior Model { diag(e t SE) } ˆ" = diag(ê t SÊ)

4 Prior Model: The Sparse Matrix Transform (SMT) Big idea: (Eigenvector matrix E) = (Sparse Matrix Transform) What is a Sparse Matrix Transform? were E k are Givens rotations E = E1 E2! Ek Each Givens rotation operates on only two coordinates Only 4 multiplies per rotation (really only 2) When k=p(p)/2, this is any orthonormal transform

5 SMT is a Generalization of the FFT and Orthonormal Wavelet Transform SMT is product of Givens rotations: E = E 1 E 2 E k where E k = So the SMT is a generalization of the FFT y 0 y 1 y 2 y 3 y 4 y 5 y 6 y 7 W W 8 0 W 8 W 0 8 W 8 0 W 8 2 FFT 0 W 8 1 W 8 2 W 8 W 3 8 y! 0 y! 1 y! 2 y! 3 y! 4 y! 5 y! 6 y! 7 y 0 y 1 y 2 y 3 y 4 y 5 y 6 y 7 E 1 E 6 E 9 E E 2 E 8 4 E 7 E 8 E 3 E 5 E 10 SMT y! 0 y! 1 y! 2 y! 3 y! 4 y! 5 y! 6 y! 7 SMT is also a generalization of orthonormal (paraunitary) wavelets

6 Design of SMT using Cost Optimization ML estimate of Eigenvectors is Ê = arg min E!SMT of order K { diag(e t SE) } where E = E 1 E 2!!! E K is an SMT transform. The algorithm: For k = 1 to K { Select most correlated coordinate pair Decorrelate the coordinate pair with rotation E k } K is choose to maximize cross-validated likelihood

7 Covariance Estimation for Hyperspectral Data # of hyperspectral bands: p = 191, # of samples: n = 80, grass class Sample Data (1928) Y (Gaussian) Y (non-gaussian)

8 Estimators Compared Shrinkage estimator estimate using cross-validation Graphic lasso (glasso) covariance estimator *:! L1 regularized ML estimate SMT estimator Use K th order SMT model Estimate k using cross-validation ˆR = (1! ") # S + " # diag(s) ˆR = arg max R: P.D. SMT-shrinkage (SMT-S) model { } log(y R)! " R!1 L1 Combine SMT covariance estimate with shrinkage ˆR = (1! ") # S + " # ˆR SMT * See Sparse inverse covariance estimation with the graphic lasso, Friedman et al. 2007

9 Eigenvalue and Eigenvector Estimation: Gaussian Case ˆ ˆ t! = diag( E SEˆ )

10 Results in Kullback-Leibler Distance: Gaussian Case Grass Water Roof Street

11 Results in Kullback-Leibler Distance * : non-gaussian Case Grass Water Roof Street * KL distance of Gaussian distributions estimated from non-gaussian samples

12 Computational Complexity Complexity CPU time (seconds) Model order Shrinkage p glasso p 3 I SMT Kp SMT-S Kp+ p I-iterations required for for glasso, K-number of Givens rotations Numerical results are based on the Guassian grass case with n = 80

13 Signal Detection Based on Covariance Estimation Formulation of signal detection : H 0 : r = x H 1 : r = x + e! t, with e not zero where: x is a background with covariance R t is a target signature e is a scalar signal strength r is the observed pixel The SNR of signal detection: where q = ˆ! 1 R t 2 ( q t t) SNR = t q Rq is the linear matching filter used to test for signal

14 Results in Signal Detection Simulation: R was chosen to be the true covariance of each class ˆR was estimated from the Gaussian sample (n = 80) Dense case: t = rand(p,1) has the uniform distribution in [0, 1] Sparse case: setting all but the largest values (>0.9) of t to zero Test was run 100 times, and the average SNRR was calculated: SNRR SNR( Rˆ ) SNR using estmated Rˆ = = SNR ( R ) opitmal SNR

15 Results in Signal Detection Dense Case SNRR SNR( Rˆ ) SNR using estmated Rˆ SNR ( R ) opitmal SNR = = The higher, the better! Grass Water Roof Street

16 Computing Eigenimages with SMT SMT Model Training n training sample images SMT Eigenimage Analysis Input Image SMT model All p eigenvalues Y= Apply Givens rotations p eigenvalues p eigenfaces p pixels Impulse at position k SMT model inverse kth eigenimage

17 Method Comparison of Traditional and SMT Eigenimages Eigenimage experiment Dataset: face image Number of samples (n) = 40 Dimensions (p) = 644 Use cross-validation to compute expected log likelihood Maximum log-likelihood PCA+Shrinkage Δ K max Full SMT Fast SMT(8) Fast SMT(16) Fast SMT(32) Diagonal Δlog likelihood= SMT produces much better fit to image data SMT can produce all eigenimages

18 SMT versus Traditional Eigenfaces Face dataset: Traditional (PCA) eigenfaces: Full SMT: SMT(8pt neighborhood):