Towards a Regression using Tensors

Transcription

1 February 27, 2014

2 Outline Background 1 Background Linear Regression Tensorial Data Analysis 2 Definition Tensor Operation Tensor Decomposition 3 Model Attention Deficit Hyperactivity Disorder Data Analysis 4

3 Classical Linear Regression Linear Regression Tensorial Data Analysis Predict e.g. speed,road conditions, weather traffic accidents rates Identify the key predictors e.g mental disease status the regions of brain

4 Linear Regression Tensorial Data Analysis Multi-Dimensional Array Data (Tensors) Neuroscience EEG data: (time frequency electrodes) fmri data: (time x axis y axis z axis) Vision image (video) data: (pixel illumination expression viewpoints) Chemistry fluorescence excitation-emission data: (samples emission excitation)

5 Background Linear Regression Tensorial Data Analysis Brain Imaging Data Analysis Mental health disorders are difficult to diagnose and treat Physiology of brain is not well understood Neuroimaging can explain the brain physiology Several types of neuroimaging EEG MRI fmri

6 Linear Regression Tensorial Data Analysis Brain Imaging Data Analysis using Regression Goal is to find association between brain images and clinical outcomes. Formulate as regression problem clinical outcome as response brain image (multi-dimensional array) as tensor predictor

7 Limitation of Classical Regression Linear Regression Tensorial Data Analysis Naive approach: turning an image array as vector predictor e.g. a fmri image: 4D array with size yields a huge number of parameters (167 millions!) ignores spatial and temporal correlation New method: treat each fmri observation as one tensor predictor in regression model

8 What is Tensor? Background Definition Tensor Operation Tensor Decomposition

9 What is Tensor? con t Definition Tensor Operation Tensor Decomposition

10 What is Tensor? con t Definition Tensor Operation Tensor Decomposition A tensor is formally denoted as X R I 1 I 2... I N generalization of vector and matrix represented as multi-dimensional array

11 Fibers Background Definition Tensor Operation Tensor Decomposition

12 Slices Background Definition Tensor Operation Tensor Decomposition

13 Matricization (Unfolding) Definition Tensor Operation Tensor Decomposition Convert a tensor to a matrix Tube fibers are rearranged into the columns of a matrix

14 Matricization (Unfolding) Example Definition Tensor Operation Tensor Decomposition

15 The n-mode Multiplication Definition Tensor Operation Tensor Decomposition Let X R I J K, B R M J, the 2-mode product of X with B is defined by Y = X 2 B R I M K Elementwise y imk = j x ijk b mj In matrix form Y (2) = BX (2)

16 Definition Tensor Operation Tensor Decomposition The n-mode Multiplication Example

17 Rank-1 Tensor Background Definition Tensor Operation Tensor Decomposition 3-way outer product X = a b c Elementwise x ijk = a i b j c k

18 Definition Tensor Operation Tensor Decomposition CANDECOMP/PARAFAC Decomposition R X λ r a r b r c r r=1

19 Definition Tensor Operation Tensor Decomposition CANDECOMP/PARAFAC Decomposition con t Define factor matrix A R I R, B R J R and C R K R R X λ r a r b r c r [λ; A, B, C] r=1 R x ijk λ r a ir b jr c kr r=1

20 Tucker decomposition Definition Tensor Operation Tensor Decomposition Defined by factor matrix A R I R, B R J S and C R K T, and core tensor G R R S T X G 1 A 2 B 3 C [G; A, B, C] R S T x ijk = g rst a ir b js c kt r=1 r=1 r=1

21 Generalized Linear Regression Model Model Attention Deficit Hyperactivity Disorder Data Analysis The standard linear regression model x R p, y = β T x + α + ε, ε N (0, σ 2 ) can be written where µ = E(Y x) µ = β T x + α y N (µ, σ 2 ) A generalized linear regression model (GLM) extends this to g(µ) = β T x + α y EF(µ, φ) EF(µ, φ) is any exponential family distribution (e.g. Normal, Poisson, Binomial) g( ) is any smooth monotonic link function β T x + α(= η) is the linear predictor

22 Generalized Linear Regression Model con t Model Attention Deficit Hyperactivity Disorder Data Analysis In classical GLM Y belongs to an exponential family with PMF yθ b(θ) p(y θ, φ) = exp{ + c(y, φ)} a(φ) The GLM relates x R p to the mean µ = E(Y x) by g(µ) = η = α + β T x The GLM for the matrix predictor X given by g(µ) = η = α + γ T z + β T 1 Xβ 2

23 Model Attention Deficit Hyperactivity Disorder Data Analysis Generalized Linear Regression Model for Tensor Predictor The GLM with the systematic part for tensor predictor given by g(µ) = η = α + γ T z+ < B, X > D-dimensional tensor predictor X R p 1 p D D-dimensional coefficient tensor B R p 1 p D B has D d=1 p d parameters, which is ultrahigh dimensional and far exceeds sample size

24 Generalized Linear CP Tensor Regression Model Attention Deficit Hyperactivity Disorder Data Analysis Univariate outcome Y belongs to exponential family Tensor covariate X R p 1 p D Assume coefficient tensor B has a rank-r decomposition [B 1,..., B D ] where B d R p d R Generalized linear CP tensor regression model (Zhou et al. 2013) with the systematic part given by g(µ) = η = α + γ T z+ < R r=1 β (r) 1 β (r) D, X > = α + γ T z+ < (B D B 1 )1 R, vec(x ) >

25 Model Attention Deficit Hyperactivity Disorder Data Analysis Generalized Linear CP Tensor Regression con t Generalized linear CP tensor regression model given by g(µ) = η = α + γ T z+ < (B D B 1 )1 R, vec(x ) > substantial reduction in dimensionality to the scale of R D d=1 p d e.g For a 128-by-128-by-128 MRI image, the dimensionality reduce from 2,097,157 to 1157 using rank-3 decomposition Zhou et al.(2013) showed that this low rank tensor model could provide a sound recovery of many low rank signals

26 Estimation Background Model Attention Deficit Hyperactivity Disorder Data Analysis Given n iid data {(y i, X i, z i ), i = 1,..., n} the log-likelihood l(α, γ, B 1,..., B D ) = n i=1 y i θ b(θ) a(φ) + n c(y i, φ) i=1 find the parameters (α, γ, B 1,..., B D ) that maximizes this function

27 Estimation con t Background Model Attention Deficit Hyperactivity Disorder Data Analysis Generalized linear CP tensor regression model given by g(µ) = η = α + γ T z+ < (B D B 1 )1 R, vec(x ) > A key observation is although g(µ) is not linear in (B 1,..., B D ) jointly, it is linear in each B d separately When updating B d R p d R, the inner product part can be written as < B d, X (d) (B D B d+1 B d 1 B 1 ) > this yields the block relaxation algorithm, which converges to a stationary point

28 Sparsity Regularization Model Attention Deficit Hyperactivity Disorder Data Analysis Maximize a regularized log-likelihood function l(α, γ, B 1,..., B D ) D R p d d=1 r=1 i=1 P λ ( β (r) di, ρ) scalar penalty function P λ ( β, ρ) power family P λ ( x, ρ) = ρ β λ, λ (0, 2] in particular lasso (λ = 1)

29 ADHD-200 Data Results Model Attention Deficit Hyperactivity Disorder Data Analysis [taken from (Zhou et al. 2013)]

30 Background Extending the linear CP/Tucker tensor regression model to the linear H-Tucker tensor regression model like CP model, the number of parameters is free from exponential dependence on D preserve the flexibility of Tucker model Comparing the performance of different tensor regression models (Tucker, H-Tucker) when applying different regularization approaches (sparsity regularization, trace norm regularization)

31 con t Finding the appropriate model and algorithm to address the multi-block tensor regression problems Combing the kernel concept and partial least squares (PLS) techniques to deal with tensor (multi-block tensor) regression problem Applying tensor regression approaches listed above to the applications such as neuroimaing data analysis, brain signal data analysis to test if the improved performance can be achieved