LOGISTIC REGRESSION IN DEPRESSION CLASSIFICATION

Transcription

1 LOGISIC REGRESSIO I DEPRESSIO CLASSIFICAIO J. Kual,. V. ran, M. Bareš KSE, FJFI, CVU v Praze PCP, CS, 3LF UK v Praze Abstrat Well nown logisti regression and the other binary response models an be used in the area of statistis, biomedial data analysis and artifiial intelligene. Beginning with basi properties of proposed model, we use lielihood ratio testing as a tool for the pruning of omplete model to produe the best sub-model. We are foused to optimum and automated pruning whih is an alternative to traditional stepwise logisti regression. he general methodology of logisti regression with optimum pruning was developed together with a library of funtions in the Matlab environment. he library was used to the analysis of neuropsyhologial and biomedial data related to the systemati researh of resistant depressions. he aim of the paper is in the seletion of optimum sub-model (set of patient properties) whih will help with the deision whether the patient has resistant or non-resistant form of the depression. Introdution here are many statistial methods whih seem to be also powerful tools in the area of artifiial intelligene (luster analysis, linear and nonlinear regression, ernel methods et.). One of them is logisti regression [4] and its generalization to binary response index model [3]. he nonlinearities of these models are very similar to models of artifiial neurons. So, the logisti regression and the testing of sub-models an be useful in designing of multilayer pereptron (MLP) networs. he model and sub-model differene an be useful for the deision, whether given input or a group of inputs plays signifiant role in hierarhial deision proess inside A. hus, statistial testing based on lielihood ratio (LR) an help to eliminate redundant weights (A pruning) or to generate hidden layer of MLP. Binary Response Index Model We suppose a model [3] with m real inputs x and single binary output y in the form y = h( x β + e) () where for z > 0 h( z ) = () 0 for z 0 is Heaviside s unit step funtion, x, β R m+, x 0 =, e is a ontinuous random variable with positive and symmetri probability density funtion g(z) around zero, of ourse. Its umulative distribution funtion is z = g( u) du (3) So, the output y is also of stohasti nature and it an be desribed via probability p( x ) = prob( y = x) = prob( x β + e > 0) = prob( e > x β) = G( x β) = G( x β) (4)

2 whih is well nown formula [3, 4] for binary model related to logisti regression. here is neessary to satisfy 0 < G(z) < for all real arguments z. Fortunately, it implies from g(z) > 0 everywhere, whih was delared above. 3 Speial Cases of Binary Model Binary model was first published by Bliss [] in 934 as probit model with nonlinearity z = Φ ( z) = π u exp du and orresponding density funtion of standard normal distribution as (5) u g( z ) = exp (6) π Lately in 944, Berson [] published logit model (logisti regression) with nonlinearity = + (7) and orresponding density funtion of logisti distribution as g( z) = (8) ( + ) his model is frequently used in many appliations. he logit model approahes its asymptotes less rapidly then probit one. he other models are also possible to use. he density of Cauhy (t ) distribution g( z) = π( + z ) (9) generates the nonlinearity = + artan z (0) π with several amazing properties orresponding to heavy tail effet of Cauhy distribution. he variety of nonlinear harateristis an help us to hoose a binary model and its parameters with the best possible quality of fitting. 4 Parameter Estimation he estimation of model parameters is frequently performed via maximization of lielihood funtion or its logarithm respetively. Resulting point estimate (if exists) has a very good asymptoti properties [3, 4]. Let be number of observations and (x, y ) be individual observation for =,,. hus, the density of y for individual x is y [ G( x β) ] [ G( x )] y f( y x, β) = β () he logarithmi lielihood funtion over all observations is defined as ( y log G( x β) + ( y )log( G( x ))) = L( β ) = β ()

3 he point estimate b of parameter vetor is obtainable via maximization of objetive funtion L on losed onvex domain D as b = βˆ arg max L( β) (3) β D he existene but not uniqueness of the estimate b is guaranteed in this ase. Replaing D by opened set R n will ause problems when the observations are separable, whih is ideal ase for lassifier tuning but not for parameter estimation. he analysis of asymptoti variane begins with matrix g( x b) x x U = (4) G( x b)) = G( x b)( When the matrix U is regular, it is also positive definite and the asymptoti variane of estimate b is Avar( b ) = V = U (5) he asymptoti standard error of estimate b is Astd( b ) = / = s diag( V ) (6) and orresponding approximate 95% CI (onfidene interval) is β b.96s, b +. 96s (7) he onfidene interval is rather useful for final report then for signifiane testing due to its sensitivity to singularity of matrix U. 5 Hypothesis esting here are many approahes to binary model testing. One of them is lielihood ratio (LR) test. It is based on the omparison of given model and its sub-models. Let r = (, r (8) m+,..., r m ) {0, } be seletion vetor, whih desribes, whether adequate omponents of vetor x will be used in the model. he vetor r deomposes the vetor x to two parts: ative input vetor u R K + and eliminated input vetor v R. Here, K is the number of real inputs (exluding x 0 ), is the number of eliminated inputs, K, 0, K + = m. Analogial deomposition of parameter vetor β omes to vetors µ R K +, η R. When = 0, then K= m and we obtain omplete model as p( x ) = G( x β) = G( u µ + v η) (9) with optimum log-lielihood value L ompl. Anyway, > 0, then K < m and we obtain restrited model as sub-model in the form p( u ) = G( u µ ) (0) with optimum log-lielihood value L restr. he traditional lielihood ratio test (LR-test) is about hypothesis : η = 0 against alternative H A : η 0. he testing riterion LR = ( L ompl L ) () restr has limiting distribution χ with degrees of freedom. he adequate p value is diretly obtained as H 0

4 p value = F ( LR) () where F is umulative distribution funtion of hi-squared distribution. he omparison of the model and its sub-model via LR-test brings a view on the signifiane of the model and its parameters. here are two useful partiular ases. When K = 0, then = m and we obtain onstant model as p( u ) = G( µ 0) = p (3) Maximum lielihood estimation proedure (3) omes to trivial estimate p = y = Log-lielihood value of this onstant model is L onst ( p log p + ( p )log( p )) = (5) Comparing omplete model with onstant one, we an measure the model signifiane (its quality) via probability p = Fm (( L )) (6) 0 ompl Lonst Lower value of p 0 indiates the higher signifiane of given omplete model and various models an be ordered aording to p 0. In the seond ase, we ompare the omplete model and a sub-model with eliminated th input, where > 0. So, we have K = m, = and adequate log-lielihood value an be denoted as L. We an easily measure the signifiane of th input, in the meaning of hypothesis H 0 : β = 0 against alternative H A : β 0, via probability p = F (( L L )) (7) ompl Stepwise strategy of parameter elimination is frequently used to obtain the model, whih every parameter is signifiant in the meaning of p < α, together with p 0 < α. If the model is not unique, we prefer the model with minimum value of p 0. (4) 6 Biomedial Appliation he theory of optimum logistis sub-model ame to the realization of library in the Matlab environment. hen we used the library to the analysis of biomedial data about resistant depressions. he data onsists of 4 patients with 93 properties. hree properties are output ones (positive response to treatment of two wees, four wees and total response). Seven properties are based on EEG ordane measurement whih is very effiient indiator for responder-resistant deision. he rest of 83 properties are basi, biohemial, psyhologial and psyhiatri harateristis of patients. he set of 4 patients onsists of 89 total responders and 5 resistants and it was analyzed via logisti regression with optimum pruning tehnique of the best sub-model finding. he best sub-model is mainly oriented to the ordane based properties and will be used for patient lassifiation. 7 Conlusion Logisti regression was employed to form the best sub-models in the ase of logisti regression. he general methodology of logisti regression with optimum pruning was developed together with a library of funtions in the Matlab environment. he library was used to the analysis of neuropsyhologial and biomedial data related to the systemati researh of resistant depressions. he researh will help with the deision whether the patient has resistant or non-resistant form of the depression.

5 Anowledgement he paper was reated under the support of grant OHK4-65/ CU in Prague. Referenes [] Bliss C.I., he method of probits, Siene, Vol.79, o.037, 934, pp [] Berson J., Appliation of the logisti funtion to bio-assay, J Am Stat Asso, Vol.39, 944, pp [3] Wooldridge J. M., Eonometri Analysis of Cross Setion and Panel Data, Cambridge, MA: MI Press, 00 [4] Hilbe J. M., Logisti Regression Models, Chapman & Hall/CRC Press, 009 Author ontats: jaromir.ual@fjfi.vut.z tran@vse.z bares@pp.lf3.uni.z