GENERALIZED STATISTICAL METHODS FOR UNSUPERVISED MINORITY CLASS DETECTION IN MIXED DATA SETS. Uwe F. Mayer

Transcription

1 GENERALIZED STATISTICAL METHODS FOR UNSUPERVISED MINORITY CLASS DETECTION IN MIXED DATA SETS Cécile Levasseu Jacobs School of Engineeing Univesity of Califonia, San Diego La Jolla, CA, USA Uwe F. Maye Fai Isaac Copoation San Diego, CA, USA Bandon Budge, Ken Keutz-Delgado Jacobs School of Engineeing Univesity of Califonia, San Diego La Jolla, CA, USA ABSTRACT Minoity class detection is the poblem of detecting the occuence of ae ey events diffeing fom the majoity of a data set. This pape consides the poblem of unsupevised minoity class detection fo multidimensional data that ae highly nongaussian, mixed continuous and/o discete), noisy, and nonlinealy elated, such as occus, fo example, in faud detection in typical financial data. A statistical modeling appoach is poposed which is a subclass of gaphical model techniques. It exploits the popeties of exponential family distibutions and genealizes techniques fom classical linea statistics into a famewo efeed to as Genealized Linea Statistics GLS). The methodology exploits the split between the data space and the paamete space fo exponential family distibutions and solves a nonlinea poblem by using classical linea statistical tools applied to data that has been mapped into the paamete space. A faud detection technique utilizing low-dimensional infomation leaned by using an Iteatively Reweighted Least Squaes IRLS) based appoach to GLS is poposed in the paamete space fo data of mixed type. ROC cuves fo an initial simulation on synthetic data ae pesented, which gives pedictions fo esults on actual financial data sets. Index Tems Minoity class detection, genealized linea models, exponential family distibutions, gaphical models, dimensionality eduction. 1. INTRODUCTION Minoity class detection consides a binay class situation whee a minoity class is disciminated fom a majoity class. It aims at diffeentiating ae ey events belonging to the minoity class fom the emainde of the data belonging to the majoity class. Many impotant is assessment system applications depend on the ability to accuately detect the occuence of ae ey events given a lage data set of obsevations. This poblem aises in dug discovey Do the molecula desciptos associated with nown dugs suggest that a new, candidate dug will have low toxicity and high effectiveness? ); and health cae Do the desciptos associated with a medical docto pofessional behavio suggest that he/she is an outlie in the categoy he/she was assigned to? ). The wo poposed hee is specifically concened with the poblem of minoity class detection; fo example, in cedit cad faud detection Given the data fo a lage set of cedit cad uses does the usage patten of this paticula cad indicate that it might have been stolen? ). In many domains, no o little a pioi nowledge exists egading the tue souces of any causal elationships that may occu between vaiables of inteest. In these situations, meaningful infomation egading the ey events must be extacted fom the data itself. The poblem of unsupevised data-diven minoity class ae event) detection is one of elating popety desciptos of a lage unlabeled database of objects to measued popeties of these objects, and then using these empiically detemined elationships to detect the popeties of new objects. Hee, the ultimate goal is to coectly chaacteize the new objects as eithe belonging to the minoity class o not. This wo assumes that minoity class and majoity class objects constitute two distinct, well-sepaated classes of objects in a latent vaiable space the paamete space ) to be defined below. In the case of a ae occuence of objects to be detected, as is typically the case in cedit cad faud detection, thee is the belief that modeling the total unlabeled database allows one to discen the statistical stuctue of the majoity class of objects. This wo consides measued object popeties that ae nongaussian, mixed compised of continuous and discete data), vey noisy, and highly nonlinealy elated fo which the esulting minoity class detection poblem is vey difficult. The difficulties ae futhe compounded because the descipto space is of high dimension. Many of the classical tools fo unsupevised featue extaction and analysis such as Pinciple Components Analysis PCA), Independent Component Analysis ICA) and classical Facto Analysis FA) ae all tied togethe by shaing a common geneal diected gaph stuctue, and diffe only in

2 cetain assumptions about the type discete o continuous) of latent vaiables and the fom of node pobability distibutions [1]. One of the ey assumptions made by these appoaches is that the components of the obseved node all shae the same fom of conditional pobability distibution. In contast, the poposed appoach allows fo the components to have diffeing paametic foms; using exponential family distibutions the components can model a lage vaiety of mixed data types. A ey aspect of ou method, efeed to as Genealized Linea Statistics GLS), is that the paamete of the exponential family distibutions is constained to a lowe dimensional latent vaiable subspace. This models the belief that the intinsic dimensionality of the data is smalle than the obseved dimensionality of the data space. The unsupevised minoity class detection technique poposed hee is pefomed in the paamete space athe than in the data space as done in moe classical appoaches. As an example, a synthetic data set is investigated, whee a single latent vaiable subspace is leaned by using the GLS based statistical modeling on an unlabeled taining set. Given a new data point, that point is pojected to its image in the paamete space on the leaned subspace and minoity class detection is pefomed by compaing its distance fom the taining set mean-image to a theshold. The pesented example shows that thee ae domains fo which the classical linea techniques, such as PCA, used in the data space pefom fa fom optimal compaed to the new poposed paamete space techniques. ROC cuves ae geneated to assess the pefomance of the poposed minoity class detection method. 2. GENERALIZED LINEAR STATISTICS GLS) The poposed statistical modeling appoach is a subclass of gaphical model techniques. It is a genealization and amalgamation of techniques fom classical linea statistics, Logistic Regession, Pincipal Component Analysis PCA), latent vaiable analysis and Genealized Linea Models GLMs) as well as ou pevious wo [2] into a unified famewo we efe to analogously to GLMs theoy) as Genealized Linea Statistics GLS). This is actually a nonlinea methodology which exploits the split that occus fo exponential family distibutions between the data space also nown as the expected value space) and the natual) paamete space as soon as one leaves the domain of puely Gaussian andom vaiables. The point is that although the poblem is now nonlinea, it can be attaced by using classical linea and othe standad) statistical tools applied to data which has been mapped into the paamete space, which still has a natual, flat Euclidean space stuctue. Fo example, in the paamete space one can pefom egession esulting in the technique of logistic egession and othe GLMs methods [3]), PCA esulting in a vaiety of genealized PCA methods [4]), o clusteing [5]. This famewo is used to develop algoithms capable of minoity class detection in domains involving highly heteogeneous data types and unlabeled data sets. Specifically, this wo consides data ecods which have both continuous e.g., exponential and Gaussian) and discete e.g., count and binay) components. It focusses on the development of unsupevised minoity class detection algoithms which can be tained using unlabeled taining data sets. The unsupevised case is vey difficult and taes one out of the domain of the standad supevised appoaches, such as neual netwos. To motivate theoetical developments, a geneal gaphical model fo hidden vaiables is consideed, cf. Fig. 1. The ow vecto x = [x 1,..., x d ] R d consists of obseved featues in a d-dimensional space. categoical o count data) and continuous values. Following the pobabilistic Genealized Latent Vaiable GLV) fomalism descibed in [6], it is assumed that taining points can be dawn fom populations having classconditional pobability density functions, px θ) = p 1 x 1 θ 1 )... p d x d θ d ), 1) whee, when conditioned on the andom paamete vecto θ = [θ 1,..., θ d ] R d, the components of x ae independent. It is futhe assumed that θ can be witten as θ = av + b 2) with the hidden o latent vaiable a = [a 1,..., a q ] R q andom with q < d and ideally q d), V R q d and b R d deteministic. Conditioning on the andom vecto θ is equivalent to conditioning on the low-dimensional andom vecto a. In a pobabilistic sense, all of the infomation which is mutually contained in the data vecto x must be contained in the latent vaiable a. As noted in [6], equations 1) and 2) genealize the classical facto analysis model to the case when the maginal densities p i x i θ i ) ae nongaussian. Indeed, the subscipt i on p i ) seves to indicate that the maginal densities can all be diffeent, allowing fo the possibility of x containing categoical, discete, and continuous valued components. It is futhe assumed that the maginal densities ae each one- a 1 a 2 a q 3 x 1 x 2 x 3 x d Fig. 1. Gaphical model fo GLS paamete exponential family densities, allowing the use of a ich and poweful theoy of such densities to be utilized, and it is commonly the case that θ i is taen to be the natual paamete o some simple bijective function of it) of the exponential family density p i. Hence, each component density p i x i θ i ) in 1) fo x i X i, i = 1,..., d, is of the fom px i θ i ) = exp θ i x i Gθ i ) ), 3)

3 whee G ) is the cumulant geneating function defined as Gθ i ) = log exp ) θ i x i νdxi ), X i with ν ) a σ-finite measue that geneates the exponential family. It can be shown, using Fubini s theoem [1], that Gθ) = d i=1 Gθ i). In both the GLV theoy descibed in [6] and the andomand Mixed-effects Genealized Linea Models MGLMs) liteatue [3], V and b ae deteministic while a and hence θ) is teated as a andom vecto. The diffeence is that in GLV, all of the quantities V, b, and a ae unnown, and hence need to be identified, wheeas in MGLMs, V is a nown matix of egesso vaiables and only the deteministic vecto b and the unnown ealizations of the andom effect vecto a must be estimated. In both GLV and MGLMs, it is assumed that in the θ-paamete space the linea elationship 2) holds and at least conceptually) that the tools of linea and statistical invese theoy ae applicable o insightful. The MGLMs theoy is a genealization of the classical theoy of linea egession, while the GLV theoy is a genealization of the classical theoy of facto analysis and PCA. In both cases the genealization is based on a move fom the data/desciption space containing the measuement vecto x to the paamete space containing θ via a geneally nonlinea tansfomation nown as a lin function [3]), and it is in the latte space that the linea elationship 2) is assumed to hold. Because both the Genealized Linea Models GLMs) and the Genealized Latent Vaiable GLV) methodologies exploit the linea stuctue 2), they can be viewed as special cases of a Genealized Linea Statistics GLS) appoach to data analysis. With the obsevational conditional distibutions descibed, attention tuns to the maginal distibution of paent nodes a 1,..., a q. Fo motivation, note that the nonconditional) density px) equies a geneally intactable integation ove the paametes, px) = px θ)πθ)dθ = d p i x i θ i )πθ)dθ, 4) i=1 whee πθ) is the pobability density function of θ = av+b. Given the obsevation matix X = [ x[1] T,..., x[n] ] T T in R n d composed of n iid statistical samples, each assumed to be stochastically equivalent to the andom ow vecto x, px) = px[]) = d p i x i [] θ i )πθ)dθ 5) i=1 with θ = av + b. Fo specified exponential family densities p i ), i = 1,..., d, maximum lielihood identification of the model 4) coesponds to identifying πθ), which, unde the condition θ = av + b, coesponds to identifying the matix V, the vecto b, and a density function, µa), on a via a maximization of the data lielihood function px) with espect to V, b, and µa). This is geneally a quite difficult poblem [3] and is usually attaced using appoximation methods which coespond to eplacing the integal in 4)/5) by a sum [7]: d px) = px θ j )π j = p i x i θ j,i )π j 6) px) = j=1 j=1 i=1 j=1 i=1 d p i x i [] θ j,i )π j 7) ove a finite numbe of suppot points θ j, equivalently, āj ), j = 1,..., m, with point-mass pobabilities π j πθ = θ j ) = πa = āj ). As clealy descibed in [7], this appoximation is justified eithe as a Gaussian quadatue appoximation to the integal in 5) [3] o by appealing to the fact that the NonPaametic Maximum Lielihood NPML) estimate of the mixtue density πθ) yields a solution which taes a finite numbe m) of points of suppot [5, 8]. With θ = av + b, with V, b fixed and a andom, the lielihood function 6) is equal to px) = px θ j )π j = px āj V + b)π j, 8) j=1 j=1 and the data lielihood function 7) is equal to px) = j=1 px[] āj V + b)π j. 9) The combined poblem of maximum lielihood estimation MLE) of the paametes V, b, the suppot points āj and the point-mass pobability estimates π j, j = 1,..., m, as appoximations to the unnown, and possibly continuous density µa)) is nown as the NPML estimation poblem [8]. It can be attaced by using the Expectation-Maximization EM) algoithm [6] as done in [5], o, as done in [4], by simply consideing the special case of unifom point-mass pobabilities, i.e., π j = 1/m fo j = 1,..., m, fo which the numbe of suppot points equals the numbe of data samples, i.e., m = n. The goal hee is to fit a pobability model of the fom 9) using exponential family densities to labeled when available) o unlabeled data to develop algoithms fo deciding if a new measuement belongs to the minoity class o not. Fo example, if an adequate fit of a paameteized pobability distibution has only been found to the single, labeled majoity class, the question is whethe the new data point fits well with this distibution o whethe it should be flagged as a potential membe of the minoity class. Altenatively, if classconditional distibutions can be fitted to minoity and majoity class labeled data, a Bayes-optimal lielihood atio test can

4 be computed [9]. Class-conditional density-based tests can be equivalently posed as disciminant functions which ae functions of sufficient statistics of the densities when they exist) and which, in tun, define decision sufaces in featue space. Of couse, the most difficult situation aises when the taining samples ae unlabeled. Howeve, even in this case, sometimes the single-class model can still be effective fo minoity class detection. Fo example, if the atio of minoity class data to majoity class data is vey small, then the unlabeled data points ae appoximately distibuted lie the majoity class data, and the simple single-class model might be effectively assumed and utilized. This condition can be satisfied in pactice; faudulent cedit cad tansactions ae typically appoximately one tenth of one pecent of all tansactions. 3. MINORITY CLASS DETECTION The minoity class detection technique poposed hee is pefomed in the paamete space athe than in the data space as done in moe classical appoaches, and exploits the low dimensional infomation povided by the latent vaiables ā j, j = 1,..., m. The poposed technique consides the special case of unifom point-mass pobabilities, π j = 1/m fo j = 1,..., m, fo which the numbe of suppot points equals the numbe of data samples, i.e., m = n. Hence, the pointmass pobabilities do not need to be estimated and the EM algoithm is unnecessay. Then, to each vecto x coesponds a vecto ā and they can shae the same index = 1,..., n. Fo sae of simplicity, the b-tem in 2) is absobed in the standad manne into the matix V using the homogenous coodinates. The simultaneous estimation of the paametes V R q d and A = [ ā T 1,..., ] ā T T n R n q is pefomed by minimizing the negative log-lielihood function. Using 9), the loss function is expessed as LV, A) = log px) = log px[] ā V) = Gā V) ā Vx[] T } = using the exponential family definition in 3). Lā, V), 1) It can be shown that the loss function 1) is convex in eithe of its aguments with the othes fixed [4]. Hence, its minimization is attaced by using an iteative appoach. The Newton-Raphson method is used fo the iteative minimization. The fist step in the l + 1) th iteation consists of the update A l+1) = ag min A LA, V l) ), with V l) the update obtained at the end of the l th iteation. The Newton-Raphson technique solves this poblem by using the update ā l+1) =āl) αl+1) a 2 al ā l), Vl))) 1 a L 11) ā l), Vl)) fo = 1,..., n, whee L ) is the gadient of the function L ), 2 L ) its Hessian matix and α l+1) the so-called step size. It is easily shown that, fo = 1,..., n, a L ā l), Vl)) = V l) G ā l) Vl)) x[] T ), whee ) G ā V l)) = G θ θ Futhemoe, fo = 1,..., n, θ = ā V l), θ ā = V l). 2 al ā l), Vl)) = V l) G ā l) Vl)) V l),t, whee G ā l) Vl)) is a d d)-diagonal matix with the diagonal tems equal 2 G θ )/ θ,i 2, i = 1,..., d. Note that the diagonal stuctue of G ) is exact and not an appoximation.) Similaly, the second step in the iteative minimization method consists of the update V l+1) = ag min V LA l+1), V). This update taes the fom v l+1) =v l) fo = 1,..., q, whee v L ā l+1), V l)) = 2 vl ā l+1), V l)) = α v l+1) 2 vl ā l+1), V l))) 1 v L ā l+1), V l)) 12) āl+1) G ā l+1) V l)) x[] T }, ā l+1) ) 2G ā l+1) V l)). Fo exponential family distibutions and canonical lin functions, it can be shown that the update equations coespond to nomal equations in a least squaes envionment [11]. Theefoe the minimization poblem coesponds to an Iteatively Reweighted Least Squaes IRLS) algoithm. developed fo a lage set of exponential family distibutions Gaussian, exponential fo the continuous distibutions, Benoulli, binomial and Poisson fo the discete distibutions). IRLS-based iteative updates exist fo a lage set of exponential family distibutions, including Gaussian, exponential, Benoulli, binomial and Poisson Positivity constaints Fo the exponential and invese Gaussian distibutions, an additional positivity constaint on the natual paamete values has to be taen into account in ode to fully comply with thei definition. Thee altenative ways to deal with the positivity constaint ae: 1) the use of Lagange multiplies and Kuhn-Tuce theoy; 2) the use of penalty functions; and 3) the use of a non-canonical lin function which enables one to

5 wo in an unconstained paamete space. The latte option is investigated. Hee, the non-canonical lin function is chosen to be the composition of the canonical lin function with the absolute value function, and the loss function becomes: LV, A) = G ā V ) + ā V x[] T }. The iteative minimization algoithm based on IRLS is then used on LV, A) as done peviously on LV, A) Mixed data The case of hybid o mixed data occus fo a poblem in which diffeent types of distibutions can be used fo diffeent desciptos. Fo simplicity of pesentation, two types of exponential family distibution, p 1) and p 2), ae discussed hee. The matix of obsevations becomes X = X 1) X 2) ), the paametes matix Θ = AV = Θ 1) Θ 2) ), the lowe dimensional subspace basis matix V = V 1) V 2) ). Howeve, the matix of pincipal components A emains common to both Θ 1) and Θ 2). Then, the loss function 1) taes the following fom: LV, A) = L 1) V 1), A) + L 2) V 2), A) = G 1) ā V 1)) ā V 1)) } x 1) [] T + G 2) ā V 2)) ā V 2)) x 2) [] T }. 13) As done peviously the loss 13) is minimized using the Newton-Raphson appoach. In ode to avoid confusion, the step supescipts l) and l+1) ae not bold wheeas the mixtue supescipts 1) and 2) ae. Fo the fist step, the update equations fo = 1,..., n ae: ā l+1),t α l+1) a = āl),t V 1)l) G 1) ā l) V1)l)) V 1)l),T + V 2)l) G 2) } ā l) V2)l)) V 2)l),T 1 V 1)l) G 1) ) ā l) V1)l)) x[] 1),T + V 2)l) G 2) ā l) V2)l)) x[] 2),T )}. Fo the second step, the two sets of ow vectos v 1) } q =1 } q =1 and v 2) ae updated sepaately. Fo the sae of simplicity, } the following deivations ae made fo the set q v indistinctively of the mixed data supescipt. The =1 update equations can then be used fo v 1) } q and =1 v 2) } q =1 by changing v to v 1), espectively to v 2), G ), G ), and G ) to G 1) ), G 1) ), and G 1) ), espectively to G 2) ), G 2) ), and G 2) ). Then, the update equations ae as follows fo = 1,..., q: v l+1),t = v l),t α l+1) v n n āl+1), ā l+1), ) 2G ) 1 ā l+1) V l)) G ā l+1) V l)) x[] T } This appoach can be natually genealized to any numbe s of exponential family distibutions, esulting in a single update equation fo A and s independent update equations fo V Minoity class detection algoithm The minoity class detection technique using the IRLS-based leaning algoithm in the paamete space wos as follows: fist, given the taining set x[]} n, we lean the diection of pojection in the paamete space, namely V, by using the IRLS-based iteative algoithm, and compute the taining set mean-image in the paamete space, namely 1 n n ā. Then, the new data point is moved fom the data space to the paamete space using the lin function. We poject the obtained point onto the leaned diection of pojection and compute its distance to the taining set mean-image. Finally, we compae the obtained distance to a given theshold to mae a decision. If the distance is geate than the theshold, then the new point is declaed to belong to the minoity class, othewise it is declaed to belong to the majoity class. 4. SIMULATION RESULTS The IRLS-based leaning algoithm has been implemented fo a dictionay of exponential family distibutions: Gaussian, exponential distibution fo continuous data, Benoulli, binomial and Poisson distibution fo discete/count data. Fig. 2 below shows an example of synthetic theedimensional mixed data d = 3), with each data sample compised of a binomial component with values between and 5, an exponential distibution component, and a Gaussian component. The data ae geneated by two diffeent classes, a minoity and a majoity one, and fo each class the paametes ae assumed to be constained to lie on a diffeent) onedimensional subspace of the paamete space q = 1). To assess the unsupevised minoity detection pefomance, we conside a situation whee the minoity class is a ae occuence 1 pecent of 1, taining points), and we pefom the detection algoithm descibed above. The poposed technique is compaed to classical PCA used in the data space with a theshold test pefomed on new data pojected along the fist pincipal axis, as well as to a supevised Bayes minimum ate) detecto fo the sae of an optimal benchma. ).

6 Fig. 3 shows a compaison between the supevised Bayes detecto, the minoity class detecto based on the utilization of the poposed algoithm to pefom detection in the paamete space, and the minoity class detecto based on classical PCA. This illuminating example shows that thee ae domains fo which classical PCA pefoms fa fom optimal Expectation Paamete Space An Iteatively Reweighted Least Squaes algoithm was pesented fo leaning distibution paametes of obseved nodes, as well as a nonpaametic density estimation method fo hidden nodes. Detection was then pefomed using disciminant thesholding in the paamete space, instead of the data space as in taditional methods. In contast to classical methods, the poposed method allows fo each data component to have its own paametic fom and enables unsupevised minoity class detection in the case of a ae occuence of minoity class objects. Initial esults on synthetic data ae encouaging, and they allow fo the pediction of quality esults on financial data sets which is now undeway. Futhemoe, the possibility of utilizing novel hybid detection techniques that wo patially in data space and patially in paamete space is being investigated Fig. 2. Data samples of a 3-dimensional data having binomial, exponential and Gaussian components cicles fo one class and squaes fo the othe class). Pobability of detection, P D Pobability of false alam, P FA Fig. 3. Compaison of supevised Bayes optimal top with pentagams), poposed GLS technique middle with squaes) and classical PCA bottom with cicles) ROC cuves. 5. CONCLUSION A gaphical model appoach fo minoity class detection in an unsupevised leaning context fo data of mixed type was poposed and efeed to as Genealized Linea Statistics GLS) REFERENCES [1] M. I. Jodan and T. J. Sejnowsi, Gaphical Models: Foundations of Neual Computation, MIT Pess, 21). [2] C. Levasseu, K. Keutz-Delgado, U. Maye and G. Gancaz, Data-patten discovey methods fo detection in nongaussian high-dimensional data sets, Asiloma Confeence on Signals, Systems and Computes, 25. [3] C.E. McCulloch, Genealized Linea Mixed Models, Institute of Mathematical Statistics, 23). [4] M. Collins, S. Dasgupta and R. Shapie, A genealization of pincipal component analysis to the exponential family, Neual Infomation Pocessing Systems, 21. [5] Sajama and A. Olitsy, Semi-paametic exponential family PCA, Neual Infomation Pocessing Systems, 24. [6] D. J. Batholomew and M. Knott, Latent Vaiable Models and Facto Analysis, Oxfod Univesity Pess, 2nd edition, 1999). [7] M. Aitin, A maximum lielihood analysis of vaiable components in genealized linea models, Biometics, 55, 1999, [8] B. G. Lindsay, Mixtue Models: Theoy, Geomety, and Applications, Institute of Mathematical Statistics, 1995). [9] R. O. Duda, P. E. Hat and D. G. Sto, Patten Classification, John Wiley, 2nd edition, 21). [1] E. L. Lehmann and G. Castella, Theoy of Point Estimation, Spinge, 2nd edition, 1998). [11] L. Fahmei and G. Tutz, Multivaiate Statistical Modelling Based on Genealized Linea Models, Spinge- Velag, 2nd edition, 21).