UNSUPERVISED LEARNING WITH IMBALANCED DATA

Transcription

1 Workho track - ICLR 6 UNSUPERVISED LEARNING WITH IMBALANCED DATA VIA STRUCTURE CONSOLIDATION LATENT VARIABLE MODEL Fariba Youefi, Zhenwen Dai, Carl Henrik Ek, Neil Lawrence, Univerity of Sheffield, Sheffield, UK Inferentia Ltd, UK Univerity of Britol, Britol, UK, {f.youefi, n.lawrence}@heffield.ac.uk zhenwendai@gmail.com carlhenrik.ek@britol.ac.uk ABSTRACT Unuervied learning on imbalanced data i challenging becaue, when given imbalanced data, current model i often dominated by the major category and ignore the categorie with mall amount of data. We develo a latent variable model that can coe with imbalanced data by dividing the latent ace into a hared ace and a rivate ace. Baed on Gauian Proce Latent Variable Model, we rooe a new kernel formulation that enable the earation of latent ace and derive an efficient variational inference method. The erformance of our model i demontrated with an imbalanced medical image dataet. INTRODUCTION In many medical alication, e.g. athology, negatively labelled data i extremely eay to obtain (e.g. healthy cell). Poitive label, on the other hand, can be harder to acquire (e.g. articular dieae morhologie). Thee maively unbalanced roblem are challenging for mot algorithm becaue the negative cla tend to dominate the objective function and the reulting model erform oorly. In ractice it i often better to throw away much of the negative data and rebalance the data et. Unuervied learning ha been attracting a lot of attention a it ha the otential to erve a an underinning technology for a range of challenge uch a generative modeling, miing data imutation and coing with multile data modalitie. Unuervied learning can alo be alied to a wider range of data et, becaue it doe not rely on having carefully labelled data available. In thi aer we exlore the oibility of uing a variant unuervied learning algorithm to olve the roblem of label balance. We build latent variable model that can imultaneouly accommodate a very large number of negative examle, haring their characteritic aroriately with the oitive cla, while imultaneouly allowing the model to characterie the manner in which the oitive cla i differently characteried through reerved (or rivate) latent ace that are earately learned for each cla. The reulting model doe not uffer from the tandard challenge in thi domain. We comare with a variant of the dicriminative GP-LVM (the model that underinned GauianFace) and how ignficantly imroved erformance. Our robabilitic latent variable model divide it latent ace into a hared ace of all the categorie and a rivate ace for each category (Damianou et al., ). The hared ace account for caturing the common regularitie among categorie (e.g. oitive and negative cla) and the rivate ace i dedicated to model the variance ecific to individual categorie. Becaue the modelling of the rivate ace i category ecific, there i no domination of it characteritic by the larger category. Thu the data in each category can be modeled aroriately while the common regularitie are till exloited.

2 Workho track - ICLR 6 We imlement the idea of hared and rivate ace in the framework of Gauian Proce latent variable model (GPLVM, Lawrence, 5) by deriving a articular covariance function (kernel) that enable uch earation. We exloit cloed form variational lower bound of the log marginal likelihood of the rooed model, which to rovide an efficient aroximation inference method. The erformance of our model i evaluated with a real image dataet, in which the oitive and negative data are extremely imbalanced. We how our model till can learn from imbalanced data and erform well in both generative and dicriminative tak. STRUCTURE CONSOLIDATION LATENT VARIABLE MODEL We aume the dataet i rereented a a et of fixed length vector Y R N D, where N i the number of data oint and D i the dimenionality of individual data oint. Additionally, a label of category i aociated with each data oint, c = (c,..., c N ), c i {,..., C}, where C indicate the number of categorie in the dataet. We aim at building a robabilitic model (Y) that i robut when the number of data in different categorie are highly imbalanced. We aume the data aociated with a et of latent rereentation X R N Q, where Q i the dimenionality of the latent ace. The latent rereentation are related to the oberved data through an unknown maing function f and f follow a rior ditribution that i defined a a Gauian roce, y = f(x) ɛ, f GP(, k), () where ɛ N (, σ I) denote the obervation noie and k i the kernel function. Given the oberved data Y, we wih to obtain a a oterior etimate for both the latent rereentation X and the unknown maing function f( ). In our model we earate the latent ace into a hared ace with the dimenionality Q and a rivate ace with the dimenionality Q. Therefore, a latent rereentation can be denoted a x = [x, x ], x R Q, x R Q, where x and x are the latent rereentation in hared and rivate ace reectively. With the earated latent rereentation, we define the kernel function in our model a k((x, c x ), (x, c x )) = k (x, x ) k ((x, c x ), (x, c x )), () where k i the kernel function for the hared ace and k i the kernel function for the rivate ace. The hared kernel can be any kernel function built on a vector ace from the literature. However, the rivate kernel i defined to take the following form: k ((x, c x ), (x, c x )) = { k ((x, c x ), (x, c x )), c x = c x,, c x c x, () where k i the kernel function choen to calculate the covariance and c x i the label of category for the data oint x. We give a unit Gauian rior ditribution to latent rereentation x N (, I). The log marginal likelihood for the rooed model can be derived a log (Y) = log (Y X)(X)dX. There i no analytical olution for thi marginal likelihood. We aly variational inference and derive a cloed form lower bound of the log marginal likelihood, by following a are Gauian roce aroximation (Titia & Lawrence, ): log (Y) D F d (q) KL(q(X) (X)), () d= [ K uu F d (q) = log (πσ ) N βψ K uu ] e y d W y d ψ σ σ Tr(K uu Ψ ), (5)

3 Workho track - ICLR 6 where W = σ I σ Ψ ( σ Ψ K uu ) Ψ, and ψ, Ψ, Ψ are the exectation of covariance matrice w.r.t. the variational oterior q(x). In our model, thee exectation are derived a ψ = N n= (Ψ ) nm = (Ψ ) mm = N n=, x (n) k (x (n), z (m ) q(x (n) ), x (n) ) q(x (n) ) q(x (n) ), z (m ), z (m ) where z i the variational arameter known a inducing inut. (6) (7), z (m ) q(x (n) ) (8) EXPERIMENT Mitoi detection i a tage in tumour aement that involve determining whether individual cell are in mitoi (dividing to reroduce). Thee cell are rare. We ued data from the aement of mitoi detection algorithm (AMIDA, Veta et al., 5) challenge which i ublicly available. The main goal of the challenge i to find roer mitoi detection method that can be automatic or emi-automatic. We ue the training et from the challenge, which conit of tiue image of atient and are annotated by human exert. We reroce the tiue image with the algorithm by Snell () and focu on the generated candidate image atche. The reulting image et contain 6,56 grey-cale image atche ( ixel), of which 55 are oitive (mitoi) according to manual annotation. We randomly take 8% of oitive image and 5, negative image a the training data. Thi give in total 5, image. Some examle of the training data i hown in Figure a. We aly for SCLVM to thi dataet and ued an exonentiated quadratic kernel for both the hared and rivate ace and et the dimenionality of both hared and rivate ace to five. Both the latent rereentation and kernel arameter are otimized until convergence. The reulting latent ace i viualized in Figure. The oitive and negative image reent imilar tructure in the hared ace, which demontrate the dicovered common regularitie, and their rivate ace are ignificantly different from each other. To demontrate the ability of SCLVM in balancing the modeling caabilitie between imbalanced categorie, we draw amle from the learned latent ace of SCLVM for both oitive and negative categorie (ee Figure b). The generated amle from oitive and negative categorie are clearly different from each other and they cature ome characteritic of their own categorie. We further evaluate the learned latent ace by erforming claification on tet et (the ret of oitive examle lu randomly amled negative examle, in total image). We comare SCLVM with BGPLVM (Titia & Lawrence, ) and DG- PLVM (Urtaun & Darrell, 7) with ten tet et. We aly weighted SVM with an exonentiated quadratic kernel on the latent ace from the BGPLVM and DGPLVM. The reult are hown in Table. Note that BGPLVM require an additional claification model to be learned. Thi doe not rovide robabilitie over the clae other than in the ad-hoc manner that an SVM will. Similarly DGPLVM doe learn a ace that reflect the cla information, but it doe not rovide mean to get the oterior over the clae. Our model i the only one that learn the claification jointly with the model and rovide a rinciled way of getting robabilitie over the clae. CONCLUSION We reented a robabilitic latent variable model that can coe with imbalanced data. We develoed a kernel that earate the latent ace into a hared are and a rivate ace. An efficient variational inference method i rooed by deriving a cloed form lower bound of marginal likelihood. Beyond Due to it comlexity O(N ), we only ue image for training ( oitive and 56 negative).

4 Workho track - ICLR 6 neg o hared ace rivate ace for negative rivate ace for oitive latent ace from BGPLVM neg o Figure : The viualization of the training data in learned latent ace. The firt figure how the oitive and negative data in two of the hared dimenion. The econd and third figure how the two of the rivate dimenion for the negative and oitive data reectively. The fourth figure how the learned latent ace from BGPLVM. (a) (b) Figure : (a) Some examle in the data et. (b) Samle generated from the trained SCLVM. In both figure, the firt two row correond to oitive image and the lat two row correond to negative image. Table : Claification erformance.the mean and tandard deviation from ten tet et are hown. SCLVM BGPLVM (SVM) DGPLVM (SVM) reciion.6 ±..6 ±.. ±.8 recall.555 ±.7.87 ±.. ±. F core.8 ±.5.7 ±..9 ±. the hown examle, the ability of jointly modeling multile data categorie and handling imbalanced dataet can be linked to many other area uch a tranfer learning. REFERENCES Andrea Damianou, Carl Henrik Ek, Michali K. Titia, and Neil D. Lawrence. Manifold relevance determination. In John Langford and Joelle Pineau (ed.), Proceeding of the International Conference in Machine Learning, volume 9, San Francico, CA,. Morgan Kauffman. Neil D. Lawrence. Probabilitic non-linear rincial comonent analyi with Gauian roce latent variable model. Journal of Machine Learning Reearch, 6:78 86, 5. Violet Snell. Shae and Texture Recognition for Automated Analyi of Pathology Image. PhD thei, Centre for Viion, Seech and Signal Proceing, Univerity of Surrey, Surrey, UK,. Michali K. Titia and Neil D. Lawrence. Bayeian Gauian roce latent variable model. In Yee Whye Teh and D. Michael Titterington (ed.), Proceeding of the Thirteenth International Workho on Artificial Intelligence and Statitic, volume 9,. 8 85, Chia Laguna Reort, Sardinia, Italy, -6 May. JMLR W&CP 9. Raquel Urtaun and Trevor Darrell. Dicriminative gauian roce latent variable model for claification. In Proceeding of the th international conference on Machine learning, ACM, 7.

5 Workho track - ICLR 6 Mitko Veta, Paul J Van Diet, Stefan M Willem, Haibo Wang, Anant Madabhuhi, Angel Cruz-Roa, Fabio Gonzalez, Ander BL Laren, Jacob S Vetergaard, Ander B Dahl, et al. Aement of algorithm for mitoi detection in breat cancer hitoathology image. Medical image analyi, ():7 8, 5. 5