Formal Concept Sampling for Counting and Threshold-Free Local Pattern Mining

Transcription

1 Formal Conept Sampling for Counting an Threshol-Free Loal Pattern Mining Mario Boley an Thomas Gärtner an Henrik Grosskreutz Fraunhofer IAIS Shloss Birlinghoven Sankt Augustin, Germany {mario.boley, thomas.gaertner, Abstrat We esribe a Metropolis-Hastings algorithm for sampling formal onepts, i.e., lose (item-) sets, aoring to any esire stritly positive istribution. Important appliations are (a) estimating the number of all formal onepts as well as (b) isovering any number of interesting, non-reunant, an representative loal patterns. Setting (a) an be use for estimating the runtime of algorithms examining all formal onepts. An appliation of setting (b) is the onstrution of ata mining systems that o not require any user-speifie threshol like minimum frequeny or onfiene. 1 Introution We esribe a sampling algorithm that ombines two iretions of ata mining researh: the well establishe task of fining lose (item-)sets of a transational atabase [3, 10, 24] an the reent tren to sample patterns instea of listing them exhaustively [13, 14, 23]. The avantage of onsiering lose patterns rather than all patterns is that they represent non-reunant information about the ataset. The avantage of sampling rather than listing is that it allows to effiiently generate exatly the esire number of patterns aoring to a ontrolle target istribution. Our algorithm has appliations in ata mining as well as formal onept analysis. In ata mining a set is alle lose if eah of its supersets is ontaine in less transations than itself. A lose set an hene be seen as a (unique) maximal representer of the set of transations that it is ontaine in. In terms of formal onept analysis, lose sets orrespon to formal onepts, i.e., they are the olumns of maximal all-1-retangles of a given binary matrix. Fousing on lose sets avois generating sets that represent the same transations, i.e., it avois generating reunant knowlege ompare to traitional pattern mining approahes (see [3] an [10]). The motivation for sampling instea of listing is that it is often infeasible to ompletely list all (frequent) lose patterns. In this senario, aborte listing algorithms proue subsets C C of the lose patterns C epening on their internal searh orer that usually oes not reflet any ata mining interestingness measure. In ontrast, with our sampling approah lose patterns an be generate effiiently aoring to some ontrolle target istribution π : C [0, 1]. This istribution an be hosen freely aoring to some interestingness measure q : C R, i.e., π( ) = q( )/Z with a normalizing onstant Z. This approah has the further avantage that it oes not rely on a well-hosen value for a threshol parameter (like minimum support), whih is often har to etermine in pratie. In aition, when using the uniform target istribution, sampling an be use to ount the number of lose patterns/formal onepts in polynomial time. 1.1 Outline an Contributions We isuss more etaile motivations an relate work in the remainer of this setion. Then, after having introue basi notation an onepts (Setion 2), we give our main results: For a given transational atabase, we efine a Markov hain that has the family of lose sets as its state spae, allows an effiient omputation of a single simulation step, an onverges to any esire stritly positive target istribution (in Setion 3). While the worst-ase mixing time of these hains an grow exponentially in the atabase size, we propose a heuristi polynomially boune funtion for the number of simulation steps. This heuristi results in suffiient loseness to the target istribution for a number of benhmark atabases. We show how onept sampling an be use to buil an approximation sheme for ounting the 177 Copyright by SIAM.

2 number of all onepts (in Setion 4). Finally, we emonstrate how to use the sampling approah for generating any number of nonreunant loal patterns that are representative for a given istribution refleting interestingness without the speifiation of any threshol parameters (in Setion 5). In a onluing Setion we isuss, among other issues, omplexity results suggesting that no worst-ase polynomial time sampling algorithm for our problem exists. 1.2 Motivation A typial senario when mining for loal patterns of a real-worl atabase is the following. While the omplete set of patterns is overwhelmingly large, only a small fration of these patterns is neee as input for subsequent analysis or moeling steps. For example onsier the assoiation rule base outlier etetion metho LERAD [19] a two-pass algorithm that first generates a large set of rules an then filters the set until about 50 to 100 goo rules are left. This two phase approah with a listing/generation step an a subsequent filtering step is use within a wie range of appliations an has been formalize to the so-alle Lego-framework [18]. Its first step, however, an be problemati, beause the omputation time of an exhaustive listing algorithm is at least proportional to the number of patterns it proues. Thus, whenever the number of unfiltere patterns is too large the whole approah beomes infeasible. Dataset #rows #ols. #attr. size Eletion MB Questionnaire MB Census (30k) MB Table 1: Databases use in motivational experiment. We now emonstrate that these senarios are not mere worst-ase onstruts but that they an appear in pratie alreay on small to mi-size atabases. Our goal is to isover minimal non-reunant assoiation rules (see Setion 2 for preise efinitions) in three soioeonomi atasets that are summarize in Table 1. Two of them are from internal projets: Eletion ontains the result of the 2004 ounil eletions in the ity of Cologne along with esriptive attributes for eah of 800 polling istrits; Questionnaire is the result of a soio-eonomi questionnaire with a total of partiipants. We omplement them by the publily available US Census 1990 ataset (Census) from the UCI mahine learning repository [2]. In orer to speeup our experiments we generate a sample of 30K rows of this last ataset. For Figure 1: Computation time of exhaustively listing all minimal non-reunant assoiation rules. all three atasets we onverte their nominal olumns into binary attributes. For the sake of simpliity we are only intereste in exat rules, i.e., rules with a onfiene of 1. Figure 1 shows the require omputation time for several minimum frequeny threshols using JClose [21] as a representative exhaustive metho. Although there are more reent an potentially faster algorithms, all of them exhibit a similar behavior: the wellknown exponential inrease of the require time with ereasing threshols. Note that this explosion ours alreay very early on the frequeny sale. For Eletion the metho beomes problemati for threshols lower than The situation for the other two atasets is even worse. For Questionnaire the require time for a threshol as high as 0.7 is more than five ays! This emonstrates that even for moerately-size atasets an moerate minimum support threshols, fining assoiation rules an be impossible. With a support threshol of 0.7 or higher, however, loal pattern isovery beomes in fat global pattern isovery whih is not esire in many appliations [12]. There are even appliations where one is intereste exlusively in low support patterns (for instane see the stuy of Chow et al. on eteting privay leaks using assoiation rules [7]). Altogether, this motivates the evelopment of algorithms that are apable of fining a esignate small number of interesting patterns in feasible time. In partiular, these patterns must not be restrite to high support patterns but shoul rather be selete aoring to appliation epenent quality measures. 1.3 Relate Work an Challenges Exhaustive listing algorithms oul be aapte to our task by (a) applying a post-proessing step to selet a subset of a given omplete pattern olletion that was previously 178 Copyright by SIAM.

3 liste (see [1, 26] for representative methos), by (b) aborting the pattern enumeration after the esire number of patterns have been generate, or by () iretly restriting the searh to high quality patterns (e.g., [20, 27]). As isusse in Setion 1.2, approah (a) an be infeasible, approah (b) usually leas to a nonrepresentative set of patterns that epens on the internal searh orer of the listing sheme, an approah () usually suffers from omputational omplexity issues. For instane, fining k patterns of maximum frequeny having a minimum length [24] or fining a pattern that is infrequent in one atabase but frequent in another [25] an neither exatly nor approximately be solve in polynomial time (unless P=NP). This is ue to the har threshols respetively optimality requirements involve. Sampling algorithms an irumvent these negative omplexity results by replaing har threshols with guaranteeing that the probability of a pattern to appear in the output is proportional to its quality. As a positive sie-effet, the user is free of the often troublesome task of fining appropriate threshols. Al Hasan et al. [13] were the first to propose to irumvent the listing bottlenek by using ranomize sampling. Their algorithm aims to generate a representative list of maximal frequent subgraphs of a given graph atabase by a twostep approah: first a number of maximal frequent subgraphs is ranomly generate, an then a representative subset of this sample is selete base on a similarity measure. The sampling phase of this algorithm, however, provies no ontrol over the target istribution. In ontrast, the algorithm from [14] essentially simulates a Markov hain with uniform stationary istribution. While the proess esribe there is also use to sample maximal frequent subgraphs, its atual state spae is the set of all frequent subgraphs. A similar hain has been use in [5] for the purpose of quikly ounting the number of frequent sets without listing them. There is no straightforwar way of aapting these proesses to sampling lose sets. Using a set sampler within a rejetion sampling approah oes not lea to an effiient algorithm beause the number of sets an grow exponentially in the number of lose sets. Consequently, the probability of suessfully rawing a lose set with rejetion sampling an be very small; for a small atabase like msweb from the UCI mahine learning repository [2] with 285 items an 32K transations alreay as small as Similarly, mapping a rawn itemset to its smallest lose superset, is not a viable option: the varying number of generators among the lose set woul introue an unontrollable bias. This motivates the onstrution of a ranom walk algorithm that iretly uses the lose sets as state spae. 2 Bakgroun This setion realls efinitions from formal onept analysis (FCA) an lose set mining as well as Markov hains. FCA an lose set mining are strongly relate. In this work we prefer the notions of FCA beause of its symmetri treatment of items an transations, whih simplifies the subsequent tehnial isussion. For more information on FCA we refer to Wille s textbook [9], for Markov hains to Ranall s survey [22] an referenes therein. 2.1 Formal onept analysis Let X be some set. We enote the power set of X by P(X). A mapping σ : P(X) P(X) is alle a losure operator if it satisfies for all A B X extensivity, i.e., A σ(a), monotoniity, i.e., σ(a) σ(b), an iempotene, i.e., σ(a) = σ(σ(a)). The set of all lose sets of σ, i.e., its fixpoints, is enote by σ(p(x)). A (formal) ontext is a tuple (A, O, D) with A an O finite sets referre to as attributes an objets, respetively, an D A O a binary relation. In lose set mining, A, O, an D are referre to as items, transations, an atabase, respetively. The maps O[ ] : P(A) P(O) an A[ ] : P(O) P(A) are O[X] = {o O : a X, (a, o) D} an A[Y ] = {a A: o Y, (a, o) D}, respetively. An orere pair C = (I, E) P(A) P(O) is alle a (formal) onept, enote I, E, if O[I] = E an A[E] = I. The set I is alle the intent of C an E is alle its extent. The set of all onepts for a given ontext is enote C(A, O, D) or just C when there is no ambiguity. It is partially orere by the binary relation efine by I, E I, E if an only if I I (or equivalently E E ). For the minimal respetively maximal elements of C with respet to we write for A[O], O an for A, O[A]. It is well-known that a) the maps A[ ] an O[ ] form an (orer-reversing) Galois onnetion, i.e., X A[Y ] Y O[X], b) their ompositions φ = A[O[ ]] an ψ = O[A[ ]] form losure operators on P(A) an P(O), respetively, ) an all onepts C = I, E C are of the form C = φ(x), A[φ(X)] for some X A respetively C = O[ψ(Y )], ψ(y ) for some Y O, i.e., the intents of all onepts are the lose sets of φ an the extents the lose sets of ψ. 179 Copyright by SIAM.

4 2.2 Markov hains Monte Carlo Methos A (isrete) Markov hain on state spae Ω is a sequene of isrete ranom variables (X t ) t N with omain Ω satisfying the Markov onition, i.e., that P[X t+1 = x X 1 = x 1,..., X t = x t ] is equal to P[X t+1 = x X t = x t ] for all t N an x, x 1,..., x t Ω satisfying P[X 1 = x 1,..., X t = x t ] > 0. The uniform istribution on Ω is enote u(ω). In this artile we only onsier time-homogeneous Markov hains on finite state spaes. Given a probability istribution on the initial state, the joine istribution of (X t ) t N is ompletely speifie by the state transition probabilities p(x, y) = P[X t+1 = y X t = x] of all x, y Ω, whih o not epen on t. Let p t (x, y) enote the t-step probability, i.e., p t (x, y) = P[X n+t = y X n = x]. We all a state y Ω reahable from a state x Ω if there is a t N suh that p t (x, y) > 0. The hain (X t ) t N is alle aperioi if for all x, y Ω with x is reahable from y there is a t 0 N suh that for all t t 0 it hols that p t (x, y) > 0, an it is alle irreuible if any two states are reahable from one another. Finally, (X t ) t N is alle ergoi if it is irreuible an aperioi. Any ergoi Markov hain has a unique limiting stationary istribution π : Ω [0, 1], i.e., for all states x, y Ω it hols that lim t p t (x, y) = π(y). Moreover, if there is a istribution π : Ω [0, 1] satisfying the etaile balane onition (2.1) x, y Ω, π (x)p(x, y) = π (y)p(y, x) an p is irreuible then π is a stationary istribution an p is alle time-reversible. A Markov hain Monte Carlo metho generates an element from Ω aoring to π by simulating (X t ) t N for some time starting in an arbitrary state X 0 = x 0 Ω. Clearly, for that approah to be effetive one has to (a) be able to effiiently generate a neighbor y p(x, ) for a given state x an (b) know after how many simulation steps t the istribution of X t is lose enough to π. The istane from the t-step istribution of a Markov hain with X 0 = x to its stationary istribution an be measure by the total variation istane p t (x, ), π tv = 1/2 y Ω pt (x, y) π(y). Using this efinition the mixing time of (X t ) t N is efine by τ(ɛ) = max x Ω min{t 0 N: t t 0, p t (x, ), π tv ɛ} as the minimum number of steps one has to simulate (X t ) t N until the resulting istribution is guarantee to be ɛ-lose to its stationary istribution. 2.3 Close Loal Pattern Mining Finally, we fix notions an notations of lose loal pattern mining an their evaluation metris. This inlues sores use in unsupervise settings, e.g., assoiation rule isovery, as well as sores use in supervise esriptive rule inution tasks suh as emerging pattern mining [25], ontrast set mining [4], or subgroup isovery [6]. For oherene we efine all notions from the perspetive of FCA. Let (A, O, D) be a ontext. We onsier two types of loal patterns: simple sets of attributes F A, an assoiation rules X Y with X, Y A. There are many measures in the literature for ranking patterns aoring to their interestingness. The support of a set F A is the relative size of its extent, i.e., q supp (F ) = O[F ] / O. A measure that is base on the minimum esription length priniple is the area funtion [11], i.e., q area (F ) = F O[F ]. The support of a rule X Y is efine as the support of X Y, an its onfiene is q onf (X Y ) = O[X Y ] / O[X]. We sometimes enote the onfiene of a rule above its -symbol, i.e., X Y. Rules with a onfiene of 1 are alle impliations or exat rules. In orer to efine supervise evaluation measures one assumes that there are assoiate binary labels l(o) {+, } for all objets o O. Patterns F A an then be ranke aoring to the istributional unusualness of these labels on the pattern s extent, respetively aoring to their support ifferene between the positive an the negative portion of the objets. A representative measure is the binomial quality funtion q bino (F ) = q supp (F ) ( O + [F ] O[F ] O+ O where (A, O +, D + ) enotes the sub-ontext ontaining only the objets with positive labels, i.e., O + = {o O : l(o) = +} an D + = {(a, o) D : o O + }. For both kins of evaluation measures unsupervise an supervise it has been shown that by onsiering only lose sets, i.e., sets F A with F = φ(f ), one an fous on non-reunant rule families. In partiular for assoiation rules, a rule X Y is alle minimal non-reunant if there is no rule X Y with X X, Y Y an the same support. The following result of [3] relates lose sets to exat minimal non-reunant assoiation rules. Proposition 1. (Bastie et al.) All exat minimal non-reunant rules X Y of a ontext (A, O, D) are of the form Y = φ(y ) with X being a minimal set with φ(x) = Y. 3 Sampling Conepts We are now reay to esribe our onept sampling algorithm. First, we show how the state spae, i.e., ) 180 Copyright by SIAM.

5 the onept lattie of a given ontext, an be onnete in a way that allows to effiiently selet a ranom element from the neighborhoo of some onept. We ahieve this by using the losure operators inue by the ontext. Then, in a seon step, we apply to this initial iea the Metropolis-Hastings algorithm an show that the resulting Metropolis proess is irreuible. It an therefore be use to sample a onept aoring to any esire stritly positive istribution. 3.1 Generating Elements In orer to onstrut a stohasti proess on the onept lattie of a given ontext we will exploit its assoiate losure systems. More speifially, one an move from one lose set of a losure operator to another by a single element augmentation an a subsequent losure operation. In this ontext, we refer to the involve single elements as generating elements that an be efine as follows. Definition 1. (Generating Element) Let σ be a losure operator on P(X) an C, C σ(p(x)) be lose sets. We say that an element x X generates C from C with respet to σ if σ(c {x}) = C. The set of all suh generating elements is enote by G σ (C, C ). These generating elements an be use to efine a irete graph on the family of lose sets in whih two verties C, C are joine by an ar if there is at least one element generating C from C. This graph is impliitly traverse by several lose set listing algorithms. Definition 2. (Generator Graph) Let σ be a losure operator on P(X). The generator graph G σ = (C, E σ, l σ ) of σ is the irete labele graph on the lose sets C = σ(p(x)) as verties with eges E σ = {(C, C ) C C: G σ (C, C ) }, an with ege labels l σ : E P(X) equal to the generating elements, i.e., l σ (C, C ) = G σ (C, C ). It is easy to see that G φ is a) ayli exept for self-loops an b) roote in σ( ): statement a) follows by observing that (C, C ) E σ implies C C, an for b) note that for all lose sets C = {x 1,..., x k } the sequene x 1,..., x k orrespons to an ege progression (walk) from σ( ) to C ue to extensivity an monotoniity of σ. A ranom walk on the generator graph an be performe by the following proeure: in a urrent lose set Y σ(p(x)) raw an element x u(x), an then move to σ(y {x}). The transition probability from a set Y to a set Z is then iretly proportional to the number of generating elements G σ (Y, Z) an an be ompute effiiently. Now, if one ientifies onepts with their intents respetively with their extents, there are two assoiate D a b Table 2: Example ontext with attributes {a, b,, } an objets {1, 2, 3, 4, 5}. generator graphs to a given ontext the one inue by φ an the one inue by ψ. For the example ontext from Table 2 these graphs are rawn in Figure 2. Note that the ar relation G φ is generally not iential to the transitive reution of, whih is usually use within iagrams illustrating a onept lattie: the onepts a, 234 an ab, 4 are joine by an ar although they are not iret suessors with respet to. The basi iea for our sampling algorithm is to perform a ranom walk on these generator graphs. Taking just one of them, however, oes not result in an irreuible hain. Either or woul be an absorbing state for suh a proess, in whih any ranom walk will result with a probability onverging to 1 for an inreasing number of steps. A first iea to ahieve irreuibility might be to take one of the orresponing generator graphs, say G φ, a all inverse eges of E φ as aitional possible state transitions, an perform a ranom walk on the resulting strongly onnete graph. Unfortunately, with this approah, rawing a neighbor of a given onept (as require for an effiient hain simulation) is as har as the general problem of sampling a onept. Proposition 2. Given a ontext (A, O, D) an a onept I, E C, rawing a preeessor of I, E in G φ uniformly at ranom, i.e., a onept I, E C with φ(i a) = I for some a A, is as har as generating an arbitrary element of C uniformly at ranom. Proof. A given ontext (A, O, D) with onepts C an be transforme into a ontext (A, O, D ) with onepts C as follows: set A = A {a } with a A, O = O {o } with o O, an D = D {(a, o ): a A }. Then C = C \{ A, {o } } an for all I, E C, it hols that φ (I {a }) = A, i.e., all C C are preeessors of A, {o } in G φ. An alternative approah is to use the a ranom walk on the union of both generator graphs as stohasti 181 Copyright by SIAM.

6 A {} 3 A {} ab 4 a, 2 ab 4 5 b, 3 ab 34 b a 234 a a 2 b 134 a 12 b 1 b a,b 45 ab 34 2 a ,4 5 a 2 b , b a b {} O 5 1,5 2,5 {} O 3,4,5 1,2,3 (a) (b) Figure 2: Generator graphs G φ an G ψ (rawn without self-loops) for the ontext from Table 2. proess. Tehnially, this an be realize as follows: in a urrent state flip a fair oin an then, base on the outome, hoose either G φ or G ψ to proee to the next state as esribe above. This leas to the state transition probabilities G φ (I, I ) /(2 A ), if C C q(c, C ) = G ψ (E, E ) /(2 O ), if C C I /(2 A ) + E /(2 O ), if C = C. for onepts C = I, E an C = I, E. It is easy to see that the resulting stohasti proess is an ergoi Markov hain. Thus, it has a stationary istribution to whih it onverges. Generally, however, we o not know anything about this istribution. In the next subsetion we show how the hain an be moifie suh that it onverges to a istribution that is known an esire. 3.2 Metropolis-Hastings Algorithm Let π : C [0, 1] be the esire target istribution, aoring to whih we woul like to sample onepts. For tehnial reasons that will beome lear shortly we require π to be stritly positive, i.e., π(c) > 0 for all C C. In pratie this an be ahieve easily. If our ergoi preliminary hain woul have symmetri state transition probabilities, we oul simply use it as proposal hain as follows: in a urrent onept C, propose a suessor C aoring to q, an then aept the proposal with probability π(c )/π(c). This is the lassi Metropolis algorithm, an for the resulting Markov hain it is easy to hek that it satisfies the etaile balane onition for π grante that q is symmetri as well as ergoi an that π is stritly positive. However, the example in Figure 2 shows that the union of the two generator graphs orresponing to a given ontext oes not neessarily inue symmetri state transition probabilities. In fat, the resulting proess is in general not even time-reversible see for instane onepts an b, 1 in Figure 2, for whih we have q(, b, 1 ) = 0 but q( b, 1, ) > 0. As a solution one an fator the quotient of the proposal probabilities into the aeptane probabilities. The resulting state transitions are { p(c, C q(c, C ) min{α π(c ) π(c) ) =, 1}, if q(c, C ) > 0 0, otherwise where α = q(c, C)/q(C, C ). This is the Metropolis- Hastings algorithm [15], an the unerlying proess is alle the Metropolis-proess of q an π. For the example from Table 2 its state transition probabilities are rawn in Figure 3. It is important to note that, in orer to simulate the Metropolis proess, one oes not nee the exat probabilities π(c ) an π(c). As the aeptane probability only epens on their quotient, it is suffiient to have aess to an unnormalize potential f : C R suh that there is a onstant α with f(c) = απ(c) for all C C. For instane, for the uniform target istribution one an hoose any onstant funtion f. Algorithm 1 is a Markov hain Monte Carlo implementation of the Metropolis proess. It takes as input a ontext, a number of iterations s, an an orale for the unnormalize potential f. Then, after hoosing an initial state X 0 uniformly among an, it simulates the proess for s steps, an returns the state it has reahe by that time, i.e., its realization of X s. It is easy to see that the algorithm inee uses the 182 Copyright by SIAM.

7 1/ / / / /8 Figure 3: Resulting state transitions for Table 2 an f( ) 1; remaining probabilities are assigne to selfloops; noes ontain stationary probability (blak) an apx. probability after five steps with X 0 =, O (grey). state transition probabilities p. So far, however, we omitte an important aspet of its orretness: while π satisfies the etaile balane onition for this hain, this omes at the ost of setting p(c, C ) = 0 for some pairs of onepts that have q(c, C ) > 0. Thus, the irreuibility of q is not iretly implie by the irreuibility of p. It is, however, guarantee by the losure properties of φ an ψ, as shown in the proof of the following summarizing theorem. Algorithm 1 Metropolis-Hastings Conept Sampling Input : ontext (A, O, D), number of iterations s, orale of map f : C R+ Output : onept I, E 1. init I, E u({, }) an i 0 2. i i raw u({up, own}) 4. if = up then 5. raw a u(a) 6. I, E φ(i {a}), O[φ(I {a})] 7. α ( G ψ (E, E) A ) / ( G φ (I, I ) O ) 8. else 9. raw o u(o) 10. I, E A[ψ(E {o})], ψ(e {o}) 11. α ( G φ (I, I) O ) / ( G ψ (E, E ) A ) 12. raw x u([0, 1]) 13. if x < αf(i ) / f(i) then I, E I, E 14. if i = s then return I, E else goto /8 1/8 Theorem 3. On input ontext (A, O, D), step number s, an stritly positive funtion f, Algorithm 1 proues in time O(s D ) a onept C C(A, O, D) aoring to a istribution p s f suh that lim s ps f, π tv = 0 where π is the istribution on C resulting from normalizing f, i.e., π( ) = f( )/ C C f(c). Proof. The time omplexity is easy to see. Also, by onstrution, it an iretly be heke that π an p satisfy the etaile balane onition (Eq. 2.1). It remains to show irreuibility, i.e., p t ( I, E, I, E ) > 0 for some t an all pairs of onepts I, E, I, E C. It is suffiient to onsier the ase I, E I, E : for other states reahability follows then via I I, A[I I ] I, E, I I, A[I I ] I, E, an the transitivity of reahability. Moreover, as the onsiere state spaes C are finite, it suffies to onsier iret suessors I, E. I, E. For suh onepts it is easy to show that I is a maximal proper subset of I in φ(p(a)) an E is a maximal proper subset of E in ψ(p(o)). Let a I \ I an o E \ E. It follows by the losure operator properties that φ(i {a}) = I an ψ(e {o}) = E. Thus, the proposal probabilities between I, E an I, E are non-zero in both iretions, an together with the fat that f is stritly positive, it follows for the transition probability of the Metropolisproess that p( I, E, I, E ) > 0 as require. Now that we know that our algorithm asymptotially raws samples as esire, it remains to isuss, for how many steps one has to simulate the proess until the istribution is lose enough to the target istribution. 3.3 Number of Iterations Unfortunately, in the worst ase the mixing an be infeasible slow, i.e., require a number of steps that an grow exponentially in the input size. The following statement on the mixing time hols. Proposition 4. Let ɛ > 0 be fixe an τ n (ɛ) enote the worst-ase mixing time of Algorithm 1 for the uniform target istribution an an input ontext of size n. Then τ n (ɛ) Ω(2 n/2 ). Proof. We prove the laim by showing that for all n with an even integral square root there is a ontext (A, O, D) with A = O = {1,..., n} an D = n/2 n suh that τ(ɛ) 2 n/2 3 log(1/(2ɛ)). It is well-known (see for instane [22]) that the mixing time of a Markov hain with state spae C an stationary istribution π is lower boune by (3.2) τ(ɛ) 1/(4Φ) log(1/ɛ), 183 Copyright by SIAM.

8 where Φ is the onutane efine as the minimum number Φ S over all S C with π(s) 1/2 where Φ S is equal to (1/π(S)) x S,y C\S π(x)p(x, y). Choose D = {(i, j): i j, (i, j n/2 i, j > n/2)}. Then the set of onepts is a isjoint union C(A, O, D) =. C 1 C2 with C 1 ={ I, O[I] : I {1,..., n/2}} C 2 ={ I, O[I] : I { n/2 + 1,..., n}} { } an if π is the uniform istribution, it hols that π(c 1 ) = 1/2. Moreover, the only state pairs that ontribute to Φ C1 are the n pairs {, C} an {C, } of the form C { I, E : I {{ n/2 + 1},..., { n}}} C { A \ {a}, E : a {1,..., n/2}}. Consequently, 1/2 n/2 1 = Φ C1 Φ an with (3.2), τ(ɛ) 2 n/2 3 log(1/(2ɛ)) as require. Thus, even the stritest theoretial worst-ase boun on the mixing time for general input ontexts woul not lea to an effiient algorithm. The proof of the proposition is base on the observation that the hain an have a very small onutane, i.e., a relatively large portion of the state spae (growing exponentially in n) an only be onnete via a linear number of states to the rest of the state spae. We assume that real-worl atasets o not exhibit this behavior. In our appliations we therefore use the following heuristi polynomially boune funtion for assigning the number of simulation steps: steps((a, O, D), ɛ) = 4n ln(n) ln(ɛ) where n = min{ A, O }. The motivation for this is as follows (see also [5], where a similar heuristi is use). Assume without loss of generality that A < O. As long as only = up is hosen in line 3 of the algorithm the expete number of steps until all elements of A have been rawn at least one is n ln(n) + O(n) with a variane that is boune by 2n 2 (oupon olletor s problem). It follows that asymptotially after 2n ln(n) all elements have been rawn with probability at least 3/4. Consequently, as = up is hosen with probability 1/2, we an multiply with an aitional fator of 2 to know that with high probability there was at least one step upwar an one step ownwar for eah attribute a A if one assumes a large onutane this suffies to reah every state with appropriate probability. The final fator of ln(ɛ) is motivate by the fat that the total variation istane usually eays exponentially in the number of simulation steps. Figure 4: Distribution onvergene. This an also be observe in our experiments with four real-worl atabases 1 an two target istributions: the uniform istribution an the istribution proportional to the area quality measure π( ) = q area ( )/Z. The results are illustrate in Figure 4. Note that the plots iffer in their semantis. While for both target istributions a point (x, y) means that the x-step istribution has a total variation istane of y from the target istribution, for the heuristi the inverse of the step-heuristi is shown, i.e., the value of ɛ on the y-axis reflets the esire auray that leas to a partiular number of steps. Thus, the heuristi was orret if its orresponing plot ominates the two total variation istane plots. As we an see, this was the ase for both istributions on all four atasets. In fat, the heuristi was rather onservative. Note that the y-axis has a logarithmi sale. The effet of the target istribution on the mixing time is unlear. On the one han, eviation from the uniform istribution an reate bottleneks, i.e., low onutane sets, where there have been non before. On the other han, it an also happen that bottleneks of the uniform istribution are softene. This an be observe for mushroom an hess. For that reason we o not reflet the target istribution in our heuristi. 1 Databases are from UCI Mahine Learning Repository [2]. In orer to expliitly ompute the state transition matries we ha to use only a sample of the transations for three of the atabases. The sample size is given in brakets. 184 Copyright by SIAM.

9 4 Conept Counting In this setion we highlight the onnetion of onept sampling to onept ounting. This onnetion is important beause it ties the omplexities of these two tasks together. In partiular, we show how onept sampling an be use to esign a ranomize approximation sheme for the number of onepts of a given ontext, i.e., a ranomize algorithm that proues for an input ontext (A, O, D) an ɛ (0, 1/2] a number N suh that (1 ɛ) (A, O, D) N (1 + ɛ) (A, O, D) with probability at least 3/4. Besie theoretial insight, the motivation for this is that suh an algorithm an be use to quikly hek the feasibility of an potentially exponential time exhaustive listing of all onepts (see also [5] where the same problem is isusse for frequent itemsets). Let (A, O, D) be a ontext an o 1..., o m some orering of the elements of O. For i {0,..., m} efine the ontext Ci = (A, O i, D i ) with O i = {o j : j i}, D i as the restrition of D to O i, φ i as the orresponing attribute losure operator, an I i = φ i (P(A)) the onept intents. Note that in general a onept intent I I i+1 is not an intent of a onept with respet to the ontext Ci, i.e., not a fixpoint of φ i. The following two properties, however, hol. Lemma 5. For all ontexts (A, O, D) an all i {0,..., O } it hols that (i) I i I i+1 an (ii) 1/2 I i / I i+1. Proof. For (i) let I I i. In ase o i+1 I[O i+1 ], it is I[O i+1 ] = I[O i ]. Otherwise, per efinition we know that all a A that satisfy (a, o) D for all o O i are also satisfying (a, o) D for all o O i+1. Thus, in both ases φ i+1 (I) = A[O i+1 [I]] = A[O i [I]] = φ i (I) = I as require. We prove (ii) by showing that the restrition of the losure operator φ i to I i+1 \ I i is an injetive map into I i, i.e., I i+1 \ I i I i. Together with (i) this implies the laim. Assume for a ontraition that there are istint intents X, Y I i+1 \ I i suh that φ i (X) = φ i (Y ). Then O i [X] = O i [Y ], an, as X an Y are both lose with respet to φ i+1 but not lose with respet to φ i, it must hol that o i+1 O i+1 [X] O i+1 [Y ]. It follows that O i+1 [X] = O i [X] {o i+1 } as well as O i+1 [Y ] = O i [Y ] {o i+1 }. This implies O i+1 [X] = O i+1 [Y ] an in turn X = Y ontraiting the assumption that the intents are istint. Thus, the I i efine an inreasing sequene of lose set families with I 0 = {A} = 1 an I m = C(A, O, D) that allows to express the number of onepts of the omplete ontext as (4.3) C(A, O, D) = ( I o 1 m I i 1 / I i ). i=1 For i {1,..., m} let Z i (I) enote the binary ranom variable on measure spae (I i, π i ) that takes on value 1 if I I i 1 an 0 otherwise. Inepenent simulations of these ranom variables an be use to ount the number of onepts via Equation 4.3 using the prout estimator Z = m 1 Z i where Z i = (Z (1) i + + Z (t) i )/t with Z (j) i inepenent opies of Z i. Using stanar reasoning (see, e.g., [17]) an Lemma 5 one an show that Z is ɛ-lose to C(A, O, D) with probability at least 3/4 if (i) the total variation istane of the π i to the uniform istribution on I i is not greater than ɛ/(12 O ) an (ii) t 12 O /ɛ 2. See Algorithm 2 for a pseuooe simulating this estimator. Observing that the roles of A an O are interhangeable, i.e., the number of onepts of (A, O, D) is equal to the number of onepts of the transpose ontext, we an onlue: Theorem 6. There is a ranomize approximation sheme for the number of onepts C of a given ontext (A, O, D) with time omplexity O ( nɛ 2 T S (ɛ/(12n)) ) for auray ɛ where n = min( A, O ) an T S (ɛ ) is the time require to sample a onept almost uniformly, i.e., aoring to a istribution with a total variation istane of at most ɛ from uniform. Algorithm 2 Conept Counting Input : ontext (A, O, D), auray ɛ (0, 1 2 ] Require: p steps(ci,ɛ ) i,1, u(c i ) tv ɛ for all i an ɛ Output : q with P[(1 ɛ) C q (1+ɛ) C ] 3/4 1. t 12 O /ɛ 2 2. for i = 1,..., O o 3. r i 0 4. for k = 1,..., t o 5. I, E sample(ci, steps(ci, ɛ/(12 O )), 1) 6. if φ i 1 (I) = I then r i r i r i r i /t 8. return O i=1 r 1 i To evaluate the auray of this ranomize ounting approah, we exeute a series of runs on the hess ataset an ompare the estimate ompute by our ranomize algorithm with the exat number of onepts. We i not onsier the whole ataset but instea use samples of ifferent size. We use an auray of ɛ = 0.5. The result is shown in Figure 4. On the 185 Copyright by SIAM.

10 Eletion Questio. Census sampling wrt. q supp 2.1 se. 21 min. 7 min. sampling wrt. q area 2.2 se. 21 min. 7 min. JClose min m min.... with support Table 3: Time for assoiation rule generation. Figure 5: Estimate number of onepts on samples of the hess ataset. x-axis, we give the size of the sample, while the y-axis shows the exat number of onepts ( exat ount ), the upper an lower 0.5 eviation limits ( upper limit an lower limit ), as well as the result of the ranomize algorithm in a series of 10 runs per sample ( estimate ount ). The figure shows that in all ranomize runs, the approximate result lies within the given eviation boun. This is a further positive evaluation of the step heuristi evelope in Setion Appliation to Loal Pattern Mining In this setion, we will present exemplary appliations of the sampling algorithm to loal pattern mining. This revisits the motivations from Setion Assoiation Rule Mining First, we illustrate how the sampling algorithm an be use in the ontext of minimal non-reunant assoiation rule isovery (see Setion 2). Again, for the sake of simpliity, we onsier exat assoiation rules, i.e., rules with a onfiene of 1. Sampling a rule with onfiene below 1 an be one, for instane, by sampling the onsequene of the rule in a first step; an then sampling the anteeent aoring to a istribution that is proportional to the onfiene an support of the resulting rule. Base on Proposition 1, we an generate exat minimal non-reunant assoiation rules by first sampling a onept Y, E, an then alulating a minimal generator X of the onept s intent, i.e., a minimal set with φ(x) = Y. For the alulation of the minimal generators, we use a simple greey algorithm that is essentially equivalent to the greey algorithm for the set over problem (see [6] for a isussion). In fat, with this algorithm we generate an approximation to a shortest generator an not only a minimal generator with respet to set inlusion. Thus, we are aiming for partiularly short rule anteeents. With this metho we generate assoiation rules for the three atasets Eletion, Questionnaire, an Census from Setion 1.2 (see Table 1) for two ifferent target istributions: the one proportional to the rule s support an the one proportional to their area (aing the onstant 1 to ensure a stritly positive istribution). That is, we use f = q supp respetively f = q area (see Setion 2) as parameter for Algorithm 1. As esire loseness to the target istribution we hoose ɛ = Note that, in ontrast to exhaustive listing algorithms, for the sampling metho we o not nee to efine any threshol like minimum support. Table 3 shows the time require to generate a single rule aoring to q supp an q area. In aition, the table shows the runtime of JClose for exemplary support threshols. Comparing the runtimes for sampling with that of JClose, we an see that the uniform generation of a rule from the Eletion ataset takes about 1/1609 of the time neee to exhaustively list non-reunant assoiation rules with threshol For Questionnaire an threshol 0.7, the ratio is 1/340; finally for Census with threshol 0.7, the ratio is 1/247. As isusse in Setion 1.2, all exhaustive listing algorithms exhibit a similar asymptoti performane behavior. Thus, for methos that outperform JClose on these partiular atasets, we woul en up with similar ratios for slightly smaller support threshols. Moreover, it is not unommon that one is intereste in a small set of rules having a substantially lower support than the threshols onsiere above. For suh ases the ratios woul inrease ramatially, beause the time for exhaustive listing inreases exponentially whereas the time for sampling remains onstant. Thus, for suh appliations sampling has the potential to provie a signifiant speeup. In fat, it is appliable in senarios where exhaustive listing is hopeless. 5.2 Supervise Desriptive Rule Inution As seon appliation we onsier supervise esriptive rule inution, i.e., loal pattern mining onsiering labele ata suh as emerging pattern mining, ontrast set mining, or subgroup isovery (see Setion 2). This an be one by hoosing the binomial quality q bino as parameter for Algorithm 1. In fat, we ap 186 Copyright by SIAM.

11 ataset #rows #ols. #attr. size target sik K sik soyb K br.-spot lung K 1 Table 4: Databases for the supervise esriptive rule inution experiments. sik soyb. lung. sampling time listing time Table 5: Time (in seons) of ranom pattern generation aoring to q bino ompare to exhaustive listing. q bino from below by a small onstant to ensure a stritly positive target istribution, i.e., we use f( ) = max{q bino ( ), }. With the binomial quality funtion we have an evaluation metri that has shown to aequately measure interestingness in several appliations. Thus, in aition to the omputation time, in this experiment we an also evaluate the quality of the sample patterns. We onsier three labele atasets from the UCI mahine learning repository (liste in Table 4). As baseline metho we use exhaustive lose subgroup isovery base on generator graph traversal [6]. This algorithm explores the same state spae as the sampling algorithm an outperforms other (non-lose) listing algorithms on the onsiere atasets. Table 5 gives the time neee to generate a pattern by the ranomize algorithm ompare to the time for exhaustively listing all patterns with a binomial quality of at least. This threshol is approximately one thir of the binomial quality of the best patterns in all three atasets. As observe in the unsupervise setting, the ranomize approah allows to generate a pattern in a fration of the time neee for the exhaustive omputation. In orer to evaluate the pattern quality, we ompare the best patterns within a set of 100 samples to the globally best patterns (Table 6). Although the quality of the ranomly generate patterns o not reah the optimum, we observe the antiipate an esire result: the ollete subgroups form a high quality sample relatively to the omplete pattern spae, from whih they are rawn. 6 Disussion We presente a Metropolis-Hastings algorithm able to effetively generate onepts aoring to some esire target istribution. In several exemplary appliations we emonstrate how this algorithm an be use for sik soybean lung. best quality sample quality Table 6: Best quality among all patterns versus best quality among sample patterns. knowlege isovery an approximate ounting. It is important to note that although we restrite the presentation to lose sets, the approah is not limite to this senario. In fat it an be applie to all pattern lasses having a similar Galois onnetion between patterns an transations. Furthermore, the metho an be ombine with any anti-monotone or monotone onstraint. It is for instane easy to see that the algorithm is still orret when the hain is restrite to the set of all frequent onepts. Although the heuristi polynomial step number leas to suffiient results in our experiments, learly one woul prefer a polynomial sampling algorithm with provable worst-ase guarantee. There is, however, some eviene that no suh algorithm exists or that at least esigning one is a iffiult problem: as we have shown in Setion 4, an algorithm that an sample onepts uniformly in polynomial time an be use to esign a fully polynomial ranomize approximation sheme (FPRAS) for ounting the number of onepts for a given ontext. This problem is equivalent to ounting the number of maximal bipartite liques of a given bipartite graph, whih is as har as a omplexity lass alle #RHΠ 1 #P (harness result an omplexity lass both were introue in [8]). Despite muh interest for suh algorithms in several researh ommunities, there is no known FPRAS for any #RHΠ 1 -har problem. In the absene of feasible a priori bouns on the mixing time, there are several tehniques that an be use not only to etet mixing but even to raw perfet samples, i.e., generate exatly aoring to the stationary istribution (see [16] an referenes therein). The most popular variant of these perfet sampling tehniques is oupling from the past (CFTP). It an be applie effiiently if the hain is monotone in the following sense: the state spae is partially orere, ontains a global maximal as well as a global minimal element, an if two simulations of the hain that use the same soure of ranom bits are in state x an y with x y then also the suessor state of x must be smaller than the suessor state of y. Inee, at first glane it appears that CFTP may be applie to our hain beause its state spae is partially orere by an always ontains a global minimal element 187 Copyright by SIAM.

12 as well as a global maximal element. Even for the uniform target istribution, however, the hain is not monotone. This an be observe in the example of Figure 2: enote by su,5,0 ( I, E ) the suessor of I, E when the ranom bits use by the omputation inue the eisions = own, o = 5, an p = 0. Then b, 1 A, whereas su,5,0 ( b, 1 ) = b, 1, 45 = su,5,0 ( A, ). It is an open problem, how to efine a Markov hain on the onept lattie that has monotone transition probabilities. Aknowlegements Part of this work was supporte by the German Siene Founation (DFG) uner the referene number GA 1615/1-1. Referenes [1] Foto N. Afrati, Aristies Gionis, an Heikki Mannila. Approximating a olletion of frequent sets. In KDD, pages ACM, [2] A. Asunion an D.J. Newman. UCI mahine learning repository, [3] Yves Bastie, Niolas Pasquier, Rafik Taouil, Ger Stumme, an Lotfi Lakhal. Mining minimal nonreunant assoiation rules using frequent lose itemsets. In Computational Logi - CL 2000, pages Springer, [4] Stephen D. Bay an Mihael J. Pazzani. Deteting group ifferenes: Mining ontrast sets. Data Min. Knowl. Disov., 5(3): , [5] Mario Boley an Henrik Grosskreutz. A ranomize approah for approximating the number of frequent sets. In ICDM, pages IEEE Computer Soiety, [6] Mario Boley an Henrik Grosskreutz. Non-reunant subgroup isovery using a losure system. In ECML/PKDD (2), pages Springer, [7] Rihar Chow, Philippe Golle, an Jessia Staon. Deteting privay leaks using orpus-base assoiation rules. In KDD 2008, pages ACM, [8] Martin Dyer, Leslie Ann Golberg, Catherine Greenhill, an Mark Jerrum. The relative omplexity of approximate ounting problems. Algorithmia, 38(3): , [9] Bernhar Ganter an Ruolf Wille. Formal Conept Analysis: Mathematial Founations. Springer Verlag, [10] Gemma C. Garriga, Petra Kralj, an Naa Lavrač. Close sets for labele ata. J. Mah. Learn. Res., 9: , [11] Floris Geerts, Bart Goethals, an Taneli Mielikäinen. Tiling atabases. In DS, pages , [12] Davi Han. Pattern etetion an isovery. In Pattern Detetion an Disovery, volume 2447 of LNAI, pages Springer, [13] Mohamma Al Hasan, Vineet Chaoji, Saee Salem, Jérémy Besson, an Mohamme Javee Zaki. Origami: Mining representative orthogonal graph patterns. In ICDM, pages IEEE Computer Soiety, [14] Mohamma Al Hasan an Mohamme Javee Zaki. Musk: Uniform sampling of k maximal patterns. In SDM, pages SIAM, [15] W. K. Hastings. Monte arlo sampling methos using markov hains an their appliations. Biometrika, 57(1):97 109, [16] Mark Huber. Perfet simulation with exponential tails on the running time. Ranom Strut. Alg., 33(1):29 43, [17] Mark Jerrum an Alistair Sinlair. The Markov hain Monte Carlo metho: an approah to approximate ounting an integration, pages PWS Publishing Co., Boston, MA, USA, [18] A. Knobbe, B. Crémilleux, J. Fürnkranz, an M. Sholz. From loal patterns to global moels: the lego approah to ata mining. In Pro. of the ECML PKDD 2008 LEGO Workshop, [19] Matthew V. Mahoney an Philip K. Chan. Learning rules for anomaly etetion of hostile network traffi. In ICDM, page 601, Washington, DC, USA, IEEE Computer Soiety. [20] Shinihi Morishita an Jun Sese. Traversing itemset lattie with statistial metri pruning. In PODS, pages ACM, [21] Niolas Pasquier, Rafik Taouil, Yves Bastie, Ger Stumme, an Lotfi Lakhal. Generating a onense representation for assoiation rules. Journal of Intelligent Information Systems, 24(1):29 60, January [22] Dana Ranall. Rapily mixing markov hains with appliations in omputer siene an physis. Computing in Siene an Engineering, 8(2):30 41, [23] Leaner Shietgat, Fabrizio Costa, Jan Ramon, an Lu De Raet. Maximum ommon subgraph mining: A fast an effetive approah towars feature generation. In MLG, [24] J. Wang, J. Han, Y. Lu, an P. Tzvetkov. Tfp: an effiient algorithm for mining top-k frequent lose itemsets. Knowlege an Data Engineering, IEEE Transations on, 17(5): , May [25] Lusheng Wang, Hao Zhao, Guozhu Dong, an Jianping Li. On the omplexity of fining emerging patterns. Theoretial Computer Siene, 335(1):15 27, [26] Dong Xin, Jiawei Han, Xifeng Yan, an Hong Cheng. Mining ompresse frequent-pattern sets. In VLDB, pages VLDB Enowment, [27] Xifeng Yan, Hong Cheng, Jiawei Han, an Philip S. Yu. Mining signifiant graph patterns by leap searh. In SIGMOD, pages ACM, Copyright by SIAM.