Efficient algorithms for for clone items detection

Transcription

1 Efficient algoritms for for clone items detection Raoul Medina, Caroline Noyer, and Olivier Raynaud Raoul Medina, Caroline Noyer and Olivier Raynaud LIMOS - Université Blaise Pascal, Campus universitaire LIMOS -des Université Cézeaux, Blaise Clermont-Ferrand, Pascal, France Campus universitaire {medina, Cézeaux, raynaud}@isima.fr Clermont-Ferrand, France {medina, raynaud}@isima.fr Abstract. Tis paper presents efficient algoritms for clone items detection in a binary relation. Best implementation of tese algoritms as O( J. M ) time complexity, wit J corresponding to te set of items of te relation and M corresponding to te size of te relation. Tis result improves te previous algoritm given in [3] wic is O( J 2. M ). Clone items ave been introduced by Medina and Nourine to explain wy, sometimes, te number of rules of a minimum cover of a relation is exponential wit te number of items of te relation. 1 Introduction Te clone items notion as been introduced by Medina and Nourine in [3]. Aim of teir paper was to understand te combinatorial explosion tat migt arise in a minimum basis of a relation. Most famous example of suc exponential minimum basis is given by Manilla and Räiä in [2]. Medina and Nourine noticed tat in tis example some items play symetrical role in te basis. Indeed, for eac rule containing a given item a in te antecedent tere is a symetrical rule were item a is replaced by item b. Suc symetrical items are said to be clone items. Te clone notion is an equivalence relation and tus classes of clone items can be found in a binary relation. In [3], te autors sow ow to compute suc clone classes and, from tose classes, ow to reduce te binary relation in order to obtain a minimum basis wit no clone items. Te detection algoritm as well as te reduction algoritm ave bot polynomial time complexities. Te obtained minimum basis is smaller tan te original one (in te case of te Mannila and Räiä example it reduces to a single rule) and tis later can be reconstructed from te clone free minimum basis and te clone classes information in polynomial time. Recently, Gely et al. (in [1]) extended te notion of clone items (wic are defined on closed sets) to te notion of P-clone items (defined on pseudo-closed sets) and to te notion of A-clone items (defined on wat tey call an atomized context). Teir approac could be generalized to any generation problem were, from a given relation, one wants to compute a (potentially exponential) collection of sets verifying a property (e.g. te sets must be closed, or te sets must be ideals, or te sets are pseudo-closed sets, etc...). Te idea is ten to reduce te combinatorial explosion of te wanted collection by representing it by a clone free collection and te classes of clone items of tis collection. Te problem is Radim Bělolávek, Václav Snášel (Eds.): CLA 2005, pp , ISBN

2 Efficient algoritms for clone items detection 71 ten to be able to compute te clone classes of te (potentially exponential) collection witout generating its sets. To solve tis problem in polynomial time, te general metod could be as follows: Let M be te (potentially exponential) collection of sets verifying a property over a given binary relation R. Compute in polynomial time a collection M suc tat: 1. te size of M is polynomial in te size of R, 2. items a and b are clone in M if and only if tey are clone items in M. Detect te clone classes in M. Our paper focuses on te detection pase of tis approac. Te clone classes computation algoritm given in [3] as an O( J 2. M ) time complexity, were J is te set of items and M is te size of te input collection. In oter words, M = m M m. In tis paper, we present different algoritms to solve te clone classes computation. Te best complexity we obtain is in O( J. M ). Tis paper is organized as follows: section 2 formally defines te problem in terms of collection of sets and introduces te corresponding Abstract Data Type wic will be used by our algoritms. Section 3 describes tree computation strategies and te corresponding time complexities are studied. Section 4 sows ow to take advantage of tose algoritms in order to compute te clone items classes as defined in [3]. 2 General context and definitions In tis section, we first formally define te studied problem. Ten we present te Abstract Data Structure called Map used in our algoritms and discuss on its possible implementations. 2.1 Clone items in a Sets Collection Let J be a set of items {x 1,..., x J } and M a sets collection on J. We denote by ϕ a,b : 2 J 2 J te mapping wic associates to any subset of J its image by swapping items a and b. More formally : (m \ {a}) {b} if b m and a m ϕ a,b (m) = (m \ {b}) {a} if a m and b m m oterwise Definition 1. Let M be a collection of sets defined on J. We say tat items a and b are clone items in M if and only if m M, ϕ a,b (m) M. Te clone items concept is a binary one. To te question are a and b clone items?, only te answer true or false is possible. It could be interesting to ave more precisions wen te negative response is given. Are a and b very far from being clone items or wy are not tey clone? For tis purpose, we introduce a measure to qualify te clone property. Tis measure will represent

3 72 Raoul Medina, Caroline Noyer, Olivier Raynaud a distance between two items a and b, based on te definition of te clone items property. Tis distance is exactly te number of elements m of M wic do not ave an image in M wen applying te swapping function ϕ a,b (m). Wen tis distance is zero, a and b are clone items. Te greater te distance is, te farter a and b are to be clone. More formally: Definition 2. Let M be a sets collection on J and let (a, b) in J 2. We call distance between a and b, denoted by d M (a, b), te mapping : J 2 N d M (a, b) {m M ϕa,b (m) M} Tanks to definition 2, clone items could be caraterized as follows : Proposition 1. Let M be a sets collection on J and (a, b) in J 2, a and b are clone items if and only if d M (a, b) = 0. Te problem we study in tis paper is te computation of distances between all pair of items of J: Problem 1 (Distance). Data : a sets collection M on J; Result : te matrix d M. Here, we present te main property on wic rely our algoritms. It caracterizes a couple (m, m ) of te sets collection M suc tat m = ϕ a,b (m ). Proposition 2. Let M be a sets collection defined over J, m and m two distinct sets of M and (a, b) two items of J suc tat a m and b m. Ten te following assertions are equivalent: 1. m = ϕ a,b (m ) 2. m = ϕ a,b (m) 3. m = m and m \ m = {a} and m \ m = {b} 4. m \ {a} = m \ {b} Tis property states tat two sets m and m are teir respective images by te swapping function ϕ if and only if tey ave same size t and sare t 1 items. Tis property follows directly from te definition of te ϕ mapping. 2.2 Abstract Data Type : Key Mapping Interface. We use a Map abstract data type similar to te Map interface of Java language. Tis data structure maps keys to values. In our case, te keys are te sets of te collection. Te values mapped by te keys depend on te algoritm. Tis abstract data type supplies te following operators: new() operator: creates a Map object and returns an empty map.

4 Efficient algoritms for clone items detection 73 get(e) operator: returns te value associated to te key e if tis key maps a value, or Nil oterwise. put(e,value) operator: inserts set e in te map and associates value to it. Time complexities of tose operators deeply rely on te data structure used for te implementation of te Map data type. We propose an implementation wic takes advantage of te type of te keys, i.e. sets. To implement te Map type we propose a lexicograpic tree: itemsets are represented by brances of te tree. Implementation: lexicograpic tree. We first give a formal definition of a lexicograpic tree corresponding to a sets collection. Definition 3. Let M be a sets collection defined over J, wit a total order on J denoted by < J. A unique lexicograpic tree is associated to M suc tat: Eac edge of te tree is labeled wit an element of J; To eac marked node of te tree corresponds a set of M; To eac set m of M correspond a unique pat in te tree (starting from root and ending wit a marked node) suc tat te union of labels in tis pat corresponds exactly to te set m. For any pat from te root to any node, te order of te successive labels respects te order defined by < J ; Te order of edges leaving a node respects te order defined by < J. Figure 1 gives an example of collection and its associated lexicograpic order. b a e c d {a,d} c d c b d {e} {c,d} e {a,b,c} e {b,c,d,e} {c,d,e} Fig. 1. Te lexicograpic tree corresponding to te collection {abc, ad, bcde, cd, cde, e}, wit J = {a, b, c, d, e} and < J being te alpabetical order of te items. Circled nodes are te marked nodes corresponding to sets of te collection. A boolean can be associated to a node of te tree in order to indicate if te node is marked (i.e. corresponds to a set of te collection and tus to a key) or

5 74 Raoul Medina, Caroline Noyer, Olivier Raynaud not. Any field type can be associated to te node for storing te value associated to te key (on marked nodes only). Tere are two ways of implementing te list of cildren of a node in te lexicograpic tree: Using a List structure, eac entry of te list containing te label of te associated edge and a reference to te cild; Using an Array structure being indexed by te labels and te entries containing eiter NIL or reference to te cild. Complexities of te put and get operators rely on te cosen implementation. Proposition 3. If te lexicograpic tree is implemented wit Lists ten: Te put(m,value) operator as an O( J ) time complexity; Te get(m) operator as an O( J ) time complexity; Access complexity is due to te fact tat an item appears only once in a set. Cost of a node creation is done in constant time. Proposition 4. If te lexicograpic tree is implemented wit Arrays ten: Te put(m,value) operator as an O( m J ) time complexity; Te get(m) operator as an O( m ) time complexity; Acces complexity is due to te fact tat we ave direct access to a cild labeled by a given item. Creation of a node of te tree costs J since we need to create and initialize a J sized array. 3 Strategies and algoritms In tis section, we study tree different strategies for solving te Distance problem. All strategies rely on te same basic idea: first te distance matrix is initialized wit te maximum possible values for eac distance d M (a, b). Ten, eac time te algoritm finds m and m suc tat m = ϕ a,b (m ), te distance d M (a, b) is decreased by one. Main difference between te tree strategies is te way tey detect tat m = ϕ a,b (m ). Te first strategy is te one given in [3]. We call it set existence cecking. Main idea of te algoritm is, given m M, to compute all possible sets ϕ a,b (m) and ten ceck if tese sets belong to te collection. Second strategy is called ϕ a,b relation cecking. Its principle is, for eac pair (m, m ), to ceck if tere exist items a and b suc tat m = ϕ a,b (m ). Tis test is done using Property 2. Tird strategy is called classes computation. Based on Property 2, it computes classes C of itemsets suc tat for any pair (m, m ) of C, tere exist items a and b suc tat m = ϕ a,b (m ). We first discuss about te initialization of te distance matrix since tis is a common ground for all strategies.

6 Efficient algoritms for clone items detection Discussion on te distance matrix Since te notion of distance d M (a, b) is a symetrical one, te distance matrix is also symmetrical. We tus coose to represent it by a triangular matrix suc tat: { dm (a, a) = 0 d M (a, b) = d M (b, a) In tis section we discuss te different strategies for te initialization and te update of te distance matrix, as well as teir respective costs. Let M = M ab + M ab + M ab + M ab, wit: - M ab : te sets m of M suc tat a m and b m - M ab : te sets m of M suc tat a m and b m - M ab : te sets m of M suc tat a m and b m - M ab : te sets m of M suc tat a m and b m. Incrementation or decrementation strategy? Tere are two ways of computing te distance matrix: Eiter by first initializing all distances to 0 and ten by increasing by 1 te distance d M (a, b) eac time we find m M suc tat ϕ a,b (m) M, or by first initializing te distances by a maximal value and ten decreasing by 1 te distance d M (a, b) eac time we find m and m M suc tat m = ϕ a,b (m ). Wat is te maximal value tat can be taken by d M (a, b)? Clearly te answer is M ab + M ab. Indeed, for any item m M ab M ab we ave m = ϕ a,b (m) and tus m cannot increase te distance between a and b. Tis represents te maximum number of basic operations (eiter incrementation or decrementation) needed to compute te distance matrix in te worst case. Tus, watever strategy we coose, te number of basic operations will be te same in te worst case. Wat is te best time complexity for bot strategies? We consider tat te basic operations can be done in O(1). Tus te overall complexity will be: O( M ab + M ab ). a J b J Now, consider m M. In te worst case, m will be taken into account in at most m J \ m distances d M (a, b). Indeed, for m to be taken into account in d M (a, b) eiter a or b belongs to m but not bot. Tus, we ave: O( M ab + M ab ) = O( m J \ m ). a J m M b J m M m M Tis can be rewritten as follows: O( m ( J m )) = O(( m J ) ( m m )). m M

7 76 Raoul Medina, Caroline Noyer, Olivier Raynaud And since m m is lesser or equal tan m J, we obtain te worst case complexity: O( m J ) = O( J M ). m M Tus, watever update strategy is cosen, time complexity cannot be less tan O( J M ) (upon te ypotesis tat te basic operations increment or decrement by 1). In tis paper we coose to adopt te decrementation strategy since our algoritms are based on Property 2. We now discuss te complexity of te initialization of te distance matrix. Initializing te distance matrix. We initialize eac distance d M (a, b) wit te maximal value possible, i.e. wit M ab + M ab. Initially, te distances are equal to 0. Tis can be done in O( J J ). Ten, for eac m M we increment by 1 te distances d M (a, b), wit a m and b m. As sown in previous subsection, tis can be done in O( J M ). Algoritm 1: InitDistance(M) Data : A sets collection M defined over J. Result : Te distance matrix d M suc tat for all a and b in J 2 we ave d M(a, b) = M ab + M ab. begin foreac a J do foreac b J do d M(a, b) = 0; foreac m M do foreac a m do foreac b m do d M(a, b) + +; return d M; end 3.2 Set Existence Cecking Strategy For eac set m of M and for all pair (a, b) of J 2, we ceck if ϕ a,b (m) belongs to M. In order to ceck te existence of ϕ a,b (m), we coose to store te collection M in te Map structure presented before. Te sets of M are te keys wile teir associated value is "present". Beware tat since a distance d M (a, b) is initialized wit M ab + M ab, tis distance sould be decremented only wen m ϕ a,b (m). Indeed, if m = ϕ a,b (m) ten m M ab M ab.

8 Efficient algoritms for clone items detection 77 Algoritm 2: ComputeDistance(M) : Set Existence Cecking Strategy Data : A sets collection M defined over J. Result : Te distance matrix d M. begin d M = InitDistance(M); T = new Map(); foreac m M do T.put(m, present ); 1 foreac m M do 2 foreac couple(a, b) J 2 do if T.get(m) NIL and m ϕ a,b (m) ten d M(a, b) ; return d M; end Proposition 5. Te Set Existence Cecking Strategy as an O( J 2 M ) time complexity. Proof. Initialization of te matrix is done in O( J M ). We suppose tat te T Map structure is implemented using Arrays. Tus, te initialization of T is done in O( m M m J ) = O( J M ). Loop on line 1 does M iterations wile loop of line 2 does J 2 iterations. Te ϕ a,b (m) operation as well as te test m ϕ a,b (m) and te get(m) operation can be done in O( m ) time complexity. Overall complexity is tus, O( m M J 2 m ) = O( J 2 M ). Tis strategy is a sligter improvement of te one presented in [3]. Note tat it can be improved a little more by coosing a in m and b in J \ m. 3.3 ϕ a,b Relation Cecking Strategy For any pair of elements (m, m ) of M, we ceck if tere exist a and b suc tat m = ϕ a,b (m ). According to Property 2, items a and b exist if and only if m and m ave same size and sare m 1 items. In tis case, d M (a, b) is decremented by 1. Proposition 6. Te ϕ a,b relation cecking strategy as an O(( J + M ) M ) time complexity. Proof. Initialisation of te matrix is done in O( J M ). External loop (line 1) does M 2 iterations. All tests of line 2 can be done in O( m ) provided tat te sets m and m are stored sorted according to < J. Te complexity of te loop is in O( M M ). Overall complexity is tus in O(( J + M ) M ). 3.4 Classes Computation Strategy Tis strategy also relies on Property 2. Let consider m, m and m be sets of M suc tat m = ϕ a,b (m ) and m = ϕ a,c (m ). Ten, according to Property

9 78 Raoul Medina, Caroline Noyer, Olivier Raynaud Algoritm 3: ComputeDistance(M): ϕ a,b Relation Cecking Strategy Data : A set collections M defined over J. Result : Te distance matrix d M. begin d M = InitDistance(M); 1 foreac (m, m ) M 2 do 2 if m = m and m \ m = 1 and m \ m = 1 ten d M(m \ m, m \ m) ; return d M; end 2 we ave m \ {a} = m \ {b} = m \ {c}. And tus, m = ϕ b,c (m ). Idea of te algoritm is to compute classes of sets m i of M aving m i 1 common items. Tus, a class C can be represented by te set of common items and we memorize in a set Union all te extra items x i wic are not common. In te Map structure we use, te set C will be te key wile te set Union will be te value associated to te key. Ten, for any m C and for any (a, b) Union, we know tat according to Property 2 we ave ϕ a,b (m) M and m ϕ a,b (m). And tus, d M (a, b) as to be decremented. Note tat a set m can belong to at most m classes. Te algoritm is quite straigtforward using te Map structure. For all sets m of M we insert eac of its m subsets of size m 1 in te Map structure. If te key was already present, we just append te extra item of m to te set Union and update all te necessary entries in te distance matrix. Oterwise, a new key m \ {x} is present in te Map structure and its associated Union value is initialized wit {x}. Proposition 7. Classes computation strategy as O( J M ) time complexity. Proof. Initialization of te distance matrix is done in O( J M ). We suppose tat te Map structure is implemented using a lexicograpic tree wit Lists. Te line 1 loop does M iterations. Loop in line 2 does m iterations. In line 3, te retrieval is done in O( J ). Te loop in line 4 does at most J iterations and te update of te matrix takes constant time. Te insertion of line 5 is done in O( J ). Tus, te overall complexity is in O( m M m J ) = O( J M ). Let us illustrate Algoritm 4 wit an example. Te considered collection is M = {m 1 = {a, e, f, }, m 2 = {b, e, f, }, m 3 = {b, d, f, }}. Te resulting lexicograpic tree obtained wit Algoritm 4 is sown in Figure 2. From tis tree, we conclude tat m 2 and m 3 sare te items {b, f, } and tat m 2 = ϕ d,e (m 3 ). Tus, d M (d, e) sould be decremented. For te same reason, te distance d M (a, b) sould also be decremented since m 1 = ϕ a,b (m 2 ) (tey sare te items {e, f, }).

10 Efficient algoritms for clone items detection 79 Algoritm 4: ComputeDistance(M): Classes Computation Strategy Data : A set collections M defined over J. Result : Te distance matrix d M. begin d M = InitDistance(M); T = new Map(); 1 foreac m M do 2 foreac x m do C = m \ {x} 3 Union = T.get(C) if Union Nil ten 4 foreac y Union do d M(x, y) ; Union = Union {x}; T.put(C, Union); else 5 T.put(C, {x}) return d M; end 3.5 Memory usage In tis section we discuss te space complexity required by eac strategy and sow ow to reduce te memory usage wen possible. First, it is obvious tat te distance matrix sould be present in memory. It requires O( J 2 ) memory space. Concerning te Set Existence Cecking Strategy, a lexicograpic tree implemented wit arrays is used. Te number of nodes is clearly bounded by M. And since eac node requires a J sized array, tis strategy uses O( J M ) memory space. For te ϕ a,b Relation Cecking Strategy, no lexicograpic tree is used. However, te collection M needs to be present in memory. Tus, tis strategy requires O( M ) memory space. Now, for te Classes Computation Strategy, te used lexicograpic tree is implemented using lists. Suc implementation requires O( M ) memory space. But M is not te collection stored in te tree. Indeed, for eac m M, we store its m subsets of size m 1. And since m is bounded by J we conclude tat tis strategy requires O( J M ) memory space. Te conclusion seems to be tat watever strategy is used, at least O( M ) memory space will be needed. But one can notice tat, according to Property 2, not all sets in M need to be present in memory at te same time. Indeed, if m = ϕ a,b (m ), ten m and m ave same size. Te idea is ten to do a partitionning of M according to te size of te sets. Tus, only sets of same size need to be present in memory at te same time. Computations done for

11 80 Raoul Medina, Caroline Noyer, Olivier Raynaud a b d e e f d e f f f f f f () (f) (e) () (f) () (f) (d,e) (b) (a,b) Fig. 2. Te lexicograpic tree corresponding to te collection M = {{a, e, f, }, {b, e, f, }, {b, d, f, }}. Circled nodes correspond to sets of M. Te Union value associated to tese node is presented in parentesis. tose sets is totally independent of computations done for sets wit a different size. Note tat te partitionning of M can be done in O( M ) time complexity, wit a single scan of te collection. Tus, te partitionning does not cange te overall complexity of te different strategies. Anoter remark is tat since computations by size are totally independent, one could easily implement a distributed version of te strategies. Tis could eventually speed up te computation of te distance matrix. But note tat in tis case, te distance matrix sould also be distributed. Best solution sould be to initialize local distance matrices wit te local partition. After local computation (for sets wit same size), all te distance matrices sould be merged togeter (by adding all te obtained values) in order to obtain te global distance matrix. 4 Clone classes computation Te problem stated in [1] and in [3] is te computation of clone classes. We tus give te algoritm wic computes clone classes from a distance matrix. First, let us recall tat two items a and b are clone if and only if d M (a, b) = 0. Since, te clone relation is an equivalence relation, it defines a partition of te set J. Principle of our algoritm 5 is te following. Let a J be an item wic as still not been assigned to a class. We ten searc all remaining items b wic distance wit a is null. All tose items will form a clone class wit a and tus are removed from te list of items wic are not assigned to a class. Te class of a is ten stored in a list L. Total complexity of te clone classes computation is in O( J M ) time and space complexity. 5 Conclusion We ave seen tat if te problem is te computation of te distance matrix and only basic incrementation or decrementation by 1 are allowed, ten te minimal

12 Efficient algoritms for clone items detection 81 Algoritm 5: ComputeCloneClasses(M) Data : A sets collection M defined over J. Result : Te list L of clone classes. begin d M = ComputeDistance(M); L = ; temp = J; wile temp do 1 foreac a temp do l a = newlist(); l a = l a {a}; temp = temp \ {a}; 2 foreac b temp do 3 if d M(a, b) = 0 ten 4 l a = l a {b}; 5 temp = temp \ {b}; return L; end L = L + l a; time complexity for te update of te matrix is in O( J M ). Tus, under tis ypotesis, our algoritm is optimal. Figure 3 gives te different time and space complexities obtained wit our algoritms. Algoritm Time complexity Space complexity Set Existence Cecking J 2 M J M ϕ a,b Relation Cecking ( J + M ) M M Classes Computation J M J M Optimal matrix computation J M? Fig. 3. Complexities of algoritms presented in te paper. An open question is to know weter or not te distance matrix could be incremented (or decremented) by more tan 1 at eac step. Tis could be te only way of improving our algoritm. Anoter open question is to know if tere is a more efficient algoritm to compute te clone classes (for instance witout computing te distance matrix). Tose are two questions we are investigating. References 1. A. Gely, R. Medina, L. Nourine, and Y. Renaud. Uncovering and reducing idden combinatorics in guigues-duquenne covers. In ICFCA 05, H. Manilla and K.J. Räiä. On te complexity of inferring functionnal dependencies. Discret Applied Matematics, 40(2): , R. Medina and L. Nourine. Clone items: a pre-processing information for knowledge discovery. submitted.