Analyses and Validation of Conditional Dependencies with Built-in Predicates

Transcription

1 Analyses and Validation of Conditional Deendencies with Built-in Predicates Wenguang Chen 1, Wenfei Fan 2,3, and Shuai Ma 2 1 Peking University, China 2 University of Edinburgh, UK 3 Bell Laboratories, USA Abstract. This aer rooses a natural extension of conditional functional deendencies (cfds [14]) and conditional inclusion deendencies (s [8]), denoted by cfd s and s, resectively, by secifying atterns of data values with, <,, > and redicates. As data quality rules, cfd s and s are able to cature errors that commonly arise in ractice but cannot be detected by cfds and s. We establish two sets of results for central technical roblems associated with cfd s and s. (a) One concerns the satisfiability and imlication roblems for cfd s and s, taken searately or together. These are imortant for, e.g., deciding whether data quality rules are dirty themselves, and for removing redundant rules. We show that desite the increased exressive ower, the static analyses of cfd s and s retain the same comlexity as their cfds and s counterarts. (b) The other concerns validation of cfd s and s. We show that given a set Σ of cfd s and s on a database D, a set of sql queries can be automatically generated that, when evaluated against D, return all tules in D that violate some deendencies in Σ. This rovides commercial dbms with an immediate caability to detect errors based on cfd s and s. Key words: functional deendency, inclusion deendency, data quality 1 Introduction Extensions of functional deendencies (fds) and inclusion deendencies (inds), known as conditional functional deendencies (cfds [14]) and conditional inclusion deendencies (s [8]), resectively, have recently been roosed for imroving data quality. These extensions enforce atterns of semantically related data values, and detect errors as violations of the deendencies. Conditional deendencies are able to cature more inconsistencies than fds and inds [14, 8]. Conditional deendencies secify constant atterns in terms of equality (=). In ractice, however, the semantics of data often needs to be secified in terms of other redicates such as, <,, > and, as illustrated by the examle below. Examle 1. An online store maintains a database of two relations: (a) item for items sold by the store, and (b) tax for the sale tax rates for the items, excet artwork, in various states. The relations are secified by the following schemas: item (id: string, name: string, tye: string, rice: float, shiing: float, sale: bool, state: string) tax (state: string, rate: float)

2 2 Wenguang Chen, Wenfei Fan, and Shuai Ma id name tye rice shiing sale state t 1: b1 Harry Potter book T WA t 2: c1 Snow White CD F NY t 3: b2 Catch-22 book F DL t 4: a1 Sunflowers art 5m 500 F DL (a) An item relation Fig. 1. Examle instance D 0 of item and tax state rate t 5: PA 6 t 6: NY 4 t 7: DL 0 t 8: NJ 3.5 (b) tax rates where each item is secified by its id, name, tye (e.g., book, cd), rice, shiing fee, the state to which it is shied, and whether it is on sale. A tax tule secifies the sale tax rate in a state. An instance D 0 of item and tax is shown in Fig. 1. One wants to secify deendencies on the relations as data quality rules to detect errors in the data, such that inconsistencies emerge as violations of the deendencies. Traditional deendencies (fds, inds; see, e.g., [1]) and conditional deendencies (cfds, s [14, 8]) on the data include the following: cfd 1 : item (id name, tye, rice, shiing, sale) cfd 2 : tax (state rate) cfd 3 : item (sale = T shiing = 0) These are cfds: (a) cfd 1 assures that the id of an item uniquely determines the name, tye, rice, shiing, sale of the item; (b) cfd 2 states that state is a key for tax, i.e., for each state there is a unique sale tax rate; and (c) cfd 3 is to ensure that for any item tule t, if t[sale] = T then t[shiing] must be 0; i.e., the store rovides free shiing for items on sale. Here cfd 3 is secified in terms of atterns of semantically related data values, namely, sale = T and shiing = 0. It is to hold only on item tules that match the attern sale = T. In contrast, cfd 1 and cfd 2 are traditional fds without constant atterns, a secial case of cfds. One can verify that no sensible inds or s can be defined across item and tax. Note that D 0 of Fig. 1 satisfies cfd 1, cfd 2 and cfd 3. That is, when these deendencies are used as data quality rules, no errors are found in D 0. In ractice, the shiment fee of an item is tyically determined by the rice of the item. Moreover, when an item is on sale, the rice of the item is often in a certain range. Furthermore, for any item sold by the store to a customer in a state, if the item is not artwork, then one exects to find the sale tax rate in the state from the tax table. These semantic relations cannot be exressed as cfds of [14] or s of [8], but can be exressed as the following deendencies: fd 1 : item (sale = F & rice 20 shiing = 3) fd 2 : item (sale = F & rice > 20 & rice 40 shiing = 6) fd 3 : item (sale = F & rice > 40 shiing = 10) fd 4 : item (sale = T rice 2.99 & rice < 9.99) ind 1 : item (state; tye art ) tax (state; nil) Here fd 2 states that for any item tule, if it is not on sale and its rice is in the range (20, 40], then its shiment fee must be 6; similarly for fd 1 and fd 3. These deendencies extend cfds [14] by secifying atterns of semantically related data values in terms of redicates <,, >, and. Similarly, fd 4 assures that for any item tule, if it is on sale, then its rice must be in the range [2.99, 9.99). Deendency ind 1 extends s [8] by secifying atterns with : for any item

3 Conditional Deendencies with Built-in Predicates 3 tule t, if t[tye] is not artwork, then there must exist a tax tule t such that t[state] = t [state], i.e., the sale tax of the item can be found from the tax relation. Using fd 1 fd 4 and ind 1 as data quality rules, we find that D 0 of Fig. 1 is not clean. Indeed, (a) t 2 violates fd 1 : its rice is less than 20, but its shiing fee is 2 rather than 3; similarly, t 3 violates fd 2, and t 4 violates fd 3. (b) Tule t 1 violates fd 4 : it is on sale but its rice is not in the range [2.99, 9.99). (c) The database D 0 also violates ind 1 : t 1 is not artwork, but its state cannot find a match in the tax relation, i.e., no tax rate for WA is found in D 0. None of fd 1 fd 4 and ind 1 can be exressed as fds or inds [1], which do not allows constants, or as cfds [14] or s [8], which secify atterns with equality (=) only. While there have been extensions of cfds [7, 18], none of these allows deendencies to be secified with atterns on data values in terms of built-in redicates, <,, > or. To the best of our knowledge, no revious work has studied extensions of s (see Section 6 for detailed discussions). These highlight the need for extending cfds and s to cature errors commonly found in real-life data. While one can consider arbitrary extensions, it is necessary to strike a balance between the exressive ower of the extensions and their comlexity. In articular, we want to be able to reason about data quality rules exressed as extended cfds and s. Furthermore, we want to have effective algorithms to detect inconsistencies based on these extensions. Contributions. This aer rooses a natural extension of cfds and s, rovides comlexity bounds for reasoning about the extension, and develos effective sql-based techniques for detecting errors based on the extension. (1) We roose two classes of deendencies, denoted by cfd s and s, which resectively extend cfds and s by suorting, <,, >, redicates. For examle, all the deendencies we have encountered so far can be exressed as cfd s or s. These deendencies are caable of caturing errors in realworld data that cannot be detected by cfds or s. (2) We establish comlexity bounds for the satisfiability roblem and the imlication roblem for cfd s and s, taken searately or together. The satisfiability roblem is to determine whether a set Σ of deendencies has a nonemty model, i.e., whether the rules in Σ are consistent themselves. The imlication roblem is to decide whether a set Σ of deendencies entails another deendency ϕ, i.e., whether the rule ϕ is redundant in the resence of the rules in Σ. These are the central technical roblems associated with any deendency language. We show that desite the increased exressive ower, cfd s and s do not increase the comlexity for reasoning about them. In articular, we show that the satisfiability and imlication roblems remain (a) n-comlete and con-comlete for cfd s, resectively, (b) in O(1)-time (constant-time) and extime-comlete for s, resectively, and (c) are undecidable when cfd s and s are taken together. These are the same as their cfds and s counterarts. While data with linearly ordered domains often makes our lives harder (see, e.g., [21]), cfd s and s do not comlicate their static analyses. (3) We rovide sql-based techniques to detect errors based on cfd s and s.

4 4 Wenguang Chen, Wenfei Fan, and Shuai Ma (1) ϕ 1 = tax (state rate, T 1) (2) ϕ 2 = item (sale shiing, T 2) state rate sale shiing T 1: T 2: = T = 0 (3) ϕ 3 = item (sale, rice shiing, T 3) (4) cfd ϕ 4 = item (sale rice, T 4) sale rice shiing sale rice = F > 20 = 6 = T 2.99 T 3: = F 40 = 6 T 4: = T < 9.99 Fig. 2. Examle cfd s Given a set Σ of cfd s and s on a database D, we automatically generate a set of sql queries that, when evaluated on D, find all tules in D that violate some deendencies in Σ. Further, the sql queries are indeendent of the size and cardinality of Σ. No revious work has been studied error detection based on s, not to mention cfd s and s taken together. These rovide the caability of detecting errors in a single relation (cfd s) and across different relations ( s) within the immediate reach of commercial dbms. Organizations. Sections 2 and 3 introduce cfd s and s, resectively. Section 4 establishes comlexity bounds for reasoning about cfd s and s. Section 5 rovides sql techniques for error detection. Related work is discussed in Section 6, followed by toics for future work in Section 7. 2 Incororating Built-in Predicates into CFDs We now define cfd s, also referred to as conditional functional deendencies, by extending cfds with redicates (, <,, >, ) in addition to equality (=). Consider a relation schema R defined over a finite set of attributes, denoted by attr(r). For each attribute A attr(r), its domain is secified in R, denoted as dom(a), which is either finite (e.g., bool) or infinite (e.g., string). We assume w.l.o.g. that a domain is totally ordered if <,, > or is defined on it. Syntax. A cfd ϕ on R is a air R(X Y, T ), where (1) X, Y are sets of attributes in attr(r); (2) X Y is a standard fd, referred to as the fd embedded in ϕ; and (3) T is a tableau with attributes in X and Y, referred to as the attern tableau of ϕ, where for each A in X Y and each tule t T, t [A] is either an unnamed variable that draws values from dom(a), or o a, where o is one of =,, <,, >,, and a is a constant in dom(a). If attribute A occurs in both X and Y, we use A L and A R to indicate the occurrence of A in X and Y, resectively, and searate the X and Y attributes in a attern tule with. We write ϕ as (X Y, T ) when R is clear from the context, and denote X as LHS(ϕ) and Y as RHS(ϕ). Examle 2. The deendencies cfd1 cfd3 and fd1 fd4 that we have seen in Examle 1 can all be exressed as cfd s. Figure 2 shows some of these cfd s: ϕ 1 (for fd cfd 2 ), ϕ 2 (for cfd cfd 3 ), ϕ 3 (for fd 2 ), and ϕ 4 (for fd 4 ). Semantics. Consider cfd ϕ = (R : X Y, T ), where T = {t 1,..., t k }. A data tule t of R is said to match LHS(ϕ), denoted by t[x] T [X], if for each tule t i in T and each attribute A in X, either (a) t i [A] is the wildcard

5 Conditional Deendencies with Built-in Predicates 5 (which matches any value in dom(a)), or (b) t[a] o a if t i [A] is o a, where the oerator o (=,, <,, > or ) is interreted by its standard semantics. Similarly, the notion that t matches RHS(ϕ) is defined, denoted by t[y ] T [Y ]. Intuitively, each attern tule t i secifies a condition via t i [X], and t[x] T [X] if t[x] satisfies the conjunction of all these conditions. Similarly, t[y ] T [Y ] if t[y ] matches all the atterns secified by t i [Y ] for all t i in T. An instance I of R satisfies the cfd ϕ, denoted by I = ϕ, if for each air of tules t 1, t 2 in the instance I, if t 1 [X] = t 2 [X] T [X], then t 1 [Y ] = t 2 [Y ] T [Y ]. That is, if t 1 [X] and t 2 [X] are equal and in addition, they both match the attern tableau T [X], then t 1 [Y ] and t 2 [Y ] must also be equal to each other and they both match the attern tableau T [Y ]. Observe that ϕ is imosed only on the subset of tules in I that match LHS(ϕ), rather than on the entire I. For all tules t 1, t 2 in this subset, if t 1 [X] = t 2 [X], then (a) t 1 [Y ] = t 2 [Y ], i.e., the semantics of the embedded fds is enforced; and (b) t 1 [Y ] T [Y ], which assures that the constants in t 1 [Y ] match the constants in t i [Y ] for all t i in T. Note that here tules t 1 and t 2 can be the same. An instance I of R satisfies a set Σ of cfd s, denoted by I = Σ, if I = ϕ for each cfd ϕ in Σ. Examle 3. The instance D 0 of Fig. 1 satisfies ϕ 1 and ϕ 2 of Fig. 2, but neither ϕ 3 nor ϕ 4. Indeed, tule t 3 violates (i.e., does not satisfy) ϕ 3, since t 3 [sale] = F and 20 < t 3 [rice] 40, but t 3 [shiing] is 20 instead of 6. Note that t 3 matches LHS(ϕ 3 ) since it satisfies the condition secified by the conjunction of the attern tules in T 3. Similarly, t 1 violates ϕ 4, since t 1 [sale] = T but t 1 [rice] > Observe that while it takes two tules to violate a standard fd, a single tule may violate a cfd. Secial cases. (1) A standard fd X Y [1] can be exressed as a cfd (X Y, T ) in which T contains a single tule consisting of only, without constants. (2) A cfd (X Y, T ) [14] with T = {t 1,..., t k } can be exressed as a set {ϕ 1,..., ϕ k } of cfd s such that for i [1, k], ϕ i = (X Y, T i ), where T i contains a single attern tule t i of T, with equality (=) only. For examle, ϕ 1 and ϕ 2 in Fig. 2 are cfd s reresenting fd cfd2 and cfd cfd3 in Examle 1, resectively. Note that all data quality rules in [10, 18] can be exressed as cfd s. 3 Incororating Built-in Predicates into CINDs Along the same lines as cfd s, we next define s, also referred to as conditional inclusion deendencies. Consider two relation schemas R 1 and R 2. Syntax. A ψ is a air (R 1 [X; X ] R 2 [Y ; Y ], T ), where (1) X, X and Y, Y are lists of attributes in attr(r 1 ) and attr(r 2 ), resectively; (2) R 1 [X] R 2 [Y ] is a standard ind, referred to as the ind embedded in ψ; and (3) T is a tableau, called the attern tableau of ψ defined over attributes X Y, and for each A in X or Y and each attern tule t T, t [A] is either an

6 6 Wenguang Chen, Wenfei Fan, and Shuai Ma (1) ψ 1 = (item [state; tye] tax [state; nil], T 1), (2) ψ 2 = (item [state; tye, state] tax [state; rate], T 2) tye nil tye state rate T 1: T 2: art art = DL = 0 Fig. 3. Examle s unnamed variable that draws values from dom(a), or o a, where o is one of =,, <,, >, and a is a constant in dom(a). We denote X X as LHS(ψ) and Y Y as RHS(ψ), and searate the X and Y attributes in a attern tule with. We use nil to denote an emty list. Examle 4. Figure 3 shows two examle s: ψ 1 exresses ind 1 of Examle 1, and ψ 2 refines ψ 1 by stating that for any item tule t 1, if its tye is not art and its state is DL, then there must be a tax tule t 2 such that its state is DL and rate is 0, i.e., ψ 2 assures that the sale tax rate in Delaware is 0. Semantics. Consider ψ = (R 1 [X; X ] R 2 [Y ; Y ], T ). An instance (I 1, I 2 ) of (R 1, R 2 ) satisfies the ψ, denoted by (I 1, I 2 ) = ψ, iff for each tule t 1 I 1, if t 1 [X ] T [X ], then there exists a tule t 2 I 2 such that t 1 [X] = t 2 [Y ] and moreover, t 2 [Y ] T [Y ]. That is, if t 1 [X ] matches the attern tableau T [X ], then ψ requires the existence of t 2 such that (1) t 1 [X] = t 2 [Y ] as required by the standard ind embedded in ψ; and (2) t 2 [Y ] must match the attern tableau T [Y ]. In other words, ψ is conditional since its embedded ind is alied only to the subset of tules in I 1 that match T [X ], and moreover, the attern T [Y ] is enforced on the tules in I 2 that match those tules in I 1. As remarked in Section 2, the attern tableau T secifies the conjunction of atterns of all tules in T. Examle 5. The instance D 0 of item and tax in Fig. 1 violates ψ 1. Indeed, tule t 1 in item matches LHS(ψ 1 ) since t 1 [tye] art, but there is no tule t in tax such that t[state] = t 1 [state] = WA. In contrast, D 0 satisfies ψ 2. We say that a database D satisfies a set Σ of s, denoted by D = Σ, if D = ϕ for each ϕ Σ. Safe CIND s. We say a (R 1 [X; X ] R 2 [Y ; Y ], T ) is unsafe if there exist attern tules t, t in T such that either (a) there exists B Y, such that t [B] and t [B] are not satisfiable when taken together, or (b) there exist C Y, A X such that A corresonds to B in the ind and t [C] and t [A] are not satisfiable when taken together; e.g., t [rice] = 9.99 and t [rice] Obviously unsafe s do not make sense: there exist no nonemty database that satisfies unsafe s. It takes O( T 2 )-time in the size T of T to decide whether a is unsafe. Thus in the sequel we consider safe only. Secial cases. Observe that (1) a standard (R 1 [X] R 2 [Y ]) can be exressed as a (R 1 [X; nil] R 2 [Y ; nil], T ) such that T is simly a emty set; and (2) a (R 1 [X; X ] R 2 [Y ; Y ], T ) with T = {t 1,..., t k } can be exressed as a set {ψ 1,..., ψ k } of s, where for i [1, k], ψ i = (R 1 [X; X ] R 2 [Y ; Y ], T i ) such that T i consists of a single attern tule t i of T defined in terms of equality (=) only.

7 Conditional Deendencies with Built-in Predicates 7 4 Reasoning about CFD s and CIND s The satisfiability roblem and the imlication roblem are the two central technical questions associated with any deendency languages. In this section we investigate these roblems for cfd s and s, searately and taken together. 4.1 The Satisfiability Analysis The satisfiability roblem is to determine, given a set Σ of constraints, whether there exists a nonemty database that satisfies Σ. The satisfiability analysis of conditional deendencies is not only of theoretical interest, but is also imortant in ractice. Indeed, when cfd s and s are used as data quality rules, this analysis hels one check whether the rules make sense themselves. The need for this is articularly evident when the rules are manually designed or discovered from various datasets [10, 18, 15]. The satisfiability analysis of CFD s. Given any fds, one does not need to worry about their satisfiability since any set of fds is always satisfiable. However, as observed in [14], for a set Σ of cfds on a relational schema R, there may not exist a nonemty instance I of R such that I = Σ. As cfds are a secial case of cfd s, the same roblem exists when it comes to cfd s. Examle 6. Consider cfd ϕ = (R : A B, T ) such that T = {( = a), ( = a)}. Then there exists no nonemty instance I of R that satisfies ϕ. Indeed, for any tule t of R, ϕ requires that both t[b] = a and t[b] a. This roblem is already n-comlete for cfds [14]. Below we show that it has the same comlexity for cfd s desite their increased exressive ower. Proosition 1. The satisfiability roblem for cfd s is n-comlete. Proof sketch: The lower bound follows from the n-hardness of their cfds counterarts [14], since cfds are a secial case of cfd s. The uer bound is verified by resenting an n algorithm that, given a set Σ of cfd s defined on a relation schema R, determines whether Σ is satisfiable. It is known [14] that the satisfiability roblem for cfds is in time when the cfds considered are defined over attributes that have an infinite domain, i.e., in the absence of finite domain attributes. However, this is no longer the case for cfd s. This tells us that the increased exressive ower of cfd s does take a toll in this secial case. It should be remarked that while the roof of Proosition 1 is an extension of its counterart in [14], the result below is new. Theorem 2. In the absence of finite domain attributes, the satisfiability roblem for cfd s remains n-comlete. Proof sketch: The roblem is in n by Proosition 1. Its n-hardness is shown by reduction from the 3SAT roblem, which is n-comlete (cf. [17]). The satisfiability analysis of CIND s. Like fds, one can secify arbitrary inds or s without worrying about their satisfiability. Below we show that s also have this roerty, by extending the roof of its counterart in [8].

8 8 Wenguang Chen, Wenfei Fan, and Shuai Ma Proosition 3. Any set Σ of s is always satisfiable. Proof sketch: Given a set Σ of s over a database schema R, one can always construct a nonemty instance D of R such that D = Σ. The satisfiability analysis of CFD s and CIND s. The satisfiability roblem for cfds and s taken together is undecidable [8]. Since cfd s and s subsume cfds and s, resectively, from these we immediately have: Corollary 4. The satisfiability roblem for cfd s and s is undecidable. 4.2 The Imlication Analysis The imlication roblem is to determine, given a set Σ of deendencies and another deendency φ, whether or not Σ entails φ, denoted by Σ = φ. That is, whether or not for all databases D, if D = Σ then D = φ. The imlication analysis hels us remove redundant data quality rules, and thus imrove the erformance of error detection and reairing based on the rules. Examle 7. The cfd s of Fig. 2 imly cfd s ϕ = item (sale, rice shiing, T ), where T consists of a single attern tule (sale = F, rice = 30 shiing = 6). Thus in the resence of the cfd s of Fig. 2, ϕ is redundant. The imlication analysis of CFD s. We first show that the imlication roblem for cfd s retains the same comlexity as their cfds counterart. The result below is verified by extending the roof of its counterart in [14]. Proosition 5. The imlication roblem for cfd s is con-comlete. Proof sketch: The lower bound follows from the con-hardness of their cfds counterart [14], since cfds are a secial case of cfd s. The con uer bound is verified by resenting an n algorithm for its comlement roblem, i.e., the roblem for determining whether Σ = ϕ. Similar to the satisfiability analysis, it is known [14] that the imlication analysis of cfds is in time when the cfds are defined only with attributes that have an infinite domain. Analogous to Theorem 2, the result below shows that this is no longer the case for cfd s, which does not find a counterart in [14]. Theorem 6. In the absence of finite domain attributes, the imlication roblem for cfd s remains con-comlete. Proof sketch: It is in con by Proosition 5. The con-hardness is shown by reduction from the 3SAT roblem to its comlement roblem, i.e., the roblem for determining whether Σ = ϕ. The imlication analysis of CIND s. We next show that s do not make their imlication analysis harder. This is verified by extending the roof of their s counterart given in [8]. Proosition 7. The imlication roblem for s is extime-comlete.

9 Σ Conditional Deendencies with Built-in Predicates 9 General setting Infinite domain only Satisfiability Imlication Satisfiability Imlication cfds [14] n-comlete con-comlete time time cfd s n-comlete con-comlete n-comlete con-comlete s [8] O(1) extime-comlete O(1) sace-comlete s O(1) extime-comlete O(1) extime-comlete cfds + s [8] undecidable undecidable undecidable undecidable cfd s + s undecidable undecidable undecidable undecidable Table 1. Summary of comlexity results Proof sketch: The imlication roblem for s is extime-hard [8]. The lower bound carries over to s since s subsume s. The extime uer bound is shown by resenting an extime algorithm that, given a set Σ {ψ} of s over a database schema R, determines whether Σ = ψ. It is known [8] that the imlication roblem is sace-comlete for s defined with infinite-domain attributes. Similar to Theorem 6, below we resent a new result showing that this no longer holds for s. Theorem 8. In the absence of finite domain attributes, the imlication roblem for s remains extime-comlete. Proof sketch: The extime uer bound follows from Proosition 7. The extime-hardness is shown by reduction from the imlication roblem for s in the general setting, in which finite-domain attributes may be resent; the latter is known to be extime-comlete [8]. The imlication analysis of CFD s and CIND s. When cfd s and s are taken together, their imlication analysis is beyond reach in ractice. This is not surrising since the imlication roblem for fds and inds is already undecidable [1]. Since cfd s and s subsume fds and inds, resectively, from the undecidability result for fds and inds, the corollary below follows immediately. Corollary 9. The imlication roblem for cfd s and s is undecidable. Summary. The comlexity bounds for reasoning about cfd s and s are summarized in Table 1. To give a comlete icture we also include in Table 1 the comlexity bounds for the static analyses of cfds and s, taken from [14, 8]. The results shown in Table 1 tell us the following. (a) Desite the increased exressive ower, cfd s and s do not comlicate the static analyses: the satisfiability and imlication roblems for cfd s and s have the same comlexity bounds as their counterarts for cfds and s, taken searately or together. (b) In the secial case when cfd s and s are defined with infinite-domain attributes only, however, the static analyses of cfd s and s do not get simler, as oosed to their counterarts for cfds and s. That is, in this secial case the increased exressive ower of cfd s and s comes at a rice.

10 10 Wenguang Chen, Wenfei Fan, and Shuai Ma 5 Validation of CFD s and CIND s If cfd s and s are to be used as data quality rules, the first question we have to settle is how to effectively detect errors and inconsistencies as violations of these deendencies, by leveraging functionality suorted by commercial dbms. More secifically, consider a database schema R = (R 1,..., R n ), where R i is a relation schema for i [1, n]. The error detection roblem is stated as follows. The error detection roblem is to find, given a set Σ of cfd s and s defined on R, and a database instance D = (I 1,..., I n ) of R as inut, the subset (I 1,..., I n) of D such that for each i [1, n], I i I i and each tule in I i violates at least one cfd or in Σ. We denote the set as vio(d, Σ), referred to it as the violation set of D w.r.t. Σ. In this section we develo sql-based techniques for error detection based on cfd s and s. The main result of the section is as follows. Theorem 10. Given a set Σ of cfd s and s defined on R and a database instance D of R, where R = (R 1,..., R n ), a set of sql queries can be automatically generated such that (a) the collection of the answers to the sql queries in D is vio(d, Σ), (b) the number and size of the set of sql queries deend only on the number n of relations and their arities in R, regardless of Σ. We next resent the main techniques for the query generation method. Let Σcfd i be the set of all cfd s in Σ defined on the same relation schema R i, and Σ (i,j) the set of all s in Σ from R i to R j, for i, j [1, n]. We show the following. (a) The violation set vio(d, Σcfd i ) can be comuted by two sql queries. (b) Similarly, vio(d, Σ (i,j) ) can be comuted by a single sql query. (c) These sql queries encode attern tableaux of cfd s ( s) with data tables, and hence their sizes are indeendent of Σ. From these Theorem 10 follows immediately. 5.1 Encoding CFD s and CIND s with Data Tables We first show the following, by extending the encoding of [14, 7]. (a) The attern tableaux of all cfd s in Σcfd i can be encoded with three data tables, and (b) the attern tableaux of all s in Σ (i,j) can be reresented as four data tables, no matter how many deendencies are in the sets and how large they are. Encoding CFD s. We encode all attern tableaux in Σcfd i with three tables enc L, enc R and enc, where enc L (res. enc R ) encodes the non-negation (=, <,, >, ) atterns in LHS (res. RHS), and enc encodes those negation ( ) atterns. More secifically, we associate a unique id cid with each cfd s in Σcfd i, and let enc L consist of the following attributes: (a) cid, (b) each attribute A aearing in the LHS of some cfd s in Σcfd i, and (b) its four comanion attributes A >, A, A <, and A. That is, for each attribute, there are five columns in enc L, one for each non-negation oerator. Similarly, enc R is defined. We use an enc tule to encode a attern A c in a cfd, consisting of cid, att, os, and val, encoding the cfd id, the attribute A, the osition ( LHS or RHS ),

11 Conditional Deendencies with Built-in Predicates 11 (1) enc L (2) enc R (3) enc cid sale rice rice > rice 2 T null null null 3 F T null null null cid shiing rice rice rice < 2 0 null null null 3 6 null null null 4 null cid os att val Fig. 4. Encoding examle of cfd s and the constant c, resectively. Note that the arity of enc L (enc R ) is bounded by 5 R i + 1, where R i is the arity of R i, and the arity of enc is 4. Before we oulate these tables, let us first describe a referred form of cfd s that would simlify the analysis to be given. Consider a cfd ϕ = R(X Y, T ). If ϕ is not satisfiable we can simly dro it from Σ. Otherwise it is equivalent to a cfd ϕ = R(X Y, T ) such that for any attern tules t, t in T and for any attribute A in X Y, (a) if t [A] is o a and t [A] is o b, where o is not, then a = b, (b) if t [A] is then so is t [A]. That is, for each non-negation o (res. ), there is a unique constant a such that t [A] = o a (res. t [A] = ) is the only o (res. ) attern aearing in the A column of T. We refer to t [A] as T (o, A) (res. T (, A)), and consider w.l.o.g. cfd s of this form only. Note that there are ossibly multile t [A] c atterns in T, We oulate enc L, enc R and enc as follows. For each cfd ϕ = R(X Y, T ) in Σ i cfd, we generate a distinct cid id ϕ for it, and do the following. Add a tule t 1 to enc L such that (a) t[cid] = id ϕ ; (b) for each A X, t[a] = if T (, A) is, and for each non-negation redicate o, t[a o ] = a if T (o, A) is o a ; (c) we let t[b] = null for all other attributes B in enc L. Similarly add a tule t 2 to enc R for attributes in Y. For each attribute A X Y and each a attern in T [A], add a tule t to enc such that t[cid] = id ϕ, t[att] = A, t[val] = a, and t[os] = LHS (res. t[os] = RHS ) if attribute A aears in X (res. Y ). Examle 8. Recall from Fig. 2 cfd s ϕ 2, ϕ 3 and ϕ 4 defined on relation item. The three cfd s are encoded with tables shown in Fig. 4: (a) enc L consists of attributes: cid, sale, rice, rice > and rice ; (b) enc R consists of cid, shiing, rice, rice and rice < ; those attributes in a table with only null attern values do not contribute to error detection, and are thus omitted; (c) enc is emty since all these cfd s have no negation atterns. One can easily reconstruct these cfd s from tables enc L, enc R and enc by collating tules based on cid. Encoding CIND s. All s in Σ (i,j) can be encoded with four tables enc, enc L, enc R and enc. Here enc L (res. enc R ) and enc encode non-negation atterns on relation R i (res. R j ) and negation atterns on relations R i or R j, resectively, along the same lines as their counterarts for cfd s. We use enc to encode the inds embedded in s, which consists of the following attributes: (1) cid reresenting the id of a, and (2) those X attributes of R i and Y attributes of R j aearing in some s in Σ (i,j). Note that the number of attributes in enc is bounded by R i + R j + 1, where R i is the arity of R i.

12 12 Wenguang Chen, Wenfei Fan, and Shuai Ma (1) enc (2) enc L (3) enc R (4) enc cid state L state R cid tye state 1 null 2 DL cid rate 1 null 2 0 Fig. 5. Encoding examle of s cid os att val 1 LHS tye art 2 LHS tye art For each ψ = (R i [A 1... A m ; X ] R j [B 1... B m ; Y ], T ) in Σ (i,j), we generate a distinct cid id ψ for it, and do the following. Add tules t 1 and t 2 to enc L and enc R based on attributes X and Y, resectively, along the same lines as their cfd counterart. Add tules to enc in the same way as their cfd counterarts. Add tule t to enc such that t[cid] = id ψ. For each k [1, m], let t[a k ] = t[b k ] = k, and t[a] = null for the rest attributes A of enc. Examle 9. Figure 4 shows the coding of s ψ 1 and ψ 2 given in Fig. 3. We use state L and state R in enc to denote the occurrences of attribute state in item and tax, resectively. In tables enc L and enc R, attributes with only null atterns are omitted, for the same reason as for cfd s mentioned above. Putting these together, it is easy to verify that at most O(n 2 ) data tables are needed to encode deendencies in Σ, regardless of the size of Σ. Recall that n is the number of relations in database R. 5.2 SQL-based Detection Methods We next show how to generate sql queries based on the encoding above. For each i [1, n], we generate two sql queries that, when evaluated on the I i table of D, find vio(d, Σcfd i ). Similarly, for each i, j [1, n], we generate a single sql query Q (i,j) that, when evaluated on (I i, I j ) of D, returns vio(d, Σ (i,j) ). Putting these query answers together, we get vio(d, Σ), the violation set of D w.r.t. Σ. Below we show how the sql query Q (i,j) is generated for validating s in Σ (i,j) ), which has not been studied by revious work. For the lack of sace we omit the generation of detection queries for cfd s, which is an extension of the sql techniques for cfds discussed in [14, 7]. The query Q (i,j) for the validation of Σ (i,j) is given as follows, which caitalizes on the data tables enc, enc L, enc R and enc that encode s in Σ (i,j). select R i. from R i, enc L L, enc N where R i.x L and R i.x N and not exists ( select R j. from R j, enc H, enc R R, enc N where R i.x = R j.y and L.cid = R.cid and L.cid = H.cid and R j.y R and R j.y N) Here (1) X = {A 1,..., A m1 } and Y = {B 1,..., B m2 } are the sets of attributes of R i and R j aearing in Σ (i,j), resectively; (2) R i.x L is the conjunction of

13 Conditional Deendencies with Built-in Predicates 13 L.A k is null or R i.a k = L.A k or (L.A k = and (L.A i> is null or R i.a k > L.A i> ) and (L.A i is null or R i.a k L.A k ) and (L.A k< is null or R i.a k < L.A k< ) and (L.A i is null or R i.a k L.A i )) for k [1, m 1 ]; (3) R j.y R is defined similarly for attributes in Y ; (4) R i.x N is a shorthand for the conjunction below, for k [1, m 1 ]: not exists (select from N where L.cid = N.cid and N.os = LHS and N.att = A k and R i.a k = N.val); (5) R j.y N is defined similarly, but with N.os = RHS ; (6) R i.x = R j.y reresents the following: for each A k (k [1, m 1 ]) and each B l (l [1, m 2 ]), (H.A k is null or H.B l is null or H.B l H.A k or R i.a k = R j.b l ). Intuitively, (1) R i.x L and R i.x N ensure that the R i tules selected match the LHS atterns of some s in Σ (i,j) ; (2) R j.y R and R j.y N check the corresonding RHS atterns of these s on R j tules; (3) R i.x = R j.y enforces the embedded inds; (4) L.cid = R.cid and L.cid = H.cid assure that the LHS and RHS atterns in the same are correctly collated; and (5) not exists in Q ensures that the R i tules selected violate s in Σ (i,j). Examle 10. Using the coding of Fig. 5, an sql query Q for checking s ψ 1 and ψ 2 of Fig. 3 is given as follows: select R 1. from item R 1, enc L L, enc N where (L.tye is null or R 1.tye = L.tye or L.tye = ) and not exist ( select * from N where N.cid = L.cid and N.os = LHS and N.att = tye ) and (L.state is null or R 1.state = L.state or L.state = ) and not exist ( select * from N where N.cid = L.cid and N.os = LHS and N.att = state and R 1.state =N.val) and not exists ( select R 2. from tax R 2, enc H, enc R R where (H.state L is null or H.state R is null or H.state L! = H.state R or R 2.state = R 1.state) and L.cid = H.cid and L.cid = R.cid and (R.rate is null or R 2.rate = R.rate or R.rate = ) and not exist ( select * from N where N.cid = R.cid and N.os = RHS and N.att = rate and R 2.rate =N.val)) The sql queries generated for error detection can be simlified as follows. As shown in Examle 10, when checking atterns imosed by enc, enc L or enc R, the queries need not consider attributes A if t[a] is null for each tule t in the table. Similarly, if an attribute A does not aear in any tule in enc, the queries need not check A either. From this, it follows that we do not even need to generate those attributes with only null atterns for data tables enc, enc L or enc R when encoding s or cfd s. 6 Related Work Constraint-based data cleaning was introduced in [2], which roosed to use deendencies, e.g., fds, inds and denial constraints, to detect and reair errors

14 14 Wenguang Chen, Wenfei Fan, and Shuai Ma in real-life data (see, e.g., [11] for a comrehensive survey). As an extension of traditional fds, cfds were develoed in [14], for imroving the quality of data. It was shown in [14] that the satisfiability and imlication roblems for cfds are n-comlete and con-comlete, resectively. Along the same lines, s were roosed in [8] to extend inds. It was shown [8] that the satisfiability and imlication roblems for s are in constant time and extime-comlete, resectively. sql techniques were develoed in [14] to detect errors by using cfds, but have not been studied for s. This work extends the static analyses of conditional deendencies of [14, 8], and has established several new comlexity results, notably in the absence of finite-domain attributes (e.g., Theorems 2, 6, 8). In addition, it is the first work to develo sql-based techniques for checking violations of s and violations of cfd s and s taken together. Extensions of cfds have been roosed to suort disjunction and negation [7], cardinality constraints and synonym rules [9], and to secify atterns in terms of value ranges [18]. While cfd s are more owerful than the extension of [18], they cannot exress disjunctions [7], cardinality constraints and synonym rules [9]. To our knowledge no extensions of s have been studied. This work is the first full treatment of extensions of cfds and s by incororating builtin redicates (, <,, >, ), from static analyses to error detection. Methods have been develoed for discovering cfds [10, 18, 15] and for reairing data based on either cfds [13], traditional fds and inds taken together [5], denial constraints [4, 12], or aggregate constraints [16]. We defer the treatment of these toics for cfd s and s to future work. A variety of extensions of fds and inds have been studied for secifying constraint databases and constraint logic rograms [3, 6, 19, 20]. While the languages of [3, 19] cannot exress cfds, constraint-generating deendencies (cgds) of [3] and constrained tule-generating deendencies (ctgds) of [20] can exress cfd s, and ctgds can also exress s. The increased exressive ower of ctgds comes at the rice of a higher comlexity: both their satisfiability and imlication roblems are undecidable. Built-in redicates and arbitrary constraints are suorted by cgds, for which it is not clear whether effective sql queries can be develoed to detect errors. It is worth mentioning that Theorems 2 and 6 of this work rovide lower bounds for the consistency and imlication analyses of cgds, by using atterns with built-in redicates only. 7 Conclusions We have roosed cfd s and s, which further extend cfds and s, resectively, by allowing atterns on data values to be exressed in terms of, <,, > and redicates. We have shown that cfd s and s are more owerful than cfds and s for detecting errors in real-life data. In addition, the satisfiability and imlication roblems for cfd s and s have the same comlexity bounds as their counterarts for cfds and s, resectively. We have also rovided automated methods to generate sql queries for detecting errors based on cfd s and s. These rovide commercial dbms with an immediate caability to cature errors commonly found in real-world data.

15 Conditional Deendencies with Built-in Predicates 15 One toic for future work is to develo a deendency language that is caable of exressing various extensions of cfds (e.g.,cfd s, ecfds [7] and cfd c s [9]), without increasing the comlexity of static analyses. Second, we are develoing effective algorithms for discovering cfd s and s, along the same lines as [10, 18, 15]. Third, we lan to extend the methods of [5, 13] to reair data based on cfd s and s, instead of using cfds [13], traditional fds and inds [5], denial constraints [4, 12], and aggregate constraints [16]. Acknowledgments. Fan and Ma are suorted in art by EPSRC E029213/1. Fan is a Yangtze River Scholar at Harbin Institute of Technology. References 1. S. Abiteboul, R. Hull, and V. Vianu. Foundations of Databases. Addison-Wesley, M. Arenas, L. E. Bertossi, and J. Chomicki. Consistent query answers in inconsistent databases. In PODS, M. Baudinet, J. Chomicki, and P. Woler. Constraint-Generating Deendencies. J. Comut. Syst. Sci., 59(1):94 115, L. E. Bertossi, L. Bravo, E. Franconi, and A. Loatenko. The comlexity and aroximation of fixing numerical attributes in databases under integrity constraints. Inf. Syst., 33(4-5): , P. Bohannon, W. Fan, M. Flaster, and R. Rastogi. A cost-based model and effective heuristic for reairing constraints by value modification. In SIGMOD, P. D. Bra and J. Paredaens. Conditional deendencies for horizontal decomositions. In ICALP, L. Bravo, W. Fan, F. Geerts, and S. Ma. Increasing the exressivity of conditional functional deendencies without extra comlexity. In ICDE, L. Bravo, W. Fan, and S. Ma. Extending deendencies with conditions. In VLDB, W. Chen, W. Fan, and S. Ma. Incororating cardinality constraints and synonym rules into conditional functional deendencies. IPL, 109(14): , F. Chiang and R. J. Miller. Discovering data quality rules. In VLDB, J. Chomicki. Consistent query answering: Five easy ieces. In ICDT, J. Chomicki and J. Marcinkowski. Minimal-change integrity maintenance using tule deletions. Inf. Comut., 197(1-2):90 121, G. Cong, W. Fan, F. Geerts, X. Jia, and S. Ma. Imroving data quality: Consistency and accuracy. In VLDB, W. Fan, F. Geerts, X. Jia, and A. Kementsietsidis. Conditional functional deendencies for caturing data inconsistencies. TODS, 33(2), W. Fan, F. Geerts, L. V. Lakshmanan, and M. Xiong. Discovering conditional functional deendencies. In ICDE, S. Flesca, F. Furfaro, and F. Parisi. Consistent query answers on numerical databases under aggregate constraints. In DBPL, M. Garey and D. Johnson. Comuters and Intractability: A Guide to the Theory of NP-Comleteness. W. H. Freeman and Comany, L. Golab, H. J. Karloff, F. Korn, D. Srivastava, and B. Yu. On generating nearotimal tableaux for conditional functional deendencies. In VLDB, M. J. Maher. Constrained deendencies. TCS, 173(1): , M. J. Maher and D. Srivastava. Chasing Constrained Tule-Generating Deendencies. In PODS, R. van der Meyden. The comlexity of querying indefinite data about linearly ordered domains. JCSS, 54(1), 1997.