A Magic Approach to Optimizing Incremental Relational Expressions

Transcription

1 A Magic Approach to Optimizing Incremental Relational Expressions Andreas Behrend Institute of Computer Science III, University of Bonn Römerstr. 164, D Bonn, Germany ABSTRACT This paper is concerned with a transformation-based approach to update propagation in an extended version of Codd s relational algebra which allows for defining derived relations (even recursively). It is shown that the desired optimization effects of update propagation may be lost if no generalized selection pushing strategy is employed to the transformed algebra expressions. A possible solution is the application of the Magic Sets rewriting but this may lead to unstratifiability of the incremental expressions. For the efficient evaluation of Magic Sets transformed algebra expressions we propose to use the soft stratification approach because of the simplicity and efficiency of this technique. Categories and Subject Descriptors H.2.3 [Database Management]: Systems Relational databases General Terms Design, Performance Keywords Incremental Evaluation, Update Propagation, Relational Algebra, Deductive Databases 1. INTRODUCTION Update Propagation (UP) is a well-established research topic which has been intensively studied for many years mainly in the context of integrity checking and materialized view maintenance [11, 14, 18, 21, 25]. Nowadays, UP plays an important role in the context of the view-based analysis of data streams [7, 13] and forms the basis for the incremental evaluation of continuous queries (e.g. in Oracle [28]). The aim of UP is the computation of implicit changes of derived relations resulting from explicitly performed updates of the extensional fact base. As in most cases an update will affect Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. IDEAS , September 16-18, Cetraro, Calabria [Italy] Editor: Bipin C. DESAI Copyright 2009 ACM /09/09...$5.00. only a small portion of the database, it is rarely reasonable to compute the induced changes by comparing the entire old and new database state. Instead, the implicit modifications should be iteratively computed by propagating the individual updates through the possibly affected views and computing their consequences. In a transformation-based approach to UP each view definition is transformed already at schema design time to a socalled delta view, a specialized version of the view referring to changes in the underlying tables, only. In the literature, SQL, Datalog, and relational algebra (RA) have been used for specifying delta views. Using RA as a low-level language has the advantage that optimization effects can be directly identified [15]. As soon as more than one algebra operator is applied in a view definition, however, the optimization effect of using specialized delta views may be limited if no generalized selection pushing strategy is considered. As an example consider the following algebra expression for defining the derived relation P (x) based on the relations Q(x), R(x), S(x, y) and T (x, z): P (Q R) π x(s T ) The following rule would yield the induced insertions P + of P resulting from insertions Q + into Q: P + (Q + R) π x(s T ) Despite of the focus on changes with respect to Q, no optimization effect is achieved with respect to the evaluation of the right-hand argument of the set difference operator. A possible reordering of operations by using classical rules of algebraic optimization cannot provide a better focus on the changes of Q either. However, another way is to use the small number of tuples in Q + already for determining all matching join partners by introducing two semi-joins: P + (Q + R) π x((q + S) (Q + T )) Under the assumption that S and T are quite large in comparison to the size of Q + and there is a low selectivity of the tuples in Q +, the argument sizes of the join and difference operator are considerably reduced. Thus, the resulting incremental algebra expression provides a much better focus on the changes to Q. This improved delta expression results from applying the Magic Sets (MS) rewriting technique [4] which basically represents a generalized selection pushing strategy. In this paper, we will show how the application of MS generally allows for enhancing the focus of (hierarchical and recursive) delta views. Although the benefits of

2 using MS in a non-recursive system have been already shown in [23], the problem is that this rewriting technique may turn non-recursive expressions into recursive ones. Additionally, in database systems supporting stratifiable recursion in their view definitions, the application of MS may lead to unstratifiable delta views. Although it is possible to avoid this increased complexity [16], we aim at providing a general solution for evaluating such transformed delta views. Therefore, we propose to use the soft stratification approach for the efficient evaluation of MS transformed delta expressions [6]. The resulting method generalizes and enhances former methods to UP providing a much better focus on the induced changes. Additionally, it may serve as a valuable basis for developing UP methods in database systems supporting recursive view definitions. Employing soft stratification for optimizing propagation rules has been already proposed by the author in [8] using Datalog. Actually, incorporating these techniques into a DBMS requires the reformulation and transformation of this method into an RA context. In this way optimization effects can be easily identified for further refinement of the general approach. This paper is organized as follows: In Section 2 the syntax and semantics of an extended version of Codd s relational algebra is introduced which allows for recursive RA rules. Afterwards we recall the basic rules of differential calculus for generating delta views. Section 4 discusses the MS method in the context of relational algebra. Afterwards, its application to the afore determined delta views is discussed and the correctness of the resulting magic delta rules is proven. In Section 6 it is shown that the resulting magic delta rules are at least softly stratifiable and can be efficiently evaluated using the soft stratification approach. 2. EXTENDED RELATIONAL ALGEBRA We assume the reader to be familiar with the basic operators of Codd s relational algebra. The following discussion is limited to duplicate-free expressions with no null values and no aggregate functions. Note that these restrictions are imposed for clarifying the underlying approach, only. The well-established relational algebra, however, is extended by allowing RA rules and recursion. 2.1 Syntax A relation R is generally defined as a subset of the Cartesian product of n domains. The schema of relation R consists of a relation name along with a list a 1,..., a n of distinct attributes names. The state of R is a finite set of tuples. Note that we will use the same notation for a relation, its name and its state but it should be clear from the context what is meant. Let E 1 and E 2 be relational expressions. We will represent the selection of E 1 based on condition C by σ C(E 1), the projection on attribute list x 1, x 2,... of E 1 by π x1,x 2,...(E 1), and the Cartesian product, the union, the intersection, and the set difference of E 1 and E 2 by E 1 E 2, E 1 E 2, E 1 E 2, and E 1 E 2, respectively. The natural join, left semi join, and left anti-join of E 1 and E 2 will be represented by E 1 E 2, E 1 E 2, and E 1 E 2, respectively. If a join operation is to be performed with respect to a given condition C, this boolean condition will be denoted below the respective join operator. This algebra is extended by allowing so-called RA rules which are relational expression of the form R E where R denotes a derived relation and E is a relational expression. We will always provide the schema of the derived relation R. If a renaming of variables in E is needed to fit the schema of R, it is assumed to be implicitly performed. There may be more than one RA rule for defining a relation R, each of them specifying a certain subset of tuples in R. Note that this implicit union is not necessary because RA has an explicit union operator. However, several rules are allowed for defining a derived relation as we need this implicit union for decomposing relational expressions later in Subsection 2.3 where normalized rules are considered. One advantage of using RA rules is that they can be used as a direct counterpart to conventional view definitions (in SQL or Datalog, e.g.). In addition, they quite naturally allow for defining recursive views where the defined relation R occurs within the relational expression E, too. As an example, the transitive closure P ath(y 1, y 2) of relation Edge(x 1, x 2) could be defined using RA rules as follows: P ath Edge P ath π x1,y 2 (Edge x 2 =y 1 P ath) Because of the different schema, a renaming would be necessary in both rules which is not explicitly stated. Using RA rules represents a very general way of specifying recursive views in relational algebra allowing non-linear and even unstratifiable recursion (cf. Subsection 2.2). A different way has been proposed in [2] where the author introduced the operator Alpha for specifying positive linear recursion. But Alpha would not be sufficient for expressing recursion resulting from the application of MS, so we propose a more general kind of recursive specifications. For a relation P, we use rel(p ) for referring to the set containing the relation name of P. For a set of relations P, rel(p) is defined as p Prel(p). This mapping is defined for RA rules in an analogous way, ie., rel(p E) := rel(p ) and rel(r) := r Rrel(r). Definition 1. A database D is a tuple B, R where B is a finite set of base relations and R a finite set of RA rules such that rel(b) rel(r) =. A relation name p is called derived (view), if p rel(r). The relation p is called extensional (or base), if p rel(b). For simplicity of exposition, and without loss of generality, we assume that a relation is either base or derived, but not both. Based on these notions, we will define the semantics of a database. Since we allow for recursive views, a fixpoint semantics is used for specifying the set of derivable tuples. 2.2 Semantics In this section, the well-known fixpoint semantics for deductive databases (e.g. [1]) is translated to the RA context. To this end, we first recall the notion stratification introduced for handling negation in views. A stratification is used for partitioning a given set of RA rules such that no recursive relation negatively depends on itself (cf. [1]). In the context of RA, a negative dependency is introduced by any non-monotonic RA operator, such as difference, division, anti-join, and outer join variants. Basically, a stratification λ on D = B, R is a mapping from the set of all relation symbols rel(b) rel(r) to the set of positive integers IN inducing a partition of the given set of rules such that all positive derivations of

3 relations can be determined before any non-monotonic operator is applied to one of the relations. For every partition P = P 1... P n induced by a stratification, the condition rel(p i) rel(p j) = with i j must necessarily hold. Based on these notions we will now define the semantics of relational databases using the immediate consequence operator as proposed by van Emden and Kowalski [33] but transferred into the RA context. This operator is defined on a set of facts which are tuples typed by the name of the relation to which they belong. Definition 2 (Immediate Consequence Operator). Let R be a set of RA rules and F an arbitrary set of facts. The accumulative consequence operator T R is a mapping on sets of facts and is defined as T R(F) = r R {eval(r, F)} F where eval(r, F) is a function returning the part of the derived relation rel(r) which can be determined by applying the RA rule r to F. The immediate consequence operator T R returns the set of facts which can be determined by the simultaneous application of all RA rules in R to the input facts F (while retaining all of F, too). As T R is monotonic for semipositive RA rules, i.e., rules in which non-monotonic operations reference no derived relations, its least fixpoint lfp (T R, B) exists, where lfp (T R, B) denotes the least fixpoint of T R containing all facts from the relations in B. We use relation(f) to denote the set of relations which can be constructed from a given set of facts F. Given a database D = B, R with semi-positive RA rules, its semantics M D is defined by relation(lfp (T R, B)). For stratifiable databases, however, the semantics is defined by using the iterated fixpoint which can be constructed as follows: Let D = B, R be a stratifiable database and λ a stratification on D. The partition P 1... P n of R defined by λ induces a sequence of least fixpoints M 1,... M n: M 1 := relation(lfp (T P1, B)), M 2 := relation(lfp (T P2, M 1)),... M n := relation(lfp (T Pn, M n 1)). The semantics M D of D is given by M n. For illustrating the notations introduced above, consider the following example of a stratifiable database D = B, R : R: B : O π y1,y 2 (P ) π y2,y 1 (P ), E : {(1, 2), (2, 1), (2, 3)} P E, P π x1,y 2 (E x 2 =y 1 P ) Relation P represents the transitive closure of relation E while relation O selects all P -tuples where Y is reachable from X but not vice versa. A stratification induces (in this case) the unique partition P = P 1 P 2 with P 1 comprising the two RA rules for relation P while P 2 includes the defining rule for O. The computation of lfp(t s P, B) then induces the following sequence of fixpoints M 1 := relation(lfp (T P1, B)) = {P } B with P = {(1, 2), (2, 1), (2, 3), (1, 1), (2, 2), (1, 3)} M 2 := relation(lfp (T P2, M 1)) = {O} M 1 with O = {(1, 3), (2, 3)} The iterated fixpoint M 2 = {O, P, E} is the set of all relations (base or derived) whose tuples are given in B or can be determined by the RA rules in R and coincides with the semantics M D of D. 2.3 Relational Normal Form Before considering the specialization of a given set of RA rules in the next section, we first simplify the relational expressions occurring. To this end, a given set of rules is decomposed into several new ones such that each of them corresponds to a single operator only. Note that the implicit union operator for RA rules defining the same relation allows us to decompose even recursive definitions. In the following, a set of relational expressions is called to be in normal form if every rule refers to one algebra operation only. This normal form can be systematically derived from an expression E by successively removing top-level operations from E and applying them to auxiliary relations which refer to the remaining part of E. As an example, consider the following set of RA rules for the relations P, Q, R, S and T all of them having the same schema variables x and y: P π x(q (R S)) Q σ x>1(t U) The rules are decomposed top-down by starting with the highest operation within the corresponding operation tree. For the first rule the operation to be considered is the projection on x which forms the first RA rule in normal form P π x(h 1) using the new auxiliary relation H 1 for the remaining part of the original relational expression H 1 (Q (R S)). In the next step this procedure is repeated for H 1 until every rule, the old as well as new ones, are in normal form. In our example, this would lead to the following set of rules: P π x(h 1) H 1 (Q H 2). H 2 (R S)). Q σ x>1(h 3) H 2 (T U) Note that the normal form R nf of a given set of RA rules R always exists and can be easily obtained. In the following the mapping norm R(R) is used for generating the normal form R nf of R while its inverse norm 1 R (Rnf ) can be used to obtain the original rule set w.r.t. the relations in R again. This transformation is usually called fold/unfold in logic programming. 3. DIFFERENTIAL CALCULUS As early as 1981, Koenig and Paige proposed to avoid costly calculation of derived expressions by using incremental counterparts instead [19]. Since then, many methods for incremental recomputation of relational expressions have been proposed in the literature. Although most approaches essentially apply the same techniques, they mainly differ in the way they have been implemented and in the granularity of the computed induced updates. With respect to implementation, e.g. authors used the database languages SQL (including triggers) [11], Datalog [14, 18, 21, 25] and relational algebra [2, 9, 10, 12, 15, 27] for specifying their delta rules - just to mention a few.

4 The different granularities considered result from integrity checking methods in which the computation of induced changes may be simplified with respect to integrity rules. As this restricts the range of applications of update propagation, we will consider the smallest granularity of updates, so called true updates, only [14]. Moreover, in contrast to [11, 26] we will use passive incremental delta rules resulting from the transformation of the original algebra expression but with a focus on the changes to the base relations. The actual propagation process is initiated and realized by a fixpoint computation in which the transformed delta rules are employed. In the following, we will recall well-known transformations for deriving specialized delta rules from a given set of RA rules. In order to achieve completeness, socalled transition rules are considered afterwards which allow to simulate the old and new state of a relation. 3.1 Delta Rules In the context of RA, various authors introduced delta views in similar, but not always identical manner. Rather than presenting laws for iteratively transforming a complex RA expression into its delta version (as [26] does), we use a style similar to that of [21, 30] where delta views are directly defined. We do not consider bags (as [11, 15] do) but allow for recursion. In the following, for each relation name P, two delta relations P +, P are used for representing the insertions and deletions induced on P by an update of the underlying database. The delta relations defined for a relation P have the same schema and type as P, i.e., if P is extensional respectively derived, then P + and P are extensional respectively derived as well. The state of a relation P before the changes given in the delta relations have been applied is denoted by P old whereas the new state is represented by P new. The delta sets and the corresponding state relations are related in the following way: P + = P new P old P = P old P new The first operator we discuss is the intersection operator. If P is defined as P Q R, the corresponding delta sets are obtained via the rules: P + (Q + R new ) (Q new R + ) P (Q R old ) (Q old R ). Using a characterization of preserved tuples, we can eliminate a common subterm occurring when evaluating these expressions characterizing the delta sets for intersections. The resulting optimized versions are P + (Q + R o ) (Q o R + ) (Q + R + ) P (Q R o ) (Q o R ) (Q R ) where R o := R old R and Q o := Q old Q represent the preserved tuples that are not deleted from R and Q, respectively. As this more accurate specification, however, is usually more complex than the former incremental one, the elimination of common subterms (e.g., as proposed in [21]) will not be considered for the following operators and the less complex incremental specifications will be used instead. The next operator we want to discuss is projection. The delta sets of P π x (Q) can be determined by using the incremental rules P + π x (Q + ) P old P π x (Q ) P new. This time a residue expression has to be included in both rules in order to eliminate induced insertions for which a different derivation already existed in the old state, or to eliminate induced deletions for which another derivation will exist in the new state, respectively. The union operator also requires this kind of effectiveness test because of duplicate derivations generated. In addition, as soon as several rules (in normal form) exist for defining a relation, a similar effectiveness test has to be included in every delta rule because of alternative derivations coming from one of the other RA rules. In a similar way, delta rules for all other relational operators can be easily found which form a basis for most transformation-based approaches to update propagation: Definition 3 (Delta Rules ϕ(r)). Let R be a set of stratifiable RA rules, R nf := norm R(R) the corresponding set in normal form and eff(p, State) a mapping on relation names P returning the state relation name P State of P if there is more than one rule for P in R nf and the empty set otherwise. The set of delta rules for true updates with respect to R nf is denoted R and is defined as the smallest set satisfying the following conditions: 1. For each rule of the form P (Q R) R nf the following two delta rules are in R : P + ((Q + R new ) (R + Q new )) eff(p, old) P ((Q R old ) (R Q old )) eff(p, new). 2. For each rule of the form P π x (Q) R nf the following two delta rules are in R : P + π x (Q + ) P old P π x (Q ) P new. 3. For each rule of the form P σ F (Q) R nf the following two delta rules are in R : P + σ F (Q + ) eff(p, old) P σ F (Q ) eff(p, new). 4. For each rule of the form P (Q R) R nf the following two delta rules are in R : P + (Q + R old ) (R + Q old ) eff(p, old) P (Q R new ) (R Q new ) eff(p, new). 5. For each rule of the form P (Q R) R nf the following two delta rules are in R : P + ((Q + R new ) (R Q new )) eff(p, old) P ((Q R old ) (R + Q old )) eff(p, new). 6. For each rule of the form P (Q R) R nf the following two delta rules are in R : P + ((Q + R new ) (R + Q new )) eff(p, old) P ((Q R old ) (R Q old )) eff(p, new). The set of delta rules for the original rule set R is denoted ϕ(r) and is defined as norm 1 R (R ).

5 A similar definition of delta rules can be found in [21]. The application of these delta rules is now shown by means of an example. Suppose the following set of rules R is to be incrementally maintained: r 1 : P π x(t Q) r 2 : T σ x>1(b C) In the first step, R is transformed R nf : P π x(h 1) H 1 (T Q) T σ x>1(h 2) H 2 (B C) Subsequently, the transformation rules from above are applied to each rule in R nf yielding the following set of delta views R (here w.r.t induced deletions, only): P π x(h 1 ) P new H 1 (T Q old ) (Q + T old ) T σ x>1(h 2 ) H 2 (B C new ) (C B new ) In the third step, the delta expressions of auxiliary relations in R are unfolded by using norm 1 R (R ) such that the original rule structure of R is obtained again r 1 : P π x((t Q old ) (Q + T old )) P new r 2 : T σ x>1(b C new ) (C B new ) The resulting delta rules represent ϕ(r). Note that the new state P new of P is needed for compensating side effects due to possible alternative projection results. Additionally, the old state of Q and T as well as the new state of B and C is referenced within the incremental rules. In the next section the question is considered how these state references can be incrementally evaluated using so-called transition rules. 3.2 Transition Rules Generally, for computing true updates references to both the old and new database state are necessary. In [17] it is assumed that all views are materialized which simplifies the state evaluation process but seems to be unrealistic in practice. Therefore, the possibility has been investigated of dropping the explicit references to one of the states by deriving it from the other one and the given updates. The benefit of such a state simulation is that the database system is not required to store both states explicitly but may work on one state only. Rules for state simulation will be called transition rules according to the naming in [25]. Although both directions are possible, we will concentrate on a somehow pessimistic approach, the simulation of the new state while the old one is actually given. The following discussion, however, can be easily transferred to the case of simulating the old state. In principle, transition rules can be distinguished by the way how far induced updates are considered for simulating the other database state. A very straightforward approach is to use so-called naive transition rules which employ the delta relations for simulating the new state of the base relations, but not for the new state of derived relations. In the above example, the new state is (naively) simulated using the transition rules P new π x(t new Q new ) T new σ x>1(b new C new ) B new (B B + ) B Q new (Q Q + ) Q C new (C C + ) C whereas the old state of the derived relations P and T is accessible via the original rules r 1 and r 2. Note that we omit the indices old as we currently work on this state. The disadvantage of these transition rules is that each derivation with respect to the new state has to go back to the extensional delta relations and hence makes no use of the implicit updates already derived during the course of propagation. In the Internal Events Method [25] as well as in [21] it has been proposed to improve state simulation by employing not only the extensional delta relations but the derived ones as well, leading to so-called incremental transition rules. However, the union of the original, the delta and the incremental transition rules is not always stratifiable. For instance, the negative delta rule for the projection operator P π x (Q) and the corresponding incremental transition rule for P new form an unstratifiable cycle: P π x (Q ) P new P new (P P + ) P Therefore, different incremental transition rules have been proposed in [25, 21] which employ one step of indirection. In our example, this indirection is achieved by using π x (Q new ) instead of P new within the effectiveness test of P. The resulting reference Q new can be incrementally maintained without stratification problems. However, this solves the stratification problems for hierarchical rules, only. If the database includes recursively defined relations, incremental transition rules can lead to unstratifiability again which may not even represent the true induced update anymore under the well-founded semantics [14]. This problem has been fixed in [14] by using a certain combination of incremental and naive transition rules. However, this more efficient form will not be considered in this paper but rather the naive one for simplifying the following exposition. Nevertheless, similar optimization effects resulting from the application of MS can be achieved if incremental transition rules are used. What is important is that a stratifiable database augmented with delta rules and naive transition rules remains stratifiable as well. Definition 4 (Naive Transition Rules τ(r)). Let D = B, R be a stratifiable database. Then the set of naive transition rules for true updates and new state simulation with respect to R is denoted τ(r) and is defined as the smallest set satisfying the following conditions: 1. For each extensional relation P rel(b), the following direct transition rule is in τ(r): P new (P P + ) P 2. For each rule R E R (E representing an arbitrary RA expression) an indirect transition rule of the form R new E[P i P new i ] is in τ(r) where E[P i Pi new ] denotes E with every relation name P i in E substituted by P new i. The following proposition shows that if delta and transition rules are generated according to Definition 3 and 4, the delta relations will correctly represent the corresponding induced update.

6 Proposition 1 (Correctness of ϕ(r) and τ(r)). Let D = B, R be a stratifiable database and u D = { P + 1,..., P n +, Q 1,..., Q m } an update with rel(u D) rel(b). Let D a = B u D, R ϕ(r) τ(r) be the augmented deductive database of D. Then the delta relations defined by the delta rules ϕ(r) and transition rules τ(r) correctly represent the induced changes of D. Hence, for each relation p rel(r) the following conditions hold: p + ( t ) M D a p( t ) M D new M D old p ( t ) M D a p( t ) M D old M D new. Proof. cf [14]. It is easy to see that a stratifiable database augmented with delta and transition rules for true updates continues to be stratifiable. Consequently, the iterated fixpoint computation from Subsection 2.2 can be used as constructive method for evaluating delta and transition rules. In this way, the incremental evaluation of (stratifiable) non-linear recursive RA expressions can be very naturally implemented. 4. MAGIC SETS AND RA Query optimization is a well-established database research topic which is divided into logical and physical optimization. Logical optimization rules include a selection pushing strategy which allows for shifting selection condition to lowerlevel operands within a given operator tree such that irrelevant answer tuples are eliminated as early as possible. As soon as recursive RA expressions are allowed, however, the classical selection pushing strategy cannot deal with new selection constants introduced during the course of evaluation anymore. For solving this problem, the rewriting technique MS has been proposed in [4] which uses auxiliary relations (so-called Magic Sets) in order to store and to dynamically apply those generated constants. As an example, consider again the transitive closure P ath(y 1, y 2) of relation Edge(x 1, x 2) from Subsection 2.1 and the query? σ y1 =1(P ath(y 1, y 2)) asking for all paths reachable from node 1. In order to restrict the relation P ath to those tuples transitively connected to node 1, the magic relation m P ath bf (y 1) is introduced which is recursively defined as follows: m P ath bf π Edge.x2 (m P ath bf Edge) Starting from the magic seed fact m P ath bf (1) the rule accumulates all relevant nodes and stores the corresponding selection constants in m P ath bf (y 1). The adornment string bf indicates for which attributes of P ath constants are stored by the magic relation. The symbol b represents a bound attribute, while f indicates a free argument position. The adorned rule set is derived from the original database with respect to the adorned query P ath bf (y 1, y 2) and a choice of sideways information strategy (SIPS) specifying for each rule the order in which a derived relation is determined. Each derived relation then represents a subquery to be called with the unique binding pattern encoded in its adornment. The queried relation P ath is now restricted to those constants occurring in the MS relation m P ath bf (y 1) by using semi-joins: P ath m P ath bf Edge P ath m P ath bf π x1,y 2 (Edge P ath) Edge.x 2 =P ath.y 1 Recursive query optimization by means of rule rewriting has been intensively studied and MS has been accepted as a kind of standard in this field. Over the last 20 years many approaches have been presented dealing with the refinement of this method and even investigate the applicability of MS to non-recursive databases [16, 23, 24] or systems involving stratifiable recursion [6]. Although the benefits of using MS in non-recursive systems have been shown in [16, 23, 24], this method does not always lead to an improved evaluation. Generally, its optimization effects strongly depend on relation sizes, selectivities and the chosen SIP strategy. As an example of its application to a non-recursive database consider the sample rule from Section 1 again. The application of MS using a left-to-right sip strategy would yield the following rule P (Q R) π x((m H bf S) (m H bf T ) m H bf Q R using the MS transformed auxiliary relation H(x, y) for representing the right argument S T whereas a right-to-left SIPS yields P ((m H b Q) (m H b R)) π x(s T ) m H b π x(s T ) based on a transformed auxiliary relation H(x) introduced for representing Q R. If the expression Q R provides significantly less tuples than S T, the evaluation of the first magic rules is more efficient than the one based on the right-to-left SIPS and vice versa. If both expressions Q R and S T specify around the same number of tuples, no optimization effect is achieved by using MS independent of the chosen SIPS. Under the assumption that delta relations are considerably smaller than state relations, however, the focus on selection constants in delta sets by using MS provides notable optimization effects. 5. MAGIC UPDATES REWRITING In the context of pure bottom-up materialization of RA expressions, the benefit of delta rules is that their evaluation can be restricted to the values of the currently propagated update such that the entire propagation process is very naturally limited to the actually affected derived relations. On the other hand, a bottom-up approach like this requires to materialize the simulated state of derived relations completely in order to determine true updates. By contrast, if update propagation were based on a pure top-down approach, as proposed, e.g., in [25] and in [18], the simulation of the opposite state can be easily restricted to the relevant part by querying the relevant portion of the database. A pure top-down approach, however, has the disadvantage that the induced changes can only be determined by querying all existing derived relations, although most of them will probably not be affected by the update. We combine the advantages of top-down and bottom-up propagation by applying the Magic rewriting to delta rules. To this end, the tuples of delta relations are considered as abstract queries on the state relations occurring within the same relational expression. Evaluating these queries with respect to the augmented database has the advantage that the respective state relations will only be partially materialized with a focus on the derived delta facts. We will now formally introduce what we call the Magic Updates (MU) rewriting. It is applied to an augmented

7 database (cf. Proposition 1) and results in a set of delta rules that contains references to relevant portions of state relations, only. Afterwards we prove the correctness of this rewriting technique. Definition 5 (Magic Predicates). Let A P ad ( x) be a positive literal with relation name P, adornment ad and bd( x) the sequence of variables within x indicated as bound in the adornment ad. Then the magic predicate of A is defined as magic(a) := m p ad (bd( x)). Given a set of RA rules R and an adorned query Q P ad ( x) with P rel(r), the Magic Sets transformed rules with respect to Q are denoted by ms Q(R). Definition 6 (Magic Updates Rewriting). Let R be a set of stratifiable RA rules, R nf := norm R(R) the corresponding set in normal form, and R the set of delta rules with respect to R nf. The MU rewriting of R nf yields the magic rule set R magic := R R query R state where R query and R state are defined as follows: 1. For each delta rule P E R and each subexpression of the form (R π S ς ) E where R π and S ς is a delta respectively derived state relation with π {+, }, {, } and ς {new, old} the following subquery rule is in R query magic(s ς ad ) Rπ. For each delta rule P E R and each subexpression of the form (R π S ς ) E where R π and S ς is a delta respectively derived state relation with π {+, } ς {new, old} the following subquery rule is in R query magic(s ς ad ) Rπ. For each delta rule P (E S ς ) R where E is a compound RA expression and S ς a derived state relation with ς {new, old} and for each delta relation R π E with π {+, } the following subquery rule is in R query magic(s ς ad ) Rπ. No other rules are in R query. 2. From the set R n := R nf τ(r nf ) we derive the rule set R state : For each relation symbol magic(l ς ad ) rel(r query ) the corresponding MS transformed rule set ms Q(R n ) is in R state where Q L ς ad represents an adorned query with rel(l ς ) rel(r n ). No other rules are in R state. The MU rewriting of the original rule set R is denoted µ(r) and is defined as norm 1 R (Rmagic ). Before proving the correctness of the MU rewriting we first illustrate its application by means of an example. Reconsider the rule from Section 1 defining the derived relation P with schema P (x) P (Q R) π x(s T ) based on the relations Q(x), R(x), S(x, y), and T (x, z). At first, the normalized form R nf of R is determined: P (H 1 H 2) H 2 π x(h 3) H 1 (Q R) H 3 (S T ) Afterwards, the rules of differential calculus are applied to R nf with respect to insertions Q + into Q yielding the set of delta rules R : P + H + 1 Hnew 2 H + 1 (Q+ R) The required transition rules are given by τ(r nf ): H new 2 π x(h new 3 ) H new 3 (S new T new ) The delta relations H + 1 and Q+ in R are now considered to be subqueries providing selection constants for the respective state relations H new and R. Since the old state R represents a base relation, however, solely the following subquery rule for H new is added to R query : m H new 2 b H + 1 The adorned query Q H new 2 b in R query is used to determine the MS transformed rules ms Q(R n ) with R n := R nf τ(r nf ): H2 new m H2 new b π x(h3 new ) H3 new (m H3 new bff S new ) (m H3 new bff T new )) m H new 3 bff m H new 2 b Folding back the introduced auxiliary relations H i within R magic := R R query ms Q(R n ) by using norm 1 R(R magic ) (and employing some standard laws of algebraic optimization) finally yields µ(r) P + (Q + R)) π x(((q + R) S) ((Q + R) T ) which provides the desired focus on the changes in Q. The resulting expression remains to be hierarchical and thus, is stratifiable. The following Theorem 1 shows that a rule set resulting from the MU rewriting correctly represents the induced updates defined by the underlying augmented database. Theorem 1 (Correctness of µ(r)). Let D = B, R be a stratifiable database and u D = { P + 1,..., P n +, Q 1,..., Q m } an update with rel(u D) rel(b). Let D m = B u D, µ(r) be the Magic Updates transformed deductive database of D. Then the delta relations defined by the delta rules in µ(r) correctly represent the induced changes of D. Hence, for each relation p rel(r) the following conditions hold: p + ( t ) M D m p( t ) M D new M D old p ( t ) M D m p( t ) M D old M D new. Proof. The correctness of the MU rewriting with respect to R is shown by proving it to be equivalent to a specific MS transformation of the augmented rule set R a := R ϕ(r) τ(r) which is known to be sound and complete. A MS transformation starts with the adornment phase which basically depicts information flow between literals in a database according to a chosen sip strategy. In [3]

8 it is shown that the MS approach is also sound for socalled partial sip strategies which may pass on only a certain subset of captured variable bindings or even no bindings at all. Let us assume we have chosen a sip strategy in such a way that no bindings are passed to delta relations, i.e. their adornments are strings only consisting of f symbols representing unbounded attributes. Additionally, let Rá = R a {h P π 1 πn 1 }... {h Pn } be an extended augmented rule set with rules for defining an auxiliary 0-ary relation h with h / rel(ϕ(r)), π i {+, } and {P π 1 1,..., P n πn } = rel(ϕ(r)) distinct relation names. Relation h references all derived delta relations in ϕ(r) as they are potentially affected by a given base update. Note that since R a is assumed to be stratifiable, Rá must be stratifiable as well. The MS rewriting of Rá with respect to the query H h using a partial sip strategy as proposed above yields the rule set ms H(Rá) which is basically equivalent to the rule set µ(r) resulting from the MU rewriting. The rule set ms H(Rá) differs from µ(r) by the answer rules of the form h m h P π 1 1 ff...,..., h m h P πn n ff... for the additional relation h, by subquery rules of the form m P π 1 1 ff... m h,..., m P πn n ff... m h, by subquery rules of the form m P π i i ff... m P π j j ff... with i, j {1,..., n}, and by the usage of m P π i i ff... relations in delta rule bodies for defining a corresponding delta relation P π i i ff.... It is obvious that these rules and relations can be removed from ms H(Rá) without changing the semantics of the derived delta relations in ms H(Rá). The remaining rules coincide with the MU rules µ(r). Using Proposition 1, it can be concluded that all delta relations defined in Rá correctly represent the induced update. Thus, the MS transformed rule set ms H(Rá) must be sound and complete as well. As the MU rules µ(r) can be derived from ms H(Rá) in the way described above, they must correctly represent the induced update as well. 6. SOFT UPDATE PROPAGATION Although MS indeed provides an improved focus on the induced changes, its application may turn hierarchical RA rules into recursive ones and stratifiable recursive rules into unstratifiable ones. In [16], a method has been proposed for preserving nonrecursiveness while [3] introduces an approach for handling unstratifiable recursion resulting from the application of MS. However, both methods introduce some computational overhead to the system foiling the optimization effects of MS. In this section, we present a general solution to this problem. To this end, it is shown that the MU transformed rules are always softly stratifiable such that the soft consequence operator from [6] can be used for their efficient evaluation. This approach for computing true updates will be denoted soft update propagation. 6.1 Soft Update Propagation by Example First we illustrate the application of MU rewriting to stratifiable non-recursive rules and consequently its application to stratifiable recursive delta rules Transformation of Hierarchical Rules Let us first consider a slightly modified version of our first example from Section 1 R 1: K (Q R) H H π X(S T ) with the schema K(x), Q(x),R(x), H(x), S(x,y) and T(x,z). This time the original rule has been decomposed into two rules such that the optimization effects resulting from the MU rewriting become more apparent. Suppose the following fact base B 1 is given B 1: Q: {(1),(2)} R: {(2), (3)} S: {(5,5), (6,6),...,(104,104)} T: {(5,6), (6,7),...,(204,205))} The derived relation K consists of 3 tuples while H contains 100 tuples. Let u D1 = {Q + } be an update containing some new insertion with respect to relation Q. The augmented database D a 1 of D 1 is given by D a 1 = B 1 {Q + }, R 1 ϕ(r 1) τ(r 1) while the delta rule in ϕ(r 1) looks as follows: K + (Q + R) H new Note that the new state of H coincides with its old state so no differentiation for the underlying relations S and T is needed. Thus, the set of transition rules τ(r 1) consists of the single rule H new π X(S T ). Suppose the update u D1 = {Q + } contains the new tuples (4) and (5) for the database D 1 = R 1, B 1. The computation of the induced updates using the augmented database D a 1 leads to the generation of 100 new state facts for relation H and to the single induced insertion K + (4) for K. The application of the MU rewriting, however, provides a much better focus avoiding the complete determination of H new. Using a left-to-right SIPS leads to the following subquery rule m H new b (Q + R) and the following specialized transition rule in ms Q(R n ) with Q = {Hb new } (as already stated in previous sections) H new b π X((m H new b S) (m H new b T ) The resulting database D1 m = B 1 u D1, µ(r 1) remains to be stratifiable (this time). Computing the iterated fixpoint M D m 1 of D1 m using the update {Q + (4), Q + (5)} leads to the generation of two corresponding subquery facts m Hb new (4) and m Hb new (5). These facts restrict the computation of new state facts such that only the one tuple H new (5) instead of 100 H new -tuples is generated. Thus, the application of MU rewriting provides indeed a much better focus on the required state facts Transformation of Stratifiable Rules We will now show how the application of MU rewriting turns a stratifiable database into an unstratifiable one. To this end, let us consider the deductive database D 2 = B 2, R 2 with R 2 consisting of the rules for the transitive closure P (x 1, x 2) of relation E(y 1, y 2):

9 R 2: B 2: P E P π E.x1,P.y 2 (E P ) E.x 2 =P.y 1 E: {(1,2), (1,4), (3,4)} {(10,11), (11,12),..., (98,99), (99,100)} Note that the derived relation P consists of 4098 tuples. For maintaining readability, we restrict our attention to the computation of true insertions into P. The corresponding delta rules ϕ(r 2) are given by P + E + P P + π E.x1,P.y 2 ( (E + P new ) E +.x 2 =P new.y 1 (P + E new )) P P +.y 1 =E new.x 2 while the transition rules τ(r 2) for state simulation are P E P π E.x1,P.y 2 (E P ) E.x 2 =P.y 1 P new E new P new π E.x1,P.y 2 (E new P new ) E new.x 2 =P new.y 1 E new (E E ) E + Let u D2 = {E + } be an update where E + consists of the new tuple (2, 3) to be inserted into D 2. The resulting augmented database D a 2 is given by D a 2 = B 2 {E + }, R 2 ϕ(r 2) τ(r 2). Computing the induced update by evaluating the stratifiable database D a 2 would lead to the generation of 94 new state facts for relation E, 4098 old state facts for P and new state facts for P. The entire number of generated facts is 8296 for computing the three induced insertions P + : {(1, 3), (2, 3), (2, 4)} with respect to P. So no optimization effects are achieved for evaluating this stratifiable recursion despite using delta rules. We will now show that the application of the MU rewriting using a left-to-right SIPS provides a much better focus on the changes to E. Within its application, the following subquery rules m P bb E + m Pbf new π x2 (E + ) m Efb new π y1 (P + ) m P bb π x1,y 2 ((E + P new ) (P + E new )) as well as the following set of specialized transition rules ms Q(R n 2 ) with Q = {Efb new, P bb, Pbf new } are generated: P m P bb E P m P bb π E.x1,P.y 2 (E P ) E.x 2 =P.y 1 P new m Pbf new Ebf new P new m Pbf new π E.x1,P.y 2 (E new P new ) E new (m Ebf new E new (m Efb new (E E )) (m Ebf new E + ) (E E )) (m Efb new E + ) m P bb π x2,y 2 (m P bb y 1 =x 1 E) m Ebf new m p new bf m Pbf new π x2 (m Pbf new y 1 =x 1 E new ) Note the large number of rules is due to the fact that the transformation takes potential deletions from E into account. Quite similar to the MS approach, the MU rewriting may result in an unstratifiable rule set. This is the case for the above rule set, too, where the following negative cycle can be found in the corresponding dependency graph of µ(r 2): P + pos pos m P bb P neg P + However, the resulting rule set is softly stratifiable such that the soft consequence operator could be used for determining the corresponding semantics. 6.2 The Soft Update Propagation Approach A stratification on D = B, R is a mapping from the set of all relation names rel(b) rel(b) to the set of positive integers IN inducing a partition P = P 1... P n of R which satisfies the condition rel(p i) rel(p j) = with i j. In contrast to this, a soft stratification λ s on D is a mapping from the set of all rules in R to the set of positive integers IN inducing a partition of the given set of RA rules for which the condition above does not necessarily hold [6]. A soft stratification is solely defined for MS transformed rule sets including unstratifiable ones. Based on the the immediate consequence operator from Subsection 2.2, we recall the soft consequence operator from [6] which may serve as the basic operator for determining the semantics of stratifiable as well as softly stratifiable databases. The basic idea is to integrate the iteration over the given partition components into the consequence operator itself. Definition 7 (Soft Consequence Operator). Let D = B, R be a (potentially unstratifiable) database, P = P 1... P n a partition of R and F an arbitrary set of facts. The soft consequence operator TP s is a mapping on sets of facts and is defined as F if T Pj (F) = F forall j {1,..., n} TP(F) s := T Pi (F) with i = min{j T Pj (F) F}, otherwise. As the soft consequence operator TP s is monotonic for stratifiable or softly stratifiable databases, its least fixpoint lfp (TP, s B) exists, where lfp (TP, s B) denotes the least fixpoint of operator TP s containing all facts from the relations in B and with respect to a stratified or softly stratified partition of R. Lemma 1. Let D = F, R be a (softly) stratifiable database and (λ s ) λ a (soft) stratification of R inducing the partition P of R. The semantics M D of B, R coincides with the least fixpoint of TP, s i.e., M D = relation(lfp(t s P, B)). Proof. cf [6]. In Theorem 1, the correctness of MU rewritten rules µ(r) has been proven by showing them to form a subset of the MS transformed rules ms H(Rá) which are known to be sound and complete. Since ms H(Rá) is softly stratifiable according to [6], the MU rules must be softly stratifiable, too. For instance, the partition P = P 1 P 2 of the MU transformed rule set mu(r 2) of our example from Section with

10 P new E new P P new (3, 4) E new (3, 4) E new (1, 2) P(1, 4) m P new bf m E new bf m E new fb m P bb m P new bf (3) m E new bf (3) m E new fb (1) m P bb(1, 3) m P new bf (4) m E new bf (4) m E new fb (2) m P bb(1, 4) m P bb(2, 3) m P bb(2, 4) m P bb(4, 3) m P bb(4, 4) Table 1: Generated state relation facts using soft update propagation P 1: P m P bb E P m P bb π E.x1,P.y 2 (E P ) E.x 2 =P.y 1 m P bb E + m P bb π x1,y 2 ((E + P new ) (P + E new )) m P bb π x2,y 2 (m P bb y1 =x 1 E) and with partition P 2 consisting of all other MU rules, i.e. P 2 := mu(r 2) \ P 1, satisfies the condition of soft stratification. Using the soft consequence operator for the determination of lfp(tp, s B 2 {E + }) then yields the correct semantics of D2 m = B 2 {E + }, µ(r 2). The fixpoint computation of TP s leads to the generation of two new state facts for relation E, one old state fact and one new state fact for P. The entire number of generated facts is 19 in contrast to 8296 for computing the three induced insertions with respect to relation P. Table 1 summarizes the generated state relation facts with respect to corresponding rules in µ(r 2). The reason for the small number of facts is that only relevant state relation facts are derived. In the example above, this excludes the E-facts {e(10, 11), e(11, 12),..., e(98, 99), e(99, 100)} and the corresponding P -facts as they are not affected by the insertion E + and thus, do not have to be considered during the update propagation process. 7. COMPARISON AND CONCLUSION In this paper, we have presented a new method for computing the implicit changes of derived relations resulting from explicitly performed updates of the extensional fact base. We use the MS method to combine the advantages of top-down and bottom-up propagation approaches in order to restrict the computation of true updates to the affected part of the database only. The incorporation of query evaluation via MS allows for handling non-linear recursion and the reuse of intermediate query results. Additionally, the underlying set-oriented fixpoint process is well-suited for being transferred into the SQL context and can be easily combined with other relational optimization techniques. The term Magic Updates has been coined in [20], where the closeness between query answering using Magic Sets and update propagation using Magic Updates has been printed out first. To apply the Magic Sets transformation to delta rules has been first proposed in our group as well [14], where structured update propagation has been introduced as computation method for the potentially unstratifiable magic propagation rules. Structured update propagation is based on the alternating fixpoint computation [32] in order to determine the well-founded model of the possibly unstratifiable magic propagation rules correctly. The application of the alternating fixpoint computation, however, is not really efficient as the specific reason for unstratifiability (namely the application of the MS transformation to a stratified rule set) is not taken into account. Other methods in Datalog are usually based on SLDNF resolution and thus cannot guarantee termination for recursively defined predicates. In addition, a set-oriented evaluation technique is preferred in the database context. Approaches formulated in relational algebra or SQL so far are also not capable of handling recursion, the latter usually based on transformed views or specialized triggers. Transformed SQL-views directly correspond to our proposed method in the non-recursive case. The application of triggers (e.g. as proposed by Ceri/Widom) does not allow the reuse of intermediate results obtained by querying the state relations within effectiveness tests. The propagation rules proposed in this paper are restricted to the propagation of insertions and deletions of base facts in stratifiable databases. However, up till now, several approaches have been proposed dealing with further kinds of updates or additional language concepts. As far as the latter are concerned, update propagation in the presence of built-ins and (numerical) constraints has been discussed while views possibly containing duplicates are considered in [11, 15]. Aggregates and updates have been investigated in [5, 17]. As for the various types of updates, methods have been introduced for dealing with the modification of individual tuples, e.g. [11, 31], the insertion and deletion of rules (respectively view definitions) and constraints, e.g. [22, 29]. All these techniques may allow for enhancing the propagation rules introduced in this section in order to provide a more elaborate approach to update propagation. 8. REFERENCES [1] Krzysztof R. Apt, Howard A. Blair, and Adrian Walker. Towards a theory of declarative knowledge. Foundations of Deductive Databases and Logic Programs, pages , M. Kaufmann, [2] Agrawal R. Alpha: An Extension of Relational Algebra to Express a Class of Recursive Queries. IEEE TSE 14(7): (1988). [3] I. Balbin, G. S. Port, K. Ramamohanarao, and K. Meenakshi. Efficient bottom-up computation of queries. JLP, 11(3&4): , October [4] Bancilhon, F., Maier, D., Sagiv, Y., Ullman,J. D.: Magic Sets and Other Strange Ways to Implement Logic Programs. PODS 1986: [5] E. Baralis and J. Widom. A rewriting technique for using delta relations to improve condition evaluation in active databases. Technical Report CS , Stanford University, November [6] Andreas Behrend. Soft stratification for magic set based query evaluation in deductive databases. PODS 2003, New York, June 9 12, pages [7] A. Behrend, C. Dorau, R. Manthey, and G. Schüller: Incremental View-Based Analysis of Stock Market Data Streams. IDEAS, 2008: [8] A. Behrend and R. Manthey: Update Propagation in Deductive Databases Using Soft Stratification. ADBIS 2004: