Environment (Parallelizing Query Optimization)

Transcription

1 Advanced d Query Optimization i i Techniques in a Parallel Computing Environment (Parallelizing Query Optimization) Wook-Shin Han*, Wooseong Kwak, Jinsoo Lee Guy M. Lohman, Volker Markl Kyungpook National University IBM Almaden Research Center 1

2 2

3 Introduction to Our Recent Query Optimization Research Efforts Progressive optimization in a shared-nothing parallel database (SIGMOD 2007) Parallelizing li i query optimization i (VLDB 2008) Time-series Database & Ranking Ranked Subsequence Matching (VLDB 2007) XML Query Processing StreamTX: Extracting Tuples from Streaming XML Data (VLDB 2008) Mapping-driven XML transformation (WWW 2007) The Dynamic Predicate: Integrating ng Access Control with Query Processing in XML Databases (VLDB Journal 2007) Moving Object Databases Cost-Based Predictive Spatio-Temporal Join (IEEE TKDE 2008 (accepted)) 3

4 My Necessary conditions for your papers p to be accepted in SIGMOD/VLDB/ICDE COND1) Solve REAL problems This gives a very strong motivation! COND2) Solve a SMALL but NOVEL problem Your paper must be focused and distinguishable! COND3) Do NOT BLAME reviewers Your faults, not reviewers! Do not ignore their even seemingly absurd (usually, only to you!) reviews! 4

5 Success of Relational Database Management Systems Standardization d of the SQL query language Development of sophisticated query optimizers Enumerate many alternative query execution plans (QEPs) using dynamic programming (DP) Estimate the cost of each Choose the least expensive plan to execute 5

6 Motivation As # of joins increases, # of alternative QEPs considered by DP grow exponentially Although approximate search algorithms reduce the enumeration time, this can result in sub-optimal plans 6

7 * Motivation (Cont d) New wave of multi-core processors Thus, it is obvious to speed up CPU-bound query optimization by parallelizing it! In the typical parallel DBMS, only a single coordinator node optimizes the query! * 7

8 Problem Definition Parallelize l DP-based query optimization to exp loit multi-core processor architectures the DP algorithm used in join enumeration belongs to the non-serial polyadic DP class [8] 8

9 Contributions first framework for parallel l DP optimization i that generates optimal plans a parallel join enumeration algorithm, along with various allocation skip vector array with algorithms speeding up our parallel join enumeration algorithm formal analysis of why the various allocation schemes generate different sizes of search spaces among threads perform extensive experiments to show parallel algorithms allocate the work to threads evenly Enhanced algorithm using the skip vector array outperforms the conventional DP algorithm by up to orders of magnitude 9

10 A Running Example 10

11 Overview of Serial DP-based Optimization QEPs for single tables QEPs for joins 11

12 Overview of Our Solution Goals partition the search space evenly among threads process each partition independently without any dependencies among threads Key insights By partitioning sub-problems by their sizes, sub-problems of the same resulting size are mutually independent As the number of quantifiers increases, the number of subproblems of the same size grows exponentially Each sub-problem of size S (=smallsz+largesz) is constructed using any combination of one smaller sub-problem of size smallsz and another sub-problem of size largesz 12

13 Overview of Our Solution (Cont d) Transform the enumeration problem into multiple theta joins, multiple plan joins (MPJs), disjoint and connectivity filters as join conditions Each MPJ is then parallelized using multiple threads without any dependencies between the threads. By judiciously allocating to threads portions of the search space for MPJ, we can achieve linear speed-up. 13

14 Plan partitions Example P 4 = (q 1,q 1 q 2 q 3 ) (q 1,q 1 q 2 q 4 ) (q 1,q 1 q 3 q 4 ) (q 2,q 1 q 2 q 3 ) (q 3,q 1 q 2 q 3 ) (q 2,q 1 q 2 q 4 ) (q 3,q 1 q 2 q 4 ) (q 2,q 1 q 3 q 4 ) P 1 P 3 thread 1 P 2 P 2 (q 1 q 2,q 1 q 2 ) (q 1 q 2,q 1 q 3 ) (q 3,q 1 q 3 q 4 ) (q 4,q 1 q 2 q 3 ) (q 4,q 1 q 2 q 4 ) (q 4,q 1 q 3 q 4 ) (q 1 q 2,q 1 q 4 ) (q 1 q 3,q 1 q 2 ) (q 1 q 3,q 1 q 3 ) (q 1 q 3,q 1 q 4 ) thread 2 (q 1 q 4,q 1 q 2 ) (q 1 q 4,q 1 q 3 ) (q 1 q 4,q 1 q 4 ) 14

15 Search Space Allocation Schemes Total Sum Allocation Scheme Stratified Allocation Equi-depth allocation Round-robin outer allocation Round-robin inner allocation 15

16 Example: Round-Robin Outer Allocation thread 1 thread 2 (q 1,q 1 q 2 q 3 ) (q 1,q 1 q 2 q 4 ) (q 1,q 1 q 3 q 4 ) (q 2,q 1 q 2 q 3 ) (q 2,q 1 q 2 q 4 ) (q 2,q 1 q 3 q 4 ) P 1 P 3 (q 3,q 1 q 2 q 3 ) (q 3,q 1 q 2 q 4 ) (q 3,q 1 q 3 q 4 ) (q 4,q 1 q 2 q 3 ) (q 4,q 1 q 2 q 4 ) (q 4,q 1 q 3 q 4 ) (q 1 q 2,q 1 q 2 ) (q 1 q 2,q 1 q 3 ) (q 1 q 2,q 1 q 4 ) P 2 P 2 (q 1 q 3,q 1 q 2 ) (q 1 q 3,q 1 q 3 ) (q 1 q 3,q 1 q 4 ) (q 1 q 4,q 1 q 2 ) (q 1 q 4,q 1 q 3 ) (q 1 q 4,q 1 q 4 ) 16

17 Basic MPJ Use block-nested loop join to be cache-conscious To represent an allocated search space for each thread, use search space description vector (SSDV) 17

18 Motivation for Skip Vector Array Considerable overhead in calling exponential number of disjoint filter calls! 18

19 Skip Vector Array P QS PlanList SV QS PlanList SV 1 P 3 1 q q 1 q 2 q q q 1 q 2 q q q 1 q 2 q q q 1 q 2 q q q 1 q 3 q q q 1 q 4 q q q 1 q 4 q q q 2 q 5 q q 4 q 7 q 8 SKIP! NOTE: Skip Vector Array can be built in one scan of quantifier sets. 19

20 How to allocate SVAs to threads? Use a pair of partitioned ii SVAs as the unit of allocation to threads First, divide each plan partition into sub-partitions Second, build the SVAs for all sub-partitions QS q 1 q 2 q 3 q 1 q 2 q 4 q 1 q 2 q 5 q 1 q 2 q 6 q 1 q 3 q 4 q 1 q 4 q 7 q 1 q 4 q 8 q 2 q 5 q 6 q 4 q 7 q 8 PlanList P {3,1} P {3,3} P 3 QS PlanList SV Equi-depth partitioning q 1 q 2 q 3 q 1 q 2 q 4 & building SVAs P {3,2} P {3,4} QS PlanList SV q 1 q 3 q 4 q 1 q 4 q 7 QS PlanList SV QS PlanList SV q 1 q 2 q 5 q 1 q 4 q 8 q 1 q 2 q 6 q 2 q 5 q 6 q 4 q 7 q 8 20

21 Formal Analysis About Various Refer to the paper allocation schemes 21

22 Experiments Environment Windows Vista PC with two Intel Xeon Quad Core E GHz CPUs and 8 GB of RAM All serial/parallel algorithms prototyped in PostgreSQL 22

23 Overall Comparisons of different algorithms 17 hours to less then 2 minutes using only 8 threads! 23

24 Sensitivity Analysis for Basic MPJ (8 threads) 24

25 Sensitivity Analysis for MPJ with SVA (8 threads) 25

26 For more experimental results Refer to the paper 26

27 Conclusions proposed a novel framework for parallelizing li i query optimization to exploit the coming wave of multi-core processor architectures By viewing enumeration as a join, we devised a way to partition the search space cleanly into independent problems To minimize mnmz unnecessary calls to the disjoint testing, we proposed the skip vector array We also formally analyzed why our various allocation schemes generate differently-sized search spaces among threads, to ensure even allocation of the work among threads 27

28 Thank you! Any yquestions? 28

29 Backup slides 29

30 30

31 Skip Vector Array Avoid unnecessary invocations of the disjoint filter Augment each row in the plan partition with a Skip Vector for the quantifier set in that row Plan partition is sorted in lexicographical order to skip large groups of quantifier sets = 31

32 Equi-depth Allocation 32

33 Round-Robin Robin Inner Allocation 33