CS 347 Distributed Databases and Transaction Processing Notes03: Query Processing
|
|
|
- Angel Short
- 5 years ago
- Views:
Transcription
1 CS 347 Distributed Databases and Transaction Processing Notes03: Query Processing Hector Garcia-Molina Zoltan Gyongyi CS 347 Notes 03 1
2 Query Processing! Decomposition! Localization! Optimization CS 347 Notes 03 2
3 Decomposition! Same as in centralized system! Normalization! Eliminating redundancy! Algebraic rewriting CS 347 Notes 03 3
4 Normalization! Convert from general language to a standard form (e.g., relational algebra) CS 347 Notes 03 4
5 Example select A, C from R, S where (R.B=1 and S.D=2) or (R.C>3 and S.D=2) # A, C! (R.B=1 v R.C>3) " S.D=2 R S conjunctive normal form CS 347 Notes 03 5
6 Also: Detect invalid expressions E.g.: select * from R where R.A=3! R does not have A attribute E.g.: select R.A from R, S, T where R.B=S.C CS 347 Notes 03 6
7 Note: Query graph useful for detecting last problem R.B=S.C R S T R.A Result CS 347 Notes 03 7
8 Note: Query graph useful for detecting last problem R.B=S.C R S T join graph R.A Result disconnected query graph CS 347 Notes 03 8
9 Eliminate redundancy E.g.: in conditions (S.A=1) " (S.A>5) $ false (S.A<10) " (S.A<5) $ S.A<5 CS 347 Notes 03 9
10 E.g.: common sub-expressions U U S!cond!cond T S!cond T R R R CS 347 Notes 03 10
11 Algebraic rewriting E.g.: push conditions down!cond3!cond!cond1!cond2 R S R S CS 347 Notes 03 11
12 ! After decomposition:!one or more algebraic query trees on relations! Localization:!Replace relations by corresponding fragments CS 347 Notes 03 12
13 Localization steps (1) Start with query (2) Replace relations by fragments (3) Push % up #,! down (use CS 245 rules) (4) Simplify eliminate unnecessary operations CS 347 Notes 03 13
14 Notation for fragment [R: cond] fragment conditions its tuples satisfy CS 347 Notes 03 14
15 Example A (1)!E=3 R CS 347 Notes 03 15
16 (2)!E=3 % [R1: E < 10] [R2: E & 10] CS 347 Notes 03 16
17 (3) %!E=3!E=3 [R1: E < 10] [R2: E & 10] CS 347 Notes 03 17
18 (3) %!E=3!E=3 [R1: E < 10] [R2: E & 10] $ Ø CS 347 Notes 03 18
19 (4)!E=3 [R1: E < 10] CS 347 Notes 03 19
20 Rule 1 A B!C1[R: c2] $!C1[R: c1 " c2] [R: False] $ Ø CS 347 Notes 03 20
21 In example A:!E=3[R2: E&10] $ [!E=3 R2: E=3 " E&10] $ [!E=3 R2: False] $ Ø CS 347 Notes 03 21
22 Example B (1) R A S A = common attribute CS 347 Notes 03 22
23 (2) A % % [R1: A<5] [R2: 5 ' A ' 10] [R3: A>10] [S1: A<5] [S2: A & 5] CS 347 Notes 03 23
24 (3) % A A A [R1:A<5][S1:A<5] [R1:A<5][S2:A&5] [R2:5'A'10][S1:A<5] A A A [R2:5'A'10][S2:A&5] [R3:A>10][S1:A<5] [R3:A>10][S2:A&5] CS 347 Notes 03 24
25 (3) % A A A [R1:A<5][S1:A<5] [R1:A<5][S2:A&5] [R2:5'A'10][S1:A<5] A A A [R2:5'A'10][S2:A&5] [R3:A>10][S1:A<5] [R3:A>10][S2:A&5] CS 347 Notes 03 25
26 (4) % A A A [R1:A<5][S1:A<5] [R2:5'A'10][S2:A&5] [R3:A>10][S2:A&5] CS 347 Notes 03 26
27 Rule 2 [R: C1] [S: C2] $ [R S: C1 " C2 " R.A = S.A] A A CS 347 Notes 03 27
28 In step 4 of Example B: [R1: A<5] [S2: A & 5] A $ [R1 S2: R1.A < 5 " S2.A & 5 " A R1.A = S2.A ] $ [R1 S2: False] $ Ø A CS 347 Notes 03 28
29 ! Localization with vertical fragmentation Example C (1) ( A R1(K, A, B) R R2(K, C, D) CS 347 Notes 03 29
30 (2) ( A K R1 R2 (K, A, B) (K, C, D) CS 347 Notes 03 30
31 (3) ( A K ( K,A ( K,A R1 R2 (K, A, B) (K, C, D) CS 347 Notes 03 31
32 (4) ( A R1 (K, A, B) CS 347 Notes 03 32
33 Rule 3! Given vertical fragmentation of R: Ri = ( A i (R), Ai ) A! Then for any B ) A: ( B (R) = ( B [ Ri B * Ai + Ø ] i CS 347 Notes 03 33
34 Summary - Query Processing! Decomposition!! Localization!! Optimization!Overview!Tricks for joins + other operations!strategies for optimization CS 347 Notes 03 34
35 Optimization Process: Generate query plans Estimate size of intermediate results Estimate cost of plan ($,time, ) P1 P2 P3 Pn C1 C2 C3 Cn pick minimum CS 347 Notes 03 35
36 Differences from centralized optimization:! New strategies for some operations (semi-join, range-partitioning, sort, )! Many ways to assign and schedule processors CS 347 Notes 03 36
37 Parallel/distributed sort Input: (a) relation R on single site/disk (b) R fragmented/partitioned by sort attribute (c) R fragmented/partitioned by other attribute CS 347 Notes 03 37
38 Output: (a) sorted R on single site/disk (b) fragments/partitions sorted F1 F2 F3 CS 347 Notes 03 38
39 Basic sort! R(K, ); sort on K! Fragmented (range partitioned) on K Vector: [ k0, k1,, kn ] 7 3 k0 k CS 347 Notes 03 39
40 ! Algorithm:!Sort each fragment independently!if necessary, ship results CS 347 Notes 03 40
41 $ Same idea on different architectures: Shared nothing F1 P1 M Net P2 M F2 Shared memory sorts F1 sorts F2 P1 P2 F1 M F2 CS 347 Notes 03 41
42 Range partitioning sort! R(K, ); sort on K! R located at one or more sites/disks not fragmented on K CS 347 Notes 03 42
43 ! Algorithm: (a) Range partition on K (b) Basic sort RA R1 Local sort R 1 RB k0 R2 Local sort R 2 Result k1 R3 Local sort R 3 CS 347 Notes 03 43
44 Selecting a good partition vector RA RB RC CS 347 Notes 03 44
45 Example! Each site sends to coordinator:!min sort key!max sort key!number of tuples! Coordinator computes vector and distributes to sites (also decides # of sites for local sorts) CS 347 Notes 03 45
46 Sample scenario Coordinator receives: RA: Min=5 Max=10 # = 10 tuples RB: Min=7 Max=17 # = 10 tuples CS 347 Notes 03 46
47 Sample scenario Coordinator receives: RA: Min=5 Max=10 # = 10 tuples RB: Min=7 Max=17 # = 10 tuples Expected tuples: k0? (assuming we want to sort at 2 sites) CS 347 Notes 03 47
48 Expected tuples: k0? Expected tuples Total tuples with key < = k0 2 2(k0 5) + (k0 7) = 10 3 k0 = = 27 k0 = 9 (assuming we want to sort at 2 sites) CS 347 Notes 03 48
49 Variations! Send more info to coordinator!partition vector for local site E.g., RA: # tuples local vector!histogram CS 347 Notes 03 49
50 ! More than one round E.g.: (1) Sites send range and # tuples (2) Coordinator returns preliminary vector V0 (3) Sites tell coordinator how many tuples in each V0 range (4) Coordinator computes final vector Vf CS 347 Notes 03 50
51 ! Can you come up with a distributed algorithm? (no coordinator) CS 347 Notes 03 51
52 Parallel external sort-merge! Same as range-partitioning sort, except sort first RA Local sort RA R1 RB Local sort RB k0 R2 Result In order k1 R3 Merge CS 347 Notes 03 52
53 Parallel/distributed join Input: Relations R, S May or may not be partitioned Output: R S Result at one or more sites CS 347 Notes 03 53
54 Partitioned join (equi-join) Local join RA R1 S1 SA RB R2 S2 SB f(a) R3 Result S3 f(a) SC CS 347 Notes 03 54
55 Notes! Same partition function f is used for both R and S (applied to join attribute A)! f can be range or hash partitioning! Local join can be of any type (use any CS 245 optimization)! Various scheduling options, e.g., (a)! partition R; partition S; join (b)! partition R; build local hash table for R; partition S and hash-join CS 347 Notes 03 55
56 More notes! We already know why partition-join works: % % $ R1 R2 R3 S1 S2 S3 R1 S1 R2 S2 R3 S3 %! Useful to give this type of join a name, because we may want to partition data to make partition-join possible (especially in a parallel DB system) CS 347 Notes 03 56
57 Even more notes! Selecting a good partition function f is very important:!number of fragments!hash function!partition vector CS 347 Notes 03 57
58 ! Good partition vector!goal: Ri + Si the same!can use coordinator to select CS 347 Notes 03 58
59 Asymmetric fragment + replicate join Local join RA R1 S SA RB R2 S SB Partition R3 S Result Union CS 347 Notes 03 59
60 Notes:! Can use any partition function f for R (even round robin)! Can do any join, not just equi-join e.g.: R S R.A < S.B CS 347 Notes 03 60
61 General fragment and replicate join RA R1 R1 RB R2 R2 R3 R3 Partition n copies of each fragment CS 347 Notes 03 61
62 ! S is partitioned in similar fashion R1 R2 R3 S1 S1 S1 R1 R2 R3 S2 S2 S2 All n x m pairings of R, S fragments Result CS 347 Notes 03 62
63 Notes! Asymmetric F+R join is special case of general F+R! Asymmetric F+R may be good if S small! Works for non-equi-joins CS 347 Notes 03 63
64 Semi-join! Goal: reduce communication traffic! R S $ (R S) S or A A R (S R) or A A (R S) (S R) A A A A CS 347 Notes 03 64
65 Example: R S R A B A C 2 a 10 b 25 c 30 d S 3 x 10 y 15 z 25 w 32 x CS 347 Notes 03 65
66 Example: R R Ans: R S S A B A C 2 a 10 b 25 c 30 d (A R = [2,10,25,30] CS 347 Notes S S R = 3 x 10 y 15 z 25 w 32 x A C 10 y 25 w
67 R A B A C 2 a 10 b 25 c 30 d Computing transmitted data in example! with semi-join R (S R): T = 4 A + 2 A + C + result! with join R S: T = 4 A + B + result S 3 x 10 y 15 z 25 w 32 x better if B is large CS 347 Notes 03 67
68 ! In general:! Say R is smaller relation! (R S) S better than R S if A A A size ((A S) + size (R A S) < size (R) CS 347 Notes 03 68
69 ! Similar comparisons for other semi-joins! Remember: only taking into account transmission cost CS 347 Notes 03 69
70 ! Trick: Encode (A S (or (A R ) as a bit vector key in S one bit/possible key CS 347 Notes 03 70
71 Three way joins with semi-joins Goal: R S T CS 347 Notes 03 71
72 Three way joins with semi-joins Goal: R S T Option 1: R S T where R = R S; S = S T Option 2: R S T where R = R S ; S = S T CS 347 Notes 03 72
73 ! Many options! Number of semi-join options is exponential in # of relations in join CS 347 Notes 03 73
74 Privacy-preserving join! Site 1 has R(A,B)! Site 2 has S(A,C)! Want to compute R! Site 1 should NOT discover any S info not in the join! Site 2 should NOT discover any R info not in the join S R S site 1 site 2 CS 347 Notes 03 74
75 Semi-join does not work! If site 1 sends (A R to site 2, site 2 learns all keys of R R A B a1 b1 a2 b2 a3 b3 a4 b4 site 1 (A R = [a1, a2, a3, a4] S A C a1 c1 a3 c2 a5 c3 a7 c4 site 2 CS 347 Notes 03 75
76 Fix: send hashed keys! Site 1 hashes each value of A before sending! Site 2 hashes (same function) its own A values to see what tuples match R A B a1 b1 a2 b2 a3 b3 a4 b4 site 1 (A R = [h(a1), h(a2), h(a3), h(a4)] Site 2 sees it has h(a1), h(a3) [(a1, c1), (a3, c3)] S A C a1 c1 a3 c2 a5 c3 a7 c4 site 2 CS 347 Notes 03 76
77 What is problem? R A B a1 b1 a2 b2 a3 b3 a4 b4 (A R = [h(a1), h(a2), h(a3), h(a4)] Site 2 sees it has h(a1), h(a3) S A C a1 c1 a3 c2 a5 c3 a7 c4 site 1! Dictionary attack! Site 2 takes all keys a1, a2, a3,... and checks if h(a1), h(a2), h(a3), matches what site 1 sent! Site 1 will never notice [(a1, c1), (a3, c3)] site 2 CS 347 Notes 03 77
78 Adversary model! Honest but curious!observes the rules of the interaction (e.g., sending incorrect keys not possible)!may analyze received data in hope of gaining additional information CS 347 Notes 03 78
79 One solution [Agrawal+03: Information Sharing across Private Databases]! Use commutative encryption function!e i (x) = x encryption using site i s private key!e 1 (E 2 (x)) = E 2 (E 1 (X))!Shorthand for example: E 1 (x) is x E 2 (x) is x E 1 (E 2 (x)) is x CS 347 Notes 03 79
80 Solution R A B a1 b1 a2 b2 a3 b3 a4 b4 site 1 (a1, a2, a3, a4) (a1, a2, a3, a4) (a1, a3, a5, a7) S A C a1 c1 a3 c2 a5 c3 a7 c4 site 2 computes [a1, a3, a5, a7], intersects with [a1, a2, a3, a4] [(a1, b1), (a3, b3)] CS 347 Notes 03 80
81 ! Why does this solution work?!could one site perform dictionary attack without the other noticing? CS 347 Notes 03 81
82 Other parallel operations! Duplicate elimination!sort first (in parallel) then eliminate duplicates in result!partition tuples (range or hash) and eliminate locally! Aggregates!Partition by grouping attributes; compute aggregate locally CS 347 Notes 03 82
83 Example Ra Rb! sum(sal) group by dept CS 347 Notes 03 83
84 Example Ra Rb! sum(sal) group by dept CS 347 Notes 03 84
85 Example Ra sum Rb sum! sum(sal) group by dept CS 347 Notes 03 85
86 Example less data! Ra sum sum Rb sum sum! sum(sal) group by dept CS 347 Notes 03 86
87 Enhancements for aggregates! Perform aggregate during partition to reduce data transmitted! Does not work for all aggregate functions Which ones? CS 347 Notes 03 87
88 Selection! Range or hash partition! Straightforward But what about indexes? CS 347 Notes 03 88
89 Indexing! Can think of partition vector as root of distributed index: k0 k1 Local indexes site 1 site 2 site 3 CS 347 Notes 03 89
90 Index on non-partition attribute k0 k1 Index sites Tuple sites CS 347 Notes 03 90
91 Notes! If index is not too big, it may be better to keep whole and make copies! If updates are frequent, can partition update work How do we handle the split of B-tree pages? CS 347 Notes 03 91
92 ! Extensible or linear hashing R1 f R2 R3 R4 add CS 347 Notes 03 92
93 ! How do we adapt schemes?! Where do we store directory, set of participants,?! Which one is better for a distributed environment?! Can we design a hashing scheme with no global knowledge (P2P)? CS 347 Notes 03 93
94 Summary: query processing! Decomposition and localization!! Optimization!Overview!!Tricks for joins, sort,!!tricks for inter-operations parallelism!strategies for optimization CS 347 Notes 03 94
95 Inter-operation parallelism! Pipelined! Independent CS 347 Notes 03 95
96 Pipelined parallelism site 2 site 1 R!c S site 2 site 1 Tuples Join Probe Result R matching!c S CS 347 Notes 03 96
97 Independent parallelism site 1 site 2 R S T V (1) temp1, R S; temp2, T V (2) result, temp1 temp2 CS 347 Notes 03 97
98 ! Pipelining cannot be used in all cases E.g., hash join Stream of R tuples Stream of S tuples CS 347 Notes 03 98
99 Summary! As we consider query plans for optimization, we must consider various tricks:!for individual operations!for scheduling operations CS 347 Notes 03 99
CS 347 Parallel and Distributed Data Processing
CS 347 Parallel and Distributed Data Processing Spring 2016 Notes 3: Query Processing Query Processing Decomposition Localization Optimization CS 347 Notes 3 2 Decomposition Same as in centralized system
P Q1 Q2 Q3 Q4 Q5 Tot (60) (20) (20) (20) (60) (20) (200) You are allotted a maximum of 4 hours to complete this exam.
Exam INFO-H-417 Database System Architecture 13 January 2014 Name: ULB Student ID: P Q1 Q2 Q3 Q4 Q5 Tot (60 (20 (20 (20 (60 (20 (200 Exam modalities You are allotted a maximum of 4 hours to complete this
CS 347 Parallel and Distributed Data Processing
CS 347 Parallel and Distributed Data Processing Spring 2016 Notes 4: Query Optimization Query Optimization Cost estimation Strategies for exploring plans Q min CS 347 Notes 4 2 Cost Estimation Based on
2.6 Complexity Theory for Map-Reduce. Star Joins 2.6. COMPLEXITY THEORY FOR MAP-REDUCE 51
2.6. COMPLEXITY THEORY FOR MAP-REDUCE 51 Star Joins A common structure for data mining of commercial data is the star join. For example, a chain store like Walmart keeps a fact table whose tuples each
Correlated subqueries. Query Optimization. Magic decorrelation. COUNT bug. Magic example (slide 2) Magic example (slide 1)
Correlated subqueries Query Optimization CPS Advanced Database Systems SELECT CID FROM Course Executing correlated subquery is expensive The subquery is evaluated once for every CPS course Decorrelate!
CSE 562 Database Systems
Outline Query Optimization CSE 562 Database Systems Query Processing: Algebraic Optimization Some slides are based or modified from originals by Database Systems: The Complete Book, Pearson Prentice Hall
Relational Algebra on Bags. Why Bags? Operations on Bags. Example: Bag Selection. σ A+B < 5 (R) = A B
Relational Algebra on Bags Why Bags? 13 14 A bag (or multiset ) is like a set, but an element may appear more than once. Example: {1,2,1,3} is a bag. Example: {1,2,3} is also a bag that happens to be a
CS 347 Parallel and Distributed Data Processing
CS 347 Parallel and Distributed Data Processing Spring 2016 Notes 2: Distributed Database Design Logistics Gradiance No action items for now Detailed instructions coming shortly First quiz to be released
Multiple-Site Distributed Spatial Query Optimization using Spatial Semijoins
11 Multiple-Site Distributed Spatial Query Optimization using Spatial Semijoins Wendy OSBORN a, 1 and Saad ZAAMOUT a a Department of Mathematics and Computer Science, University of Lethbridge, Lethbridge,
α-acyclic Joins Jef Wijsen May 4, 2017
α-acyclic Joins Jef Wijsen May 4, 2017 1 Motivation Joins in a Distributed Environment Assume the following relations. 1 M[NN, Field of Study, Year] stores data about students of UMONS. For example, (19950423158,
Relational operations
Architecture Relational operations We will consider how to implement and define physical operators for: Projection Selection Grouping Set operators Join Then we will discuss how the optimizer use them
Quiz 2. Due November 26th, CS525 - Advanced Database Organization Solutions
Name CWID Quiz 2 Due November 26th, 2015 CS525 - Advanced Database Organization s Please leave this empty! 1 2 3 4 5 6 7 Sum Instructions Multiple choice questions are graded in the following way: You
CS 347. Parallel and Distributed Data Processing. Spring Notes 11: MapReduce
CS 347 Parallel and Distributed Data Processing Spring 2016 Notes 11: MapReduce Motivation Distribution makes simple computations complex Communication Load balancing Fault tolerance Not all applications
Database Design and Implementation
Database Design and Implementation CS 645 Data provenance Provenance provenance, n. The fact of coming from some particular source or quarter; origin, derivation [Oxford English Dictionary] Data provenance
Flow Algorithms for Parallel Query Optimization
Flow Algorithms for Parallel Query Optimization Amol Deshpande amol@cs.umd.edu University of Maryland Lisa Hellerstein hstein@cis.poly.edu Polytechnic University August 22, 2007 Abstract In this paper
Lifted Inference: Exact Search Based Algorithms
Lifted Inference: Exact Search Based Algorithms Vibhav Gogate The University of Texas at Dallas Overview Background and Notation Probabilistic Knowledge Bases Exact Inference in Propositional Models First-order
NP-COMPLETE PROBLEMS. 1. Characterizing NP. Proof
T-79.5103 / Autumn 2006 NP-complete problems 1 NP-COMPLETE PROBLEMS Characterizing NP Variants of satisfiability Graph-theoretic problems Coloring problems Sets and numbers Pseudopolynomial algorithms
Factorized Relational Databases Olteanu and Závodný, University of Oxford
November 8, 2013 Database Seminar, U Washington Factorized Relational Databases http://www.cs.ox.ac.uk/projects/fd/ Olteanu and Závodný, University of Oxford Factorized Representations of Relations Cust
Databases 2011 The Relational Algebra
Databases 2011 Christian S. Jensen Computer Science, Aarhus University What is an Algebra? An algebra consists of values operators rules Closure: operations yield values Examples integers with +,, sets
Introduction to Cryptography Lecture 13
Introduction to Cryptography Lecture 13 Benny Pinkas June 5, 2011 Introduction to Cryptography, Benny Pinkas page 1 Electronic cash June 5, 2011 Introduction to Cryptography, Benny Pinkas page 2 Simple
communication complexity lower bounds yield data structure lower bounds
communication complexity lower bounds yield data structure lower bounds Implementation of a database - D: D represents a subset S of {...N} 2 3 4 Access to D via "membership queries" - Q for each i, can
CS425: Algorithms for Web Scale Data
CS425: Algorithms for Web Scale Data Most of the slides are from the Mining of Massive Datasets book. These slides have been modified for CS425. The original slides can be accessed at: www.mmds.org Challenges
7 RC Simulates RA. Lemma: For every RA expression E(A 1... A k ) there exists a DRC formula F with F V (F ) = {A 1,..., A k } and
7 RC Simulates RA. We now show that DRC (and hence TRC) is at least as expressive as RA. That is, given an RA expression E that mentions at most C, there is an equivalent DRC expression E that mentions
A Worst-Case Optimal Multi-Round Algorithm for Parallel Computation of Conjunctive Queries
A Worst-Case Optimal Multi-Round Algorithm for Parallel Computation of Conjunctive Queries Bas Ketsman Dan Suciu Abstract We study the optimal communication cost for computing a full conjunctive query
Query answering using views
Query answering using views General setting: database relations R 1,...,R n. Several views V 1,...,V k are defined as results of queries over the R i s. We have a query Q over R 1,...,R n. Question: Can
GAV-sound with conjunctive queries
GAV-sound with conjunctive queries Source and global schema as before: source R 1 (A, B),R 2 (B,C) Global schema: T 1 (A, C), T 2 (B,C) GAV mappings become sound: T 1 {x, y, z R 1 (x,y) R 2 (y,z)} T 2
CSE 4502/5717 Big Data Analytics Spring 2018; Homework 1 Solutions
CSE 502/5717 Big Data Analytics Spring 2018; Homework 1 Solutions 1. Consider the following algorithm: for i := 1 to α n log e n do Pick a random j [1, n]; If a[j] = a[j + 1] or a[j] = a[j 1] then output:
Logic and Databases. Phokion G. Kolaitis. UC Santa Cruz & IBM Research Almaden. Lecture 4 Part 1
Logic and Databases Phokion G. Kolaitis UC Santa Cruz & IBM Research Almaden Lecture 4 Part 1 1 Thematic Roadmap Logic and Database Query Languages Relational Algebra and Relational Calculus Conjunctive
High Performance Computing
Master Degree Program in Computer Science and Networking, 2014-15 High Performance Computing 2 nd appello February 11, 2015 Write your name, surname, student identification number (numero di matricola),
SPATIAL DATA MINING. Ms. S. Malathi, Lecturer in Computer Applications, KGiSL - IIM
SPATIAL DATA MINING Ms. S. Malathi, Lecturer in Computer Applications, KGiSL - IIM INTRODUCTION The main difference between data mining in relational DBS and in spatial DBS is that attributes of the neighbors
Jeffrey D. Ullman Stanford University
Jeffrey D. Ullman Stanford University 3 We are given a set of training examples, consisting of input-output pairs (x,y), where: 1. x is an item of the type we want to evaluate. 2. y is the value of some
CS246: Mining Massive Datasets Jure Leskovec, Stanford University
CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu 2/26/2013 Jure Leskovec, Stanford CS246: Mining Massive Datasets, http://cs246.stanford.edu 2 More algorithms
Lecture 11: Hash Functions, Merkle-Damgaard, Random Oracle
CS 7880 Graduate Cryptography October 20, 2015 Lecture 11: Hash Functions, Merkle-Damgaard, Random Oracle Lecturer: Daniel Wichs Scribe: Tanay Mehta 1 Topics Covered Review Collision-Resistant Hash Functions
CS341 info session is on Thu 3/1 5pm in Gates415. CS246: Mining Massive Datasets Jure Leskovec, Stanford University
CS341 info session is on Thu 3/1 5pm in Gates415 CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu 2/28/18 Jure Leskovec, Stanford CS246: Mining Massive Datasets,
Multi-join Query Evaluation on Big Data Lecture 2
Multi-join Query Evaluation on Big Data Lecture 2 Dan Suciu March, 2015 Dan Suciu Multi-Joins Lecture 2 March, 2015 1 / 34 Multi-join Query Evaluation Outline Part 1 Optimal Sequential Algorithms. Thursday
Financial. Analysis. O.Eval = {Low, High} Store. Bid. Update. Risk. Technical. Evaluation. External Consultant
ACM PODS98 of CS Dept at Stony Brook University Based Modeling and Analysis of Logic Workows Hasan Davulcu* N.Y. 11794, U.S.A. * Joint with M. Kifer, C.R. Ramakrishnan, I.V. Ramakrishnan Hasan Davulcu
Lecture and notes by: Alessio Guerrieri and Wei Jin Bloom filters and Hashing
Bloom filters and Hashing 1 Introduction The Bloom filter, conceived by Burton H. Bloom in 1970, is a space-efficient probabilistic data structure that is used to test whether an element is a member of
INTRODUCTION TO RELATIONAL DATABASE SYSTEMS
INTRODUCTION TO RELATIONAL DATABASE SYSTEMS DATENBANKSYSTEME 1 (INF 3131) Torsten Grust Universität Tübingen Winter 2017/18 1 THE RELATIONAL ALGEBRA The Relational Algebra (RA) is a query language for
DRAFT. Algebraic computation models. Chapter 14
Chapter 14 Algebraic computation models Somewhat rough We think of numerical algorithms root-finding, gaussian elimination etc. as operating over R or C, even though the underlying representation of the
Exam 1. March 12th, CS525 - Midterm Exam Solutions
Name CWID Exam 1 March 12th, 2014 CS525 - Midterm Exam s Please leave this empty! 1 2 3 4 5 Sum Things that you are not allowed to use Personal notes Textbook Printed lecture notes Phone The exam is 90
Foundations of Databases
Foundations of Databases (Slides adapted from Thomas Eiter, Leonid Libkin and Werner Nutt) Foundations of Databases 1 Quer optimization: finding a good wa to evaluate a quer Queries are declarative, and
Lecture 21: Algebraic Computation Models
princeton university cos 522: computational complexity Lecture 21: Algebraic Computation Models Lecturer: Sanjeev Arora Scribe:Loukas Georgiadis We think of numerical algorithms root-finding, gaussian
Lecture 11: Data Flow Analysis III
CS 515 Programming Language and Compilers I Lecture 11: Data Flow Analysis III (The lectures are based on the slides copyrighted by Keith Cooper and Linda Torczon from Rice University.) Zheng (Eddy) Zhang
You are here! Query Processor. Recovery. Discussed here: DBMS. Task 3 is often called algebraic (or re-write) query optimization, while
Module 10: Query Optimization Module Outline 10.1 Outline of Query Optimization 10.2 Motivating Example 10.3 Equivalences in the relational algebra 10.4 Heuristic optimization 10.5 Explosion of search
Mining Data Streams. The Stream Model. The Stream Model Sliding Windows Counting 1 s
Mining Data Streams The Stream Model Sliding Windows Counting 1 s 1 The Stream Model Data enters at a rapid rate from one or more input ports. The system cannot store the entire stream. How do you make
DESIGN THEORY FOR RELATIONAL DATABASES. csc343, Introduction to Databases Renée J. Miller and Fatemeh Nargesian and Sina Meraji Winter 2018
DESIGN THEORY FOR RELATIONAL DATABASES csc343, Introduction to Databases Renée J. Miller and Fatemeh Nargesian and Sina Meraji Winter 2018 1 Introduction There are always many different schemas for a given
The Query Containment Problem: Set Semantics vs. Bag Semantics. Phokion G. Kolaitis University of California Santa Cruz & IBM Research - Almaden
The Query Containment Problem: Set Semantics vs. Bag Semantics Phokion G. Kolaitis University of California Santa Cruz & IBM Research - Almaden PROBLEMS Problems worthy of attack prove their worth by hitting
Robust Network Codes for Unicast Connections: A Case Study
Robust Network Codes for Unicast Connections: A Case Study Salim Y. El Rouayheb, Alex Sprintson, and Costas Georghiades Department of Electrical and Computer Engineering Texas A&M University College Station,
Module 10: Query Optimization
Module 10: Query Optimization Module Outline 10.1 Outline of Query Optimization 10.2 Motivating Example 10.3 Equivalences in the relational algebra 10.4 Heuristic optimization 10.5 Explosion of search
Conceptual Explanations: Radicals
Conceptual Eplanations: Radicals The concept of a radical (or root) is a familiar one, and was reviewed in the conceptual eplanation of logarithms in the previous chapter. In this chapter, we are going
FIRST-ORDER QUERY EVALUATION ON STRUCTURES OF BOUNDED DEGREE
FIRST-ORDER QUERY EVALUATION ON STRUCTURES OF BOUNDED DEGREE INRIA and ENS Cachan e-mail address: kazana@lsv.ens-cachan.fr WOJCIECH KAZANA AND LUC SEGOUFIN INRIA and ENS Cachan e-mail address: see http://pages.saclay.inria.fr/luc.segoufin/
Mining Data Streams. The Stream Model Sliding Windows Counting 1 s
Mining Data Streams The Stream Model Sliding Windows Counting 1 s 1 The Stream Model Data enters at a rapid rate from one or more input ports. The system cannot store the entire stream. How do you make
Extending Oblivious Transfers Efficiently
Extending Oblivious Transfers Efficiently Yuval Ishai Technion Joe Kilian Kobbi Nissim Erez Petrank NEC Microsoft Technion Motivation x y f(x,y) How (in)efficient is generic secure computation? myth don
Lecture Lecture 25 November 25, 2014
CS 224: Advanced Algorithms Fall 2014 Lecture Lecture 25 November 25, 2014 Prof. Jelani Nelson Scribe: Keno Fischer 1 Today Finish faster exponential time algorithms (Inclusion-Exclusion/Zeta Transform,
Stat Mech: Problems II
Stat Mech: Problems II [2.1] A monatomic ideal gas in a piston is cycled around the path in the PV diagram in Fig. 3 Along leg a the gas cools at constant volume by connecting to a cold heat bath at a
Fast Sorting and Selection. A Lower Bound for Worst Case
Presentation for use with the textbook, Algorithm Design and Applications, by M. T. Goodrich and R. Tamassia, Wiley, 0 Fast Sorting and Selection USGS NEIC. Public domain government image. A Lower Bound
Branch Prediction based attacks using Hardware performance Counters IIT Kharagpur
Branch Prediction based attacks using Hardware performance Counters IIT Kharagpur March 19, 2018 Modular Exponentiation Public key Cryptography March 19, 2018 Branch Prediction Attacks 2 / 54 Modular Exponentiation
Cut-and-Choose Yao-Based Secure Computation in the Online/Offline and Batch Settings
Cut-and-Choose Yao-Based Secure Computation in the Online/Offline and Batch Settings Yehuda Lindell Bar-Ilan University, Israel Technion Cryptoday 2014 Yehuda Lindell Online/Offline and Batch Yao 30/12/2014
Lecture 19: Interactive Proofs and the PCP Theorem
Lecture 19: Interactive Proofs and the PCP Theorem Valentine Kabanets November 29, 2016 1 Interactive Proofs In this model, we have an all-powerful Prover (with unlimited computational prover) and a polytime
A Dichotomy. in in Probabilistic Databases. Joint work with Robert Fink. for Non-Repeating Queries with Negation Queries with Negation
Dichotomy for Non-Repeating Queries with Negation Queries with Negation in in Probabilistic Databases Robert Dan Olteanu Fink and Dan Olteanu Joint work with Robert Fink Uncertainty in Computation Simons
A. Incorrect! Replacing is not a method for solving systems of equations.
ACT Math and Science - Problem Drill 20: Systems of Equations No. 1 of 10 1. What methods were presented to solve systems of equations? (A) Graphing, replacing, and substitution. (B) Solving, replacing,
Finally the Weakest Failure Detector for Non-Blocking Atomic Commit
Finally the Weakest Failure Detector for Non-Blocking Atomic Commit Rachid Guerraoui Petr Kouznetsov Distributed Programming Laboratory EPFL Abstract Recent papers [7, 9] define the weakest failure detector
Unit 4 Systems of Equations Systems of Two Linear Equations in Two Variables
Unit 4 Systems of Equations Systems of Two Linear Equations in Two Variables Solve Systems of Linear Equations by Graphing Solve Systems of Linear Equations by the Substitution Method Solve Systems of
Quantum Mechanics - I Prof. Dr. S. Lakshmi Bala Department of Physics Indian Institute of Technology, Madras. Lecture - 16 The Quantum Beam Splitter
Quantum Mechanics - I Prof. Dr. S. Lakshmi Bala Department of Physics Indian Institute of Technology, Madras Lecture - 16 The Quantum Beam Splitter (Refer Slide Time: 00:07) In an earlier lecture, I had
CORRECTNESS OF A GOSSIP BASED MEMBERSHIP PROTOCOL BY (ANDRÉ ALLAVENA, ALAN DEMERS, JOHN E. HOPCROFT ) PRATIK TIMALSENA UNIVERSITY OF OSLO
CORRECTNESS OF A GOSSIP BASED MEMBERSHIP PROTOCOL BY (ANDRÉ ALLAVENA, ALAN DEMERS, JOHN E. HOPCROFT ) PRATIK TIMALSENA UNIVERSITY OF OSLO OUTLINE q Contribution of the paper q Gossip algorithm q The corrected
Optimizing Joins in a Map-Reduce Environment
Optimizing Joins in a Map-Reduce Environment Foto Afrati National Technical University of Athens, Greece afrati@softlab.ece.ntua.gr Jeffrey D. Ullman Stanford University, USA ullman@infolab.stanford.edu
Lecture 2: Divide and conquer and Dynamic programming
Chapter 2 Lecture 2: Divide and conquer and Dynamic programming 2.1 Divide and Conquer Idea: - divide the problem into subproblems in linear time - solve subproblems recursively - combine the results in
A Tight Lower Bound for Dynamic Membership in the External Memory Model
A Tight Lower Bound for Dynamic Membership in the External Memory Model Elad Verbin ITCS, Tsinghua University Qin Zhang Hong Kong University of Science & Technology April 2010 1-1 The computational model
Lecture 16 Oct 21, 2014
CS 395T: Sublinear Algorithms Fall 24 Prof. Eric Price Lecture 6 Oct 2, 24 Scribe: Chi-Kit Lam Overview In this lecture we will talk about information and compression, which the Huffman coding can achieve
Information Systems for Engineers. Exercise 8. ETH Zurich, Fall Semester Hand-out Due
Information Systems for Engineers Exercise 8 ETH Zurich, Fall Semester 2017 Hand-out 24.11.2017 Due 01.12.2017 1. (Exercise 3.3.1 in [1]) For each of the following relation schemas and sets of FD s, i)
Lecture 10: Data Flow Analysis II
CS 515 Programming Language and Compilers I Lecture 10: Data Flow Analysis II (The lectures are based on the slides copyrighted by Keith Cooper and Linda Torczon from Rice University.) Zheng (Eddy) Zhang
Lecture 11: Non-Interactive Zero-Knowledge II. 1 Non-Interactive Zero-Knowledge in the Hidden-Bits Model for the Graph Hamiltonian problem
CS 276 Cryptography Oct 8, 2014 Lecture 11: Non-Interactive Zero-Knowledge II Instructor: Sanjam Garg Scribe: Rafael Dutra 1 Non-Interactive Zero-Knowledge in the Hidden-Bits Model for the Graph Hamiltonian
Relational Algebra 2. Week 5
Relational Algebra 2 Week 5 Relational Algebra (So far) Basic operations: Selection ( σ ) Selects a subset of rows from relation. Projection ( π ) Deletes unwanted columns from relation. Cross-product
Database Design and Normalization
Database Design and Normalization Chapter 11 (Week 12) EE562 Slides and Modified Slides from Database Management Systems, R. Ramakrishnan 1 1NF FIRST S# Status City P# Qty S1 20 London P1 300 S1 20 London
On Two Class-Constrained Versions of the Multiple Knapsack Problem
On Two Class-Constrained Versions of the Multiple Knapsack Problem Hadas Shachnai Tami Tamir Department of Computer Science The Technion, Haifa 32000, Israel Abstract We study two variants of the classic
Differential Privacy
CS 380S Differential Privacy Vitaly Shmatikov most slides from Adam Smith (Penn State) slide 1 Reading Assignment Dwork. Differential Privacy (invited talk at ICALP 2006). slide 2 Basic Setting DB= x 1
Part 1: You are given the following system of two equations: x + 2y = 16 3x 4y = 2
Solving Systems of Equations Algebraically Teacher Notes Comment: As students solve equations throughout this task, have them continue to explain each step using properties of operations or properties
Verifying Computations in the Cloud (and Elsewhere) Michael Mitzenmacher, Harvard University Work offloaded to Justin Thaler, Harvard University
Verifying Computations in the Cloud (and Elsewhere) Michael Mitzenmacher, Harvard University Work offloaded to Justin Thaler, Harvard University Goals of Verifiable Computation Provide user with correctness
Shared on QualifyGate.com
CS-GATE-05 GATE 05 A Brief Analysis (Based on student test experiences in the stream of CS on 8th February, 05 (Morning Session) Section wise analysis of the paper Section Classification Mark Marks Total
CS475: Linear Equations Gaussian Elimination LU Decomposition Wim Bohm Colorado State University
CS475: Linear Equations Gaussian Elimination LU Decomposition Wim Bohm Colorado State University Except as otherwise noted, the content of this presentation is licensed under the Creative Commons Attribution
Benny Pinkas Bar Ilan University
Winter School on Bar-Ilan University, Israel 30/1/2011-1/2/2011 Bar-Ilan University Benny Pinkas Bar Ilan University 1 Extending OT [IKNP] Is fully simulatable Depends on a non-standard security assumption
Mathematical Logic Part Three
Mathematical Logic Part hree riday our Square! oday at 4:15PM, Outside Gates Announcements Problem Set 3 due right now. Problem Set 4 goes out today. Checkpoint due Monday, October 22. Remainder due riday,
Cryptanalysis of Achterbahn
Cryptanalysis of Achterbahn Thomas Johansson 1, Willi Meier 2, and Frédéric Muller 3 1 Department of Information Technology, Lund University P.O. Box 118, 221 00 Lund, Sweden thomas@it.lth.se 2 FH Aargau,
Clique trees & Belief Propagation. Siamak Ravanbakhsh Winter 2018
Graphical Models Clique trees & Belief Propagation Siamak Ravanbakhsh Winter 2018 Learning objectives message passing on clique trees its relation to variable elimination two different forms of belief
Flow Algorithms for Two Pipelined Filtering Problems
Flow Algorithms for Two Pipelined Filtering Problems Anne Condon, University of British Columbia Amol Deshpande, University of Maryland Lisa Hellerstein, Polytechnic University, Brooklyn Ning Wu, Polytechnic
Communication Complexity
Communication Complexity Jaikumar Radhakrishnan School of Technology and Computer Science Tata Institute of Fundamental Research Mumbai 31 May 2012 J 31 May 2012 1 / 13 Plan 1 Examples, the model, the
CS54100: Database Systems
CS54100: Database Systems Relational Algebra 3 February 2012 Prof. Walid Aref Core Relational Algebra A small set of operators that allow us to manipulate relations in limited but useful ways. The operators
Distributed Mining of Frequent Closed Itemsets: Some Preliminary Results
Distributed Mining of Frequent Closed Itemsets: Some Preliminary Results Claudio Lucchese Ca Foscari University of Venice clucches@dsi.unive.it Raffaele Perego ISTI-CNR of Pisa perego@isti.cnr.it Salvatore
5 Systems of Equations
Systems of Equations Concepts: Solutions to Systems of Equations-Graphically and Algebraically Solving Systems - Substitution Method Solving Systems - Elimination Method Using -Dimensional Graphs to Approximate
Linear Algebra March 16, 2019
Linear Algebra March 16, 2019 2 Contents 0.1 Notation................................ 4 1 Systems of linear equations, and matrices 5 1.1 Systems of linear equations..................... 5 1.2 Augmented
Lecture 3,4: Multiparty Computation
CS 276 Cryptography January 26/28, 2016 Lecture 3,4: Multiparty Computation Instructor: Sanjam Garg Scribe: Joseph Hui 1 Constant-Round Multiparty Computation Last time we considered the GMW protocol,
EE5585 Data Compression April 18, Lecture 23
EE5585 Data Compression April 18, 013 Lecture 3 Instructor: Arya Mazumdar Scribe: Trevor Webster Differential Encoding Suppose we have a signal that is slowly varying For instance, if we were looking at
Dot-Product Join: Scalable In-Database Linear Algebra for Big Model Analytics
Dot-Product Join: Scalable In-Database Linear Algebra for Big Model Analytics Chengjie Qin 1 and Florin Rusu 2 1 GraphSQL, Inc. 2 University of California Merced June 29, 2017 Machine Learning (ML) Is
Interfaces for Stream Processing Systems
Interfaces for Stream Processing Systems Rajeev Alur, Konstantinos Mamouras, Caleb Stanford, and Val Tannen University of Pennsylvania, Philadelphia, PA, USA {alur,mamouras,castan,val}@cis.upenn.edu Abstract.
1 Terminology and setup
15-451/651: Design & Analysis of Algorithms August 31, 2017 Lecture #2 last changed: August 29, 2017 In this lecture, we will examine some simple, concrete models of computation, each with a precise definition
The Complexity of a Reliable Distributed System
The Complexity of a Reliable Distributed System Rachid Guerraoui EPFL Alexandre Maurer EPFL Abstract Studying the complexity of distributed algorithms typically boils down to evaluating how the number
Discrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Discussion 6A Solution
CS 70 Discrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Discussion 6A Solution 1. Polynomial intersections Find (and prove) an upper-bound on the number of times two distinct degree
Environment (Parallelizing Query Optimization)
Advanced d Query Optimization i i Techniques in a Parallel Computing Environment (Parallelizing Query Optimization) Wook-Shin Han*, Wooseong Kwak, Jinsoo Lee Guy M. Lohman, Volker Markl Kyungpook National
x y = 2 x + 2y = 14 x = 2, y = 0 x = 3, y = 1 x = 4, y = 2 x = 5, y = 3 x = 6, y = 4 x = 7, y = 5 x = 0, y = 7 x = 2, y = 6 x = 4, y = 5
List six positive integer solutions for each of these equations and comment on your results. Two have been done for you. x y = x + y = 4 x =, y = 0 x = 3, y = x = 4, y = x = 5, y = 3 x = 6, y = 4 x = 7,
arxiv: v1 [cs.cr] 16 Dec 2015
![arxiv: v1 [cs.cr] 16 Dec 2015](/thumbs/72/67087738.jpg "arxiv: v1 [cs.cr] 16 Dec 2015")
A Note on Efficient Algorithms for Secure Outsourcing of Bilinear Pairings arxiv:1512.05413v1 [cs.cr] 16 Dec 2015 Lihua Liu 1 Zhengjun Cao 2 Abstract. We show that the verifying equations in the scheme