Priority Queues. Sanders: Algorithm Engineering 12. Juni Maintain set M of Elements with Keys. Insert: DeleteMin:

Transcription

1 Sanders: Algorithm Engineering 12. Juni Priority Queues Maintain set M of Elements with Keys Insert: DeleteMin: Applications (no additional operations): Multi-way Merging Greedy Algorithms (e.g., machine scheduling) Discrete event simulation Branch-and-Bound Building long runs Time forward processing (small) (small medium) (medium large) (large) (large) (huge)

2 Sanders: Algorithm Engineering 12. Juni Binary Heaps best comparison based flat memory algorithm + On average constant time for insertion + On average logn+o (1) key comparisons per delete-min using the bottom-up heuristics [Wegener 93]. 2log(n/M) block accesses per delete-min M Cache

3 Sanders: Algorithm Engineering 12. Juni Bottom Up Heuristics delete Min O(1) compare swap 7 9 move sift down hole log(n) Factor two faster than naive implementation sift up O(1) average

4 Sanders: Algorithm Engineering 12. Juni Der Wettbewerber fit gemacht: int i=1, m=2, t = a[1]; m += (m!= n && a[m] > a[m + 1]); if (t > a[m]) { do { a[i] = a[m]; i = m; m = 2*i; if (m > n) break; m += (m!= n && a[m] > a[m + 1]); } while (t > a[m]); a[i] = t;} Keine signifikanten Leistungsunterschiede auf meiner Maschine (heapsort von random integers)

5 Sanders: Algorithm Engineering 12. Juni Vergleich Speicherzugriffe: O (1) weniger als top down O (log n) worst case. bei effizienter Implementierung Elementvergleiche: log n weniger für bottom up (average case) aber die sind leicht vorhersagbar Aufgabe: siftdown mit worst case logn+o (loglogn) Elementvergleichen

6 Sanders: Algorithm Engineering 12. Juni Heapkonstruktion Procedure buildheapbackwards for i := n/2 downto 1 do siftdown(i) Procedure buildheaprecursive(i : N) if 4i n then buildheaprecursive(2i) buildheaprecursive(2i + 1) siftdown(i) Rekursive Funktion für große Eingaben 2 schneller! (Rekursion abrollen für 2 unterste Ebenen) Aufgabe: Erklärung

7 Sanders: Algorithm Engineering 12. Juni Medium Size Queues (km M 2 /B Insertions) k m sorted sequences k merge tournament tree data structure buffer m m insertion buffer Insert: Initially into insertion buffer. Overflow sort; merge with deletion buffer; write out largest elements. Delete-Min: Take minimum of insertion and deletion buffer. Refill deletion buffer if necessary.

8 Sanders: Algorithm Engineering 12. Juni Large Queues [Sanders 00] external swapped cached group 2 group 3 group 1 k k m k m k k k m k merge T 1 k merge T2 k merge T3 group group group buffer 1 m buffer 2 m buffer 3 m R merge deletion buffer m m insertion buffer insert: group full merge group; shift into next group. merge invalid group buffers and move them into group 1. Delete-Min: Refill. m m. nothing else

9 Sanders: Algorithm Engineering 12. Juni Example Merge insertion buffer, deletion buffer, and leftmost group buffer insert( 3) k j f v s r q l i u n m g c w tp o h e x k j g f v s r q l i u n m c w tp o h e d b d b b b a x a 3

10 Sanders: Algorithm Engineering 12. Juni Example Merge group 1 e x k j g f v s r q l i u n m c w tp o h e x k j g f v s r q l i u n m c w tp o h d b b d b b a 3 a 3

11 Sanders: Algorithm Engineering 12. Juni Example Merge group 2 e x k j g f v s r q l i u n m c w tp o h e x k j g f m n o p q l r i s w c h t u v d b b d b b a 3 a 3

12 Sanders: Algorithm Engineering 12. Juni Example Merge group buffers e x k j g f m n o p q l r i s w c h t u v e x d b b k j g f m n o p q l r i s w c h t u v d b b a 3 a 3

13 Sanders: Algorithm Engineering 12. Juni Example DeleteMin 3; DeleteMin a; e x d b b k j g f m n o p q l r i s w c h t u v e x d b b k j g f m n o p q l r i s w c h t u v a 3

14 Sanders: Algorithm Engineering 12. Juni Example DeleteMin b e x d b b k j g f m n o p q l r i s w c h t u v e x d k j m n o p q l r i s w t u v b g f h c

15 Sanders: Algorithm Engineering 12. Juni Analysis I insertions, buffer sizes m = Θ(M) merging degree k = Θ(M/B) block accesses: sort(i)+ small terms key comparisons: I logi + small terms (on average) W W R W R W R R R R W R Other (similar, earlier) [Arge 95, Brodal-Katajainen 98, Brengel et al. 99, Fadel et al. 97] data structures spend a factor 3 more I/Os to replace I by queue size.

16 Sanders: Algorithm Engineering 12. Juni Implementation Details Fast routines for 2 4 way merging keeping smallest elements in registers Use sentinels to avoid special case treatments (empty sequences,... ) Currently heap sort for sorting the insertion buffer k M/B: multiple levels, limited associativity, TLB

17 Sanders: Algorithm Engineering 12. Juni Experiments Keys: random 32 bit integers Associated information: 32 dummy bits Deletion buffer size: 32 Group buffer size: 256 Merging degree k: 128 Near optimal : performance on all machines tried! Compiler flags: Highly optimizing, nothing advanced Operation Sequence: (Insert-DeleteMin-Insert) N (DeleteMin-Insert-DeleteMin) N Near optimal performance on all machines tried!

18 Sanders: Algorithm Engineering 12. Juni MIPS R10000, 180 MHz (T(deleteMin) + T(insert))/log N [ns] bottom up binary heap bottom up aligned 4-ary heap sequence heap N

19 Sanders: Algorithm Engineering 12. Juni Ultra-SparcIIi, 300 MHz (T(deleteMin) + T(insert))/log N [ns] bottom up binary heap bottom up aligned 4-ary heap sequence heap N

20 Sanders: Algorithm Engineering 12. Juni Alpha-21164, 533 MHz (T(deleteMin) + T(insert))/log N [ns] bottom up binary heap bottom up aligned 4-ary heap sequence heap N

21 Sanders: Algorithm Engineering 12. Juni Pentium II, 300 MHz (T(deleteMin) + T(insert))/log N [ns] bottom up binary heap bottom up aligned 4-ary heap sequence heap N

22 Sanders: Algorithm Engineering 12. Juni (insert (deletemin insert) s ) N (deletemin (insert deletemin) s ) N (T(deleteMin) + T(insert))/log N [ns] s=0, binary heap s=0, 4-ary heap s=0 s=1 s=4 s= N

23 Sanders: Algorithm Engineering 12. Juni Methodological Lessons If you want to compare small constant factors in execution time: Reproducability demands publication of source codes (4-ary heaps, old study in Pascal) Highly tuned codes in particular for the competitors (binary heaps have factor 2 between good and naive implementation). How do you compare two mediocre implementations? Careful choice/description of inputs Use multiple different hardware platforms Augment with theory (e.g., comparisons, data dependencies, cache faults, locality effects... )

24 Sanders: Algorithm Engineering 12. Juni Open Problems Integrate into STL Dependence on size rather than number of insertions Parallel disks Space efficient implementation Multi-level cache aware or cache-oblivious variants

25 Sanders: Algorithm Engineering 12. Juni Adressable Priority Queues Procedure build({e 1,...,e n }) M:= {e 1,...,e n } Function size return M Procedure insert(e) M:= M {e} Function min return minm Function deletemin e:= min M; M:= M \ {e}; return e Function remove(h : Handle) e:= h; M:= M \ {e}; return e Procedure decreasekey(h : Handle,k : Key) assert key(h) k; Procedure merge(m ) M:= M M key(h):= k

26 Sanders: Algorithm Engineering 12. Juni Quick-Heaps Idea: Partition M into subsets M 1,...,M k Invariant: 1 i < j k,e i M i,e j M j : e i e j M i is approximately growing geometrically with i

27 Sanders: Algorithm Engineering 12. Juni Quick-Heaps Insert Procedure insert(e) for i := k downto 1 while e < minm i do ;// define minm 0 = if i = 0 then i:= 1; minm 1 := e M i := M i {e} decreasekey: similar

28 Sanders: Algorithm Engineering 12. Juni Quick-Heaps deletemin Procedure deletemin e:= minm 1 M 1 := M 1 \ {e} if M 1 = /0 then (recursively) partition M 2 and renumber to reestablish invariant return e

29 Sanders: Algorithm Engineering 12. Juni Quick-Heaps Analysis Paredes: average case complexity O (log n) Experiments: with very favorable constant factors. Open Problem: Make fit for worst case inputs. O (1) expected cost for decreasekey? Idea: active rebalancing

30 Sanders: Algorithm Engineering 12. Juni Heap Ordered Forest Basic Data Structure A forest of heap-ordered trees minptr d a e g f c b h i

31 Sanders: Algorithm Engineering 12. Juni Manipulating Forests Cut: a b a<b a Link: b

32 Sanders: Algorithm Engineering 12. Juni Pairing Heaps Procedure insertitem(h : Handle) newtree(h) Procedure newtree(h : Handle) forest:= forest {h} if e < min then minptr:= h

33 Sanders: Algorithm Engineering 12. Juni Pairing Heaps Procedure decreasekey(h : Handle,k : Key) key(h):= k if h is not a root then cut(h)

34 Sanders: Algorithm Engineering 12. Juni Pairing Heaps Function deletemin : Handle m:= minptr forest:= forest \ {m} foreach child h of m do newtree(h) Perform a pairwise link of the tree roots in forest return m

35 Sanders: Algorithm Engineering 12. Juni Pairing Heaps Procedure merge(o : AdressablePQ) if minptr > o.minptr then minptr:= o.minptr forest:= forest o.forest o.forest:= /0

36 Sanders: Algorithm Engineering 12. Juni Fibonacci Heaps (A sample from the Zoo) Ranks: initially zero, increases for root of a link Union by rank: Only link roots of equal rank Mark nodes that lost a child Cascading cuts: cut marked nodes (i.e., lost two childs) Amortized complexity: O (log n) for deletemin O (1) for all other operations

37 Sanders: Algorithm Engineering 12. Juni Representing the Tree Items parent Fibonacci Heap left sibling data rank mark right sibling one child Pairing Heap left sibling or parent data right sibling one child

38 Sanders: Algorithm Engineering 12. Juni Addressable Priority Queues: Todo No recent comparison of efficient implementations No tight analysis of pairing heaps No implementation of compromises, e.g., thin heaps [Kaplan Tarjan 99] (three pointers, no mark, slightly more complicated balancing) No implementation of worst case efficient variants Study implementation tricks: two pointers per item? sentinels,... (Almost) nothing known for memory hierarchies

39 Sanders: Algorithm Engineering 12. Juni van Emde-Boas Search Trees Store set M of K = 2 k -bit integers. later: associated information K = 1 or M = 1: store directly K := K/2 { } M i := x mod 2 K : x div 2 K = i root points to nonempty M i -s top t = {i : M i /0} insert, delete, search ino (logk) time K bit veb top hash table K bit veb bot i

40 Sanders: Algorithm Engineering 12. Juni Comparison with Comparison Based Search Trees Ideally: logn loglogn Problems: Many special case tests High space consumption Time for random locate [ns] 1000 orig-stree LEDA-STree STL map (2,16)-tree n

41 Sanders: Algorithm Engineering 12. Juni Efficient 32 bit Implementation top 3 layers of bit arrays 0 63 K bit veb top hash table hash table K bit veb bot i K bit veb bot i

42 Sanders: Algorithm Engineering 12. Juni Layers of Bit Arrays t 1 [i] = 1 iff M i /0 t 2 [i] = t 1 [32i] t 1 [32i+1] t 1 [32i+31] t 3 [i] = t 2 [32i] t 2 [32i+1] t 2 [32i+31]

43 Sanders: Algorithm Engineering 12. Juni Efficient 32 bit Implementation Break recursion after 3 layers Level 1 root hash table hash table Bits Level 2 Bits 15-8 K bit veb bot i Level 2 Bits 15-8 hash table hash table Level 3 Bits 7-0 hash table

44 Sanders: Algorithm Engineering 12. Juni Efficient 32 bit Implementation root hash table root array hash table Level 1 root Bits array Level 1 root Bits Level 2 Bits 15-8 Level 2 Bits 15-8 Level 2 Bits 15-8 hash table Level 2 Bits 15-8 hash table hash table Level 3 Bits 7-0 hash table Level 3 Bits 7-0 hash table hash table

45 Sanders: Algorithm Engineering 12. Juni Efficient 32 bit Implementation Sorted doubly linked lists for associated information and range queries array Level 1 root Bits array Level 1 root Bits Level 2 Bits 15-8 Level 2 Bits 15-8 Level 2 Bits 15-8 hash table Level 2 Bits 15-8 hash table hash table Level 3 Bits 7-0 hash table hash table Level 3 Bits 7-0 hash table

46 Sanders: Algorithm Engineering 12. Juni Example v v t 3 t 2 M ={1,11,111,1111,111111} 4095 root top t root t 2 0 t v hash M ={1,11,111,1111} L2 t 2 00 t v hash M 04 ={1111} M 00 ={1,11,111} M 1 ={111111} L Element List

47 Sanders: Algorithm Engineering 12. Juni Locate High Level //return handle of minx M : y x Function locate(y : N) : ElementHandle if y > maxm then return i := y[16..31] // Level 1 if r[i] = nil y > maxm i then return minm t 1.locate(i) if M i = {x} then return x j := y[8..15] // Level 2 if r i [ j] = nil y > maxm i j then return minm i,t 1 i.locate( j) if M i j = {x} then return x return r i j [ti 1 j.locate(y[0..7])] // Level 3

48 Sanders: Algorithm Engineering 12. Juni Locate in Bit Arrays //find the smallest j i such that t k [ j] = 1 Method locate(i) for a bit array t k consisting of n bit words //n = 32 for t 1, t 2, ti 1, t1 i j ; n = 64 for t3 ; n = 8 for ti 2, t2 i j assert some bit in t k to the right of i is nonzero j := i div n // which word? a := t k [n j..n j+ n 1] set a[(i mod n)+1..n 1] to zero // n 1 i mod n 0 if a = 0 then j := t k+1.locate( j) a := t k [n j..n j+ n 1] return n j+ msbpos(a) // e.g. floating point conversion

49 Sanders: Algorithm Engineering 12. Juni Random Locate Time for locate [ns] orig-stree LEDA-STree STL map (2,16)-tree STree n

50 Sanders: Algorithm Engineering 12. Juni Random Insert Time for insert [ns] orig-stree 500 LEDA-STree STL map (2,16)-tree STree n

51 Sanders: Algorithm Engineering 12. Juni Delete Random Elements Time for delete [ns] orig-stree LEDA-STree 500 STL map (2,16)-tree STree n

52 Sanders: Algorithm Engineering 12. Juni Open Problems Measurement for worst case inputs Measure Performance for realistic inputs IP lookup etc. Best first heuristics like, e.g., bin packing More space efficient implementation (A few) more bits

53 Sanders: Algorithm Engineering 12. Juni Hashing to hash to bring into complete disorder paradoxically, this helps us to find things more easily! store set M Element. key(e) is unique for e M. support dictionary operations in O (1) time: M.insert(e : Element): M := M {e} M.remove(k : Key): M := M \ {e}, e = k M.find(k : Key): return e M with e = k; if none present (Convention: key is implicit), e.g. e = k iff key(e) = k)

54 Sanders: Algorithm Engineering 12. Juni An (Over)optimistic approach A (perfect) hash function h h maps elements of M to unique entries of table t[0..m 1], i.e., t[h(key(e))] = e M t

55 Sanders: Algorithm Engineering 12. Juni Collisions perfect hash functions are difficult to obtain h M t Example: Birthday Paradoxon

56 Sanders: Algorithm Engineering 12. Juni Collision Resolution for example by closed hashing entries: elements sequences of elements k M h t[h(k)] t < > < > <> <> <> < > <> <> < > <> <> <>

57 Sanders: Algorithm Engineering 12. Juni Hashing with Chaining Implement sequences in closed hashing by singly linked lists insert(e): Insert e at the beginning of t[h(e)]. constant time remove(k): Scan through t[h(k)]. If an element e with h(e) = k is encountered, remove it and return. find(k) : Scan through t[h(k)]. If an element e with h(e) = k is encountered, return it. Otherwise, return. O ( M ) worst case time for remove and find k M h t[h(k)] t < > < > <> <> <> < > <> <> < > <> <> <>

58 Sanders: Algorithm Engineering 12. Juni A Review of Probability s 1 s 2 s 3 binary search? s s 4 5 s 6 s 7 buckets sample space Ω events: subsets of Ω p x =probability of x Ω Example from hashing random hash functions{0..m 1} Key E 42 = {h Ω : h(4) = h(2)} uniform distr. p h = m Key P[E ] = x E p x P[E 42 ] = m 1 random variable X 0 : Ω R X = {e M : h(e) = 0}.

59 Sanders: Algorithm Engineering 12. Juni expectation E[X 0 ] = y Ω p y X(y) Linearity of Expectation: E[X +Y] = E[X]+E[Y] E[X] = M m (*) Proof of (*): Consider the 0-1 RV X e = 1 if h(e) = 0 for e M and X e = 0 else. E[X 0 ] = E[ X e ] = E[X e ] = P[X e = 1] = M 1 e M e M e M m

60 Sanders: Algorithm Engineering 12. Juni Analysis for Random Hash Functions h < > Satz 1. The expected execution time of remove(k) and find(k) iso (1) if M =O (m). M t < > <> <> <> < > <> <> < > <> <> <> t[h(k)] Proof. Constant time plus the time for scanning t[h(k)]. X := t[h(k)] = {e M : h(e) = h(k)}. Consider the 0-1 RV X e = 1 if h(e) = h(k) for e M and X e = 0 else. E[X] = E[ X e ] = E[X e ] = P[X e = 1] = M e M e M e M m =O (1) This is independent of the input set M.

61 Sanders: Algorithm Engineering 12. Juni Universal Hashing Idea: use only certain easy hash functions Definition: U {0..m 1} Key is universal if for all x, y in Key with x y and random h U, P[h(x) = h(y)] = 1 m. Satz 2. Theorem 1 also applies to universal families of hash functions. Proof. For Ω =U we still have P[X e = 1] = 1 m. The rest is as before.

62 Sanders: Algorithm Engineering 12. Juni H Ω

63 Sanders: Algorithm Engineering 12. Juni A Simple Universal Family Assume m is prime, Key {0,...,m 1} k Satz 3. For a = (a 1,...,a k { ) {0,...,m 1} k define h a (x) = a x mod m, H = h a : a {0..m 1} k}. H is a universal family of hash functions x 1 x 2 x 3 * + * + * mod m = h a (x) a 1 a 2 a 3

64 Sanders: Algorithm Engineering 12. Juni Proof. Consider x = (x 1,...,x k ), y = (y 1,...,y k ) with x j y j count a-s with h a (x) = h a (y). For each choice of a i s, i j, exactly one a j with h a (x) = h a (y): a i x i 1 i k a j (x j y j ) a i y i ( mod m) 1 i k a i (y i x i )( mod m) i j,1 i k a j (x j y j ) 1 m k 1 ways to choose the a i with i j. m k is total number of as, i.e., a i (y i x i )( mod m) i j,1 i k

65 Sanders: Algorithm Engineering 12. Juni P[h a (x) = h a (y)] = mk 1 m k = 1 m.

66 Sanders: Algorithm Engineering 12. Juni Bit Based Universal Families Let m = 2 w, Key = {0,1} k Bit-Matrix Multiplication: H = where h M (x) = Mx (arithmetics mod 2, i.e., xor, and) Table Lookup:H [] = {h M : M {0,1} w k} { h [] (t 1,...,t b ) : t i {0..m 1} {0..w 1}} where h [] (t 1,...,t b ) ((x 0,x 1,...,x b )) = x 0 L b i=1 t i [x i ] k x x 2 x 1 a a x 0 w

67 Sanders: Algorithm Engineering 12. Juni Hashing with Linear Probing Open hashing: go back to original idea. Elements are directly stored in the table. Collisions are resolved by finding other entries. linear probing: search for next free place by scanning the table. Wrap around at the end. simple space efficient h cache efficient M t

68 Sanders: Algorithm Engineering 12. Juni The Easy Part Class BoundedLinearProbing(m,m : N; h : Key 0..m 1) t=[,..., ] : Array [0..m+m 1] of Element invariant i : t[i] : j {h(t[i])..i 1} : t[i] Procedure insert(e : Element) h for i := h(e) to while t[i] do ; assert i < m+m 1 t[i] := e Function find(k : Key) : Element for i := h(e) to while t[i] do if t[i] = k then return t[i] return M m t m

69 Sanders: Algorithm Engineering 12. Juni Remove example: t = [..., x h(z),y,z,...], remove(x) invariant i : t[i] : j {h(t[i])..i 1} : t[i] Procedure remove(k : Key) for i := h(k) to while k t[i] do if t[i] = then return //we plan for a hole at i. for j := i+1 to while t[ j] do t[i] := // Establish invariant for t[ j]. if h(t[ j]) i then t[i] := t[ j] i := j // search k // nothing to do // Overwrite removed element // move planned hole // erase freed entry

70 Sanders: Algorithm Engineering 12. Juni More Hashing Issues High probability and worst case guarantees more requirements on the hash functions Hashing as a means of load balancing in parallel systems, e.g., storage servers Different disk sizes and speeds Adding disks / replacing failed disks without much copying

71 Sanders: Algorithm Engineering 12. Juni Space Efficient Hashing with Worst Case Constant Access Time Represent a set of n elements (with associated information) using space (1+ε)n. Support operations insert, delete, lookup, (doall) efficiently. Assume a truly random hash function h

72 Sanders: Algorithm Engineering 12. Juni Related Work Uniform hashing: Expected time 1 ε h 1 h 2 h 3 Dynamic Perfect Hashing, [Dietzfelbinger et al. 94] Worst case constant time for lookup but ε is not small. Approaching the Information Theoretic Lower Bound: [Brodnik Munro 99,Raman Rao 02] Space (1 + o(1)) lower bound without associated information [Pagh 01] static case.

73 Sanders: Algorithm Engineering 12. Juni Cuckoo Hashing [Pagh Rodler 01] Table of size 2+ε. Two choices for each element. Insert moves elements; rebuild if necessary. Very fast lookup and insert. Expected constant insertion time. h 1 h 2

74 Sanders: Algorithm Engineering 12. Juni d-ary Cuckoo Hashing d choices for each element. Worst case d probes for delete and lookup. Task: maintain perfect matching h 1 in the bipartite graph (L = Elements,R = Cells,E = Choices), e.g., insert by BFS of random walk. h 2 h 3

75 Sanders: Algorithm Engineering 12. Juni Tradeoff: Space Lookup/Deletion Time Lookup and Delete: d =O ( log 1 ε) probes Insert: ( ) 1 O(log(1/ε)), (experiments) O (1/ε)? ε

76 Sanders: Algorithm Engineering 12. Juni Experiments ε * #probes for insert d=2 d=3 d=4 d= space utilization

77 Sanders: Algorithm Engineering 12. Juni Open Questions and Further Results Tight analysis of insertion Two choices with d slots each [Dietzfelbinger et al.] cache efficiency Good Implementation? Automatic rehash Always correct