CS6000: Foundations of Algorithm Design and Machine Learning Sourangshu Bhattacharya
Paths in graphs Consider a digraph G = (V, E) with edge-weight function w : E R. The weight of path p = v 1 v L v k is defined to be k-1 w( p) =åw(v i,v i+1 ). i=1 November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.
Paths in graphs Consider a digraph G = (V, E) with edge-weight function w : E R. The weight of path p = v 1 v L v k is defined to be Example: k-1 w( p) =åw(v i,v i+1 ). i=1 vv11 4 vv 5 1 vv55 vv vv4 4 w(p) = November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.
Shortest paths A shortest path from u to v is a path of minimum weight from u to v. The shortestpath weight from u to v is defined as d(u, v) = min{w(p) : p is a path from u to v}. Note: d(u, v) = if no path from u to v exists. November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.4
Optimal substructure Theorem. A subpath of a shortest path is a shortest path. November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.5
Optimal substructure Theorem. A subpath of a shortest path is a shortest path. Proof. Cut and paste: November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.6
Optimal substructure Theorem. A subpath of a shortest path is a shortest path. Proof. Cut and paste: November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.7
Triangle inequality Theorem. For all u, v, x Î V, we have d(u, v) d(u, x) + d(x, v). November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.8
Triangle inequality Theorem. For all u, v, x Î V, we have d(u, v) d(u, x) + d(x, v). Proof. uu d(u, v) vv d(u, x) d(x, v) xx November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.9
Well-definedness of shortest paths If a graph G contains a negative-weight cycle, then some shortest paths may not exist. November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.10
Well-definedness of shortest paths If a graph G contains a negative-weight cycle, then some shortest paths may not exist. Example: < 0 uu vv November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.11
Single-source shortest paths Problem. From a given source vertex s Î V, find the shortest-path weights d(s, v) for all v Î V. If all edge weights w(u, v) are nonnegative, all shortest-path weights must exist. IDEA: Greedy. 1. Maintain a set S of vertices whose shortestpath distances from s are known.. At each step add to S the vertex v Î V S whose distance estimate from s is minimal.. Update the distance estimates of vertices adjacent to v. November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.1
Dijkstra s algorithm d[s] 0 for each v Î V {s} do d[v] S Æ Q V Q is a priority queue maintaining V S November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.1
Dijkstra s algorithm d[s] 0 for each v Î V {s} do d[v] S Æ Q V Q is a priority queue maintaining V S while Q ¹ Æ do u EXTRACT-MIN(Q) S S È {u} for each v Î Adj[u] do if d[v] > d[u] + w(u, v) then d[v] d[u] + w(u, v) November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.14
Dijkstra s algorithm d[s] 0 for each v Î V {s} do d[v] S Æ Q V while Q ¹ Æ do u EXTRACT-MIN(Q) S S È {u} for each v Î Adj[u] do if d[v] > d[u] + w(u, v) then d[v] d[u] + w(u, v) Q is a priority queue maintaining V S Implicit DECREASE-KEY relaxation step November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.15
Example of Dijkstra s algorithm Graph with nonnegative edge weights: A 10 B D 8 1 4 7 9 C E November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.16
Initialize: Example of Dijkstra s algorithm 0 A 10 B D 8 1 4 7 9 Q: A B C D E 0 C E S: {} November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.17
Example of Dijkstra s algorithm A EXTRACT-MIN(Q): 0 A 10 B D 8 1 4 7 9 Q: A B C D E 0 C E S: { A } November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.18
Example of Dijkstra s algorithm Relax all edges leaving A: 0 A 10 10 B D 8 1 4 7 9 Q: A B C D E 0 10 C E S: { A } November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.19
Example of Dijkstra s algorithm C EXTRACT-MIN(Q): 0 A 10 10 B D 8 1 4 7 9 Q: A B C D E 0 10 C E S: { A, C } November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.0
Example of Dijkstra s algorithm Relax all edges leaving C: 0 A 10 7 B 11 D 8 1 4 7 9 Q: A B C D E 0 10 7 11 5 C S: { A, C } E 5 November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.1
Example of Dijkstra s algorithm E EXTRACT-MIN(Q): 0 A 10 7 B 11 D 8 1 4 7 9 Q: A B C D E 0 10 7 11 5 C S: { A, C, E } E 5 November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.
Example of Dijkstra s algorithm Relax all edges leaving E: 0 A 10 7 B 11 D 8 1 4 7 9 Q: A B C D E 0 10 7 11 5 7 11 C S: { A, C, E } E 5 November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.
Example of Dijkstra s algorithm B EXTRACT-MIN(Q): 0 A 10 7 B 11 D 8 1 4 7 9 Q: A B C D E 0 10 7 11 5 7 11 C S: { A, C, E, B } E 5 November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.4
Example of Dijkstra s algorithm Relax all edges leaving B: 0 A 10 7 B 9 D 8 1 4 7 9 Q: A B C D E 0 10 7 11 5 7 11 9 C S: { A, C, E, B } E 5 November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.5
Example of Dijkstra s algorithm D EXTRACT-MIN(Q): 0 A 10 7 B 9 D 8 1 4 7 9 Q: A B C D E 0 10 7 11 5 7 11 9 C E 5 S: { A, C, E, B, D } November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.6
Correctness Part I Lemma. Initializing d[s] 0 and d[v] for all v Î V {s} establishes d[v] ³ d(s, v) for all v Î V, and this invariant is maintained over any sequence of relaxation steps. November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.7
Correctness Part I Lemma. Initializing d[s] 0 and d[v] for all v Î V {s} establishes d[v] ³ d(s, v) for all v Î V, and this invariant is maintained over any sequence of relaxation steps. Proof. Suppose not. Let v be the first vertex for which d[v] < d(s, v), and let u be the vertex that caused d[v] to change: d[v] = d[u] + w(u, v). Then, d[v] < d(s,v) d(s, u) + d(u, v) d(s,u) + w(u, v) d[u] + w(u, v) Contradiction. supposition triangle inequality sh. path specific path v is first violation November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.8
Correctness Part II Lemma. Let u be v s predecessor on a shortest path from s to v. Then, if d[u] = d(s, u) and edge (u, v) is relaxed, we have d[v] = d(s, v) after the relaxation. November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.9
Correctness Part II Lemma. Let u be v s predecessor on a shortest path from s to v. Then, if d[u] = d(s, u) and edge (u, v) is relaxed, we have d[v] = d(s, v) after the relaxation. Proof. Observe that d(s, v) = d(s, u) + w(u, v). Suppose that d[v] > d(s, v) before the relaxation. (Otherwise, we re done.) Then, the test d[v] > d[u] + w(u, v) succeeds, because d[v] > d(s, v) = d(s, u) + w(u, v) = d[u] + w(u, v), and the algorithm sets d[v] = d[u] + w(u, v) = d(s, v). November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.0
Correctness Part III Theorem. Dijkstra s algorithm terminates with d[v] = d(s, v) for all v Î V. November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.1
Correctness Part III Theorem. Dijkstra s algorithm terminates with d[v] = d(s, v) for all v Î V. Proof. It suffices to show that d[v] = d(s, v) for every v Î V when v is added to S. Suppose u is the first vertex added to S for which d[u] > d(s, u). Let y be the first vertex in V S along a shortest path from s to u, and let x be its predecessor: uu S, just before adding u. ss xx yy November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.
Correctness Part III (continued) ss S xx yy uu Since u is the first vertex violating the claimed invariant, we have d[x] = d(s, x). When x was added to S, the edge (x, y) was relaxed, which implies that d[y] = d(s, y) d(s, u) < d[u]. But, d[u] d[y] by our choice of u. Contradiction. November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.
Analysis of Dijkstra while Q ¹ Æ do u EXTRACT-MIN(Q) S S È {u} for each v Î Adj[u] do if d[v] > d[u] + w(u, v) then d[v] d[u] + w(u, v) November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.4
Analysis of Dijkstra V times while Q ¹ Æ do u EXTRACT-MIN(Q) S S È {u} for each v Î Adj[u] do if d[v] > d[u] + w(u, v) then d[v] d[u] + w(u, v) November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.5
Analysis of Dijkstra while Q ¹ Æ do u EXTRACT-MIN(Q) S S È {u} for each v Î Adj[u] V times degree(u) times do if d[v] > d[u] + w(u, v) then d[v] d[u] + w(u, v) November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.6
Analysis of Dijkstra while Q ¹ Æ do u EXTRACT-MIN(Q) S S È {u} for each v Î Adj[u] V times degree(u) times do if d[v] > d[u] + w(u, v) then d[v] d[u] + w(u, v) Handshaking Lemma Þ Q(E) implicit DECREASE-KEY s. November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.7
Analysis of Dijkstra while Q ¹ Æ do u EXTRACT-MIN(Q) S S È {u} for each v Î Adj[u] V times degree(u) times do if d[v] > d[u] + w(u, v) then d[v] d[u] + w(u, v) Handshaking Lemma Þ Q(E) implicit DECREASE-KEY s. Time = Q(V T EXTRACT -MIN + E T DECREASE -KEY) Note: Same formula as in the analysis of Prim s minimum spanning tree algorithm. November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.8
Analysis of Dijkstra (continued) Time = Q(V) T EXTRACT -MIN + Q(E) T DECREASE -KEY Q T EXTRACT -MIN T DECREASE -KEY Total November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.9
Analysis of Dijkstra (continued) Time = Q(V) T EXTRACT -MIN + Q(E) T DECREASE -KEY Q T EXTRACT -MIN T DECREASE -KEY Total array O(V) O(1) O(V ) November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.40
Analysis of Dijkstra (continued) Time = Q(V) T EXTRACT -MIN + Q(E) T DECREASE -KEY Q T EXTRACT -MIN T DECREASE -KEY Total array binary heap O(V) O(1) O(V ) O(lg V) O(lg V) O(E lg V) November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.41
Analysis of Dijkstra (continued) Time = Q(V) T EXTRACT -MIN + Q(E) T DECREASE -KEY Q T EXTRACT -MIN T DECREASE -KEY Total array binary heap Fibonacci heap O(V) O(1) O(V ) O(lg V) O(lg V) amortized O(lg V) O(1) amortized O(E lgv) O(E + V lg V) worst case November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.4
Unweighted graphs Suppose that w(u, v) = 1 for all (u, v) Î E. Can Dijkstra s algorithm be improved? November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.4
Unweighted graphs Suppose that w(u, v) = 1 for all (u, v) Î E. Can Dijkstra s algorithm be improved? Use a simple FIFO queue instead of a priority queue. November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.44
Unweighted graphs Suppose that w(u, v) = 1 for all (u, v) Î E. Can Dijkstra s algorithm be improved? Use a simple FIFO queue instead of a priority queue. Breadth-first search while Q ¹ Æ do u DEQUEUE(Q) for each v Î Adj[u] do if d[v] = then d[v] d[u] + 1 ENQUEUE(Q, v) November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.45
Unweighted graphs Suppose that w(u, v) = 1 for all (u, v) Î E. Can Dijkstra s algorithm be improved? Use a simple FIFO queue instead of a priority queue. Breadth-first search while Q ¹ Æ do u DEQUEUE(Q) for each v Î Adj[u] do if d[v] = then d[v] d[u] + 1 ENQUEUE(Q, v) Analysis: Time = O(V + E). November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.46
Example of breadth-first search aa ff hh dd bb gg ee ii cc Q: November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.47
Example of breadth-first search 0 aa ff hh dd bb gg ee ii cc 0 Q: a November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.48
Example of breadth-first search 0 aa 1 ff hh dd 1 bb gg ee ii cc 1 1 Q: a b d November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.49
Example of breadth-first search 0 aa 1 ff hh dd 1 bb cc ee 1 Q: a b d c e gg ii November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.50
Example of breadth-first search 0 aa 1 ff hh dd 1 bb gg ee ii cc Q: a b d c e November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.51
Example of breadth-first search 0 aa 1 ff hh dd 1 bb gg ee ii cc Q: a b d c e November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.5
Example of breadth-first search aa ff hh 0 1 1 bb cc dd ee Q: a b d c e g i gg ii November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.5
Example of breadth-first search 0 1 aa 1 ff hh dd bb cc ee 4 gg 4 Q: a b d c e g i f ii November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.54
Example of breadth-first search 0 1 aa bb cc 1 dd ee 4 ff gg 4 4 Q: a b d c e g i f h 4 hh ii November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.55
Example of breadth-first search 0 1 aa bb cc 1 dd ee 4 ff gg 4 Q: a b d c e g i f h 4 hh ii November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.56
Example of breadth-first search 4 4 0 aa 1 ff hh 1 bb dd gg ee ii cc Q: a b d c e g i f h November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.57
Example of breadth-first search 4 4 0 aa 1 ff hh 1 bb dd gg ee ii cc Q: a b d c e g i f h November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.58
Correctness of BFS while Q ¹ Æ do u DEQUEUE(Q) for each v Î Adj[u] do if d[v] = then d[v] d[u] + 1 ENQUEUE(Q, v) Key idea: The FIFO Q in breadth-first search mimics the priority queue Q in Dijkstra. Invariant: v comes after u in Q implies that d[v] = d[u] or d[v] = d[u] + 1. November 14, 005 Copyright 001-5 by Erik D. Demaine and Charles E. Leiserson L17.59