Computer Algorithms CISC4080 CIS, Fordham Univ. Instructor: X. Zhang Lecture 2

Transcription

1 Computer Algorithms CISC4080 CIS, Fordham Univ. Instructor: X. Zhang Lecture 2

2 Outline Introduction to algorithm analysis: fibonacci seq calculation counting number of computer steps recursive formula for running time of recursive algorithm math help: math. induction Asymptotic notations Algorithm running time classes: P, NP 2

3 Last class Review some algorithms learned in previous classes idea => pseudocode => implementation Correctness of sorting algorithms: insertion sort: gradually expand the sorted sub array/list bubble sort: bubble largest number to the right, also expand sorted sub array/list Algorithm time efficiency: via measurement: implement & instrument the code, run it in a computer 3

4 Example (Fib1: recursive) n T(n)ofFib1 F(n) 10 3e e Time (in seconds) 12 4e e e e e e e n Running time seems to grows exponentially as n increases How long does it take to Calculate F(100)?

5 Example (Fib2: iterative) 10 1e e e e e e e e e-06 Time (in seconds) n Increase very slowly as n increases 44 5

6 Take-away It s possible to perform model-fitting to find out T(n): running time of the algorithms given input size Pros of measurement based studies? Cons: time consuming, maybe too late Does not explain why? 6

7 Analytic approach Is it possible to find out how running time grows when input size grows, analytically? Does running time stay constant, increase linearly, logarithmically, quadratically, exponentially? Yes: analyze pseudocode/code, calculate total number of steps in terms of input size, and study its order of growth results are general: not specific to language, run time system, caching effect, other processes sharing computer shed light on effects of larger problem size, faster CPU, 7

8 Running time analysis Given an algorithm in pseudocode or actual program When the input size is n, how many total number of computer steps are executed? Size of input: size of an array, polynomial degree, # of elements in a matrix, vertices and edges in a graph, or # of bits in the binary representation of input Computer steps: arithmetic operations, data movement, control, decision making (if, while), comparison, each step take a constant amount of time Ignore: overhead of function calls (call stack frame allocation, passing parameters, and return values) 8

9 Let T(n) be number of computer steps needed to compute fib1(n) T(0)=1: when n=0, first step is executed T(1)=2: when n=1, first two steps are executed For n >1, T(n)=T(n-1)+T(n-2)+3: first two steps are executed, fib1(n-1) is called (with T(n-1) steps), fib1(n-2) is called (T(n-2) steps), return values are added (1 step) Can you see that T(n) > F n? How big is T(n)? Case Studies: Fib1(n) 9

10 Let T(n) be number of computer steps to compute fib1(n) T(0)=1 T(1)=2 T(n)=T(n-1)+T(n-2)+3, n>1 Analyze running time of recursive algorithm first, write a recursive formula for its running time then, recursive formula => closed formula, asymptotic result How fast does T(n) grow? Can you see that T(n) > F n? How big is T(n)? Running Time analysis 10

11 Mathematical Induction F0=0, F1=1, Fn=Fn-1+Fn-2 We will show that Fn >= 2 0.5n, for n >=6 using strong mathematical induction technique Intuition of basic mathematical induction it s like Domino effect if one push 1st card, all cards fall because 1) 1st card is pushed down 2) every card is close to next card, so that when one card falls, next one falls 11

12 Mathematical Induction Sometimes, we needs the multiple previous cards to knock down next card Intuition of strong mathematical induction it s like Domino effect: if one push first two cards, all cards fall because the weights of two cards falling down knock down the next card Generalization: 2 => k 12

13 Fibonacci numbers F0=0, F1=1, Fn=Fn-1+Fn-2 show that for all integer n >=6 using strong mathematical induction basis step: show it s true when n=6, 7 inductive step: show if it s true for n=k-1, k, then it s true for k+1 given, F F k 2 k k 1 2 k F k+1 = F k 1 + F k 2 k 1 2 k k k 2 =2 2 k 1 2 =2 1+ k 1 2 =2 k

14 Fibonacci numbers F0=0, F1=1, Fn=Fn-1+Fn-2 Fn is lower bounded by F n 2 n 2 =2 0.5n In fact, there is a tighter lower bound n Recall T(n): number of computer steps to compute fib1(n), T(0)=1 T(1)=2 T(n)=T(n-1)+T(n-2)+3, n> n T (n) >F n n 14

15 Exponential running time Running time of Fib1: T(n)> n Running time of Fib1 is exponential in n calculate F 200, it takes at least computer steps On NEC Earth Simulator (fastest computer ) Executes 40 trillion (10 12 ) steps per second, 40 teraflots Assuming each step takes same amount of time as a floating point operation Time to calculate F 200: at least 2 92 seconds, i.e., 1.57x10 20 years Can we throw more computing power to the problem? Moore s law: computer speeds double about every 18 months (or 2 years according to newer version) 15

16 Exponential algorithms Moore s law (computer speeds double about every two years) can sustain for 4-5 more years 16

17 Exponential running time Running time of Fib1: T(n)> n = n Moore s law: computer speeds double about every 18 months (or 2 years according to newer version) If it takes fastest CPU of this year 6 minutes to calculate F50, fastest CPU in two years from today can calculate F52 in 6 minutes Algorithms with exponential running time are not efficient, not scalable 17

18 Fastest Supercomputer June 2017 ranking Sunway TaihuLight, 93 petaflopts Tianhe-2 (milky way-2), 33.9 petaflops Cray XC50, Swiss, 19.6 petaflops Titan, Cray XK7, US, 17.6 petaflopts Petaflop: one thousand million million (10 15 ) floating-point operations per second Need parallel algorithms to take full advantage of these computers 18

19 Big numbers 19

20 Can we do better? Draw recursive function call tree for fib1(5) Observation: wasteful repeated calculation Idea: Store solutions to subproblems in array (key of Dynamic Programming) 20

21 Running time fib2(n) Analyze running time of iterative (non-recursive) algorithm: T(n)=1 // if n=0 return 0 +n // create an array of f[0 n] +2 // f[0]=0, f[1]=1 +(n-1) // for loop: repeated for n-1 times = 2n+2 T(n) is a linear function of n, or fib2(n) has linear running time 21

22 Alternatively How long does it take for fib2(n) finish? T(n)= n+2*60+(n-1)*800=1000n+320 // in unit of us Again: T(n) is a linear function of n Estimation based upon CPU: takes 1000us, takes 200n us each assignment takes 60us addition and assignment takes 800us Constants are not important: different on different computers System effects (caching, OS scheduling) makes it pointless to do such fine-grained analysis anyway Algorithm analysis focuses on how running time grows as problem size grows (constant, linear, quadratic, exponential?) not the actual real world time 22

23 Summary: Running time analysis Given an algorithm in pseudocode or actual program When the input size is n, how many total number of computer steps are executed? Size of input: size of an array, polynomial degree, # of elements in a matrix, vertices and edges in a graph, or # of bits in the binary representation of input Computer steps: arithmetic operations, data movement, control, decision making (if, while), comparison, Ignore: each step take a constant amount of time Overhead of function calls (call stack frame allocation, passing parameters, and return values) Different execution time for different steps 23

24 Time for exercises/examples 1. Reading algorithms in pseudocode 2. Writing algorithms in pseudocode 3. Analyzing algorithms 24

25 Algorithm Analysis: Example What s the running time of MIN? Algorithm/Function.: MIN (a[1 n]) input: an array of numbers a[1 n] output: the minimum number among a[1 n] m = a[1] for i=2 to n: if a[i] < m: m = a[i] return m How do we measure the size of input for this algorithm? How many computer steps when the input s size is n? 25

26 Algorithm Analysis: bubble sort Algorithm/Function.: bubblesort (a[1 n]) input: a list of numbers a[1 n] output: a sorted version of this list for endp=n to 2: for i=1 to endp-1: if a[i] > a[i+1]: swap (a[i], a[i+1]) return a How do you choose to measure the size of input? length of list a, i.e., n the longer the input list, the longer it takes to sort it Problem instance: a particular input to the algorithm e.g., a[1 6]={1, 4, 6, 2, 7, 3} e.g., a[1 6]={1, 4, 5, 6, 7, 9} 26

27 Algorithm Analysis: bubble sort Algorithm/Function.: bubblesort (a[1 n]) input: an array of numbers a[1 n] output: a sorted version of this array for endp=n to 2: for i=1 to endp-1: if a[i] > a[i+1]: swap (a[i], a[i+1]) return a a compute step endp=n: inner loop (for j=1 to endp-1) repeats for n-1 times endp=n-1: inner loop repeats for n-2 times endp=n-2: inner loop repeats for n-3 times endp=2: inner loop repeats for 1 times Total # of steps: T(n) = (n-1)+(n-2)+(n-3)

28 How big is T(n)? T(n) = (n-1)+(n-2)+(n-3)+ +1 Can you write big sigma notation for T(n)? Can you write simplified formula for T(n)? Can you prove the above using math. induction? 1) when n=2, left hand size =1, right hand size is 2) if the equation is true for n=k, then it s also true for n=k+1 28

29 Algorithm Analysis: Binary Search Algorithm/Function.: search (a[l R], value) input: a list of numbers a[l R] sorted in ascending order, a number value output: the index of value in list a (if value is in it), or -1 if not found if (L>R): return -1 m = (L+R)/2 if (a[m]==value): return m else: if (a[m]>value): return search (a[l m-1], value) else: return search (a[m+1 R], value) What s the size of input in this algorithm? length of list a[l R] 29

30 Algorithm Analysis: Binary Search Algorithm/Function.: search (a[l R], value) input: a list of numbers a[l R] sorted in ascending order, a number value output: the index of value in list a (if value is in it), or -1 if not found if (L>R): return -1 m = (L+R)/2 if (a[m]==value): return m else: if (a[m]>value): return search (a[l m-1], value) else: return search (a[m+1 R], value) Let T(n) be number of steps to search an list of size n best case (value is in middle point), T(n)=3 worst case (when value is not in list) provides an upper bound 30

31 Algorithm Analysis: Binary Search Algorithm/Function.: search (a[l R], value) input: a list of numbers a[l R] sorted in ascending order, a number value output: the index of value in list a (if value is in it), or -1 if not found if (L>R): return -1 m = (L+R)/2 if (a[m]==value): return m else: if (a[m]>value): return search (a[l m-1], value) else: return search (a[m+1 R], value) Let T(n) be number of steps to search an list of size n in worst case T(0)=1 //base case, when L>R T(n)=3+T(n/2) //general case, reduce problem size by half Next chapter: master theorem solving T(n)=log2n 31

32 mini-summary Running time analysis identifying input size n counting, and recursive formula: number of steps in terms of n Chapter 2: Solving recursive formula Next: Asymptotic running time 32

33 Order (Growth Rate) of functions Growth rate: How fast f(x) increases as x increases slope (derivative) f(x + x) f(x) x f(x)=2x: constant growth rate (slope is 2) f(x) =2 x : growth rate increases as x increases (see figure above) f(x) =log 2 x: growth rate decreases as x increases 33

34 Order (Growth Rate) of functions Asymptotic Growth rate: growth rate of function when slope (derivative) when x is very big The larger asym. growth rate, the larger f(x) when x 1 e.g., f(x)=2x: asymptotic growth rate is 2 : very big f(x) =2 x x 1 (Asymptotic) Growth rate of functions of n (from low to high): log(n) < n < nlog(n) < n 2 < n 3 < n 4 <.< 1.5 n < 2 n < 3 n 34

35 Compare two running times Two sorting algorithms: yours: 50nlog 2 n your friend: 2n 2 Which one is better (for large program size)? Compare ratio when n is large 50nlog 2 n 2n 2 = 25log 2n n 0, when n 0 For large n, running time of your algorithm is much smaller than that of your friends. 35

36 Rules of thumb if We say g(n) dominates f(n) n a dominates n b, if a>b e.g., n 2 dominates n any exponential dominates any polynomials e.g., 1.1 n dominates n 20 any polynomial dominates any logarithm n dominates logn 2 f(n) g(n) 0, when n 1 36

37 Algorithm Efficiency vs. Speed E.g.: sorting n numbers Friend s computer = 10 9 instructions/second Friend s algorithm = 2n 2 computer steps Your computer = 10 7 instructions/second Your algorithm = 50nlog(n) computer steps To sort n=10 6 numbers, Your friend: You: Your algorithm finishes 20 times faster More importantly, the ratio becomes larger with larger n 37

38 Compare Growth Rate of functions(2) Two sorting algorithms: yours: 2n n your friend: 2n 2 Which one is better (for large arrays)? Compare ratio when n is large 2n n 2n 2 =1+ 100n 2n 2 =1+50 n 1, when n 1 They are same In general, the lower order term can be dropped. 38

39 Compare Growth Rate of functions(3) Two sorting algorithms: yours: 100n 2 your friend: n 2 Your friend s wins. Ratio of the two functions: 100n 2 n 2 = 100, as n 1 The ratio is a constant as n increase. => They scale in the same way. Your alg. always takes 100x time, no matter how big n is. 39

40 Focus on Asymptotic Growth Rate In answering How fast T(n) grows as n grows?, leave out lower-order terms constant coefficient: not reliable info. (arbitrarily counts # of computer steps), and hardware difference makes them not important Note: you still want to optimize your code to bring down constant coefficients. It s only that they don t affect asymptotic growth rate for example: bubble sort executes steps to sort a list of n elements We say bubble sort s running time is quadratic in n, or T(n)=n 2. 40

41 Big-O notation Let f(n) and g(n) be two functions from positive integers to positive reals. f=o(g) means: f grows no faster than g, g is asymptotic upper bound of f(n) f = O(g) if there is a constant c>0 such that for all n, or Most books use notations, where O(g) denotes the set of all functions T(n) for which there is a constant c>0, such that In reference textbook (CLR), for all n>n 0, f(n) apple c g(n) 41

42 Big-O notations: Example f=o(g) if there is a constant c>0 such that for all n, e.g., f(n)=100n 2, g(n)=n 3 so f(n)=o(g(n)), or 100n 2 =O(n 3 ) Exercise: 100n 2 +8n=O(n 2 ) nlog(n)=o(n 2 ) 2 n = O(3 n ) 42

43 Big-Ω notations Let f(n) and g(n) be two functions from positive inters to positive reals. f=ω(g) means: f grows no slower than g, g is asymptotic lower bound of f f = Ω(g) if and only if g=o(f) or, if and only if there is a constant c, such that for all n, Equivalent def in CLR: there is a constant c, such that for all n>n0, 43

44 Big-Ω notations f=ω(g) means: f grows no slower than g, g is asymptotic lower bound of f f = Ω(g) if and only if g=o(f) or, if and only if there is a constant c, such that for all n, e.g., f(n)=100n 2, g(n)=n so f(n)=ω(g(n)), or 100n 2 =Ω(n) Exercise: show 100n 2 +8n=Ω(n 2 ) and 2 n =Ω(n 8 ) 44

45 Big- notations f = (g) means: f grows no slower and no faster than g, f grows at same rate as g asymptotically f = (g) if and only if f = O(g) and f = (g) i.e., there are constants c1, c2>0, s.t., c 1 g(n) apple f(n) apple c 2 g(n), for any n Function f can be sandwiched between g by two constant factors 45

46 Big- notations Show that log 2 n = (log 10 n) 46

47 mini-summary in analyzing running time of algorithms, what s important is scalability (perform well for large input) therefore, constants are not important (counting if.. then as 1 or 2 steps does not matter) focus on higher order which dominates lower order parts a three-level nested loop dominates a single-level loop In algorithm implementation, constants matters 47

48 Typical Running Time Functions 1 (constant running time): Instructions are executed once or a few times log(n) (logarithmic), e.g., binary search A big problem is solved by cutting original problem in smaller sizes, by a constant fraction at each step n (linear): linear search A small amount of processing is done on each input element n log(n): merge sort A problem is solved by dividing it into smaller problems, solving them independently and combining the solution 48

49 Typical Running Time Functions n 2 (quadratic): bubble sort Typical for algorithms that process all pairs of data items (double nested loops) n 3 (cubic) Processing of triples of data (triple nested loops) n K (polynomial) n (exponential): Fib1 2 n (exponential): Few exponential algorithms are appropriate for practical use 3 n (exponential), 49

50 Review of logarithmic functions Definition Rules Applications to CS 50

51 NP-Completeness An problem that has polynomial running time algorithm is considered tractable (feasible to be solved efficiently) Not all problems are tractable Some problems cannot be solved by any computer no matter how much time is provided (Turing s Halting problem) such problems are called undecidable Some problems can be solved in exponential time, but have no known polynomial time algorithm Can we tell if a problem can be solved in polynomial time? NP, NP-complete, NP-hard 51

52 Summary This class focused on algorithm running time analysis start with running time function, expressing number of computer steps in terms of input size Focus on very large problem size, i.e., asymptotic running time big-o notations => focus on dominating terms in running time function Constant, linear, polynomial, exponential time algorithms NP, NP complete problem 52

53 Coming up? Algorithm analysis is only one aspect of the class We will look at different algorithms design paradigm, using problems from a wide range of domain (number, encryption, sorting, searching, graph, ) 53

54 Readings Chapter 1 54