Algorithms and Complexity

Algorithms and Complexity

2.1 ALGORITHMS( 演算法 ) Def: An algorithm is a finite set of precise instructions for performing a computation or for solving a problem The word algorithm algorithm comes from the name of a Persian author, Abu Ja far Mohammed ibn Musa al Khowarizmi (c. 825 A.D.), who wrote a textbook on mathematics. Example: A computer program is an algorithm, more precisely, program = algorithm + data structure Remark: From a mathematical perspective, an algorithm represents a function. The British mathematician Alan Turing proved that some functions cannot be represented by an algorithm.

Properties of algorithms Input Output Definiteness : The steps of an algorithm must be defined precisely. Correctness : produce correct output values Finiteness : produce the desired output after a finite number of step. Generality : The procedure should be applicable for all problems of the desired form, not just for a particular set of input values Effectiveness

Pseudo-code Terminology: A good pseudocoding of an algorithm provides a clear prose representation of the algorithm and also is transformable into one or more target programming languages.

Find Maximum Find maximum from a sequence of integers: Set the temporary maximum equal to the first integer in the sequence. Compare the next integer in the sequence to the temporary maximum, and if it is larger than the temporary maximum, set the temporary maximum equal to this integer. Repeat the previous step if there are more integers in the sequence. Stop when there are no integers left in the sequence. The temporary maximum at this point is the largest integer in the sequence.

Problem Locate an element x in a list of distinct elements a 1, a 2,., a n, or determine that it is not in the list

Remark If the array were pre-sorted into ascending (or descending) order, then faster algorithms could be used.

Binary Search Example example

Remark Sorting algorithms Bubble sort Insertion sort Quick sort Merge sort etc

Sorting Algorithms Problem : Suppose that we have a list of elements, a sorting is putting these elements into a list in which the elements are in increasing order. example 7, 2, 1, 4, 5, 9 1, 2, 4, 5, 7, 9

Bubble Sort( 氣泡排序法 ) 設原 sequence 為 a 1,,a n 從 a 1, a 2 開始, 向後兩兩比較, 若 a i > a i+1 則交換, 當檢查完 a n 時,a n 必定是最大數 再從 a 1,a 2 開始向後比較, 若 a i > a i+1 則交換, 此時只需檢查到 a n-1 即可 依此類推

Example Put 3, 2, 4, 1, 5 in increasing order using the bubble sort Sol : 3 2 4 1 5 First pass (i=1) : 2 3 4 1 5 2 3 4 1 5 2 3 1 4 5 Second pass (i=2) : Third pass (i=3) : Fourth pass (i=4) : 1 2 3 4 5 2 3 1 4 5 2 3 1 4 5 2 1 3 4 5 2 1 3 4 5 2 3 1 4 5 2 1 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

Bubble Sort( 氣泡排序法 ) Algorithm The Bubble Sort procedure bubble_sort (a 1,,a n ) for i := 1 to n 1 for j := 1 to n i if a j > a j+1 then interchange a j and a j+1 { a 1,a 2,,a n is in increasing order }

Insertion Sort 設原 sequence 為 a 1,,a n 從 j = 2 開始, 將 a j 插入已排序好 a 1,,a j-1 間的位置, 使得 a 1,,a j 都由小到大排好 j 逐次遞增, 重複上一步驟至做完

Example Use insertion sort to put 3, 2, 4, 1, 5 in increasing order Solution (j=2 時,a 1 =3 可看成已經排序好的數列, 此時要插入 a 2 ) : 3 < 2 2, 3 交換 2, 3, 4, 1, 5 (j=3 時,a 1,a 2 已經排序好, 此時要插入 a 3 ) : 4 > 2, 4 > 3 4 的位置不變 2, 3, 4, 1, 5 (j=4 時,a 1,a 2,a 3 已經排序好, 此時要插入 a 4 ) : 1 < 2 將 1 插在最前面 1, 2, 3, 4, 5 (j=5 時,a 1,a 2,a 3,a 4 已經排序好, 此時要插入 a 5 ) : 5 > 1, 5 > 2, 5 > 3, 5 > 4 5 不變 1, 2, 3, 4, 5

Insertion Sort( 插入排序法 ) Algorithm The Insertion Sort procedure insertion_sort ( a 1,,a n : real numbers with n 2 ) for j := 2 to n begin i := 1 找出 a j 應插入的位置 while a j > a 最後 a i i 1 < a j <= a i i := i + 1 m := a j 將 a for k := 0 to j i 1 i, a i+1,, a j 1 全部往右移一格 a j k := a j k 1 a i := m end { a 1,a 2,,a n are sorted }

Summary Search Find maximum Binary search Sort Bubble sort Insertion sort

Remark Strategies of designing good algorithms: Greedy Divide & conquer Prune & search Dynamic programming Branch and bound Approximation Heuristics

Exercises Section 2.1. 13(b), 34, 38

2.2 The Growth of Functions To analyze the practicality of the program, we need to understand how quickly the function (number of operations used by this algorithm) grows as n (number of input elements) grows.

Asymptotical Upper Bound

BIG OH CLASSES

Def ( Big-O notation ) Let f and g be functions from the set of integers to the set of real numbers. We say that f (x) is O(g(x)) if there are constants C and K such that f (x) C g(x) whenever x > K. ( read as f (x) is big-oh of g(x) )

Figure The function f (x) is O(g(x)) Cg(x) f (x) g(x) k f (x) < C g(x) for x > k

Witness

Example Example Show that f (x) = x 2 +2x+1 is O(x 2 ) Sol : Since x 2 +2x+1 x 2 +2x 2 +x 2 = 4x 2 whenever x > 1, it follows f (x) is O(x 2 ) (take C = 4 and K =1 ) Method 2: If x > 2, we see that x 2 +2x+1 x 2 +x 2 +x 2 = 3x 2 ( take C = 3 and K = 2 )

Example Example Show that f (n)= n 2 +2n +2 is O(n 3 ) Sol : Since n 2 +2n+2 n 3 +n 3 +n 3 = 3n 3 whenever n > 1, we see that f (n) is O(n 3 ) ( take C = 3 and K = 1 ) Note. The function g in O(g) is chosen to be as small as possible.

Exercise Show that f (n)= n 2 +2n +2 is O(n 2 )

Remark Big O = asymptotic upper bound Big Ω = asymptotic lower bound Big Θ = exact order

Theorem Let f (x) = a n x n +a n-1 x n-1 + +a 1 x+a 0 where a 0, a 1,, a n are real numbers. Then f (x) is O (x n ).

Example Give big-o estimates for f (n) = n! Sol : n! = 1 2 3 n n n n = n n n! is O(n n ), taking C =1 and K =1. Example (see Figure 3 in textbook) 常見 function 的成長速度由小至大排列 : 1 < log n < n < n log n < n 2 < 2 n < n!

Theorem Suppose that f1(x) is O(g1(x)) and f2(x) is O(g2(x)), then (f1+f2)(x) is O(max( g1(x), g2(x) )), (f1 f2)(x) is O(g1(x) g2(x)).

Exercises Section 2.2 3,18

2.3 Complexity of Algorithms Question : How can the efficiency of an algorithm be analyzed? Answer : (1) time (2) memory

Complexity Def : Time complexity : an analysis of the time required to solve a problem of a particular size. ( 評量方式 : 計算 # of operations, 如 comparison 次數, 加法 或 乘法 次數等 ) Space complexity : an analysis of the computer memory required to solve a problem of a particular size. ( 是 data structure 探討範圍 )

Example Describe the time complexity of Algorithm Find Max Algorithm ( Find Max ) procedure max(a 1,,a n : integers) max := a 1 for i := 2 to n if max < a i then max := a i { max is the largest element } Sol : ( 計算 # of comparisons) i 值一開始 = 2 逐次加一, 並比較是否 >n. 當 i 變成 n+1 時因比 n 大, 故結束 for 迴圈 共有 n 次 comparison 共有 n-1 次 comparison 故整個演算法共做 2n-1 次 comparison 其 time complexity 為 O(n).

Example Describe the time complexity of the binary search algorithm. Algorithm ( Binary Search ) procedure bs ( x : integer, a 1,,a n : increasing integers ) i := 1 { left endpoint } j := n { right endpoint } while i < j /* ( k+1 次 ) begin m := ( i + j ) / 2 if x > a m then i := m+1 /* ( k 次 ) else j := m end if x = a i then location := i /* ( 1 次 ) else location := 0 Sol : 設 n = 2 k 以簡化計算 ( 若 n < 2 k, 其比較次數必小於等於 n = 2 k 的情況 ) 因 while 迴圈每次執行後整個 list 會切成兩半故最多只能切 k 次就會因 i = j 而跳出迴圈 共比較 2k+2 次 time complexity 為 O(k) = O(log n)

Table Commonly Used Terminology Complexity O(1) O(log n) O(n) O(n log n) O(n b ) O(b n ), b >1 O(n!) Terminology constant complexity Logarithmic complexity Linear complexity n log n complexity Polynomial complexity Exponential complexity Factorial complexity

COMPLEXITY JARGON def: A problem is solvable if it can be solved by an algorithm. Example: Alan Turing defined the halting problem to be that of deciding whether a computational procedure (e.g., a program) halts for all possible input. He proved that the halting problem is unsolvable.

COMPLEXITY JARGON def: A problem is in class P if it is solvable by an algorithm that runs in polynomial time. def: A problem is tractable if it is in class P. def: A problem is in class NP if an algorithm can decide in polynomial time whether a putative solution is really a solution.

Exercises Is P = NP?