Electrical & Computer Engineering University of Waterloo Canada February 26, 2007

Transcription

1 : Electrical & Computer Engineering University of Waterloo Canada February 26, 2007 We want to choose the best algorithm or data structure for the job. Need characterizations of resource use, e.g., time, space; for circuits: area, depth. Many, many approaches: Worst Case Execution Time (WCET): for hard real-time applications Exact measurements for a specific problem size, e.g., number of gates in a 64-bit addition circuit. Performance models, e.g., R, n 1/2 for latency-throughput, HINT curves for linear algebra (characterize performance through different cache regimes), etc.... :

2 Asymptotic analysis We will focus on Asymptotic analysis: a good first approximation of performance that describes behaviour on big problems Reasonably independent of: Machine details (e.g., 2 cycles for add+mult vs. 1 cycle) Clock speed, programming language, compiler, etc. : : Brief history Basic ideas originated in Paul du Bois-Reymond s Infinitärcalcül ( calculus of infinities ) developed in the 1870s. G. H. Hardy greatly expanded on Paul du Bois-Reymond s ideas in his monograph Orders of Infinity (1910) [3]. The big-o notation was first used by Bachmann (1894), and popularized by Landau (hence sometimes called Landau notation. ) Adopted by computer scientists [4] to characterize resource consumption, independent of small machine differences, languages, compilers, etc. :

3 Basic asymptotic notations Asymptotic behaviour as n, where for our purposes n is the problem size. Three basic notations: f g ( f and g are asymptotically equivalent ) : f g ( f is asymptotically dominated by g ) f g (f and g are asymptotically bounded by one another) Basic asymptotic notations f g means lim n f (n) g(n) = 1 : Example: 3x 2 + 2x + 1 3x 2. is an equivalence relation: Transitive: (x y) (y z) (x z) Reflexive: x x Symmetric: (x y) (y x). Basic idea: We only care about the leading term, disregarding less quickly-growing terms.

4 Basic asymptotic notations i.e., f (n) g(n) f g means lim sup n f (n) g(n) < is eventually bounded by a finite value. Basic idea: f grows more slowly than g, or just as quickly as g. is a preorder (or quasiorder): Transitive: (f g) (g h) (f h). Reflexive: f f fails to be a partial order because it is not antisymmetric: there are functions f, g where f g and g f but f g. Variant: g f means f g. : Basic asymptotic notations Write f g when there are positive constants c 1, c 2 such that c 1 f (n) g(n) c 2 : for sufficiently large n. Examples: n 2n n (2 + sin πn)n is an equivalence relation.

5 Strict forms Write f g when f g but f g. Basic idea: f grows strictly less quickly than g Equivalent: f g exactly when lim n f (n) g(n) = 0. Examples x 2 x 3 log x x Variant: f g means g f : Orders of growth We can use as a ruler by which to judge the growth of functions. Some common tick marks on this ruler are: log log n log n log k n n ɛ n n 2 2 n n! n n 2 2n We can always find in a dense total order without endpoints. i.e., There is no slowest-growing function; There is no fastest-growing function; If f h we can always find a g such that f g h. (The canonical example of a dense total order without endpoints is Q, the rationals.) This fact allows us to sketch graphs in which points on the axes are asymptotes. :

6 Big-O Notation Big-O is a convenient family of notations for asymptotics: O(g) {f : f g} i.e., O(g) is the set of functions f so that f g. O(n 2 ) contains n 2, 7n 2, n, log n, n 3/2, 5,... Note that f O(g) means exactly f g. A standard abuse of notation is to treat a big-o expression as if it were a term: : x 2 + 2x 1/2 + 1 }{{} x 2 = x 2 + O(x 1/2 ) The above equation should be read as there exists a function f O(x 1/2 ) such that x 2 + 2x 1/2 + 1 = x 2 + f (x). Big-O for algorithm analysis Big-O notation is an excellent tool for expressing machine/compiler/language-independent complexity properties. On one machine a sorting algorithm might take 5.73n log n seconds, on another it might take 9.42n log n + 3.2n seconds. We can wave these differences aside by saying the algorithm runs in O(n log n) seconds. O(f (n)) means something that behaves asymptotically like f (n): Disregarding any initial transient behaviour; Disregarding any multiplicative constants c f (n); Disregarding any additive terms that grow less quickly than f (n). :

7 Basic properties of big-o notation Given a choice between an sorting algorithm that runs in O(n 2 ) time and one that runs in O(n log n) time, which should we choose? : 1. Gut instinct: the O(n log n) one, of course! 2. But: note that the class of functions O(n 2 ) also contains n log n. Just because we say an algorithm is O(n 2 ) does not mean it takes n 2 time! 3. It could be that the O(n 2 ) algorithm is faster than the O(n log n) one. Additional notations To distinguish between at most this fast, at least this fast, etc. there are additional big-o-like notations: : f O(g) f g f o(g) f g f Θ(g) f g f Ω(g) f g f ω(g) f g upper bound strict upper bound tight bound lower bound strict lower bound

8 Tricks for a bad remembering day Lower case means strict: o(n) is strict version of O(n) ω(n) is strict version of Ω(n) ω, Ω (omega) is the last letter of the greek alphabet if f ω(g) then g comes after f in asymptotic ordering. f Θ(g): the line through the middle of the theta asymptotes converge : Notation: o( ) f o(g) means f g : o( ) expresses a strict upper bound. If f (n) is o(g(n)), then f grows strictly slower than g. Example: n k=0 2 k = = 2 + o(1) n o(1) indicates the class of functions for which g(n) lim n 1 = 0, which means lim n g(n) = o(1) means 2 plus something that vanishes as n If f is o(g), it is also O(g). n! = o(n n ).

9 Notation: ω( ) f ω(g) means f g : ω( ) expresses a strict lower bound. If f (n) is ω(g(n)), then f grows strictly faster than g. f ω(g) is equivalent to g o(f ). Example: Harmonic series hn = n k=0 1 k ln n + γ + O(n 1 ) hn ω(1) (It is unbounded.) hn ω(ln ln n) n! = ω(2 n ) (grows faster than 2 n ) Notation: Ω( ) f Ω(g) means f g : Ω( ) expresses a lower bound, not necessarily strict If f (n) is Ω(g(n)), then f grows at least as fast as g. f Ω(g) is equivalent to g O(f ) Example: Matrix multiplication requires Ω(n 2 ) time. (At least enough time to look at each of the n 2 entries in the matrices.)

10 Notation: Θ( ) f Θ(g) means f g Θ( ) expresses a tight asymptotic bound If f (n) is Θ(g(n)), then f (n)/g(n) is eventually contained in a finite positive interval [c 1, c 2 ]. Θ( ) bounds are very precise, but often hard to obtain. Example: QuickSort runs in time Θ(n log n) on average. (Tight! Not much faster or slower!) Example: Stirling s approximation ln n! n ln n n + O(ln n) implies that ln n! is Θ(n ln n) Don t make the mistake of thinking that f Θ(g) f (n) means lim n g(n) = k for some constant k. : Algebraic manipulations of big-o Manipulating big-o terms requires some thought always keep in mind what the symbols mean! An additive O(f (n)) term swallows any terms that are f (n): n 2 + n 1/2 + O(n) + 3 = n 2 + O(n) : The n 1/2 and 3 on the l.h.s. are meaningless in the presence of an O(n) term. O(f (n)) O(f (n)) = O(f (n)) not 0! O(f (n)) O(g(n)) = O(f (n)g(n)). Example: What is ln n + γ + O(n 1 ) times n + O(n 1/2 )? [ ln n + γ + O(n 1 ) ] [ ] n + O(n 1/2 ) = n ln n + γn + O(n 1/2 ln n) The terms γo(n 1/2 ), O(n 1/2 ), O(1), etc. get swallowed by O(n 1/2 ln n).

11 Sharpness of estimates Example: for a constant c, ( ( ln(n + c) = ln n 1 + c )) n = ln n + c n c2 2n 2 + ( ) 1 = ln n + Θ n ( = ln n + ln 1 + c ) n : (Maclaurin series) It is also correct to write ln(n + c) = ln n + O(n 1 ) ln(n + c) = ln n + o(1) since Θ(n 1 ) O( 1 n ) o(1). However, the Θ( 1 n ) error term is sharper a better estimate of the error. Sharpness of estimates & The Riemann Hypothesis Example: let π(n) be the number of prime numbers n. The Prime Number Theorem is that where Li(n) = n x=2 π(n) Li(n) (1) 1 ln x dx is the logarithmic integral, and Li(n) Note that (1) is equivalent to: n ln n : π(n) = Li(n) + o(li(n)) It is known that the error term can be improved, for example to ( n ) π(n) = Li(n) + O ln n ln n e a

12 Sharpness of estimates & The Riemann Hypothesis The famous Riemann hypothesis is the conjecture that a sharper error estimate is true: : π(n) = Li(n) + O(n 1 2 ln n) This is one of the Clay Institute millenium problems, with a $1,000,000 reward for a positive proof. Sharp estimates matter! Sharpness of estimates To maintain sharpness of asymptotic estimates during analysis, some caution is required. E.g. If f (n) = 2 n + O(n), what is log f (n)? Bad answer: log f (n) = n + O(n). More careful answer: : log f (n) = log(2 n + O(n)) = log(2 n (1 + O(n2 n ))) = log(2 n ) + log(1 + O(n2 n )) Since log(1 + δ(n)) O(δ(n)) if δ o(1), log f (n) = n + O(n2 n ) i.e., log f (n) is equal to n plus some value converging exponentially fast to 0.

13 Sharpness of estimates log f (n) = n + O(n2 n ) is a reasonably sharp estimate (but, what happens if we take 2 log f (n) with this estimate?) If we don t care about the rate of convergence we can write : f (n) = n + o(1) where o(1) represents some function converging to zero. This is less sharp since we have lost the rate of convergence. Even less sharp is f (n) n which loses the idea that f (n) n 0, and doesn t rule out things like f (n) = n + n 3/4. Asymptotic expansions An asymptotic expansion of a function describes how that function behaves for large values. Often it is used when an explicit description of the function is too messy or hard to derive. e.g. if I choose a string of n bits uniformly at random (i.e., each of the 2 n possible strings has probability 2 n ), what is the probability of getting 3 4 n 1 s? Easy to write the answer: there are ( n k) ways of arranging k 1 s, so the probability of getting 3 4n 1 s is: : P(n) = n k= 3 4 n 2 n ( ) n k This equation is both exact and wholly uninformative.

14 Asymptotic expansions Can we do better? Yes! The number of 1 s in a random bit string is a binomial distribution and is well-approximated by the normal distribution as n : n k= 1 2 n+α n ( ) n 2 n k x=α = 1 F (α) 1 2π e x2 2 dx ( ( )) where F (x) = erf x is the cumulative normal 2 distribution. Maple s asympt command yields the asymptotic expansion: ( ) 1 F (x) 1 O xe x2 2 : Asymptotic expansions example We want to estimate the probability of 3 4 n 1 s: gives α = 1 2 n + α n = 3 4 n n 4. Therefore the probability is ( ) n P(n) 1 F 4 ( O = O ( 1 ne n 32 ) ne n 32 ) : So, the probability of having more than 3 4 n 1 s converges to 0 exponentially fast.

15 Asymptotic Expansions When taking an asymptotic expansion, one writes ln n! n ln n n + O(1) rather than ln n! = n ln n n + O(1) : Writing is a clue to the reader that an asymptotic expansion is being taken, rather than just carrying an error term around. Asymptotic expansions are very important in average case analysis, where we are interested in characterizing how an algorithm performs for most inputs. To prove an algorithm runs in O(f (n)) on average, one technique is to obtain an asymptotic estimate of the probability of running in time f (n), and show it converges to zero very quickly. Asymptotic Expansions for Average-Case Analysis The time required to add two n-bit integers by a no carry adder is proportional to the longest carry sequence. It can be shown that the probability of having a carry sequence of length t(n) satisfies : t(n)+log n+o(1) Pr(carry sequence t(n)) 2 If t(n) log n, the probability converges to 0. We can conclude that the average running time is O(log n). In fact we can make a stronger statement: Pr(carry sequence log n + ω(1)) 0 Translation: The probability of having a carry sequence longer than log n + δ(n), where δ(n) is any unbounded function, converges to zero.

16 The Taylor series method of asymptotic expansion This is a very simple method for asymptotic expansion that works for simple cases; it is one technique Maple s asympt function uses. Recall that the Taylor series of a C function about x = 0 is given by: : f (x) = f (0) + xf (0) + x 2 2! f (0) + x 3 3! f (0) + To obtain an asymptotic expansion of some function F (n) as n, 1. Substitute n = x 1 into F (n). (Then n as x 0.) 2. Take a Taylor series about x = Substitute x = n Use the dominating term(s) as the expansion, and the next term as the error term. Taylor series method of asymptotic expansion: example Example expansion: F (n) = e 1+ 1 n. Obviously lim n F (n) = e, so we expect something of the form F (n) e + o(1). 1. Substitute n = x 1 into F (n): obtain F (x 1 ) = e 1+x 2. Taylor series about x = 0: : 3. Substitute x = n 1 : e 1+x = e + xe + x 2 4. Since e 1 n e 1 2n 2 e, 2 e + x 3 6 e + = e + e n + 1 2n 2 e + 1 6n 3 e + F (n) e + Θ ( ) 1 n

17 of algorithms is a key tool for algorithms and data structures: Analyze algorithms/data structures to obtain sharp estimates of asymptotic resource consumption (e.g., time, space) Possibly use asymptotic expansions in the analysis to estimate e.g. probabilities Use these resource estimates to Decide which algorithm/data structure is best according to design criteria Reason about the performance of compositions (combinations) of algorithms and data structures. : References on asymptotics Course text: [1] Asymptotic notations Concrete Mathematics, Ronald L. Graham, Donald E. Knuth and Oren Patashnik, Ch. 9 [2] Advanced: Shackell, Symbolic [6] Hardy, Orders of Infinity [3] Lightstone + Robinson, Nonarchimedean fields and asymptotic expansions [5] :

18 I [1] Thomas H. Cormen, Charles E. Leiserson, and Ronald R. Rivest. Intoduction to algorithms. McGraw Hill, bib : [2] Ronald L. Graham, Donald E. Knuth, and Oren Patashnik. Concrete Mathematics: A Foundation for Computer Science. Addison-Wesley, Reading, MA, USA, second edition, bib II [3] G. H. Hardy. Orders of infinity. The Infinitärcalcül of Paul du Bois-Reymond. Hafner Publishing Co., New York, Reprint of the 1910 edition, Cambridge Tracts in Mathematics and Mathematical Physics, No. 12. bib : [4] Donald E. Knuth. Big omicron and big omega and big theta. SIGACT News, 8(2):18 24, bib pdf [5] A. H. Lightstone and Abraham Robinson. Nonarchimedean fields and asymptotic expansions. North-Holland Publishing Co., Amsterdam, North-Holland Mathematical Library, Vol. 13. bib

19 III : [6] John R. Shackell. Symbolic asymptotics, volume 12 of Algorithms and Computation in Mathematics. Springer-Verlag, Berlin, bib