In general, we need to describe how much time or memory will be required with respect to one or more variables. The most common variables include:

Transcription

1 2.3 Asymptotic Aalysis It has already bee described qualitatively that if we ited to store othig but objects, that this ca be doe quickly usig a hash table; however, if we wish to store relatioships ad perform queries ad data maipulatios based o that relatioship, it will require more time ad memory. The words quickly ad more are qualitative. I order to make ratioal egieerig decisios about implemetatios of data structures ad algorithms, it is ecessary to describe such properties quatitatively: How much faster? ad How much more memory? The best case to be made is that a ew algorithm may be kow to be faster tha aother algorithm, but that oe word will ot be able to allow ay professioal egieer to determie whether or ot the ewer algorithm is worth the seve perso-weeks required to implemet, itegrate, documet, ad test the ew algorithm. I some cases, it may be simply easier to buy a faster computer Variables of Aalysis I geeral, we eed to describe how much time or memory will be required with respect to oe or more variables. The most commo variables iclude: 1. The umber of objects that are curretly stored i a cotaier, 2. The umber of objects m that the cotaier could possibly hold, or 3. The dimesios of a square matrix. I some cases, we may deal with multiple variables: 1. Whe dealig with objects stored i a cotaier with m memory locatios, 2. Dealig with o-square m matrices, ad 3. Dealig with sparse square matrices where oly m etries are o-zero. We will use the followig code as a example throughout the ext two topics: it fid_max( it *array, it ) { it max = array[0]; for ( it i = 1; i < ; ++i ) { if ( array[i] > max ) { max = array[i]; retur max; Fidig the maximum etry i a array requires that every etry of array to be ispected. We would expect that if we double the size of the array, we would expect it to take twice as log to fid the maximum etry. Page 1 of 12

2 If we multiply a matrix by a -dimesioal vector. Each etry of the matrix is multiplied by oe etry i the vector. Therefore, if we multiply a 2 2 matrix by a 2-dimesioal vector, we would expect four times as may multiplicatios ad therefore it should take about four times as log. The questio is, how ca we express this mathematically? Biary Search versus Liear Search As aother example, cosider searchig a array for a value. If the array is sorted, we ca do a biary search but if the array is ot sorted, we must perform a liear search checkig each etry of the array. Here are two implemetatios of these fuctios. it liear_search( it value, it *array, it ) { for ( it i = 0; i < ; ++i ) { if ( array[i] == value ) { retur i; retur -1; it biary_search( it value, it *array, it ) { it a = 0; it c = - 1; while ( a <= c ) { it b = (a + c)/2; if ( array[b] == value ) { retur b; else if ( array[b] < value ) { a = b + 1; else { c = b - 1; retur -1; I each case, there is a worst case as to how may etries of the array will be checked; however, there is also a average umber of etries that will be checked. Figure 1 shows the worst-case ad average umber of comparisos required by both the liear ad biary searches i searchig a array of size from 1 to 32. Page 2 of 12

3 Figure 1. The average ad worst-case umber of comparisos required for usig a liear search (blue) ad a biary search (red) o a array if size = 1,..., 32. It seems that eve the worst-case for a biary search is sigificatly better tha the average case for a liear search. Also, the umber of comparisos required for a liear search appears to grow, well, liearly, whereas the umber of searches required for a biary search appears to be growig at the same rate as either or l(). Which oe is it ad why? Aother issue is what do we really care about? If Algorithm A rus twice as fast as Algorithm B, it is always possible to simply purchase a faster computer i which case Algorithm B will perform just as well, if ot better tha Algorithm A. A importat questio about the liear search versus the biary search is: ca we purchase a computer fast eough so that liear search will always outperform a biary search implemeted o the 1980s-era processor? Rate of Growth of Polyomials Cosider ay two polyomials of the same degree. If the coefficiets of the leadig term are the same, while the fuctios may be very differet aroud zero, as you plot them o larger ad larger itervals, they will appear to be essetially the same. Figure 2 shows two quadratic polyomials, 2 ad , ad while they appear differet o the iterval [0, 3], o [0, 1000], they are essetially idetical. Page 3 of 12

4 Figure 2. Two quadratic polyomials plotted over [0, 3] ad [0, 1000]. Figure 3 shows two sextic policomials: 6 ad ad while they appear to be very differet o the iterval [ 2, 5], but agai, o the iterval [0, 1000], they appear to be essetially idetical. Figure3. Two sextic polyomials plotted o [ 2, 5] ad agai o [0, 1000]. If two polyomials have the same degree but the coefficiets of the leadig terms are differet, agai, while they may appear to be sigificatly differet aroud the origi, o a larger scale, oe will simply be a scalar multiple of the other Examples of Algorithm Aalysis For example, Selectio Sort Bubble Sort best case worst case describe the umber of istructios required to implemet selectio sort ad bubble sort to sort a list of size, respectively. You ca look at appedix A to see the C++ source code ad disassembled object code produced by g++. Iitially, selectio sort requires sigificatly eve more istructios tha the worst-case for bubble sort; however, for problems of size 18, selectio sort performs better tha the worst-case bubble sort. As the problem size becomes very large, selectio sort falls approximately half way betwee the best ad worst cases of bubble sort. This is show i Figure 4. Page 4 of 12

5 Figure 4. The istructios required to sort a array of size for selectio sort (blue) ad the best ad worst case scearios for bubble sort (red). As the problem size gets larger ad larger, the rate of growth of all three fuctios is similar: the best case for bubble sort will require approximately the umber of istructios for selectio sort ad the worst case will require times the umber of istructios. Because the umber of istructios ca be calculated directly, we ca also approximate the time it will take to execute this code o, for example, a computer ruig at 1 GHz: divide the umber of istructios by However, we ca also, for example, ru selectio sort o a computer that rus at 2 GHz. Figure 5 shows the best ad worst cases for bubble sort ru o a 1 GHz computer; however, the left image shows the time required by selectio sort ru o a 1 GHz computer while the right shows it ru o a 2 GHz computer. Figure 5. Time required to sort a problem of size up to oe millio for bubble sort ru o a 1 GHz computer ad selectio sort ru o 1 ad 2 GHz computers, respectively. Page 5 of 12

6 The critical poit here is that selectio sort is ot sigificatly better or worse tha bubble sort. Ay differet i speed ca be compesated by usig better hardware Big-Oh ad Big-Theta What we eed is a way of sayig that two fuctios are growig at the same rate that is, they are essetially growig at the same rate. To do this, we must recall a cocept from first year: big-oh otatio. This is oe of five Ladau symbols that we will use i this class. You will recall that i first year, you described f() = O(g()) if M > 0 ad N 0 > such that f( ) Mg < wheever > N. If we are dealig with fuctios such as liear combiatios of terms like r or r l() where r is a positive real umber where the fial result is mootoically icreasig ad positive for > 0 (behaviours we would expect from fuctios describig algorithms), the f() = O(g()) is equivalet to sayig that f lim g However, to say that two fuctios are growig at the same rate, we eed a stroger restrictio: two fuctios will be said to be growig at the same rate if ad we will write that f() = Θ(g()). <. f 0 < lim < g For example, two polyomials of the same degree are big-theta of each other because the above limit will be the ratio of the coefficiets of the leadig terms, for example, For real values p 0 ad q 0, it is true that 1. p = Θ( q ) if ad oly if p = q, ad 2. p = O( q ) if ad oly if p q Cosequece = 2 lim If f() ad g() describes either the time required or the umber of istructios required for two differet algorithms to solve a problem of size ad f() = Θ(g()), we ca always make oe fuctio ru faster tha the other by havig sufficietly better hardware. Page 6 of 12

7 2.3.6 Little-oh Cosider the two fuctios f() = ad g() = l(). It is already true that for all > 0, > l(). However, is there a sufficietly small value of c > 0 such that l() > c for all > N for some N? A little thought will probably covice you that o such positive umber exists, for o matter what umber we choose, ( 1 ) l 1 1 lim = lim = lim = 0. c c c Therefore, the logarithm grows sigificatly slower tha the fuctio. We would also like to describe whe oe fuctio f() grows sigificatly slower tha aother fuctio g(), ad therefore we will say that f() = o(g()) (little-oh) if f lim = 0. g Thus, we could write that l() = o(). A ice way of rememberig this is that the little o looks like a zero ad the limit is zero. We may cotiue the aalogy: for real values p 0 ad q 0, it is true that 1. p = Θ( q ) if ad oly if p = q, 2. p = O( q ) if ad oly if p q, ad 3. p = o( q ) if ad oly if p < q Little-omega ad Big-Omega Curret, we have the followig table: Ladau Symbol Limit Descriptio Aalogous Relatioal Operator f() = Θ(g()) 0 < c < f grows at the same rate as g = f() = O(g()) c < f grows at the same rate as or slower tha g f() = o(g()) 0 f grows sigificatly slower tha g < What we are missig are meas of describig if oe fuctios grows faster tha aother fuctio. We will add two more Ladau symbols: ω ad Ω. We will say that f() grows sigificatly faster tha g() if f lim g ad we will write this as f() = ω(g()). Note that little-omega looks like the ifiity symbol. = Page 7 of 12

8 We will also say that f() grows either at the same rate or faster tha g() if ad we will write this as f() = Ω(g()). f lim > 0 g We may cotiue the aalogy: for real values p 0 ad q 0, it is true that 1. p = ω( q ) if ad oly if p > q, ad 2. p = Ω( q ) if ad oly if p q. Thus, we have the full table: Ladau Symbol Limit Descriptio Aalogous Relatioal Operator f() = ω(g()) f grows sigificatly faster tha g > f() = Ω(g()) 0 < c f grows at the same rate as or faster tha g f() = Θ(g()) 0 < c < f grows at the same rate as g = f() = O(g()) c < f grows at the same rate as or slower tha g f() = o(g()) 0 f grows sigificatly slower tha g < Big-Theta as a Equivalece Relatio There are some iterestig characteristics of Ladau symbols with respect to the fuctios that we are iterested i: 1. f() = Θ(f()), 2. f() = Θ(g()) if ad oly if g() = Θ(f()), ad 3. If f() = Θ(g()) ad g() = Θ(h()), it follows that f() = Θ(h()). Therefore, big-theta defies a equivalece relatio o fuctios. Oe of the properties of equivalece relatios is that you ca create equivalece classes of all fuctios that are big-theta of each other. For example, all of the fuctios l() l() grow at the same rate as each other. Therefore, to describe this etire equivalece class of fuctios that grow at the same rate as a quadratic moomial, we will select o member to represet the etire class ad while we could chose , it would make more sese to choose 2 ad call this class of fuctios quadraticly growig fuctios. Page 8 of 12

9 There are specific equivalece classes that appear so ofte i the discussio of algorithm aalysis that we give them special ames. These are listed i Table 1. Table 1. Equivalece classes of equivalet rates of growth. Equivalece Class Represetative Costat 1 Logarithmic l() Liear N -log- l() Quadratic 2 Cubic 3 Expoetials 2, e, 3, etc. Note that with the expoetial fuctios, there is ot oe sigle represetative, for if a < b the a grows slower tha b : a a lim = lim 0 = b b a because 1 b < Little-oh as a Weak Orderig We have already oted that for ay real 0 p < q, it follows that p = o( q ) ad therefore, we ca order the equivalece classes. I additio to these classes, we also ote that ( ) 1 = o l, however, l() = o( p ) for ay p > 0. This ca be see by seeig that l ( 1 ) 1 1 lim lim lim 0 p p 1 p = p = p = as p > 0. Similarly, p = o( q l()) for ay 0 p q ad q l() = o( r ) for ay 0 q < r. Graphically, we ca show this weak orderig as show i Figure 6. Figure 6. The orderig of the equivalece classes of fuctios growig at the same rate. Page 9 of 12

10 Notes (beyod the scope of this class) We could also cosider fuctios that grow accordig to p l q () where p, q 0 as well as fuctios such as l(l()). There are searchig algorithms that ca ru as fast as Θ(l(l())) meaig that a problem of size = requires oly four times loger to solve tha a problem of size = 15. We are restrictig our defiitios to usig limits. To be more correct, we should use the limit supremum ad limit ifimum. Techically, we should say that f() = Θ(g()) if f ( ) f ( ) 0 < limif limsup < g( ) g( ) ad f() = O(g()) if f ( ) limsup <. g What s Next? We will use Ladau symbols to describe the ru-times ad memory usage of various data structures ad algorithms. A algorithm will be said to have polyomial time complexity if its ru time may be described by O( p ) for some p 0. I a geeral sese, problems that ca be solved with kow polyomial time algorithms are said to be efficietly solvable or tractable. Problems for which there are o algorithms that ca solve the problem i polyomial time are said to be itractable. For example, the travellig salesma problem (fid the shortest path that visits each of cities exactly oce) ca oly be solved by a algorithm that is requires Θ( 2 2 ) time: add oe more city ad the algorithm takes more tha twice as log to ru. Add 10 cities ad the algorithm takes more tha 2 10 = 1024 times to ru. Algorithms that require a expoetial amout of time are, for the most part, udesirable. Be sure to ever describe a fuctio with quadratic growth as expoetial. Page 10 of 12

11 Appedix A Source Code void bubble_sort( it *array, it ) { for ( it i = ( - 1); i >= 1; --i ) { for ( it j = 0; j < i; ++j ) { if ( array[j] > array[j + 1] ) { it tmp = array[j]; array[j] = array[j + 1]; array[j + 1] = tmp; Output of % g++ -O -c bubble_sort.cpp % objdump d bubble_sort.o 0: 83 ee 01 sub $0x1,%esi 3: 85 f6 test %esi,%esi 5: 7f 21 jg 28 7: f3 c3 repz retq 9: 8b 0c 87 mov (%rdi,%rax,4),%ecx c: 8b mov 0x4(%rdi,%rax,4),%edx 10: 39 d1 cmp %edx,%ecx 12: 7e 07 jle 1b 14: mov %edx,(%rdi,%rax,4) 17: 89 4c mov %ecx,0x4(%rdi,%rax,4) 1b: c0 01 add $0x1,%rax 1f: 39 c6 cmp %eax,%esi 21: 7f e6 jg 9 23: 83 ee 01 sub $0x1,%esi 26: je 2f 28: b mov $0x0,%eax 2d: eb da jmp 9 2f: f3 c3 repz retq Page 11 of 12

12 Soucre Code void selectio_sort( it *array, it cost ) { for ( it i = ( - 1); i >= 1; --i ) { it pos = 0; for ( it j = 1; j <= i; ++j ) { if ( array[j] > array[pos] ) { pos = j; it tmp = array[i]; array[i] = array[pos]; array[pos] = tmp; Output of % g++ -O -c selectio_sort.cpp % objdump d selectio_sort.o 0: 83 ee 01 sub $0x1,%esi 3: c6 movslq %esi,%rax 6: 85 f6 test %esi,%esi 8: 4c 8d lea (%rdi,%rax,4),%r10 c: 7e 45 jle 53 e: f9 mov %rdi,%r9 11: b mov $0x1,%ecx 16: c0 xor %r8d,%r8d 19: 0f 1f opl 0x0(%rax) 20: 41 8b mov 0x4(%r9),%eax 24: 42 3b cmp (%rdi,%r8,4),%eax 28: d1 movslq %ecx,%rdx 2b: 4c 0f 4f c2 cmovg %rdx,%r8 2f: 83 c1 01 add $0x1,%ecx 32: c1 04 add $0x4,%r9 36: 39 f1 cmp %esi,%ecx 38: 7e e6 jle 20 3a: 4a 8d lea (%rdi,%r8,4),%rdx 3e: 41 8b 0a mov (%r10),%ecx 41: 8b 02 mov (%rdx),%eax 43: mov %eax,(%r10) 46: ea 04 sub $0x4,%r10 4a: 83 ee 01 sub $0x1,%esi 4d: 89 0a mov %ecx,(%rdx) 4f: 75 bd je e 51: f3 c3 repz retq 53: f3 c3 repz retq Page 12 of 12