Algorithms - Basic Cocepts Algorithms so what is a algorithm, ayway? The dictioary defiitio: A algorithm is a well-defied computatioal procedure that takes iput ad produces output. This class will deal with how to desig algorithms ad uderstad their complexity i terms of rutime, space, ad correctess. Examples: data compressio, path-fidig, game-playig, schedulig, bi packig Example algorithm: Isertio Sort Isertio sort is how people ofte sort a had of cards. Startig from the left, go towards the right ad make sure that everythig o the left had side is correctly sorted. For example: Origial: 5 3 8 10 7 Pass 1: 3 5 8 10 7 Pass : 3 5 8 10 7 Pass 3: 3 5 8 10 7 Pass 4: 3 5 8 10 7 Pass 5: 3 5 7 8 10 Here is the pseudocode for the Isertio sort algorithm. Typically we will use pseudocode ad ot actual implemetatio code. Isertio-Sort(A) ; Assume our arrays start at idex 1 ad ot at 0 For j to legth(a) Do keya[j] i j-1 while i>0 ad A[i] > key do A[i+1] A[i] i i-1 A[i+1]key Ru through algorithm o our iput, 5 3 8 10 7 J= A=[5 3 8 10 7] Key=3 I=1 A[1]>3 so A[]=A[1] A[1]=3
J=3 J=4 A=[3 5 8 10 7] Key=8 I= A[] < 8 A[3]=8 A=[3 5 8 10 7] Key= I=3 A[3]> so A[4]=A[3] A[]> so A[3]=A[] A[1]> so A[]=A[1] A[1]= J=5 A=[ 3 5 8 10 7] The fial two passes are left as a exercise for you to verify. It looks like this algorithm works to sort the iput. But how do we aalyze it? There are also certaily may other ways to sort the data (ad we ll look at lots of them!) What makes oe algorithm better tha aother? Let s make a assumptio that we will use throughout most of this class. RAM model. This is a uiprocessor, radom access machie with o parallelism (PRAM). Ru time ad space will geerally deped o the size of the iput. Iput Size : Defiitio of depeds o the problem. It may be bits, bytes, but is usually the umber of items i the iput. For sortig, =# of items to sort. For a graph, could be the umber of odes or the umber of liks. Ruig time: defied as the umber of steps that are executed i the program. Let s cout up the rutime for this algorithm. This ca be a little tricky sice the loop does t always ru the same umber of times, but varies depedig o the iput. Let t j = the # of times the ier while loop is examied for the curret value of j. This icludes checkig for the exit coditio of the loop.
Code Cost Times For j to legth(a) C1 Do keya[j] C -1 i j-1 C3-1 j while i>0 ad A[i] > key C4 t j do j j A[i+1] A[i] C5 ( t 1) i i-1 C6 ( t 1) A[i+1]key C7-1 The statemet for the outer j loop will ru times (oce for each elemet, oce to check the exit coditio). The ier statemets ru -1 times. Sice we ve defied t j as the umber of times the ier loop is executed for a value of j, to get the total times it is ru we just add up the sum or all values of j. Our total rutime is the: C1()+C(-1)+C3(-1)+C4( t j )+C5( t j 1)+C6( j j j j t j j 1)+C7(-1) Doig a little math: C1()+C()-C+C3()-C3+C4( t j )+(C5+C6)( t j 1)+C7()-C7 j j (C1+C+C3+C7)() (C+C3+C7) + (C4)( t j j ) + (C5+C6)( ( 1) j j t ) What s the best case? If the iput is already sorted. I this case, j, t j 1. That is the ier loop oly gets executed oce sice we ll immediately fid the right spot for the last item. Rutime for this case: (C1+C+C3+C7)() (C+C3+C7) + (C4)( 1) + (C5+C6)(0) (C1+C+C3+C7)() (C+C3+C7) + (C4)(-1) (C1+C+C3+C4+C7)()-(C+C3+C7+C4) = (CONSTANT1) CONSTANT j
This is a liear fuctio of ; y=ax+b. The rutime will icrease liearly as the size of the iput icreases. rutime What s the worst case? Lets say that the iput cospires to provide the worst rutime. This would happe if the array was i reverse order, because the we have to compare with every other umber i the ier loop. 5 4 3 1 j= t j = compares j=3 t j = 3 compares j=4 t j = 4 compares So we kow that t j = j t j j = j j t j j 1 = 1 1 = j - 1 j j 1 = 1 - (-1) = = ( 1) Plug these ito our formula: (C1+C+C3+C7)() (C+C3+C7) + (C4)( (C1+C+C3+C7)() (C+C3+C7) + (C4)( t j j ) + (C5+C6)( t j j 1) 1 1) + (C5+C6)( ( 1) ) Of the form (Costat1)( ) + (Costat)() + Costat3
iput. This is a quadratic; a +b+c; the rutime will icrease based o the square of the rutime Average case is also quadratic. How much space does this take? Sort is doe i place, so just a few variables. This requires costat space. Fuctio Growth -otatio : Gives asymptotically tight boud o a fuctios growth Defiitio: ( ) ( ):,, : ( ) ( ) ( ) g f c c 0 c g f c g for all 1 0 1 0 c g() f() c 1 g() 0 F()=3 is () sice we ca fid g()=, c1=1, c=4. Isertio sort is ot ( ) sice it is liear i the best case. However, average ad worst case isertio sort is ( ).
O-otatio: Big-O otatio. This gives a asymptotic upper boud. Defiitio: O g( ) f ( ): c, : 0 f ( ) c g( ) for all 1 0 1 0 c1g() f() 0 Isertio sort is O( ). Tight upper boud i the worst case, loose i the best case. Looser: O( 3 ), O( ). Use little-o otatio if you kow that it is a loose boud. o( 3 ) if f=. - Omega otatio. Omega provides a asymptotic lower boud. Defiitio: g( ) f ( ): c, : 0 c g( ) f ( ) for all 0 1 0 1 f() c1g() 0 (1) Costat time, this is a trivial lower boud for most cases. For isertio sort, () : liear time is the best possible
Ruig Time Compariso: =10 =100 =1000 =10000 lg() 5 7 9 10 100 1000 10000 (lg()) 0 500 7000 90000 100 10000 1,000,000 100,000,000 104 1.3x10 30 huge very huge expoetial quadratic liear log For compariso: estimate of the umber of atoms i the uiverse is about 10 80 (i.e. about 70 ). Oe of the tools we will use for aalysis ad solutios is Divide ad Coquer Recursive approach to solve problems. The idea is to break the problem up ito subproblems with the same solutio, but smaller size. Solve these problems recursively, the combie the sub-solutios to solve the origial problem. Divide ito sub-problems Coquer Sub-problems recursively Combie Sub-problem solutios to origial problem Let s look at Merge Sort: Divide elemets ito two subsequeces to be sorted of size / Coquer sort subsequeces recursively with merge sort Combie merge sorted subsequeces ito big sorted aswer Need termiatio criteria for recursio Quit if sequece to sort is legth 1
Pseudocode: Merge_Sort(A,p,r) If p<r the p r q Merge_Sort(A,p,q) Merge_Sort(A,q+1,r) Merge(A,p,q,r) ; Merges A[p..q] with A[q+q..r] ito A[p..q] Call with Merge_Sort(A,1,legth(A)) q gets the media, merge left, right A[1 5 10] becomes Merge_Sort(A,1,5), Merge_Sort(A,6,10) How do we merge? A1: 1 5 8 30 31 50 A: 10 11 1 15 Combied: 1 5 8 10 11 1 15 30 31 50 Easy to do; just start at the frot of each array, compare the poiters, ad put the smallest ito a ew array ad the icremet the poiter that was the smallest. We ca t do this merge i-place though (usig the origial array storage locatio oly) so we eed to make a copy of the merged array ad the copy it back to the origial array. If is the umber of elemets to merge, the it takes () time to merge. Example: A= 1 5 3 7 11 8 4 MS(A,1,8) A=[1, 5, 3, 7, 11, 8,, 4] MS(A,1,4) MS(A,5,8) MS(A,1,) MS(A,3,4) MS(A,5,6) MS(A,7,8) MS(A,1,1) MS(A,,) MS(A,3,3) MS(A,4,4) MS(A,5,5) MS(A,6,6) MS(A,7,7) MS(A,8,8) Merge(A[1..1], A[..]) Merge(A[3..4], A[4..4]) Merge(A[5..5], A[6..6]) Merge(A[7..7], A[8..8]) A = [1, 5, 3, 7, 11, 8,, 4] Merge(A[1..], A[3..4]) A = [1, 5, 3, 7, 11, 8,, 4] A = [1, 5, 3, 7, 8, 11,, 4] Merge(A[5..6], A[7..8]) A = [1, 5, 3, 7, 8, 11,, 4] A = [1, 3, 5, 7, 11, 8,, 4] A = [1, 3, 5, 7,, 4, 8, 11] Merge(A[1..4], A[5..8]) A=[1,, 3, 4, 5, 7, 8, 11]
How much work is doe? It s the cost to split + the cost to merge. Let s defie T() to be the rutime for a problem of size(). This is called a recurrece relatio, sice it is recursively defied. ( 1), 1 Recurrece: T( ) T ( ), 1 The T(/) comes from the divide, ad the () comes from the merge. Later we will show that Merge Sort is (lg) i rutime by illustratig a umber of techiques to solve these recurrece relatios. Math Review x - floor of x, roud dow x - ceilig of x, roud up lg(x) = log x l x = log e x a b log b a log c (ab) = log c (a) + log c (b) logc a logb a logc b log b (1/a) = -log b a log b a = 1 / (log a b) log b log b a a iterated log : log(log ) = log () This grows very slowly! Almost the same as costat time. 1 k 1 3 4... ( 1) ( ) k 1 1 1 1 1 1... l( ) O( 1) (l ) Harmoic series k 1 k 3 ak ak 1 a a0 k 0 Telescopig There are some other summatios we will use, but we ll discuss them as we get to that part of the course.
Mote Carlo Methods Throughout this class we ll focus primarily o aalytic methods to characterize algorithms. However, there is aother class of statistical/probabilistic methods that are commoly referred to as Mote Carlo algorithms. From http://mathworld.wolfram.com/motecarlomethod.html : Ay method which solves a problem by geeratig suitable radom umbers ad observig that fractio of the umbers obeyig some property or properties. The method is useful for obtaiig umerical solutios to problems which are too complicated to solve aalytically. It was amed by S. Ulam Eric Weisstei's World of Biography, who i 1946 became the first mathematicia to digify this approach with a ame, i hoor of a relative havig a propesity to gamble (Hoffma 1998, p. 39). Numerical methods that are kow as Mote Carlo methods ca be loosely described as statistical simulatio methods, where statistical simulatio is defied i quite geeral terms to be ay method that utilizes sequeces of radom umbers to perform the simulatio. Mote Carlo methods have bee used for ceturies, but oly i the past several decades has the techique gaied the status of a full-fledged umerical method capable of addressig the most complex applicatios. The ame Mote Carlo was coied durig the Mahatta Project of World War II, because of the similarity of statistical simulatio to games of chace, ad because the capital of Moaco was a ceter for gamblig ad similar pursuits. Mote Carlo is ow used routiely i may diverse fields, but has gaied some of the greatest popularity i computer sciece ad physics. For example, cosider a complex physical system (solar system, uclear reactio, quatum chromodyamics, traffic patters, etc.) that has bee modeled through a series of pdf s (probability desity fuctios). The behavior ad iteractio of the model with the iput is ofte too complex to determie determiistically. Istead, the system is simulated by ruig may trials usig radom iputs selected from a proper rage. The resultig output helps us lear about the outputs of the system ad the complexity of the system. As aother example, the merge sort example we previously covered could be aalyzed usig a mote carlo method by radomly selectig iputs, ruig the algorithm, ad measurig the rutime (or the umber of comparisos). By averagig out the average rutime, we could determie that merge sort is a (lg) algorithm.