CS1800: Sequences & Sums Professor Kevin Gold
Moving Toward Analysis of Algorithms Today s tools help in the analysis of algorithms. We ll cover tools for deciding what equation best fits a sequence of numbers Which may be useful for predicting program performance given some sample running times And tools for calculating sums of sequences of numbers (series) Because we may know how much work an algorithm does every round, and may want to calculate the total work
Sequences Motivation Suppose we have some datapoints for a black box algorithm - we can t look at the code Processing 1 input takes 12ms Processing 2 inputs takes 20ms Processing 3 inputs takes 30ms Processing 4 inputs takes 42ms What s the general function that gives the running time of this algorithm? Plotting the points may look like a line, but it s not, exactly
Sequences Motivation Whether looking at real data or trying to find a pattern in a series of numbers, it can be helpful to be familiar with common sequences Identifying whether a sequence is arithmetic, quadratic, harmonic, geometric, or none of the above can help you extrapolate or bring other math facts to bear x 2 +5x+6 (quadratric)
Notation & Terminology We can denote a sequence of numbers with a letter for the sequence, and a subscript for the item number n a 1 = 12, a 2 = 20, a 3 = 30, a n = n 2 + 5n + 6 We ll follow the book and usually make 1 the first item number, not 0. (So not a 0 = 12, a 1 = 20, ) This may result in different equations from some you find on the Internet, or in other texts It has the advantage that a 1 a n really is n things instead of n+1 things (but be careful when we get to geometric series ) Try not to say a series of numbers unless you mean a summation over a sequence, a 1 + a 2 + + a n
What s the Equation? 1, 1/2, 1/3 an =? 2, 4, 6, 8, 10 an =? 1,1/2,1/4,1/8 an =? 1,3,6,10,15 an =? 1,1,2,3,5,8, an =?
Equations 1, 1/2, 1/3 an = 1/n 2, 4, 6, 8, 10 an = 2n 1,1/2,1/4,1/8 an = (1/2) n-1 1,3,6,10,15 an = an-1 + n, or n(n+1)/2 1,1,2,3,5,8, an = an-1 + an-2
Terms 1, 1/2, 1/3 an = 1/n (harmonic sequence) 2, 4, 6, 8, 10 an = 2n arithmetic sequence (evens) 1,1/2,1/4,1/8 an = (1/2) n-1 geometric sequence 1,3,6,10,15 an = n(n+1)/2 quadratic sequence ( triangular numbers 1,1+2,1+2+3 ) 1,1,2,3,5,8, an = an-1 + an-2 Fibonacci numbers (recurrence)
Arithmetic Sequences An arithmetic sequence is recognizable by the constant difference between each term and the next. a1 = 4, a2 = 7, a3 = 10, a4 = 13, a5 = 16 +3 +3 +3 +3
Arithmetic Sequences That means each term can be defined as the first term, plus some multiple of this constant difference d. a1 = 4, a2 = 7, a3 = 10, a4 = 13, a5 = 16 4 4+3 4+6 4+9 4+12
Arithmetic Sequences That means each term can be defined as the first term, plus some multiple of this constant difference d. a1 = 4, a2 = 7, a3 = 10, a4 = 13, a5 = 16 4 4+3 4+6 4+9 4+12 a1+d(n-1)
Arithmetic Sequences Represent Linear Growth a1 + d(n-1) is essentially the equation of a line, with d as the slope and (1,a1) as a point. But, we don t consider sequences to have values for inputs between integers. So it s not exactly the same as a line; it s discrete. 2 4 6 4+3(n-1)
Geometric Sequences In a geometric sequence, each term is a constant multiple of the last. Multiply by 2: 1, 2, 4, 8, 16 Multiply by 1/2: 1, 1/2, 1/4, 1/8, 1/16 Multiply by -2: 1, -2, 4, -8, 16, -32 This multiple is typically called r (ratio); above, r = 2, 1/2, and -2. The sequence need not start from 1.
Geometric Sequence Formula Just as we can sum up all the additions of d for arithmetic sequences, we can also figure out the net effect of multiplying by r repeatedly a 1 5 a 2 = a 1 * r = a 1 r 5*2 = 10 (a 1 * r) * r = a 1 r 2 10*2 = 5*2 2 = 20 (a 1 * r 2 ) * r = a 1 r 3 20*2 = 5*2 3 = 40 In other words, the formula is a n = a 1 r n-1
Geometric Sequences Represent Exponential Change If r > 1, a geometric series grows exponentially. 1, 2, 4, 8 a n = 2 n-1 If 0 < r < 1, a geometric series shrinks exponentially. 1, 1/2, 1/4, a n = (1/2) n-1 Either of these is considered very rapid change. r < 0 isn t that common, but will behave like r while bouncing back and forth between - and +
Identifying & Finding the Equations For Arithmetic & Geometric Seqs Can you add the same number d to each term to get the next? If so, it s an arithmetic sequence. Use the d you identified and the first term a1 to get the equation a1 + d(n-1). Can you always multiply by the same number r to get the next number in the sequence? If so, it s geometric. Use the first term a1 to get formula a1r n-1
Quadratic Sequences It is possible to have the difference between terms ( first difference ) increase by a changing amount - but the amount of that change ( second difference ) is constant This is the mark of a quadratic sequence, with a formula of the form xn = an 2 + bn + c. 12 20 30 42 +8 +10 +12 +14 56 +2 +2 +2
Solving for the Quadratic Sequence Equation If we have three data points, and we know the sequence is quadratic, we can solve for the equation. That s because we ll have 3 equations of the form an 2 + bn + c = d where we know n and d. The unknowns are a, b, and c. 3 equations, 3 unknowns. For example, if we know x 1 = 12, x 2 = 20, x 3 = 30, then we have a(1) 2 + b(1) + c = 12 -> a + b + c = 12 a(2) 2 + b(2) + c = 20 -> 4a+2b+ c = 20 a(3) 2 + b(3) + c = 30 -> 9a+3b+ c = 30
Example Solution/ Algebra Reminder We want to add equations together to cancel variables, until we know their values. a + b + c = 12 (Eq1) 4a+2b+ c = 20 (Eq2) 9a+3b+ c = 30 (Eq3) Subtract Eq1 from Eq2: 3a + b = 8 (Eq4) Subtract Eq2 from Eq3: 5a + b = 10 (Eq5) Subtract Eq4 from Eq5: 2a = 2 so a = 1 Substitute a = 1 into Eq4: 3 + b = 8 so b = 5 Substitute a = 1, b = 5 into Eq1: 1 + 5 + c = 12 so c = 6 Thus we recovered a n = n 2 + 5n + 6.
Another Example (The Steps Don t Need to Change Much) If we choose to use the first 3 values of the sequence, the equations won t change besides the values on the RHS - n 2 is still 1, 4, 9, for example a + b + c = 12 (Eq1) 4a+2b+ c = 25 (Eq2) 9a+3b+ c = 48 (Eq3) 12, 25, 48 Subtract Eq1 from Eq2: 3a + b = 13 (Eq4) Subtract Eq2 from Eq3: 5a + b = 23 (Eq5) Subtract Eq4 from Eq5: 2a = 10 so a = 5 Substitute a = 5 into Eq4: 15 + b = 13 so b = -2 Substitute a = 5, b = -2 into Eq1: 5 + -2 + c = 12 so c = 9 Thus we recovered a n = 5n 2-2n + 9.
Sums ( Series ) upper limit index of summation n Σ 1+3i = 4 + 7 + 10 + + (1+3n) i=1 lower limit Σ is a capital sigma, the Greek S, short for sum. (A for product would imply multiplication.)
Arithmetic Series & Gauss s Trick (Like many stories about mathematicians, this may not be true) Sum the numbers from 1 to 100, said the teacher. Young Gauss thought for a bit. 5050, he answered. How did he figure it out so fast?
Summing the Numbers from 1 to 100 With Gauss s Trick In the series 1 + 2 + 3 + + 100, the first and last numbers sum to 101 (1 + 100) the 2nd and 2nd-to-last numbers sum to 101 (2 + 99) the 3rd and 3rd-to-last numbers sum to 101 (3 + 98) And so on. Every number from 1 to 50 pairs off with some number from 100 to 51 to make 101. How many pairs are there? 100/2 = 50. 50 pairs worth 101 each. 50*101 = 5050.
Gauss s Trick, Illustrated 1 2 3 4 5 6 7 8 4+5=9 3+6=9 2+7=9 1+8=9 9*(8/2) = 36
Gauss s Trick, Even Case An arithmetic series is one formed by summing an arithmetic sequence - all numbers are the same distance d apart. The sum of the first number a 1 and last number a n is clearly a 1 + a n a 2 + a n-1 = (a 1 + d) + (a n - d) = a 1 + a n again a 3 + a n-2 = (a 1 + 2d) + (a n - 2d) = a 1 + a n again If n is even, this will again produce n/2 pairs, each worth a 1 + a n So the sum is n(a 1 + a n )/2. n If a 1 = 1 and a n = n, this is the formula Σ i = n(n+1)/2. i=1
Gauss s Trick, Odd Case/General Case The pairing argument doesn t quite make sense if n is odd. It works for all numbers besides the middle number, giving (n-1)/2 pairs worth a1 + an The middle value must be the average of a1 and an since it is equidistant from both Thus the sum is ((n-1)/2)[a1 + an] + (a1+an)/2 = (a1 + an)(n/2), identical to the even case
Example Problem Problem: find the sum of 2 + 5 + 8 + + 29. The difference d appears to be 3. How many terms are there? a n - a 1 = 29-2 = 27. Must be 27/3 = 9 steps of adding 3. Counting the start, 2 (which we didn t add a 3 to get), that s 10 numbers; n = 10. Apply Gauss s formula: (29+2) * 10/2 = 31*5 = 155.
Example Problem #2 Problem: find the sum of 3 + 8 + 13 + + 503. The difference d appears to be 5. How many terms are there?
Example Problem #2 Problem: find the sum of 3 + 8 + 13 + + 503. The difference d appears to be 5. How many terms are there? a n - a 1 = 503-3 = 500. Must be 500/5 = 100 steps of adding 5. Counting a 1 = 3, that s 101 numbers; n = 101. Apply Gauss s formula: (503+3) * 101/2 = 253*101 = 25553.
Arithmetic Series And Code Gauss s trick is how we know the total running time of a variety of algorithms that perform increasing or decreasing work as time goes on For example, Selection Sort (which we ll cover in full later) scans N items for the minimum, then N-1 items for their minimum, then N-2, all the way down to 2 items The total work done by the algorithm is then N + (N-1) + + 3 + 2 steps We know from Gauss s trick that this is a total of N(N+1)/2-1 operations - which we can compare to other sorting algorithms
Geometric Series The sum of a geometric sequence, for example: 8 Σ (1/2) n = 1 + 1/2 + 1/4 + 1/8 + + 1/256 n=0 A geometric series can come up in the analysis of algorithms that work on shrinking fractions of the input For example, an algorithm that works on N things, then N/2 things, then N/4 things N + N/2 + N/4 + = N(1 + 1/2 + 1/4 + ) and we d like to know what that sums to
Deriving a Geometric Series Equation Let S = a + ar + ar 2 + + ar n, our original sum Note there are n+1 terms here; first is ar 0 If we multiply the sum by r, we get something that looks almost identical. (This is < S if r < 1.) rs = ar + ar 2 + ar 3 + + ar n + ar n+1 Subtracting one of these from the other cancels the terms ar + ar 2 + + ar n. S - rs = a - ar n+1 S - rs = S(1-r), so divide both sides by 1-r: S = (a - ar n+1 )/(1-r). It may be easier to remember the derivation than than the equation
6 Finite Geometric Series Example Σ (1/3) k = 1 + 1/3 + 1/9 + + 1/3 6 n=0 a = 1, n = 6, r=1/3: (a - ar n+1 )/(1-r) = (1-1/3 7 )/(2/3) This comes out to 1.5 or so (1.499 ), which we could actually estimate with 1/(2/3) = 3/2 As the terms of a 0 < r < 1 series get small, they start having only a negligible impact For this reason, sometimes we will approximate a geometric series describing an algorithm to be infinite - then we won t need terms like (1/3) 7
Deriving an Infinite Geometric Series Equation Let s assume 0 < r < 1. S = a + ar + ar 2 + rs = ar + ar 2 For every ar i in one series (i >= 1), there is an ar i in the other. So they ll all cancel if we subtract: S - rs = a S(1-r) = a, so S = a/(1-r).
Example of Using the Approximation in an Analysis I have an algorithm that will scan N items, then N/2, then N/4, etc, down to 1 item. How much total work is being done? This is N(1 + 1/2 + 1/4 + 1/8 + + 1/2 i ), where I could figure out i from n, or I could just approximate as the infinite sum N(1 + 1/2 + 1/4 ) Taking the latter approach, the sum 1 + 1/2 + is a/(1-r) = 1/(1/2) = 2. So I can approximate the total work done as 2N.
Picture of the Infinite Sum 1 + 1/2 + 1/4 + 1/8 + 1 1/2 1/4 1/8 1 2 Each term gets us halfway to 2, from wherever we are The difference from 2 approaches 0 All fractional geometric series (0 < r < 1) converge, but r > 1 obviously does not, since those terms get larger and larger
Another Example of Finite & Infinite Series 9 Σ (1/4) k = 1 + 1/4 + + 1/4 9 n=0 Using our formula, (a - ar n+1 )/(1-r) = (1 - (1/4) 10 )/(3/4) (1048575/1048576)/(3/4) is roughly 1/(3/4) = 4/3 = 1.333 If we calculate it more exactly, we get 1.333332
Other Properties of Sums As suggested earlier, we can factor common terms out of all elements of the sum. n Example: Σ 2i = 2 Σ i = 2[n(n+1)/2] = n(n+1) i=1 We can also break up a formula into parts summed separately, since addition doesn t care in what order things happen. n Example: Σ (i + (1/3) i ) = Σ i + Σ (1/3) i i=0 n i=0 = n(n+1)/2 + (1-(1/3) n+1 )/(2/3) n i=0
Summing Quadratic and Cubic Equations There are formulas for the sum of squares and the sum of cubes: Combined with the strategies mentioned on the previous slide, we can tackle a sum like this: n Σ 5k 2-2k + 9 = 5 Σ k 2-2 Σ k + Σ 9 k=1 = 5 n(n+1)(2n+1)/6-2(n(n+1)/2) + 9n Which we could then simplify, substitute a value for n, etc
Additional Series: Harmonic Series The sum of a harmonic sequence: 1 + 1/2 + 1/3 + 1/4 + 1/5 + This doesn t converge - it just keeps growing. In fact, it grows at a very specific rate - roughly like ln n = log e n. (So, slowly.) This sum can come up in algorithm analysis as well as analyzing natural problems like how long it takes to collect N random game pieces
Additional Series: Telescoping This is a term for a series where some element of each term cancels out an element of the next term, leaving only part of the first and part of the last. Named after a piratey collapsing spyglass, yarrr Suppose we have Σ [1/k - 1/(k+1)]. (1-1/2) + (1/2-1/3) + (1/3-1/4) + + (1/(n-1) - 1/n) Canceling the paired terms leaves us with just 1-1/n
Advanced Series Techniques: Bounding With Continuous Calculus Overestimate Underestimate In discrete math, we actually want the rectangle areas The difference between summing series and doing continuous calculus is that our sums care only about values for integer inputs. Nevertheless, it may sometimes be easiest to approximate with continuous calculus. We can sum a function that is definitely larger than our value, or sum a function that is definitely smaller. The true value lies in between.
Aside: Discrete Calculus There is a discrete version of calculus with its own power laws and such. It s complex and not that useful, but it s kind of interesting that it exists? http://homepages.math.uic.edu/~kauffman/dcalc.pdf