SECTION 2.1 BASIC CALCULUS REVIEW

Transcription

1 Tis capter covers just te very basics of wat you will nee moving forwar onto te subsequent capters. Tis is a summary capter, an will not cover te concepts in-ept. If you ve never seen calculus before, ten grasping concepts later on in te book coul be ifficult. But certain capters won t require calculus, so eac capter will ave captions inicating capter pre-requisites. But in orer to unerstan every algoritm in te book, you ll nee to eventually know basic calculus. SECTION 2.1 BASIC CALCULUS REVIEW Calculus is te stuy of cange an is split into two brances: ifferential calculus an integral calculus. Differential calculus eals wit ow a certain variable canges wit respect to anoter variable. For example, if a car travele a certain istance in time t, we can use ifferential calculus to etermine ow te car s position canges wit respect to time. Integral calculus is simply te antitesis of ifferential calculus. By knowing at wat rate someting canges, we can use integral calculus to quantify ow muc in total someting i cange witin a certain interval. Let s formalize tis matematically now. Given a function f(x), we can etermine te rate of ow muc f(x) canges as x varies by taking te erivative of f(x). Te erivative of a function f(x) is often enote as f (x). You migt also see te ifferential operator ) being use to imply we are taking te erivative )* or ifferentiating a function. Formally, te erivative of a function f(x) is efine as: f, x f x + f(x) Anoter way to interpret te erivative is unerstaning it is te slope of a function f(x) at a particular point x (also known as te tangent or instantaneous rate of cange). If a function is increasing, te erivative will be positive. If a function is ecreasing te erivative will be negative. We can prove tis easily by using a stanar line equation you soul be familiar wit: f(x) = mx + b, were b is te intercept an m is te slope. Let s use te efinition of te erivative to fin f (x) of te line equation.

2 f, x f x + f x m x + + b mx b mx + m mx m m = m Tus, f, x = m, an we are one. We can see tat for any x, f (x) is always m. Wat about nonlinear equations like a parabola? Well, we will see tat te slope itself canges as x canges. Jon is riving is car on a roa along te prairie. His istance travele in miles is efine by a function f(x) = 6x ; were x is te time in ours. For example, if Jon s been riving for 5 ours, ten e s travele f 5 = 6 5 ; = 150 miles. We can use te formal efinition of te erivative in orer to figure out ow Jon s istance canges wit respect to time in ours. We re essentially calculating Jon s spee in miles per our. f, x f x + f x 6(x + ) ; 6x ; 6x ; + 12x + 6 ; 6x ; 6x ; + 12x + 6 ; 6x ; 12x + 6 ; 12x 12x + lim 6 = 12x + lim 6 ; Hence, Jon s spee is efine by f (x) = 12x. After 5 ours of riving, Jon s spee is 60 miles per our. If tis example mae no sense, ten I suggest spening more time reviewing your pre-calculus an calculus basics from te references I liste at te en of te capter. Tere are usually tricks an tecniques to take erivatives of polynomials, logaritmic an trigonometric function an it is in your best interest to memorize tem (Table 2.2). Finally, to complete te review of erivatives, know te four rules: power rule, cain rule, prouct rule, an quotient rule (Table 2.1).

3 Prouct Rule x f x g(x) = f, x g x + f x g (x) Quotient Rule f x x g x = fc x g x D f x g C x g x 2 Power Rule x xn = nx nd1 Cain Rule x f(g(x)) = f g(x )g (x) Table 1: Te prouct, quotient, power an cain rule. sin x = cos (x) x cos x = sin (x) x x tan x = sec2 (x) sec x = sec x tan(x) x csc x = csc x cot (x) x x cot x = csc; (x) x ex = e x x ln x = 1 x x c = 0 Table 2: Common erivatives. Integral calculus eals wit taking te antierivatives or integral of a function. As te name suggests, we can essentially o te reverse of te previous example: fining te istance

4 Jon s travele given te spee between a given interval [a, b]. Te integral is better unerstoo if we ivie te interval [a, b] into n subintervals of wit x. If we let x [ be te starting point witin te interval [x [, x [ + x], ten te integral of a function g(x) witin an interval [a, b] is: f(x) = \ g x x ] ^ [`a g(x [ ) x were x = \D] ^. g(x) an x are calle te integran, an te ifferential, respectively. Te integral can be viewe as a summation of te area of mini-rectangles of areas g(x [ ) x; ence, it is te area uner te curve. g(x) x Since te integral is an antierivative of a function, te following integrals in te table below look analogous to te erivatives in Table 2.2. a x = ax + C x n x = 1 n + 1 xnc1 + C x n x = 1 n + 1 xnc1 + C x D1 x = ln x + C e * x = e * + C cos (x) x = sin (x) + C

5 sin(x) x = cos(x) + C sec ; (x) x = tan(x) + C tan(x) x = ln sec x + C Table 3: Common Integrals Say we ave a function f x = x ; + 1. I ope everyone knows wat tis function looks like. Using te Power Rule from te Table 2.1. We easily work out te erivative to be f, x = 2x. Let s plot bot of tem on te same grap an see wat s going on. f(x) 4 4 f (x) f, ( 1) = f, (2) = f, (0) = 0-4 Figure 2: Plot of f(x) against its erivative f'(x). Tangent slope lines are rawn on te rigt plot to furter illustrate te concept bein instantaneous rate of cange. -4 As te quaratic function f(x) ecreases (re), witin te interval ( 2 < x < 0), its erivative (blue) is negative. As f(x) increases between (0 < x < 2), te erivative is positive. Te more vertical te tangent line, te larger te slope (erivative) an te more orizontal te tangent line, te lower te slope. So since te erivative is te rate of cange, ten a small erivative at a particular point inicates te function is canging slowly. A large erivative inicates te

6 function is canging fast. Note, a negative erivative implies our function is ecreasing at tat point, but it says noting about ow fast it s ecreasing. Do not confuse a negative-value erivative wit ow fast or slow a function grows. Te most important ting to notice ere is tat at one point, x = 0, te slope is zero, inicate by a orizontal tangent line. At tis point, te function is neiter increasing nor ecreasing (tink about wat tat means). SECTION 2.2 OPTIMIZATION Our essential objective in macine learning is not only to select a moel, but optimize it as well. Since our ata serves as input, an our preiction is an output, our moel can ten be efine as a function f(x). In tis section we look at ow ifferential calculus allows us to optimize our moel. Wat oes it mean to optimize? Imagine I aske you to sell a book for me at an optimal price. Your objective woul be to fin a price tat buyers woul be bot: able to affor; an not question te books legitimacy. An expensive book is problematic ue to financial constraints. An extremely ceap book woul make buyers woner wy is te book so low-value monetarily. Witout selling anyting yet, if you moele te purcases against possible prices, your moel woul look someting like te plot on te left in Figure 1. Simply looking at te peak of te curve makes it obvious to locate te optimal price x. Tis type of function, ue to its sape, can be referre to as a concave function. x Figure 3: Te left plot sows a teoretical relationsip between te price of a book an te buyers. We can see te optimal price is were te number of buyers is te igest (left). Te two graps on te rigt sow a simple example of a convex an concave curve.

7 A concave function is a function suc tat if you connecte a line between two points on te curve, ten te line woul lie below te curve. Oppositely, we say a function is convex if a connecte line between two points on te curve lies above tat curve. Te most common example of a convex function woul a quaratic function, (e.g. f x = x ; ). Also note tat te negative of a convex function is a concave function, since we re essentially flipping it. Te actual formal efinitions of te convexity/concavity conitions for a function will not be covere in tis section or book; references are liste at te en of te section. Te minimum an maximum are te smallest an largest values for a given function, respectively. Wen te entire omain space ( < x < ) of a function f(x) is concerne, te smallest value is known as te global minima; an te largest value is known as te global minima. Te smallest an largest values witin a specific interval [a, b] are known as te local minima an te local maxima, respectively. Te plot below illustrates tese concepts (Figure 3). Now back to optimization. Say we ave access to training ata wic inclues a set of N inputs efine by {x [ } a o an teir respective true values {y [ qrst } a o. Our moel f(x) will train on te input ata an attempt to preict values close to te true values. Tis is te stanar training process we iscusse in Capter 1 for supervise learning. We enote our moel s preictions for eac N ata point as {y [ urt) } a o. Ten for a given ata point x [, error can be efine as te square ifference between te moel s preiction an te true values: E x [ = (y [ qrst y [ urt) ) ;.

8 To calculate our moel s overall error, we can take te mean of te square ifference over all ata points (often calle te mean square error or MSE): E wxy = o (y qrst [ y urt) z`a [ ) ;. If tis is our error, ten our obvious goal woul be to minimize it. Well it was easy to fin te minimum of a function wen we coul visualize it, but ow o we fin it analytically? Calculus! Wen te erivative of a function is zero at a given point x = a, ten te function is not canging at tat point (f, a = 0). Tis was mae obvious from Example an Figure 2. Hol on to tat fact an imagine two ifferent scenarios: trowing a ball from te groun to te air an watcing it lan; an watcing a penulum ball swing back an fort. If we were to plot te trajectory of te motion in eac scenario, te trown ball looks like a concave function, an te penulum looks like a concave function. If we rew tangent lines, te ball reaces maximum eigt wen te tangent line is orizontal. An wat oes a orizontal tangent line imply? Te erivative is zero! We can say te same for for te penulum curve wen it reaces its minimum eigt. Tis concept is extremely important. Te erivative of a

9 smoot ifferentiable function f(x) in a close interval, is zero at global/local minimums an maximums. Now back to te mean square error. It goes witout proving tat te sum of quaratic functions is quaratic as well. For te error function E wxy = (y qrst [ y urt) [ ) ;, if we let f [ = y qrst [ y urt) o ; [, ten we may write te error function as a sum of squares: E wxy = z`a f [. Wit respect to te preiction an true values, tis mean square error is always convex 1. We will en tis capter ere for now on tat note. I ope tis capter serve as a refreser capter for tose wo ve stuie calculus or a some exposure to it. We will apply ifferential an integral calculus later on in te book. Please complete te capter review question if you feel saky wit te concepts. Next capter we will look at our first macine learning algoritm! o z`a 1 Te MSE is always convex wit respect to y [ qrst an y [ urt). Tis statement is not always true if y [ urt) is parameterize by anoter parameter tat affects it s beavior (e.g. neural networks).