LINEAR ALGEBRA - CHAPTER 1: VECTORS

LINEAR ALGEBRA - CHAPTER 1: VECTORS A game to introduce Linear Algebra In measurement, there are many quantities whose description entirely rely on magnitude, i.e., length, area, volume, mass and temperature. Alternatively, there are quantities whose magnitude do not entirely determine their nature, velocity, acceleration and force all depend on the magnitude and their direction to be described fully. Geometrically these are represented as arrows or a directed line segment of a specific length. Originally introduced in the nineteenth century, vectors have a wide application outside of mathematics, in particular in physics,engineering, economics, computer science, statistics and the life and social sciences. In this chapter we will introduce vectors and consider their geometric and algebraic properties. As an alternative viewpoint we will introduce a non-geometric application with particular utility to computer science. However as a first introduction we examine the rules for a simple game played on graph paper. This game will be a racing game, and hence requires a track with starting and finishing lines, the track may be as complicated as one likes as long as it may accommodate each player at the starting line. As an example, we will play this game with two players A and B. To start, both players begin by drawing a point on the starting line to indicate where they are at time zero. They then take turns moving to a new point subject to the rules [1]: (1) Each point and the line segment connecting it to the previous point must lie entirely within the track. (2) No two players may occupy the same point on the same turn. (3) Each new move is related to the previous move as follows: If a player moves a units horizontally and b units vertically on one move, then on the next move the player must move between a 1 and a + 1 units horizontally and between b 1 and b + 1 units vertically. 1

2 LINEAR ALGEBRA - CHAPTER 1: VECTORS Figure 1. A sample game of Race track. To better understand this game, there are some questions we can ask about it: Question Use the notation [a, b] to denote a move that is a units horizontally and b units vertically. If [3, 4] has been made, what is the region containing the points that can be reached on the next move? Question What is the net effect of two successive moves? Question Assuming player A starts at the origin (0, 0) can we express their moves using the [a, b] notation? If the axes were translated so that player A begins at (2, 3) what is the coordinate of the final point on player A s trajectory? This game actually contains the core ideas of this section, as it contains an algebraic and geometric interpretation of vectors that may be generalized to higher dimensions than a two-dimensional plane. Vectors as geometric and algebraic objects Vectors in two-dimensions: the plane. To introduce the concepts of vectors, we start by considering a familiar example, the Cartesian plane. A vector is defined as a line between two points in the plane along with a direction, i.e, a directed line segment. The vector from A to B is denoted AB - the point A is called the initial point or tail, while B is the terminal point or head. Frequently the labels for the points are omited and instead we denote a vector in bold as v or v in the hand-written figures. The collection of all points in the plane is equivalent to the set of all vectors whose tails are at the origin O = (0, 0); for each point A, there is a corresponding vector a = OA and vice versa. It is always helpful to work with coordinates, one

LINEAR ALGEBRA - CHAPTER 1: VECTORS 3 may give a precise representation of vectors. For example, in the following figure A = (1, 3) while B = (4, 5) yielding the vector AB = [3, 2], alternatively the point C = ( 2, 2) gives the vector OC = [ 2, 2]. Although the zero vector cannot be drawn, it is perfectly acceptable to include it as a vector as the zero vector, 0 = OO = [0, 0]. Figure 2. Vectors or line segments - both are acceptable. The individual coordinates are called the components of the vector; the position of the components is vital as the vectors with [a, b] and [b, a] a, b R will generically be different except in the trivial case with a = b. We say two vectors are equal if and only if their corresponding components are equal. In calculations [ ] it will a be convenient to use column vectors instead of row vectors, i.e. instead b of [a, b]. These [ two representations are related through the transpose operation, a [a, b] T =, due to this fact we will use both representations. The set of all b] vectors with two components will be denoted as R 2 where R is the set of all real numbers (the rational and irrational numbers). Returning to the race track game, we can interpret vectors whose tails are not at the origin in the context of the game. A vector [a, b] may be interpreted as a players first move from the origin where they travel a units horizontally and b units vertically. The same displacement may be applied with a different starting point; the following figure (3) shows two equivalent displacements:

4 LINEAR ALGEBRA - CHAPTER 1: VECTORS Figure 3. A vector in standard position and the same vector at different points for its tail. We say two vectors are equal if they have the same length and direction. In figure (3) AB = CD. In a geometric sense, two vectors are equal if one can translate one vector so that it overlaps the other. We will say a vector such as OB with a tail at the origin is said to be in standard position. Furthermore we now know that every vector in the plane can be drawn as a vector in standard position and any standard position vector may be translated so that its tail is at any point. Example 0.1. Q: If A = ( 1, 1) and B = (2, 3), find AB and redraw this vector in standard position and with its tail at the point C = (2, 2) A: Calculating the difference in the components AB = [2 ( 1), 3 1] = [3, 2]. Translating AB to CD: D = (3 + 2, 2 + 2) = (5, 4). Figure 4. Addition of vectors

LINEAR ALGEBRA - CHAPTER 1: VECTORS 5 One vector, two vector: new vector. If one is playing the race track game, at each turn, one must follow one vector by another. If one were to combine two moves in terms of vector notation, can we add them to get another vector? The answer is yes, in fact vector addition is one of the most basic vector operations. As an example consider u = [1, 3] and v = [3, 2] as two moves in the game, we can determine the total displacement as a third vector u + v. In figure (5) we see that the sum will be u + v = [4, 5] which may be seen geometrically From this example we may derive a simple formula for the vector sum in terms of the components: u + v = [u + v 1, u 2 + v 2 ]. Alternatively by translating u and v parallel to themselves we produce a parallelogram. Moving the vectors to standard position we have the parallelogram determined by u and v. Theorem 0.2. The Parallelogram Rule Given vectors u and v in R 2, their sum u + v is the vector in standard position along the diagonal of the parallelogram determined by the vectors. Figure 5. The Parallelogram Rule - when it looks like a square. Example 0.3. Q: If u = [2, 3] and v = [4, 2] computer their sum and draw it. A: Adding the components of the vectors, u + v = [2 + 4, 3 + ( 2)] = [6, 1]. Using the parallelogram rule we have: Figure 6. The sum of u and v.

6 LINEAR ALGEBRA - CHAPTER 1: VECTORS The next vector operation is scalar multiplication; given a vector v and a real number c, the scalar multiple cv. This is computed by multiplying each component of the vector by c: cv = c[v 1, v 2 ] = [cv 1, cv 2 ] 1 Example 0.4. Q: If v = [1, 3] what is 2v, 2 v and 1 2v. Draw these. A: These quantities are easily calculated using coordinates, 2v = [2, 6], 1 2 v = [1 2, 3 2 ], 1 2 v = [ 1 2, 3 2 ]. Figure 7. Scalar multiples of v. Notice that cv has the same direction as the original vector if c > 0 and the opposite direction if c < 0; furthermore cv is c times as long as v. In the context of vectors, constants will be called scalars. Taking into account that we may always translate vectors, we say they are parallel if and only if they are scalar multiples of each other. Combining these two operations we may now define vector subtractions as u v = u + ( 1)v Figure (8) in the following example shows u v will be the other diagonal of the parallelogram determined by u and v. Example 0.5. Q: If u = [3, 1] and v = [2, 3] what is u v? A: Choosing coordinates the sum will be u + ( v) = [3 2, 1 3] = [1, 2].

LINEAR ALGEBRA - CHAPTER 1: VECTORS 7 Figure 8. Geometric derivative of u v. Vectors in R n. By adding another component to our vectors we may extend the work done to three dimensions, by considering the ordered triples of real numbers denoted as R 3 Points and vectors may now be defined by choosing coordinates and identifying their position on the x, y and z axes. One may verify that vector addition and scalar multiplication behave as one would expect by actually drawing the usual two-dimensional immersion of R 3. If we want to work in spaces like R n with n > 3, we can no longer resort to drawing the vectors and points in space. Instead, we must resort to more symbolic calculations, we define R n as the set of all ordered n-tuples of real numbers written as either row or column vectors, [v 1, v 2,..., v n ] or v 1 v 2.... v n If i [1, n] we say v i is the i-th component. Then the addition of two vectors u and v consists of adding each component, u i + v i, while scalar multiplication is then cu i. As we can no longer draw n-dimensional vectors, we must have some sure way to calculate with vectors. To do so we explore their algebraic properties. As an example, we notice that addition is commutative that is u + v = v + u for any two vectors. In two and three dimensions this may be seen and verified directly, in higher dimensions one must resort to the formulas to prove this fact. In this manner we may prove the following theorem listing the algebraic properties of vectors. Theorem 0.6. Let u, v and w be vectors in R n, and c and d be scalars. (1) u+v= v+u(commutativity) (2) (u+ v)+w= u+(v+w) (Associativity) (3) u+ 0 = u (4) u+ (-u) = 0 (5) c(u+v)=cu+ cv(distributivity) (6) (c+d)u= cu+ du(distributivity) (7) c(du) = cd u (8) 1u= u

8 LINEAR ALGEBRA - CHAPTER 1: VECTORS + 0 1 0 0 1 1 1 0 * 0 1 0 0 0 1 0 1 Here is an example to illustrate the utility of Theorem (0.6) for algebraic computations of vectors Example 0.7. Q: Let uand vand wdenote vectors in R n. Simplify the expression, 3u + (5v 2u) + 2(v u) and solve for w from the expression 5w u = 2(u + 2w). A: In the first case by applying the above rules we find this may be written as 7v u. In the second case this first becomes 3w = 3u, then dividing by 3 yields w = u. Linear Combinations of vectors. If a vector may be expressed as a sum of scalar multiples of other known vectors, we say this is a linear combination of those vectors. More formally we have the definition Definition 0.8. A vector v is a linear combination of vectors v 1, v 2,..., v n if there are scalars c 1, c 2,..., c n such that v = c 1 v 1 + c 2 v 2 +... + c n v n. The scalars c i are called the coefficients of the vectors in the linear combination. Example 0.9. The vector 1 3 is a linear combination of 0 1, 2 3 and 1 3 as: 2 1 1 4 1 0 2 1 3 = 3 1 + 3 3. 2 1 1 4 [ [ 2 1 Example 0.10. If u = and v = we can express any vector in R 1] 4] 2 in terms [ [ 1 0 of u and v, instead of the usual vector basis e 1 = and e 0] 2 =. For example 1] if we suppose w = 2u + v we see that [ ] 3 w = = 3e 2 1 + 2e 2 Modular Arithmetic and Binary Vectors. In computer science one often encounters a vector which has no geometric interpretation. As a computer represents data in terms of 1s and 0s, binary vectors are vectors each of whose components is a 0 or a 1. In this setting the usual rules of arithmetic must be changed as any calculation must yield a 0 or a 1. The group tables for addition and multiplication are simply: Notice here that 1 + 1 = 0, this may appear odd, however if we say 0 corresponds to even and 1 to odd these tables summarize the fact the usual parity

LINEAR ALGEBRA - CHAPTER 1: VECTORS 9 + 0 1 2 0 0 1 2 1 1 2 0 2 2 0 1 * 0 1 2 0 0 0 0 1 1 1 2 2 0 2 1 rules. i.e. the sum of two odd or even numbers is even, or the sum of an even and odd number is odd. With these rules and the set {0, 1} is denoted by Z 2 and we call this the set of integers modulo 2. More generally we call the order n-tuples with components in Z 2 binary vectors of length n and denote this as Z n 2. Example 0.11. The vectors in Z 2 2 consist of [0, 0], [0, 1], [1, 0] and [1, 1]. For Z n 2 we have n 2 vectors. This idea can be extended to produce ordered n-tuples whose components are from a finite set {0, 1, 2,...k}, k > 1. To do so we recall the concept of a remainder, for any integer n we may write n = mk + l where m is some integer and 0 l < k; by ignoring the mk term, the binary operations of addition and multiplication produce elements in the original set - the operations are closed. We will denote this as Z k and call it the integers modulo k. For technical reasons which will not be stated here, we will restrict k to prime numbers to discuss vectors whose components belong to Z k, called k-ary vector of length n. Example 0.12. Consider the integers modulo 3, Z 3 = {0, 1, 2} with addition and multiplication tables: Example 0.13. Q: 16661 is the 1928th prime, what is its value in Z 3? A: Since 16661 = 5553*3+2, we conclude that this number is equivalent to 2 in Z 3. Example 0.14. Q: In Z 5 3, let u = [1, 1, 2, 1, 2] and v = [0, 1, 2, 2, 0], computer their sum. A: Adding each component and using the addition and multiplication tables, u+v = [1, 2, 1, 0, 2]. Length and Angle via the Dot Product Since vectors have a magnitude and a direction, the familiar ideas of length, distance and angle may be expressed in terms of vectors. In fact by generalizing these ideas from two and three dimensions we may define these quantities independent of dimension.

10 LINEAR ALGEBRA - CHAPTER 1: VECTORS The dot product of two vectors. The vector equivalent of length, distance and angle depend on the concept of a dot product Definition 0.15. If u T = [u 1, u 2,..., u n ] and v T = [v 1, v 2,..., v n ] then the dot product u v of u and v is defined as u v = u 1 v 1 + u 2 v 2 +... + u n v n. So the dot product of u and v is the sum of the products of the components of these two vectors. Unlike addition and scalar multiplication this operation takes two vectors and produces a scalar. Furthermore this operation is defined only for vectors with the same number of components. Example 0.16. Q: Compute u v where u = 1 1, v = 3 5. A: u v = 2 4 1( 3) + ( 1)5 + 2(4) = 0. The dot product is commutative, as the dot product of v and u in this case because v u = 0. However, this will happen if u v 0 because the components belong to R which is commutative. Again, knowing the properties of the dot product will facilitate calculations with vectors. Theorem 0.17. Let u, v and w be vectors in R n and let c be a scalar. The dot product satisfies: (1) u v = v u (2) u (v + w) = u v + u w (3) (cu) v = c(u v) (4) u u 0 and u u = 0 if and only if u = 0 Example 0.18. The dot product of the sum of two vectors may be simplified: (u + v) (u + v) = (u + v) u + (u + v) v = u u + v u + u v + v v = u u + u v + u v + v v = u u + 2u v + v v Length. To illustrate how an idea of length may be derived from the dot product, we return to R 2 and the Pythagoras theorem. Here, the length of a vector u T = [a, b] is the distance from the origin to the head of the vector at (a, b). Using Pythagoras, we see that the distance is a 2 + b 2. Noting that u u = a 2 + b 2 we have a simple definition for distance Definition 0.19. The length or norm of a vector u T = [u 1, u 2,..., u n ] in R n is the non-negative scalar u given by u = u u = u 2 1 + u2 2 +... + u2 n Example 0.20. Consider the length of the vector u T = [0, 0, 4, 3], u = 0 2 + 0 2 + 4 2 + 3 2 = 25 = 5 Using the properties of the dot product we have two helpful facts for the length of an arbitrary vector:

LINEAR ALGEBRA - CHAPTER 1: VECTORS 11 Theorem 0.21. Let u be a vector in R n and c a scalar. (1) u = 0 if and only if u = 0 (2) cu = c u We call any vector of length 1 a unit vector, in R 2 and R 3 the set of all unit vectors can be identified with the unit circle and unit sphere respectively (i.e. with radius 1 centered at the origin). Given any non-zero vector u, we may produce a unit vector v by taking the norm of the vector and dividing each component, i.e. v = 1 u u. It is easily verified that this is indeed a unit vector u = 1 u u = 1 Notice that v is in the same direction as the original vector as u > 0 We call this procedure normalizing a vector. Figure 9. Unit vectors in the plane. Example 0.22. Q: Normalize the vector, u T = [0, 3, 4] A: The norm of u is u = 5 and so v = u u = 1 [0, 3, 4]. 5

12 LINEAR ALGEBRA - CHAPTER 1: VECTORS Figure 10. Normalizing a vector As the rule describing length and its change under scalar multiplication, one wonders if length and vector addition produce a similar rule. For example, when is the identity u + v = u + v true? For almost any choice of u and v this identity will not hold. Instead if we replace the equality for an inequality: u + v u + v this identity holds true, and is called the Triangle inequality. In two and three dimensions the triangle inequality may be checked visually using geometry. To prove this rigorously for any dimension, we must introduce another inequality Theorem 0.23. The Cauchy-Schwarz Inequality: For all vectors u and v in R n, Invoking this inequality we have u v u v Theorem 0.24. The Triangle Inequality: For all vectors u and v in R n, u + v u + v Proof. Both sides of the inequality are non-negative, implying that the square of one side is less than or equal to the square of the other side. Thus we may compute u + v 2 = (u + v) (u + v) = u u + 2u v + v v u 2 + 2 u v + v 2 Taking the square root completes the proof. u 2 + 2 u v + v 2 = ( u + v ) 2 Distance. Just as how vectors are directed line segments between points, distance between two vectors is the direct analogue of the distance between two points on the real number line R or two points in R 2. In R the distance between points A and B is simply B A (distances must be positive, so we must use the absolute value).

LINEAR ALGEBRA - CHAPTER 1: VECTORS 13 Figure 11. Normalizing a vector u. In R 2 the distance between A = (a 1, a 2 ) and B = (b 1, b 2 ) is simply d = (b 1 a 2 1 + (b 2 a 2 ) 2 Figure 12. Distance on the plane. In terms of vectors u T = [a 1, a 2 ] and v T = [b 1, b 2 ] then the distance will just be the length of the vector u v. Definition 0.25. The distance d(u, v) between two vectors in R n is defined by d(u, v) = u v. Example 0.26. Q: Find the distance between u T = [ 3, 1, 3] and v T = [0, 1, 2] A: Computing (u v) T = [ 3, 0, 1] we find that d(u, v) = 3 + 0 + 1 = 2. Angles. Just as the dot product is used to calculate length and distances in terms of vectors, it may be used to calculate the angle between a pair of vectors. In two and three dimensions, the angle between two non-zero vectors will refer to the angle θ [0, 2π] between the two vectors.

14 LINEAR ALGEBRA - CHAPTER 1: VECTORS Figure 13. The angle between u and v. Figure 14. Always look for the angle θ [0, π] Consider the triangle with sides u, v and u v given in figure (13), we denote the angle between u and v as θ. Applying the law of cosines to this triangle we find u v 2 = u 2 + v 2 2 u v cosθ then by expanding the left hand side and noting v 2 = v v we obtain u 2 2(u v) + v 2 = u 2 + v 2 2 u v cosθ. Simplifying we find that u v = u v cosθ, we have found a simple formula for θ in terms of dot products of vectors. Definition 0.27. For any two non-zero vectors u and v in Rn the angle between them is defined as cosθ = u v u v Example 0.28. Q: Compute the angle between the vectors ut = [1, 1, 0] and vt = [1, 0, 0]. A: Calculating u v = 1, u = 2 and v = 1. Therefore cosθ = 12 implying that θ = π4.

LINEAR ALGEBRA - CHAPTER 1: VECTORS 15 θ 0 π 6 π 4 π 3 π 2 3 2 1 2 1 2 0 cosθ 1 Table 1. Cosines of Special Angles Orthogonal vectors. So far we have seen that vectors which are parallel to each other must be scalar multiples of each other. With the dot product we say that these vectors have θ = 0 or π. What about perpendicular vectors? This is an important tool in geometry, and so it will be helpful to generalize this concept to vectors in R n. In two and three dimensions two non-zero vectors u and v are perpendicular if the angle between them is a right angle, θ = π 2. Applying the formula for θ this implies u v u v = 0 Thus the dot product of u and v must vanish. Definition 0.29. Two vectors u and v in R n are orthogonal if u v = 0 Notice that 0 u = 0, so that every vector in R n is orthogonal to the zero vector. Example 0.30. Consider the basis vectors in R 3, e T 1 = [1, 0, 0] and e T 2 = [0, 1, 0], clearly the dot product of these two is zero. What of u = e 1 e 2 and v = e 1 + e 2? To illustrate the utility of orthogonal vectors, we easily prove Pythagoras theorem in arbitrary dimension, Theorem 0.31. For all vectors u and v in R n, u + v 2 = u 2 + v 2 if and only if u and v are orthogonal. Proof. Noting that u+v 2 = u 2 +2(u v)+ v 2, if these vectors are orthogonal the term, u v = 0 giving the desired equality. As an application of this idea, we use this to find the distance from a line to a point in R n. Projection of a Vector onto Another. In two dimensions, figure (15) summarizes the problem of finding the distance from a point B to a line L. This can always be reduced to finding the length of the perpendicular line segment from P to B, or alternatively the length of the vector AB. Picking another point A on L we may draw a right-angled triangle AP B with two new vectors AP and AB. We say AP is the projection of AB onto the line L. We will now interpret this in terms of n-dimensional vectors.

16 LINEAR ALGEBRA - CHAPTER 1: VECTORS Figure 15 Given two non-zero vectors u and v, let p be the vector obtained by dropping a perpendicular from the head of v onto u and denote θ as the angle between u p u, furthermore using simple and v as in figure (16). We may express p as p = u u v trigonometry, p = v cosθ where cosθ = u v. Combining these facts we find u v p= u, u u from this derivation we now have a helpful tool. Definition 0.32. If u and v are vectors in Rn and u 6= 0 then the projection of v onto u is the vector proju (v) defined by u v proju (v) = u. u u Returning to the question of the distance from the point B to a line `, we see that the shortest distance between B and ` is the magnitude of the vector or directed line segment between B and P in figure (16); if we could calculate its magnitude we would have the distance. Since we know v and can calculate p = AP = projd (v), The vector PB will be the difference: PB = v p, and taking the magnitude gives us the distance. Figure 16 Definition 0.33. The distance d(b, `) between a point B and a line ` in R2 is defined as the magnitude of the vector: d(b, `) = projd (v)

LINEAR ALGEBRA - CHAPTER 1: VECTORS 17 where v = AB is the directed line segment between B and another point on l and d is the direction vector of l. References [1] D. Poole, Linear Algebra: A modern introduction - 3rd Edition, Brooks/Cole (2012).