1. Vectors. notation. examples. vector operations. linear functions. complex vectors. complexity of vector computations

Transcription

1 L. Vandenberghe ECE133A (Winter 2018) 1. Vectors notation examples vector operations linear functions complex vectors complexity of vector computations 1-1

2 Vector a vector is an ordered finite list of numbers we use two types of notation: vertical and horizontal arrays; for example = ( 1.1, 0.0, 3.6, 7.2) numbers in the list are the elements (entries, coefficients, components) number of elements is the size (length, dimension) of the vector a vector of size n is called an n-vector set of n-vectors with real elements is denoted R n Vectors 1-2

3 Conventions we usually denote vectors by lowercase letters a = a 1 a 2. a n = (a 1, a 2,..., a n ) ith element of vector a is denoted a i i is the index of the ith element a i Note several other conventions exist we ll make exceptions, e.g., a i can refer to ith vector in a collection of vectors Vectors 1-3

4 Block vectors, subvectors Stacking vectors can be stacked (concatenated) to create larger vectors example: stacking vectors b, c, d of size m, n, p gives an (m + n + p)-vector a = b c d = (b 1,..., b m, c 1,..., c n, d 1,..., d p ) other notation: a = (b, c, d) Subvectors colon notation can be used to define subvectors (slices) of a vector example: if a = (1, 1, 2, 0, 3), then a 2:4 = ( 1, 2, 0) Vectors 1-4

5 Special vectors Zero vector and ones vector 0 = (0, 0,..., 0), 1 = (1, 1,..., 1) size follows from context (if not, we add a subscript and write 0 n, 1 n ) Unit vectors there are n unit vectors of size n, written e 1, e 2,..., e n ith unit vector is zero except its ith element which is 1; for n = 3, e 1 = 1 0 0, e 2 = 0 1 0, e 3 = size of e i follows from context (or should be specified explicitly) Vectors 1-5

6 Outline notation examples vector operations linear functions complex vectors complexity of vector computations

7 Location and displacement Location: coordinates of a point in a plane or three-dimensional space x 2 x 0 x 1 Displacement: shown as arrow in plane or 3-D space x 2 x x 1 other quantities that have direction and magnitude, e.g., force vector Vectors 1-6

8 Resource vector, portfolio vector Resource vector elements of n-vector represent quantities of n resources or commodities sign indicates whether quantity is held or owed, produced or consumed,... example: bill of materials gives quantities needed to create a product Portfolio n-vector represents stock portfolio or investment in n assets ith element is amount invested in asset i elements can be no. of shares, dollar values, or fractions of total dollar amount Vectors 1-7

9 Signal or time series elements of n-vector are values of some quantity at n different times hourly temperature over period of n hours xi ( F) i daily return of a stock for period of n trading days cash flow: payments to an entity over n periods (e.g., quarters) Vectors 1-8

10 Monochrome (black and white) image Images, video grayscale values of M N pixels stored as MN-vector (e.g., row-wise) x = x 1 x 2 x 3.. x 62 x 63 x 64 = Color image: 3MN-vectors with R, G, B values of the MN pixels Video: vector of size KMN represents K monochrome images of M N pixels Vectors 1-9

11 Word count vectors vector represents a document size of vector is number of words in a dictionary word count vector: element i is number of times word i occurs in the document word histogram: element i is frequency of word i in the document Example Word count vectors are used in computer based document analysis. Each entry of the word count vector is the number of times the associated dictionary word appears in the document. word in number horse document Vectors 1-10

12 Feature vectors contain values of variables or attributes that describe members of a set Examples age, weight, blood pressure, gender,..., of patients square footage, #bedrooms, list price,..., of houses in an inventory Note vector elements can represent very different quantities, in different units can contain categorical features (e.g., 0/1 for male/female) ordering has no particular meaning Vectors 1-11

13 Polynomials and generalized polynomials a polynomial of degree n 1 or less f(t) = c 1 + c 2 t + c 3 t c n t n 1 can be represented by an n-vector (c 1, c 2,..., c n ) Extensions n basis functions f 1 (t),..., f n (t) n-vector c represents the function f(t) = c 1 f 1 (t) + + c n f n (t) example: the cosine polynomial f(t) = c 1 + c 2 cos t + c 3 cos(2t) + + c n cos((n 1)t) can be represented by an n-vector (c 1, c 2,..., c n ) Vectors 1-12

14 Outline notation examples vector operations linear functions complex vectors complexity of vector computations

15 Addition and subtraction a + b = a 1 + b 1 a 2 + b 2. a n + b n, a b = a 1 b 1 a 2 b 2. a n b n commutatitive associative a + b = b + a a + (b + c) = (a + b) + c a + b a b Vectors 1-13

16 Scalar-vector and componentwise multiplication Scalar-vector multiplication: for scalar β and n-vector a, β a 1 a 2. a n = βa 1 βa 2. βa n Component-wise multiplication: for n-vectors a, b a b = a 1 b 1 a 2 b 2. a n b n (in MATLAB: a.* b) Vectors 1-14

17 Linear combination a linear combination of vectors a 1,..., a m is a sum of scalar-vector products β 1 a 1 + β 2 a β m a m the scalars β 1,..., β m are the coefficients of the linear combination a 2 = (2, 2) 3 4 a a 2 = (6, 3) a 1 = (4, 0) Vectors 1-15

18 Inner product the inner product of two n-vectors a, b is defined as a T b = a 1 b 1 + a 2 b a n b n a scalar meaning of superscript T will be explained when we discuss matrices other notation: a, b, (a b),... (in MATLAB: a * b) Vectors 1-16

19 Properties for vectors a, b, c of equal length, scalar γ a T a = a a a 2 n 0 a T a = 0 only if a = 0 commutative: a T b = b T a associative with scalar multiplication: (γa) T b = γ(a T b) distributive with vector addition: (a + b) T c = a T c + b T c Vectors 1-17

20 Simple examples Inner product with unit vector e T i a = a i Differencing (e i e j ) T a = a i a j Sum and average 1 T a = a 1 + a a n ( 1 n 1)T a = a 1 + a a n n Vectors 1-18

21 Examples Weighted sum f is vector of features w is vector of weights inner product w T f = w 1 f 1 + w 2 f w n f n is total score Cost p is vector of prices of n goods q is vector of quantities purchased inner product p T q = p 1 q 1 + p 2 q p n q n is total cost Vectors 1-19

22 Examples Discounted total c is a cash flow over n periods d is vector of discount factors assuming interest rate r 0: d = (1, r, 1 (1 + r) 2,..., 1 (1 + r) n 1) inner product d T c is discounted total or net present value of cash flow d T c = c 1 + c r + c 3 (1 + r) c n (1 + r) n 1 Vectors 1-20

23 Examples Portfolio return h is portfolio vector, with h i the dollar value of asset i held r is vector of fractional returns over the investment period: p init i and p final i r i = pfinal i p init i p init i, i = 1,..., n are the prices of asset i at the beginning and end of the period r T h = r 1 h r n h n is the total return, in dollars, over the period Polynomial evaluation c is vector of coefficients of f(t) = c 1 + c 2 t + c 3 t c n t n 1 x = (1, u, u 2,..., u n 1 ) is vector of powers of u at some given u c T x = c 1 + c 2 u + + c n u n 1 is value f(u) of polynomial at u Vectors 1-21

24 Outline notation examples vector operations linear functions complex vectors complexity of vector computations

25 Linear function a function f : R n R is linear if superposition holds: for all n-vectors x, y and all scalars α, β f(αx + βy) = αf(x) + βf(y) (1) Extension: if f is linear, superposition holds for any linear combination: f(α 1 u 1 + α 2 u α m u m ) = α 1 f(u 1 ) + α 2 f(u 2 ) + + α m f(u m ) for all scalars α 1,..., α m and all n-vectors u 1,..., u m (this follows by applying (1) repeatedly) Vectors 1-22

26 Inner product function for fixed a R n, define a function f : R n R as f(x) = a T x = a 1 x 1 + a 2 x a n x n any function of this type is linear: a T (αx + βy) = α(a T x) + β(a T y) holds for all scalars α, β and all n-vectors x, y every linear function can be written as an inner-product function: f(x) = f(x 1 e 1 + x 2 e x n e n ) = x 1 f(e 1 ) + x 2 f(e 2 ) + + x n f(e n ) line 2 follows from superposition Vectors 1-23

27 Examples in R 3 f(x) = 1 3 (x 1 + x 2 + x 3 ) is linear: f(x) = a T x with a = ( 1 3, 1 3, 1 3 ) f(x) = x 1 is linear: f(x) = a T x with a = ( 1, 0, 0) f(x) = max{x 1, x 2, x 3 } is not linear: superposition does not hold for x = 1 0 0, y = 0 0 0, α = 1, β = 1 we have f(x) = 1, f(y) = 0, f(αx + βy) = 0 αf(x) + βf(y) = 1 Vectors 1-24

28 Exercise F (t) x 1 x 2 x 7 x 9 F (t) x 3 x 4 x 6 x 8 x 10 x t unit mass with zero initial position and velocity apply piecewise-constant force F (t) during interval [0, 10): F (t) = x j for t [j 1, j), j = 1,..., 10 define f(x) as position at t = 10, g(x) as velocity at t = 10 are f and g linear functions of x? Vectors 1-25

29 Solution from Newton s law s (t) = F (t) where s(t) is the position at time t integrate twice to get final velocity and position s (10) = s(10) = 10 0 F (t) dt = x 1 + x x s (t) dt = 19 2 x x x x 10 the two functions are linear: f(x) = a T x and g(x) = b T x with a = ( 19 2, 17 2,..., 3 2, 1 ), b = (1, 1,..., 1) 2 Vectors 1-26

30 Affine function a function f : R n R is affine if it satisfies f(αx + βy) = αf(x) + βf(y) for all n-vectors x, y and all scalars α, β with α + β = 1 Extension: if f is affine, then f(α 1 u 1 + α 2 u α m u m ) = α 1 f(u 1 ) + α 2 f(u 2 ) + + α m f(u m ) for all n-vectors u 1,..., u m and all scalars α 1,..., α m with α 1 + α α m = 1 Vectors 1-27

31 Affine functions and inner products for fixed a R n, b R, define a function f : R n R by f(x) = a T x + b = a 1 x 1 + a 2 x a n x n + b i.e., an inner-product function plus a constant (offset) any function of this type is affine: if α + β = 1 then a T (αx + βy) + b = α(a T x + b) + β(a T x + b) every affine function can be written as f(x) = a T x + b with: a = (f(e 1 ) f(0), f(e 2 ) f(0),..., f(e n ) f(0)) b = f(0) Vectors 1-28

32 Affine approximation first-order Taylor approximation of differentiable f : R n R around z: ˆf(x) = f(z) + f x 1 (z)(x 1 z 1 ) + + f x n (z)(x n z n ) generalizes first-order Taylor approximation of function of one variable ˆf(x) = f(z) + f (z)(x z) ˆf is a local affine approximation of f around z in vector notation: ˆf(x) = f(z) + f(z) T (x z) where f(z) = ( f x 1 (z), f x 2 (z),..., the n-vector f(z) is called the gradient of f at z ) f (z) x n Vectors 1-29

33 Example with one variable f(x) ˆf(x) z ˆf(x) = f(z) + f (z)(x z) Vectors 1-30

34 Example with two variables f(x 1, x 2 ) = x 1 3x 2 + e 2x 1+x 2 1 Gradient f(x) = [ 1 + 2e 2x 1 +x e 2x 1+x 2 1 ] First-order Taylor approximation around z = 0 ˆf(x) = f(0) + f(0) T (x 0) = e 1 + (1 + 2e 1 )x 1 + ( 3 + e 1 )x 2 Vectors 1-31

35 Regression model ŷ = x T β + v = β 1 x β p x p + v x is feature vector elements x i are regressors, independent variables, or inputs β = (β 1,..., β p ) is vector of weights or coefficients v is offset or intercept coefficients β 1,..., β p, v are the parameters of the regression model ŷ is prediction (or outcome, dependent variable) regression model expresses ŷ as an affine function of x Example ŷ is selling price of a house in a neighborhood regressors (x 1, x 2, x 3, x 4 ) = (lot size, area, #bedrooms, #bathrooms) Vectors 1-32

36 Example: house price regression model ŷ = x x 2 ŷ is predicted selling price in thousands of dollars x 1 is area (1000 square feet); x 2 is number of bedrooms scatter plot shows sale prices for 774 houses in Sacramento 800 Predicted price ŷ Actual price y Vectors 1-33

37 Outline notation examples vector operations linear functions complex vectors complexity of vector computations

38 Complex numbers Complex number: x = α + jβ with α, β real scalars j = 1 (more common notation is i or j) α is the real part of x, denoted Re x β is the imaginary part, denoted Im x set of complex numbers is denoted C Modulus and conjugate modulus (absolute value, magnitude): x = (Re x) 2 + (Im x) 2 conjugate: x = Re x j Im x useful formulas: Re x = x + x 2 x x, Im x =, x 2 = xx 2j Vectors 1-34

39 Polar representation nonzero complex number x = Re x + j Im x can be written as x = x (cos θ + j sin θ) = x e jθ θ [0, 2π) is the argument (phase angle) of x (notation: arg x) e jθ is complex exponential: e jθ = cos θ + j sin θ imaginary axis Im x x x arg x Re x real axis Vectors 1-35

40 Complex vector vector with complex elements: a = α + jβ with α, β real vectors real and imaginary part, conjugate are defined componentwise: Re a = (Re a 1, Re a 2,..., Re a n ) Im a = (Im a 1, Im a 2,..., Im a n ) ā = Re a j Im a set of complex n-vectors is denoted C n addition, scalar/componentwise multiplication defined as in R n : a + b = a 1 + b 1 a 2 + b 2. a n + b n, γa = γa 1 γa 2. γa n, a b = a 1 b 1 a 2 b 2. a n b n Vectors 1-36

41 Complex inner product the inner product of complex n-vectors a, b is defined as b H a = b 1 a 1 + b 2 a b n a n a complex scalar meaning of superscript H will be explained when we discuss matrices other notation: a, b, (a b),... for real vectors, reduces to real inner product b T a (in MATLAB: b * a ) Vectors 1-37

42 Properties for complex n-vectors a, b, c and complex scalars γ a H a 0: follows from a H a = ā 1 a 1 + ā 2 a ā n a n = a a a n 2 a H a = 0 only if a = 0 b H a = a H b b H (γa) = γ(b H a) (γb) H a = γ(b H a) (b + c) H a = b H a + c H a b H (a + c) = b H a + b H c Vectors 1-38

43 Example: power in electric networks ṽ(t) is voltage across circuit element at time t ĩ(t) is current through element + ṽ(t) ĩ(t) p(t) = ṽ(t)ĩ(t) is instantaneous power absorbed by element at time t for n elements, with voltages ṽ k (t) and currents ĩ k (t), total power is p(t) = ṽ 1 (t)ĩ 1 (t) + + ṽ n (t)ĩ n (t) the (real) inner product of two n-vectors of voltages and currents Vectors 1-39

44 Sinusoidal voltage and current assume voltage and current are sinusoids with the same frequency ṽ(t) = V cos(ωt + α), ĩ(t) = I cos(ωt + β) (with V, I 0) can be represented by complex numbers (phasors) v = V 2 e jα, i = I 2 e jβ instantaneous power at time t is p(t) = ṽ(t)ĩ(i) = V I cos(ωt + α) cos(ωt + β) = V I (cos(α β)(1 + cos 2(ωt + α)) + sin(α β) sin 2(ωt + α)) 2 = Re(īv) (1 + cos 2(ωt + α)) + Im(īv) sin 2(ωt + α) Vectors 1-40

45 Complex power p(t) = Re(īv) (1 + cos 2(ωt + α)) }{{} average Re(īv) + Im(īv) sin 2(ωt + α) }{{} average zero P = Re(iv) is called average (or real, active) power Q = Im(iv) is reactive power P + jq = iv is complex power for n elements: n-vectors of phasors v = (v 1,..., v n ) and i = (i 1,..., i n ) i H v = Re(i H v) + j Im(i H v) Re(i H v) is total average power Im(i H v) is sum of reactive powers i H v is sum of complex powers Vectors 1-41

46 Outline notation examples vector operations linear functions complex vectors complexity of vector computations

47 Floating point operation Floating point operation (flop) the unit of complexity when comparing vector and matrix algorithms 1 flop = one basic arithmetic operation (+,,, /,,... ) in R or C Comments: this is a very simplified model of complexity of algorithms we don t distinguish between the different types of arithmetic operations we don t distinguish between real and complex arithmetic we ignore integer operations (indexing, loop counters,... ) we ignore cost of memory access Vectors 1-42

48 Complexity Operation count (flop count) total number of operations in an algorithm in linear algebra, typically a polynomial of the dimensions in the problem a crude predictor of run time of the algorithm: run time number of operations (flops) computer speed (flops per second) Dominant term: the highest-order term in the flop count 1 3 n n n n3 Order: the power in the dominant term 1 3 n3 + 10n = order n 3 Vectors 1-43

49 Examples complexity of vector operations in this lecture (for vectors of size n) addition, subtraction: n flops scalar multiplication: n flops componentwise multiplication: n flops inner product: 2n 1 2n flops these operations are all order n Vectors 1-44