ECE594I Notes set 2: Math

Size: px

Start display at page:

Download "ECE594I Notes set 2: Math"

Ariel Wood
6 years ago
Views:

1 ECE594I otes set : Math Mar Rodwell University of California, Santa Barbara rodwell@ece.ucsb.edu , fa

2 Topics Sets, probabilit y, conditiona l probabilit y, Bayes theorem, independen ce. Bernoulli trials, random variables, density and distribution functions Binomial distributions. Gaussian and oisson as limiting cases. Transforma tions.

3 Set Definitions Set a collection e.g. R C of objects... { set of all real #s} { set of cards in a dec} notation : elements { } A a, b, c, d lowercase, sets uppercase Finite set : A {,,3,4} Countably ifiit infinite set : B uncountabl y infinite set : C { 0,, 4, 6, 8,... } { real #s between and 3}

4 Set Definitions Subsets : A B A is a subset of B if all elements of A are also elements of B. Strict subsets : A B A If is and if a strict titsubset of B all elements of Aare also elements of B, additionally ly B has elements which are not in A.

5 Set operations, logic operations Sets A and B, both subsets of universal set S. **Union of Sets ** C A B : set of all elements within either A or B. Boolean logic terminology : " A or B" C A + B. **Intersection of Sets ** C A B : set of all elements within bth both A or B. Boolean logic terminology : " A and B" C AB.

6 Other set operations We can also define other set operations and hence logic operations difference of sets complement s of sets etc.,...but we will asssume that the or can figure out as necessary. student nows these,

7 Eclusive Sets Key point for statistical Two sets i.e. they have are * eclusive no common independen ce. * if A B {}the empty set, elements. In probability, the events A and B are said to be mutually eclusive.

8 robability Key to the idea of probabilit y is an * eperiment *, for which there are a set of possible * outcomes *. Each possible outcome has a numerical probabilit y. The sum of the probabilit ies of all * distinct * outcomes is one something always happens.

9 Eamples of probability Toss a coin : S { heads, tails} { H, T} H T / S H + T Shoot a photon through a polarizer S { photon received, no photon received } { R, } R / S R + :

10 Events An event is a set of possible outcomes.,, 3, 4 are possible outcomes. A and B 4 are events.

11 Rules Aioms of robability Each outcome has probability between 0 and. Given an event which is the its probabilit y is the sum of union of muntually eclusive the individual probabilit ies. events, A B C A + B C + if A, B,and C are distinct events.

12 Joint robability robabilit y that both A and B occur. A B A B A + B A B

13 Conditional robability A B : probability of an event A, given the event B. A B A B B Total probability is obtained by adding up conditional probabilities : A A B + A B +... A B + B But only if the events B, B,..., B,...are mutually eclusive, and together mae up the sample {} for i j and B B B S i.e. Bi B j... space S.

14 Bayes' Theorem Handy for receiver problems. A B B A A B This follows directly from the definition of conditional probability : A B A B B

15 Bayes' Theorem: Binary Communications Channel t 0 T0 / t T / But the channel is noisy : r 0 t 0 The channel never mae a mistae r t 0 0 when a zero is sent. r 0 t /..but maes r t / when a one is sent. mistaes

16 Bayes' Theorem: Binary Communications Channel Diagram R 0 T 0, R T 0 0 R T /, R T / 0 robability that a zero is received : R R T T + R T T / + / / 3/4

17 Bayes' Theorem: Binary Communications Channel 3 R T, R T R T /, R T / 0 If a zero is received, what is the probability that a one was sent? R T T R T R / */ 3/ / 3 Before receiving the message, we now it has message has equal probabilities of being a or a 0. After receiving a zero, the odds are now /3 :/3 that a zero vs. a was sent. Our nowledge has improved, but remains imperfect.

18 Independence Two events, A and B, are statistica lly independen t if their probabilit ies of occurence are unrelated. i.e. A B A B recall : A B A B B equivalently A B A Independen ce of A and B is written as A B

heads ote that H and H are not mutually eclusive If we assert from

19 Independence: Eample coin tosses : first toss h t h t or, second toss or There are four outcomes Define two events : H first coin heads, H nd coin heads ote that H and H are not mutually eclusive If we assert from physical arguments that t the tosses are, then all 4 outcomes have 5% probabilit y.

20 Indendence does not cascade... If we have 3 events, A, B, and C, then A B and B C does not imply A C

21 Bernoulli Trials Optical Shot oise: Discrete-Time Repeatedly peform a trial with Do the trial times. Each trial : probabilit y of event binany outcomes. A is. The number of sequences in which A occurs times is # " choose "!!! So the probabilit y of p p occurences p in q trials is q p is a standard notational shortcut.

22 Bernoulli Trials Optical Shot oise: Discrete-Time Fiber : probabilit bili y of photon transmiss ion. Event T : Transmitte r sends photons for a message "", with probability Event T0 : Transmitte r sends 0 photons for a message "0", with probability /. /. assage of each photon is a Bernoulli trial. photons photons received received sent 0 0 sent 0 p p q q q 0

23 Bernoulli Trials Optical Shot oise: Discrete-Time roblem is closely related to shot - noise - limited optical lins. We are at the receiver. What rule might we use to best decide what message was sent? If you receive photons, is it more liely that a "" or "0" was sent?

24 Bernoulli Trials Optical Shot oise: Discrete-Time o o q p q p T T T T q p q p is received photons sent given was probability that a "" the So / T T T 0 0 q p 0 0 q p q p + 0!!!! 0 0 q +

25 Bernoulli Trials Optical Shot oise: Discrete-Time This at least tells us the relative odds of a "" vs. a zero having been sent, given the message we have received. Ultimately, we must still guess, with improved odds, as to the message sent. One is tempted to choose the more probable outcome directly, but the best choice depends upon the * relative costs * of the types of possible errors in guessing.

26 Random Variables Mathematic ian' s picture : A random variable is a mapping of the sample space onto the set of real numbers. Random variables must be bounded : { X ± } 0. We wish to define the probability that X falls within some range : { < X }.

27 Discrete and Continuous Random Variables Discrete R.V.' s Only a discrete set of values : finite {,,3}, or infinite {,,3,4,...} Continuous R.V.' s Taes on a continuous range of values : eample : X { real #s} We can ointer have a continuous sample space and yet a discrete : sample space R.V. X Θ { real #s between 0 and π } {,,3,4,, } R.V.

28 Cumulative Distribution Function Random variable X. articular Events : value it might tae on : { X < } or { < X }. The probabilit y of the event cumulative distribution function { X < } is the F X X Distributi on function of the random variable X. at the particular outcome value.

29 robability Distribution Function During an eperiment, a random variable X taes on a particular value. The probability that lies between and is { f X < < } f X d is the probabilit y distributi on function. f X

30 Eample: The Gaussian Distribution The f We Gaussian will distribution : ep f X ~σ X πσ σ and define shortly the mean the standard deviation σ. Because of the * central limit i theorem*, physical random processes arising from the sum of many small effects have probabilit y distributionstions close to that of the Gaussian. Gaussians also important because linear operations produce Gaussians simplified math. on Gaussians

31 Gaussian Cumulative Distribution Function π X d d f F σ σ ' ' ep ' ' π d σ β β σ ep - form answer. no closed is There π σ σ Q F ' π β β α σ d Q ep where π α

32 Error Functions Q α can be related to the more well - nown * error function * α Q α erf, but Q α is more directly useful in communcati ons problems. I will provide a good tabulatio n of Q α, but there is a very good bound for large α : α α α ep < Q α < ep + α α π α π

33 Tabulated values of the Q-function ome values of the Q-functionare given below for reference. Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q

34 Error Functions F ep πσ ' σ d' Q ' σ where Q α π α β ep dβ Q σ gives the probabilit y of the Gaussian eceeding its mean by σ standard deviations.

35 Eample: Digital Transmission thermal noise, Gaussian distributi on, adds Receiver er decides betweeneen "0" & "" depending R T + is bigger tha n or smaller th an /. to signal. on whether f T t / δ t + / δ t t is transmitt ed signal, not time f n n n ep with 0 and n σ n πσ n σ n 0.5 say

36 Eample: Digital Transmission What is the probability of error, given that a zero was sent? error T 0 R > / T 0 > / } In this / } F / n Q Q σ 0.03 contet, the signal /noise ratio is probabilit y of error. n / and Q is the

37 Bernoulli Trials Binomial Distribution Bernoulli til trials : successes trials p note that is a discrete random variable. q Call the # of sucesses, a continuous random variable f X p q δ 0...which is just a change in notation. : Important because : - arises in problems involving counting events - leads to other distributi ons as limiting cases. shot noise

38 Binomial Gaussian Bernoulli trials il : successes trials p If apoulis 965, p. 66 npq >> and if np ~ O npq or less, then : q q p n π npq ep np npq This is a Gaussian of mean np and variance npq.

39 Binomial oisson Bernoulli trials : successes trials p q If apoulis 965, p. 7 n >>, p <<, bt but np is not >>, then : p q e p p! i.e. the distribution approaches the ossion distribution : a e a!

40 Eponential Disribution f X / aep 0 a b for > b otherwise This will show up in thermally - driven distributions Boltzmann

41 Transformations of a Random Variable Suppose Y Y X, where Y X is a -function, f and given some y < Y < y < X < where y y, y y f X. If so, dy + ε, then y y + ε d dy ε fy y ε f X d or, more clearly f y Y f X dy d

42 Transformations of a Random Variable This assumed a single range a -function, i.e. each range in Y corresponds in X. This may not always be true... to Consider y c : Then the regions ~ and ~ - both map to y ~ c. The density function for y is then f Y f X f X y + c c y / c y / c f X y / c + f y / c X cy

43 References and Citations: Sources / Citations : Kittel and Kroemer : Thermal hysics Van der Ziel : oise in Solid - State Devices apoulis : robabil ity and Random Variables hard, comprehens ive eyton Z. eebles : robabili ty, Random Variables, Random Signal rinciple s introduct ory Wozencraft & Jacobs : rinciple s of Communicat ions Engineeri ng. Motchenba er : Low oise Electroni c Design Information theory lecture notes : Thomas Cover, Stanford, circa 98 robabilit y lecture notes : Martin Hellman, Stanford, circa 98 ational Semiconductor Linear Applicatio ns otes : oise in circuits. Suggested references for study. td Van der Ziel, Wozencraft & Jacobs, eebles, Kittel and Kroemer apers by Fuui device noise, Smith & ersoni optical receiver design ational Semi. App. otes! Cover and Williams : Elements of Information Theory

ECE594I Notes set 4: More Math: Expectations of 1-2 R.V.'s

ECE594I Notes set 4: More Math: Expectations of 1-2 R.V.'s C594I Notes set 4: More Math: pectations o - R.V.'s Mark Rodwell Universit o Caliornia, Santa Barbara rodwell@ece.ucsb.edu 805-893-344, 805-893-36 a Reerences and Citations: Sources / Citations : Kittel