CSE 312 Final Review: Section AA

Transcription

1 CSE 312 TAs December 8, 2011

2 General Information

3 General Information Comprehensive Midterm

4 General Information Comprehensive Midterm Heavily weighted toward material after the midterm

5 Pre-Midterm Material

6 Pre-Midterm Material Basic Counting Principles

7 Pre-Midterm Material Basic Counting Principles Pigeonhole Principle

8 Pre-Midterm Material Basic Counting Principles Pigeonhole Principle Inclusion Exclusion

9 Pre-Midterm Material Basic Counting Principles Pigeonhole Principle Inclusion Exclusion Counting the Complement

10 Pre-Midterm Material Basic Counting Principles Pigeonhole Principle Inclusion Exclusion Counting the Complement Using symmetry

11 Pre-Midterm Material Basic Counting Principles Pigeonhole Principle Inclusion Exclusion Counting the Complement Using symmetry Conditional Probability

12 Pre-Midterm Material Basic Counting Principles Pigeonhole Principle Inclusion Exclusion Counting the Complement Using symmetry Conditional Probability P(A B) = P(AB) P(B)

13 Pre-Midterm Material Basic Counting Principles Pigeonhole Principle Inclusion Exclusion Counting the Complement Using symmetry Conditional Probability P(A B) = P(AB) P(B) Law of Total Probability: P(A) = P(A B) P(B) + P(A B) P(B)

14 Pre-Midterm Material Basic Counting Principles Pigeonhole Principle Inclusion Exclusion Counting the Complement Using symmetry Conditional Probability P(A B) = P(AB) P(B) Law of Total Probability: P(A) = P(A B) P(B) + P(A B) P(B) Bayes Theorem: P(A B) = P(B A) P(A) P(B)

15 Pre-Midterm Material Basic Counting Principles Pigeonhole Principle Inclusion Exclusion Counting the Complement Using symmetry Conditional Probability P(A B) = P(AB) P(B) Law of Total Probability: P(A) = P(A B) P(B) + P(A B) P(B) Bayes Theorem: P(A B) = Network Failure Questions P(B A) P(A) P(B)

16 Pre-Midterm Material

17 Pre-Midterm Material Independence

18 Pre-Midterm Material Independence E and F are independent if P(EF ) = P(E)P(F )

19 Pre-Midterm Material Independence E and F are independent if P(EF ) = P(E)P(F ) E and F are independent if P(E F ) = P(E) and P(F E) = P(F )

20 Pre-Midterm Material Independence E and F are independent if P(EF ) = P(E)P(F ) E and F are independent if P(E F ) = P(E) and P(F E) = P(F ) Events E 1,... E n are independent if for every subset S of events ( ) P E i = P(E i ) i S i S

21 Pre-Midterm Material Independence E and F are independent if P(EF ) = P(E)P(F ) E and F are independent if P(E F ) = P(E) and P(F E) = P(F ) Events E 1,... E n are independent if for every subset S of events ( ) P E i = P(E i ) i S i S Biased coin example from Lecture 5, slide 7

22 Pre-Midterm Material Independence E and F are independent if P(EF ) = P(E)P(F ) E and F are independent if P(E F ) = P(E) and P(F E) = P(F ) Events E 1,... E n are independent if for every subset S of events ( ) P E i = P(E i ) i S i S Biased coin example from Lecture 5, slide 7 If E and F are independent and G is an arbitrary event then in general P(EF G) P(E G) P(F G)

23 Pre-Midterm Material Independence E and F are independent if P(EF ) = P(E)P(F ) E and F are independent if P(E F ) = P(E) and P(F E) = P(F ) Events E 1,... E n are independent if for every subset S of events ( ) P E i = P(E i ) i S i S Biased coin example from Lecture 5, slide 7 If E and F are independent and G is an arbitrary event then in general P(EF G) P(E G) P(F G) For any given G, equality in the above statement means that E and F are Conditionally Independent given G

24 Distributions

25 Distributions Know the mean and variance for:

26 Distributions Know the mean and variance for: Uniform distribution

27 Distributions Know the mean and variance for: Uniform distribution Normal distribution

28 Distributions Know the mean and variance for: Uniform distribution Normal distribution Geometric distribution

29 Distributions Know the mean and variance for: Uniform distribution Normal distribution Geometric distribution Binomial distribution

30 Distributions Know the mean and variance for: Uniform distribution Normal distribution Geometric distribution Binomial distribution Poisson distribution

31 Distributions Know the mean and variance for: Uniform distribution Normal distribution Geometric distribution Binomial distribution Poisson distribution Hypergeometric distribution

32 Distributions Know the mean and variance for: Uniform distribution Normal distribution Geometric distribution Binomial distribution Poisson distribution Hypergeometric distribution Remember Linearity of Expectation and other useful facts (e.g. Var[aX + b] = a 2 Var[X ]; in general Var[X + Y ] Var[X ] + Var[Y ]).

33 Distributions Know the mean and variance for: Uniform distribution Normal distribution Geometric distribution Binomial distribution Poisson distribution Hypergeometric distribution Remember Linearity of Expectation and other useful facts (e.g. Var[aX + b] = a 2 Var[X ]; in general Var[X + Y ] Var[X ] + Var[Y ]). Remember: For any a, P(X = a) = 0 (the probability that a continuous R.V. falls at a specific point is 0!)

34 Distributions Know the mean and variance for: Uniform distribution Normal distribution Geometric distribution Binomial distribution Poisson distribution Hypergeometric distribution Remember Linearity of Expectation and other useful facts (e.g. Var[aX + b] = a 2 Var[X ]; in general Var[X + Y ] Var[X ] + Var[Y ]). Remember: For any a, P(X = a) = 0 (the probability that a continuous R.V. falls at a specific point is 0!) Expectation is now an integral: E[X ] = x f (x)dx

35 Distributions Know the mean and variance for: Uniform distribution Normal distribution Geometric distribution Binomial distribution Poisson distribution Hypergeometric distribution Remember Linearity of Expectation and other useful facts (e.g. Var[aX + b] = a 2 Var[X ]; in general Var[X + Y ] Var[X ] + Var[Y ]). Remember: For any a, P(X = a) = 0 (the probability that a continuous R.V. falls at a specific point is 0!) Expectation is now an integral: E[X ] = x f (x)dx Use normal approximation when applicable.

36 Central Limit Theorem

37 Central Limit Theorem Central Limit Theorem: Consider i.i.d. (independent, identically distributed) random variables X 1, X 2,.... Xi has µ = E[X i ] and σ 2 = Var[X i ]. Then, as n X X n nµ σ n N(0, 1) Alternatively X = 1 n n X i N i=1 ) (µ, σ2 n

38 Tail Bounds

39 Tail Bounds Markov s Inequality: If X is a non-negative random variable, then for every α > 0, we have P(X α) E[X ] α

40 Tail Bounds Markov s Inequality: If X is a non-negative random variable, then for every α > 0, we have Corollary P(X α) E[X ] α P(X αe[x ]) 1 α

41 Tail Bounds Markov s Inequality: If X is a non-negative random variable, then for every α > 0, we have Corollary P(X α) E[X ] α P(X αe[x ]) 1 α Chebyshev s Inequality: If Y is an arbitrary random variable with E[Y ] = µ, then, for any α > 0, P( Y µ α) Var[Y ] α 2

42 Tail Bounds

43 Tail Bounds Chernoff Bounds: Suppose X is drawn from Bin(n, p) and µ = E[X ] = pn Then, for any 0 < δ < 1 P(X > (1 + δ)µ) e δ2 µ 2 P(X < (1 δ)µ) e δ2 µ 3

44 Law of Large Numbers

45 Law of Large Numbers Weak Law of Large Numbers: Let X be the empirical mean of i.i.d.s X 1,..., X n. For any ɛ > 0, as n Pr ( X µ > ɛ ) 0

46 Law of Large Numbers Weak Law of Large Numbers: Let X be the empirical mean of i.i.d.s X 1,..., X n. For any ɛ > 0, as n Pr ( X µ > ɛ ) 0 Strong Law of Large Numbers: Same hypotheses ( ( )) X1 + + X n Pr lim = µ = 1 n n

47 Random Facts

48 Random Facts If X and Y are R.V.s from the same distribution, then Z = X + Y isn t necessarily from the same distribution as X and Y.

49 Random Facts If X and Y are R.V.s from the same distribution, then Z = X + Y isn t necessarily from the same distribution as X and Y. If X and Y are both normal, then so is X + Y.

50 MLEs

51 MLEs Write an expression for the likelihood.

52 MLEs Write an expression for the likelihood. Convert this into the log likelihood.

53 MLEs Write an expression for the likelihood. Convert this into the log likelihood. Take derivatives to find the maximum.

54 MLEs Write an expression for the likelihood. Convert this into the log likelihood. Take derivatives to find the maximum. Verify that this is indeed a maximum.

55 MLEs Write an expression for the likelihood. Convert this into the log likelihood. Take derivatives to find the maximum. Verify that this is indeed a maximum. See Lecture 11 for worked examples.

56 Expectation-Maximization

57 Expectation-Maximization E-step: Computes the log-likelihood using current parameters.

58 Expectation-Maximization E-step: Computes the log-likelihood using current parameters. M-step: Maximize the expected log-likelihood, changing the parameters.

59 Expectation-Maximization E-step: Computes the log-likelihood using current parameters. M-step: Maximize the expected log-likelihood, changing the parameters. Iterated until convergence is achieved.

60 Hypothesis Testing

61 Hypothesis Testing H 0 is the null hypothesis

62 Hypothesis Testing H 0 is the null hypothesis H 1 is the alternative hypothesis

63 Hypothesis Testing H 0 is the null hypothesis H 1 is the alternative hypothesis Likelihood Ratio = L 1 L 0 where L 0 is the likelihood of H 0 and L 1 is the likelihood of H 1.

64 Hypothesis Testing H 0 is the null hypothesis H 1 is the alternative hypothesis Likelihood Ratio = L 1 L 0 where L 0 is the likelihood of H 0 and L 1 is the likelihood of H 1. Saying that alternative hypothesis is 5 times more likely than the null hypothesis means that L 1 L 0 5.

65 Hypothesis Testing H 0 is the null hypothesis H 1 is the alternative hypothesis Likelihood Ratio = L 1 L 0 where L 0 is the likelihood of H 0 and L 1 is the likelihood of H 1. Saying that alternative hypothesis is 5 times more likely than the null hypothesis means that L 1 L 0 5. Decision rule: When to reject H 0.

66 Hypothesis Testing H 0 is the null hypothesis H 1 is the alternative hypothesis Likelihood Ratio = L 1 L 0 where L 0 is the likelihood of H 0 and L 1 is the likelihood of H 1. Saying that alternative hypothesis is 5 times more likely than the null hypothesis means that L 1 L 0 5. Decision rule: When to reject H 0. α = P(rejected H 0 but H 0 was true)

67 Hypothesis Testing H 0 is the null hypothesis H 1 is the alternative hypothesis Likelihood Ratio = L 1 L 0 where L 0 is the likelihood of H 0 and L 1 is the likelihood of H 1. Saying that alternative hypothesis is 5 times more likely than the null hypothesis means that L 1 L 0 5. Decision rule: When to reject H 0. α = P(rejected H 0 but H 0 was true) β = P(accept H 0 but H 1 was true)

68 Algorithms

69 Algorithms Better algorithms usually trump better hardware, but both are needed for progress

70 Algorithms Better algorithms usually trump better hardware, but both are needed for progress Some problems cannot be solved (e.g. the Halting Problem)

71 Algorithms Better algorithms usually trump better hardware, but both are needed for progress Some problems cannot be solved (e.g. the Halting Problem) Other (intractable) problems cannot (yet?) be solved in a reasonable amount of time (e.g. Integer Factorization)

72 Sequence Alignment

73 Sequence Alignment Brute force solution take at least ( 2n n ) computations of the sequence score, which is exponential time.

74 Sequence Alignment Brute force solution take at least ( 2n n ) computations of the sequence score, which is exponential time. Dynamic Programming decrease computation to O(n 2 ).

75 Sequence Alignment Brute force solution take at least ( 2n n ) computations of the sequence score, which is exponential time. Dynamic Programming decrease computation to O(n 2 ). IDEA:

76 Sequence Alignment Brute force solution take at least ( 2n n ) computations of the sequence score, which is exponential time. Dynamic Programming decrease computation to O(n 2 ). IDEA: Store prior computations so that future computations can do table look-ups.

77 Sequence Alignment Brute force solution take at least ( 2n n ) computations of the sequence score, which is exponential time. Dynamic Programming decrease computation to O(n 2 ). IDEA: Store prior computations so that future computations can do table look-ups. Backtrace Algorithm: Start at the bottom right of the matrix and find which neighboring cells could have transitioned to the current cell under the cost function, σ. Time Bound (O(n 2 ))

78 Sequence Alignment Brute force solution take at least ( 2n n ) computations of the sequence score, which is exponential time. Dynamic Programming decrease computation to O(n 2 ). IDEA: Store prior computations so that future computations can do table look-ups. Backtrace Algorithm: Start at the bottom right of the matrix and find which neighboring cells could have transitioned to the current cell under the cost function, σ. Time Bound (O(n 2 )) See Lecture 15 for a worked example.

79 P vs. NP

80 P vs. NP Some problems cannot be computed (e.g. The Halting Problem).

81 P vs. NP Some problems cannot be computed (e.g. The Halting Problem). Some problems can be computed but take a long time (e.g. SAT, 3-SAT, 3-coloring).

82 P vs. NP Some problems cannot be computed (e.g. The Halting Problem). Some problems can be computed but take a long time (e.g. SAT, 3-SAT, 3-coloring). P - a solution is computable in polynomial time

83 P vs. NP Some problems cannot be computed (e.g. The Halting Problem). Some problems can be computed but take a long time (e.g. SAT, 3-SAT, 3-coloring). P - a solution is computable in polynomial time NP - a solution can be verified in polynomial time, given a hint that is polynomial in the input length

84 P vs. NP Some problems cannot be computed (e.g. The Halting Problem). Some problems can be computed but take a long time (e.g. SAT, 3-SAT, 3-coloring). P - a solution is computable in polynomial time NP - a solution can be verified in polynomial time, given a hint that is polynomial in the input length A p B means that if you have a fast algorithm for B, you have a fast algorithm for A.

85 P vs. NP Some problems cannot be computed (e.g. The Halting Problem). Some problems can be computed but take a long time (e.g. SAT, 3-SAT, 3-coloring). P - a solution is computable in polynomial time NP - a solution can be verified in polynomial time, given a hint that is polynomial in the input length A p B means that if you have a fast algorithm for B, you have a fast algorithm for A. NP-complete - In NP and as hard as the hardest problem in NP

86 P vs. NP Some problems cannot be computed (e.g. The Halting Problem). Some problems can be computed but take a long time (e.g. SAT, 3-SAT, 3-coloring). P - a solution is computable in polynomial time NP - a solution can be verified in polynomial time, given a hint that is polynomial in the input length A p B means that if you have a fast algorithm for B, you have a fast algorithm for A. NP-complete - In NP and as hard as the hardest problem in NP Any fast solution to an NP-complete problem would yield a fast solution to all problems in NP.