The Smoothed Analysis of Algorithms. Daniel A. Spielman. Program in Applied Mathematics Yale University

Transcription

1 The Smoothed Analysis of Algorithms Daniel A. Spielman Dept. of Computer Science Program in Applied Mathematics Yale University

2 Outline Why? Definitions (by formula, by picture, by example) Examples: Perceptron Condition numbers (GaussianElimination) Simplex Method K means Decision trees What you can do, and how.

3 Why Want theorems that explain why algorithms and heuristics work in practice May be contrived examples on which they fail. fail to converge take too long return wrong answer Worst case analysis is not satisfactory.

4 Attempted fix: Average-case analysis Measure expected performance on random inputs. random graphs random point sets random signals random matrices

5 Random is not typical

6 Critique of Average-Case analysis Actual inputs might not look random. Random inputs have very special properties with very high probability.

7 Smoothed Analysis Assume randomness/noise in low order bits Randomly perturb problem Problem comes through noisy channel measurement error random sampling arbitrary circumstances (managers)

8 Hybrid of worst and average case Complexity of algorithm : inputs > time Worst case: Ave case: Smoothed: Gaussian perturbation of std dev

9 Smoothed Complexity Interpolates between worst and average case Considers neighborhood of every input If low, all high complexity is unstable

10 Complexity Landscape ru un time worst case average case input space

11 Smoothed Complexity Landscape run time smoothed complexity input space

12 Perceptron Algorithm Given points and labels Find so that for all Algorithm: iteratively find violated condition, and use it to update Eventually finds a solution, if one exists

13 Perceptron Algorithm (smoothed analysis by Blum-Dunagan 02) Given points and labels Find so that for all Perturbation: a Gaussian random vector Theorem: if solution exists Prob[# steps > ] <

14 Smoothed Margin Margin = angle separating pos from neg examples Block Novikoff: Perceptron converges in iterations. Blum Dunagan: Prob[ margin < ] <

15 Smoothed Margin (simplified) Assume data is separable by and re label when perturb Perturbation: Analysis: is unlikely that any gets close to separating plane

16 Smoothed Margin (simplified) Assume data is separable by and re label when perturb Perturbation: Analysis: is unlikely that any gets close to separating plane Prob[dist < ] < density of

17 Smoothed Margin (simplified) Assume data is separable by and re label when perturb Perturbation: Analysis: is unlikely that any gets close to separating plane Prob[dist < ] < density of

18 Explain where going from here Body text

19 Condition Numbers Measure maximum of norm(change in output) norm(change in input) Or, 1/(distance to ill posed problem) 1/margin is a condition number Perturbed problems usually have small condition number, and are not to close to ill posed problems

20 The Matrix Condition Number The ratio of largest to smallest singular values. Condition number for problem Focus on As

21 Estimating Smoothed Condition Number Approximately aspect ratio of simplex formed by vectors in columns, and origin.

22 Probability of Large Condition Number Unlikely, as large very unlikely, and small not too likely, because Gaussian point unlikely near plane

23 Smoothed Analysis of Edelman: for standard dgaussian random matrix Sankar S Teng: for 23

24 Geometry of 24

25 Geometry of (union bound) should be d 1/2 25

26 Improving bound on Lemma: For, Apply to random So conjecture 26

27 Gaussian Elimination w/ Partial Pivoting >> A = randn(2) A = >> b = randn(2,1) b = >> x = A \ b x = >> norm(a*x - b) ans = e-016

28 Gaussian Elimination w/ Partial Pivoting >> A = 2*eye(70) - tril(ones(70)); >> A(:,70) = 1; >> b = randn(70,1); >> x = A \ b; >> norm(a*x - b) ans = e+004 Failed! GEPP

29 Gaussian Elimination w/ Partial Pivoting >> A = 2*eye(70) - tril(ones(70)); >> A(:,70) = 1; >> b = randn(70,1); >> x = A \ b; >> norm(a*x - b) ans = e+004 Failed! >> Ap = A + randn(70) / 10^9; >> x = Ap \ b; >> norm(ap*x - b) ans = e-015 Perturb A

30 >> A = 2*eye(70) - tril(ones(70)); >> A(:,70) = 1; >> b = randn(70,1); >> x = A \ b; >> norm(a*x - b) ans = e+004 Gaussian Elimination w/ Partial Pivoting Failed! >> Ap = A + randn(70) / 10^9; >> x = Ap \ b; >> norm(ap*x - b) ans = e-015 Perturb A >> norm(a*x - b) ans = e-008 Solved original too!

31 Gaussian Elimination with Partial Pivoting Fast heuristic for maintaining precision, by trying to keep entries small Pivot not just on zeros, but to move up entry of largest magnitude

32 Gaussian Elimination with Partial Pivoting Gaussian elimination with partial pivoting is utterly stable in practice. In fifty yyears of computing, no matrix problems that excite an explosive instability are know to have arisen under natural circumstances Matrices with large growth factors are vanishingly rare in applications. Nick Trefethen

33 Gaussian Elimination with Partial Pivoting Gaussian elimination with partial pivoting is utterly stable in practice. In fifty yyears of computing, no matrix problems that excite an explosive instability are know to have arisen under natural circumstances Matrices with large growth factors are vanishingly rare in applications. Nick Trefethen Theorem: [Sankar-S-Teng]

34 Simplex Method for Linear Programming g max s.t. Worst-Case: exponential Average-Case: polynomial Widely used in practice

35 Smoothed Analysis of Simplex Method max s.t. max s.t. G is Gaussian Theorem: For all A, b, c, simplex method takes expected time polynomial in

36 Shadow Vertices 36

37 Another shadow 37

38 Shadow vertex pivot rule start z objective 38

39 Theorem: For every plane, the expected size of the shadow of the perturbed tope is poly(m,d,1/σ ) 39

40 z Theorem: For every er z, two-phase Algorithm runs in expected time poly(m,d,1/σ ) 40

41 A Local condition for optimality z a 2 0 z Vertex on a 1,,a d maximizes z iff z cone(a 1,,a d ) a 1 41

42 Primal a T 1 x 1 a 2 T x 1 a m T x 1 Polar ConvexHull(a 1, a 2, a m ) 42

43 43

44 Polar Linear Program z max α αz ConvexHull(a 1, a 2,..., a m ) 44

45 Opt Simplex Initial Simplex 45

46 Shadow vertex pivot rule 46

47 47

48 Count facets by discretizing to N directions, N 48

49 Count pairs in different facets [ ]< c/n Pr Different Facets So, expect c Facets 49

50 Expect cone of large angle 50

51 Distance 51

52 Isolate on one Simplex 52

53 Integral Formulation 53

54 Example: For a and b Gaussian distributed points, given that ab intersects x-axis Prob[θ < ε] = O(ε 2 ) θ a b 54

55 θ a b 55

56 a b 56

57 a b 57

58 b a 58

59 a b 59

60 b a 60

61 θ a b P ε = Pr[ θ < ε ab axis 0] = c a, b [ θ < ε ][ ab axis 0] μ ( a) μ ( b 0 1 ) dadb Claim: For ε < ε 0, P ε < ε 2 61

62 Change of variables z u θ a b v da db = (u+v)sin(θ) du dv dz dθ μ ( a ) ν ( u, z, ) 0( 0 θ μ1( b) ν1( v, z, θ ) 62

63 Analysis: For ε < ε 0 P ε < ε 2 y 0, ε sin( Pε = c ( u + v) θ ) ν 0( u, z, θ ) ν1( v, z, θ ) du dv dz dθ u, v, z θ < ε Slight change in θ has little effect on ν i for all but very rare u,v,z ε ε 0 θ 63

64 Distance: Gaussian distributed corners a 1 a 2 p a 3 64

65 Idea: fix by perturbation 65

66 Title Body text Body text Body text Body text

67 Title Body text Body text Body text Body text

68 Title Body text Body text Body text Body text

69 Title Body text Body text Body text Body text

70 Body text Title

71 Body text Title

72 Body text Title

73 Body text Title

74 Body text Title

75 Body text Title

76 Body text Title