Günter Rote. Freie Universität Berlin

Transcription

1 Algorithms for Isotonic Regression a i Günter Rote Freie Universität Berlin i

2 Algorithms for Isotonic Regression a i x i Günter Rote Freie Universität Berlin Minimize n i=1 i h( x i a i ) subject to x 1 x n

3 Minimize n i=1 Objective functions h( x i a i ) subject to x 1 x n h(z) = z: L 1 -regression h(z) = z 2 : L 2 -regression n i=1 x i a i min n i=1 (x i a i ) 2 min h(z) = z p, p : L -regression versions with weights w i > 0: n n w i x i a i, w i (x i a i ) 2, i=1 General form: n i=1 i=1 h i (x i ) min max x i a i min 1 i n max w i x i a i, 1 i n / ( ) h i convex and piecewise simple max h i(x i ) min 1 i n

4 Overview The classical Pool Adjacent Violators (PAV) algorithm dynamic programming More general constraints: with a given partial order x i x j for i j In particular, L max regression with a d-dimensional partial order Randomized optimization technique of Timothy Chan (1998)

5 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n a i x i i

6 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i i

7 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i i

8 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i 2. Find adjacent violators x i > x i+1 i

9 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i 2. Find adjacent violators x i > x i+1 3. Pool them: x i = x i+1, and solve h( x a 10 ) + h( x a 11 ) min i

10 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i 2. Find adjacent violators x i > x i+1 3. Pool them: x i = x i+1, and solve h( x a 10 ) + h( x a 11 ) min i

11 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i 2. Find adjacent violators x i > x i+1 3. Pool them: x i = x i+1, and solve 4. Repeat from step 2. i

12 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i 2. Find adjacent violators x i > x i+1 3. Pool them: x i = x i+1, and solve 4. Repeat from step 2. i

13 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i 2. Find adjacent violators x i > x i+1 3. Pool them: x i = x i+1, and solve 4. Repeat from step 2. i

14 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i 2. Find adjacent violators x i > x i+1 3. Pool them: x i = x i+1, and solve 4. Repeat from step 2. i

15 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i 2. Find adjacent violators x i > x i+1 3. Pool them: x i = x i+1, and solve 4. Repeat from step 2. i

16 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i 2. Find adjacent violators x i > x i+1 3. Pool them: x i = x i+1, and solve 4. Repeat from step 2. i

17 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i 2. Find adjacent violators x i > x i+1 3. Pool them: x i = x i+1, and solve 4. Repeat from step 2. i

18 Pool Adjacent Violators (PAV) Minimize n i=1 h( x i a i ) subject to x 1 x n 1. Relax all x i x i+1 a i x i 2. Find adjacent violators x i > x i+1 3. Pool them: x i = x i+1, and solve 4. Repeat from step 2. i

19 min s i t min s i t x = w i x a i w i (x a i ) 2 s i t s i t w i a i w i Subproblems = x = weighted median of a s,..., a t = x = weighted mean of a s,..., a t in O(1) time, after O(n) preprocessing Weighted isotonic L 2 regression is solvable in O(n) time.

20 min s i t w i x a i Subproblems = x = weighted median of a s,..., a t Ahuja and Orlin [2001]: O(n log n) algorithm based on PAV and scaling: Solve the problem for scaled (integer) data ā i := 2 a i /2. Solution for original data a i can be recovered in O(n) time. We can assume a i {1, 2,..., n}, after sorting. Quentin Stout [2008]: O(n log n) PAV implementation median queries by mergeable trees (2-3-trees, AVL trees) extended with weight information Rote [2012]: O(n log n) by dynamic programming. A priority queue is sufficient

21 { k f k (z) := min Rekursion: Dynamic programming i=1 w i x i a i : x 1 x 2 x k = z f k (z) := min{ f k 1 (x) : x z } + w k z a k } k = 0, 1,..., n; z R

22 { k f k (z) := min Rekursion: Dynamic programming i=1 w i x i a i : x 1 x 2 x k = z f k (z) := min{ f k 1 (x) : x z } + w k z a k }{{} g k 1 (z) } k = 0, 1,..., n; z R Transform f k 1 g k 1 f k

23 Recursion step 1 Transform f k 1 to g k 1 (z) := min{ f k 1 (x) : x z } y f k 1 (z) z

24 Recursion step 1 Transform f k 1 to g k 1 (z) := min{ f k 1 (x) : x z } y f k 1 (z) p k 1 g k 1(z) z Remove the increasing parts right of the minimum p k 1 and replace them by a horizontal part.

25 Recursion step 2 Transform g k 1 to f k (z) = g k 1 (z) + w k z a k Add two convex piecewise-linear functions

26 Recursion step 2 Transform g k 1 to f k (z) = g k 1 (z) + w k z a k Add two convex piecewise-linear functions Lemma. f k is a piecewise-linear convex function. The breakpoints are located at a subset of the points a i. The leftmost piece has slope k i=1 w i. The rightmost piece has slope w k.

27 y = f(x) Piecewise-linear functions y = s x + t y = s x + t y = sx + t x 0 f has a breakpoint at position x 0 with value s s. x Represent f by the rightmost slope s and the set of breakpoints (position+value). (The rightmost intercept t is not needed.)

28 Piecewise-linear functions y = f(x) y = s x + t y = s x + t y = sx + t x 0 Transformation f k 1 g k 1 : while s (value of rightmost breakpoint) 0: remove rightmost breakpoint update s update s to 0 x

29 Piecewise-linear functions y = f(x) y = s x + t y = s x + t y = sx + t x 0 Transformation g k 1 f k (z) = g k 1 (z) + w k z a k add w k to s. add a breakpoint at position a k with value 2w k. x

30 Piecewise-linear functions y = f(x) y = s x + t y = s x + t y = sx + t x 0 Transformation g k 1 f k (z) = g k 1 (z) + w k z a k add w k to s. add a breakpoint at position a k with value 2w k. x

31 Piecewise-linear functions y = f(x) y = s x + t y = s x + t y = sx + t x 0 Transformation f k 1 g k 1 : while s (value of rightmost breakpoint) 0: remove rightmost breakpoint update s update s to 0 x

32 Piecewise-linear functions y = f(x) y = s x + t y = s x + t y = sx + t x 0 Transformation f k 1 g k 1 : while s (value of rightmost breakpoint) 0: remove rightmost breakpoint update s update s to 0 x

33 y = f(x) Piecewise-linear functions Only access to the rightmost breakpoint is required. y = s priority x + t queue ordered by position y = s x + t y = sx + t x 0 Transformation f k 1 g k 1 : while s (value of rightmost breakpoint) 0: remove rightmost breakpoint update s update s to 0 x

34 The algorithm Q := ; // priority queue of breakpoints ordered by the key position; s := 0; for k = 1,..., n do // On entry, Q and s represents g k 1. Q.add(new breakpoint with position := a k, value := 2w k ); s := s + w k ; // We have computed f k. while loop true do B := Q.findmax; // rightmost breakpoint B if s B.value < 0 then exit loop; ; s := s B.value; Q.deletemax; p k := B.position; B.value := B.value s; s := 0; // We have computed g k. // Compute the optimal solution x 1,..., x n backwards: x n := p n ; for k = n 1, n 2,..., 1 do x k := min{x k+1, p k }; g k 1 f k g k

35 General objective functions Minimize n i=1 h i (x i ) Each h i is convex and piecewise simple : Summation of pieces in constant time Minimum on a sum of pieces in constant time Examples: L 3 norm: h i (x) = { (x a i ) 3, x a i (x a i ) 3 O(n log n) time, x a i L 4 norm: h i (x) = (x a i ) 4 O(n) time (no breakpoints! A stack suffices.) Linear + sinusoidal pieces: w i cos(x a i )

36 General partial orders Minimize n subject to i=1 h i (x i ) or max 1 i n h i(x i ) for a given partial order. x i x j for i j x 4 x 5 x 1 x 4 x 6 x 2 x 7 PAV can be extended to tree-like partial orders. weighted L 1 -regression for a DAG with m edges in O(nm + n 2 log n) time. [Angelov, Harb, Kannan, and Wang, SODA 2006]... and many other results

37 General partial orders Minimize n subject to i=1 h i (x i ) or max 1 i n h i(x i ) for a given partial order. x i x j for i j x 4 x 5 x 1 x 4 x 6 x 2 x 7 PAV can be extended to tree-like partial orders. weighted L 1 -regression for a DAG with m edges in O(nm + n 2 log n) time. [Angelov, Harb, Kannan, and Wang, SODA 2006]... and many other results

38 Weighted L regression Minimize z := max 1 i n h i(x i ) subject to x i x j for i j. h i (x) = w i x a i x or h i (x) = any function which increases from a minimum into both directions x

39 Weighted L regression Minimize z := max 1 i n h i(x i ) subject to x i x j for i j. h i (x) = w i x a i ε l i u i x or h i (x) = any function which increases from a minimum into both directions ε l i u i x h i (x) ε l i (ε) x u i (ε)

40 Checking feasibility Is the optimum value z ε? [ z = max i h i (x i ) ] Find values x i with l i (ε) x i u i (ε) for all i, and (1) x i x j for i j. (2) Find the smallest values x i = x low i with l i (ε) x i for all i, and Result: x low j x i x j for i j. x low j = max{l j, max{ l i i j }} Calculate in topological order: := max{l i, max{ x low i i predecessor of j }} z ε iff x low i u i for all i = O(m + n) time

41 Characterizing feasibility z ε iff for every pair i, j with i j: l i (ε) u j (ε) Theorem. Define for every pair i, j: z ij := min{ ε l i (ε) u j (ε) } h j z ij h i x Then z = max{ z ij i j }.

42 d-dimensional orders n points p = (p 1, p 2,..., p d ) R d Product order (domination order): p q (p 1 q 1 ) (p 2 q 2 ) (p d q d ) p 2 q p p 1

43 d-dimensional orders n points p = (p 1, p 2,..., p d ) R d Product order (domination order): p q (p 1 q 1 ) (p 2 q 2 ) (p d q d ) p 2 p 2 q p p 1 p 1 The digraph of the order is not explicitly given. It might have quadratic size.

44 L regression for d-dimensional orders Stout [2011]: O(n log d 1 n) space and O(n log d n) time Embed the order in a DAG with more vertices but fewer edges. divide-and-conquer strategy, recursive in the dimension.

45 L regression for d-dimensional orders Stout [2011]: O(n log d 1 n) space and O(n log d n) time Embed the order in a DAG with more vertices but fewer edges. divide-and-conquer strategy, recursive in the dimension.

46 L regression for d-dimensional orders Stout [2011]: O(n log d 1 n) space and O(n log d n) time Embed the order in a DAG with more vertices but fewer edges. divide-and-conquer strategy, recursive in the dimension.

47 L regression for d-dimensional orders Stout [2011]: O(n log d 1 n) space and O(n log d n) time Embed the order in a DAG with more vertices but fewer edges. divide-and-conquer strategy, recursive in the dimension.

48 L regression for d-dimensional orders Stout [2011]: O(n log d 1 n) space and O(n log d n) time Embed the order in a DAG with more vertices but fewer edges. divide-and-conquer strategy, recursive in the dimension.

49 L regression for d-dimensional orders Stout [2011]: O(n log d 1 n) space and O(n log d n) time Embed the order in a DAG with more vertices but fewer edges. divide-and-conquer strategy, recursive in the dimension. Recurse in each half. O(n log n) in 2 dimensions. small Manhattan networks [ Gudmundsson, Klein, Knauer, and Smid 2007 ] O(n log n) is optimal in 2 dimensions: Horton sets

50 L regression for d-dimensional orders Stout [2011]: O(n log d 1 n) space and O(n log d n) time Embed the order in a DAG with more vertices but fewer edges. divide-and-conquer strategy, recursive in the dimension. Recurse in each half. O(n log n) in 2 dimensions. small Manhattan networks [ Gudmundsson, Klein, Knauer, and Smid 2007 ] O(n log n) is optimal in 2 dimensions: Horton sets Rote [2013]: O(n) space and O(n log d 1 n) expected time

51 L regression for d-dimensional orders Rote [2013]: O(n) space and O(n log d 1 n) expected time feasibility checking without explicitly constructing the DAG randomized optimization by Timothy Chan s technique (saves a log-factor.) x low j = max{l j, max{ l i p i p j }} More general operation update(a, B): (A, B {1,..., n}) for all j B: x new j = max{x old j, max{ x old i i A, p i p j }} equivalent formulation: for all i A, j B (in any order): if p i p j then set x j := max{x j, x i }. B A

52 Partitioning the update operation d A + B + procedure update(a, B): Split A B along the last coordinate; update(a, B ); update(a, B + ); update(a +, B ); update(a +, B + ); A B 1, 2,..., d 1

53 Partitioning the update operation d A + B + procedure update(a, B): Split A B along the last coordinate; update(a, B ); update(a, B + ); update(a +, B ); update(a +, B + ); A B 1, 2,..., d 1

54 Partitioning the update operation d A + B + A procedure update(a, B): Split A B along the last coordinate; update(a, B ); update(a, B + ); update(a +, B ); update(a +, B + ); a (d 1)-dimensional order! B 1, 2,..., d 1

55 Partitioning the update operation d A + B + A procedure update(a, B): Split A B along the last coordinate; update(a, B ); update(a, B + ); update(a +, B ); update(a +, B + ); a (d 1)-dimensional order! B 1, 2,..., d 1

56 Partitioning the update operation d A + B + A B 1, 2,..., d 1 procedure update(a, B): Split A B along the last coordinate; update(a, B ); update(a, B + ); update(a +, B ); update(a +, B + ); a (d 1)-dimensional order! procedure update k (A, B): Split A B along the k-th coordinate; update k (A, B ); update k 1 (A, B + ); update k (A +, B + );

57 Partitioning the update operation (2) procedure update k (A, B): Split A B into two equal parts along the k-th coordinate; update k (A, B ); update k 1 (A, B + ); update k (A +, B + ); Initially sort along all coordinates in O(n log n) time. Splitting takes linear time. Initial call: update d (P, P ) with P = {1,..., n} Base case: update 1 (A, B) is a linear scan. Takes O(n) time. Induction: update k (A, B) in O(n log k 1 n) time. (n = A B ) Remark: update d (A, B) is always called with A = B. update k (A, B) for k < d is always called with A B =.

58 Randomized optimization technique The problem is decomposable: z = max{ z ij i j } Define z(p ) := max{ z ij i, j P, i j } If P = P 1 P 2 P 3, then z(p ) = max{z(p 1 P 2 ), z(p 1 P 3 ), z(p 2 P 3 )} (Similar problems: Diameter, closest pair) We can check feasibility: Is z(p ) ε? Lemma. (Chan 1998) = The solution can be computed in the same expected time as checking feasibility.

59 Randomized maximization Permute subproblems S 1, S 2,..., S r into random order. z := ; for k = 1,..., r do if z(s k ) > z then z := z(s k ); ( ) ( ) Proposition. The test ( ) is executed r times, and the computation ( ) is executed in expectation at most times r = H r 1 + ln r

60 L regression in d dimensions Partition P into 10 equal subsets P 1 P 10 (for example along the d-axis). Form 45 subproblems (P i, P j ) for 1 i j 10 and permute them into random order. z := ; for k = 1,..., 45 do Let (P i, P j ) the k-th subproblem. Feasibility check: Is z(p i P j ) z? ( ) if not then compute z(p i P j ) recursively ( ) and set z := z(p i P j ); T (n) = O(FEAS(n)) + H 45 T (2n/10) O(n log d 1 n) T (n/5) = O(n log d 1 n)