Randomized Smoothing Networks

Transcription

1 Randomized Smoothing Netorks Maurice Herlihy Computer Science Dept., Bron University, Providence, RI, USA Srikanta Tirthapura Dept. of Electrical and Computer Engg., Ioa State University, Ames, IA, USA A preliminary version of this article has appeared in the Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS) 2004 Corresponding Author addresses: mph@cs.bron.edu ( Maurice Herlihy ), snt@iastate.edu ( Srikanta Tirthapura ). Preprint submitted to Elsevier Science 23 March 2005

2 1 Introduction A k-smoothing netork is a distributed data structure that accepts tokens on input ires and routes them to output ires. It ensures that no matter ho imbalanced the traffic on the input ires, the numbers of tokens emitted on the output ires are approximately balanced, lying ithin k of one another, here k is a constant, independent of the number of active tokens. Smoothing netorks are ell suited for load balancing applications here tokens represent requests for service. Clients send tokens to arbitrary input ires; these tokens are routed to servers in such a ay that all servers receive approximately the same number of tokens. In a real distributed system, netork sitches may be rebooted or replaced dynamically, and it may not be feasible to determine the correct initial state for each sitch. Hence, an attractive approach to fault-recovery is simply to initialize the ne sitch to a random state, thus eliminating the need for a global coordination. In this paper, e analyze the smoothness properties of randomized balancing netorks, hose constituent balancers are initialized to random states. Randomized netorks are easy to maintain and can recover from faults ithout a global reconfiguration. We sho that randomized netorks produce outputs that are remarkably smooth. We consider the block smoothing netork initialized to a random state; e assume that an off-line adversary, one that does not kno the initial state, chooses an input sequence. We sho that the output of the netork is 2.36 log -smooth ith high probability. As a direct consequence, e sho that the output smoothness of a randomly initialized bitonic or a periodic netork is also O ( log ) ith high probability. In prior ork [7] e shoed that certain ell-knon 1-smoothing netorks, hen started in an arbitrary initial state (perhaps chosen by an adversary), produce outputs that are at orst (log )-smooth, here is the number of input and output ires. This bound is tight for each of the netorks that e considered there. Thus, our results sho that the outputs of randomized netorks are significantly smoother than the orst-case outputs of the same netorks initialized deterministically. Our results lead to extremely practical smoothing netorks. If an application requires approximate smoothing, then a block netork ith randomized balancers ill ork very ell. The block netork of idth has depth exactly log (there are no hidden constants), and the smoothness bound gros very sloly ith. For example, if the idth = 2 24, then the output smoothness is less than 12 ith overhelming probability. 2

3 Input Wires Output Wires Increasing Layers Fig. 1. A balancing netork of idth 4 and depth 3. Horizontal segments are ires and the vertical segments balancers. We conclude this article ith a brief discussion of informal but suggestive experimental results. By feeding random sequences to randomized smoothing netorks, e observed a degree of smoothness orse than constant, and consistent ith our bound. On random input sequences, e discovered that randomized netorks ere dramatically smoother than the same netorks initialized to the default initial state. 1.1 Model Balancers and Smoothing Netorks A balancer is an asynchronous sitch ith to input ires and to output ires, labeled top and bottom. A balancer accepts a stream of tokens on its input ires. A balancer has to states: it is oriented up or don. If the balancer is oriented up, then the next input token leaves on the top output ire and the balancer becomes oriented don. If the balancer is oriented don, then the next input token leaves on the bottom output ire and the balancer becomes oriented up. The above definition applies to balancers of idth to. All the balancers that e consider in this paper have a idth of to. A balancing netork is an acyclic netork of balancers here output ires of some balancers are linked to input ires of others. An example is shon in Figure 1. The netork s input ires are not linked to the output of any balancer, and similarly the netork s output ires are not linked to the input of any balancer. In this paper, e consider balancing netorks ith the same number of input and output ires, called the netork s idth. Tokens enter the netork on the input ires, typically several per ire, propagate asynchronously through the balancers, and leave on the output ires, typically several per ire. A balancing netork is quiescent if every token that has entered the netork has also left. Because balancing netorks are acyclic (as directed graphs), each balancer can be assigned a unique layer, hich is the length of the longest path from an input ire to that balancer. 3

4 Definition 1 A sequence of natural numbers X = x 0,..., x n 1 is k-smooth if x i x j k, for any 0 i, j < n. Definition 2 A balancing netork is a k-smoothing netork if, starting from its initial state, the overall distribution of output tokens across the output ires in any quiescent state is k-smooth. Note that if a netork is k-smooth, it is also l-smooth for any l > k. In this paper, e ill be concerned ith the block netork [6] (hich is isomorphic to the popular butterfly netork), the periodic netork [6,3] and the bitonic netork [4,3]. We no prove a simple but useful lemma. Lemma 3 If a sequence X = x 0,..., x n 1 is k-smooth, then the result of passing X through a balancing netork is k-smooth. PROOF. We sho that the result of passing any to elements of X through a balancer leads to a sequence that is k-smooth, and the lemma follos by induction. Let Z be the result of passing to elements of X through a balancer. The smallest element in Z is greater than or equal to the smallest element in X, and the largest element in Z is lesser than or equal to the largest element in X. Since X is k-smooth, Z is also k-smooth. Lemma 3 leads to the folloing corollary. Corollary 4 If a balancing netork N contains another netork M embedded inside it, and M is a k-smoothing netork, then N is also a k-smoothing netork Randomized Balancers and Netorks Definition 5 A randomized balancer is a balancer hich has been initialized to a random initial state, i.e. up or don ith equal probability. Definition 6 A randomized balancing netork is a balancing netork hose component balancers are independent randomized balancers. Let x denote the total number of input tokens to a randomized balancer b, and y 1, y 2 denote the number of tokens routed to the top and bottom output ires respectively. Let x b = D(x)r b, here: r b is a random variable specific to balancer b, hich can take values +1/2 or 1/2 ith equal probability. 4

5 D is the odd-characteristic function: D(x) = 1 if x is an odd number, and 0 otherise. By the definition of the randomized balancer: y 1 = x/2 + x b y 2 = x/2 x b (1) Note that if x is even, then the input tokens are divided equally among the output ires. The randomness comes into play only hen x is odd. In such a case, the excess token is routed to a randomly chosen output ire. 1.2 Our Results We ask the folloing question. What are the smoothness properties of randomized balancing netorks? We sho the folloing results in response. The output of a randomized block netork of idth is (2.36 log )- smooth ith probability at least 1 4/. We sho that the bitonic netork contains a block netork embedded inside it. The periodic netork consists of many pipelined block netorks, hence trivially contains a block netork embedded inside. Hence, the outputs of periodic and bitonic netorks of idth are also (2.36 log )-smooth ith probability at least 1 4/. 1.3 Related Work Aiello, Venkatesan and Yung [1] build smoothing netorks using a different kind of randomized balancer: odd-numbered input tokens to the balancer leave on a randomly chosen ire, hile even-numbered input tokens leave on the opposite ire from their immediate predecessors. It can be easily verified that our result for the smoothness of the block netork still holds if e replace our random balancers ith theirs. Using a combination of deterministic and randomized balancers, Aiello et al. [1] construct smoothing netorks hose outputs are 2-smooth ith high probability. In our model, hoever, an adversary could set the orientations of the deterministic balancers, and their analysis does not hold under these conditions. Klugerman and Plaxton [10] sho the existence of counting netorks (hich are 1-smoothing netorks ith other additional properties) of depth O(log ) 5

6 Analysis of idth netork Depth Balancers Global Initialization Smoothness Klugerman and Plaxton [10,9] O(log ) Det. Required 1 Aiello et al. [1] O(log ) Det. and Rand. Required 2 (.h.p.) Herlihy et al. [7] log Det. Not required log This paper log Rand. Not required 2.36 log (.h.p.) Fig. 2. Comparison of various analyses of smoothing netorks. Note:.h.p. = ith high probability, det. = deterministic and rand. = randomized. hich use deterministic balancers. They also give an explicit construction of a counting netork of depth O(c log log ) (here c is a constant) using deterministic balancers. Klugerman [9] extends this to give a polynomial time construction of an O(log ) depth counting netork of idth. The above netorks use the AKS netork [2] as a subnetork, and are hence not practical since the constants in O(log ) are huge. For a survey of ork on lo-depth counting netorks and related topics, e refer the reader to Busch and Herlihy [5]. In earlier ork [7], e shoed that the orst case output smoothness of the block, periodic and bitonic netorks of idth is exactly log hen the sitches are initialized adversarially. Our results in this paper sho that randomization helps significantly to reduce the output smoothness of the block netork. A comparison of various analyses of smoothing netorks appears in Figure 2. Roadmap: The rest of the paper is organized as follos. Section 2 contains the analysis of the block netork. Section 3 contains the analysis of the bitonic netork. Section 4 contains the experimental results, and e conclude by listing some open problems in Section 5. 2 The Periodic and Block Netorks The periodic smoothing netork [3] is isomorphic to the periodic sorting netork of Dod et al. [6]. At its heart is a component Block[] netork, defined inductively as follos. 6

7 Fig. 3. A Block[8] Netork consists of to Block[4] netorks: one ith balancers denoted by solid lines, and the other ith balancers denoted by dashed lines. The outputs of the Block[4] netorks are fed into an EvenOdd[8] netork, shon inside the box. The Block Netork: The Block[2] netork is a single balancer. The Block[2] netork is constructed from to Block[] netorks as follos. Given an input sequence X of length 2, represent each index (subscript) as a binary string. The A-cochain of X, denoted X A, is the subsequence hose indexes have lo-order bits 00 or 11. For example, the A-cochain of the sequence x 0,..., x 7 is x 0, x 3, x 4, x 7. The B-cochain x B is the subsequence hose lo-order bits are 01 and 10. The input sequence X is fed into to parallel Block[] netorks, hich e ill call the A-block and the B-block. Sequence X A goes to the A-block, and X B to the B-block. Suppose the output sequence of the A-block is Y A = y0 A, ya 1,..., ya 1 and the output sequence of the B-block is Y B = y0 B, yb 1,..., yb 1. These are fed into an EvenOdd[2] netork, hich simply balances each element of Y A ith the corresponding element of Y B. More formally, Even- Odd[2] consists of balancers numbered 0 to 1, and the ith balancer balances yi A ith yi B. Figure 3 shos a Block[8] netork. The Block[] netork consists of log layers of balancers, numbered from 1 (input layer) to log (output layer). The output ires of a balancer in layer l belong to layer l. The input ires to the netork belong to layer 0. Each layer consists of /2 balancers, so that the netork has a total of ( log )/2 balancers. The Periodic Netork: The Periodic[] netork is the cascade of (log ) Block[] netorks. Since the Periodic[] netork trivially contains a Block[] embedded inside it, because of Corollary 4, our upper bounds for the block netork directly carry over to the periodic netork. 7

8 2.1 Smoothness of the Randomized Block[] Netork Consider an execution of the randomized smoothing netork, taking it from a random initial state to a quiescent state, here a total of M tokens have entered and left the netork. We consider Block[] as a tree. Each node in this tree is a set of one or more balancers, as described belo. The root of the tree is the set of all the /2 balancers at the input layer of the netork (layer 1). This node is labeled v 1,1, since it is the first node in layer 1. For each i = 2... log, the ith layer is divided into 2 i 1 nodes, numbered from v i,1 till v i,2 i 1, ith each node consisting of /2 i balancers. Given the nodes in the (i 1)th layer, the nodes in the ith layer are defined as follos. The node v i,2k 1 consists of all the balancers that the top output ires of the balancers in node v i 1,k point to. Similarly, the node v i,2k consists of all the balancers that the bottom output ires of the balancers in node v i 1,k point to. In the tree, there is an edge from node v i 1,k to nodes v i,2k 1 and v i,2k. The leaves of the tree are the balancers at the output layer. Let m i,j denote the total number of tokens entering node v i,j. Since the tokens on the top output ires of the balancers in v i,j enter v i+1,2j 1, the number of tokens entering node v i+1,2j 1 is (using Equation 1): m i+1,2j 1 = m i,j 2 + b v i,j x b (2) We ill no express the number of tokens that are output at the top output ire of v log,1 as a function of the number of input tokens and the random variables corresponding to the different balancers. The tokens that exit from the topmost output ire must follo the path v 1,1 v 2,1 v 3,1... v log,1 and further exit on the top output ire of balancer v log,1. The number of tokens entering v 1,1 is M. Let X 1 denote the number of tokens exiting the top output ire of v log,1, hich is a single output balancer. Applying Equation 2 repeatedly, and finally Equation 1, e get: 8

9 m 2,1 = M 2 + b v 1,1 x b m 3,1 = m 2,1 2 + b v 2,1 x b X 1 = m log,1 2 + b v log,1 x b Combining the above, e get: X 1 = M + b v 1,1 x b b v + 2,1 x b b v log 1,1 x b + x b /2 /4 2 b v log,1 Reriting the above: X 1 = M/ + V here V = log i=1 2i x b b v i,1 (3) We ill no analyze V. The main problem is that V is not the sum of independent random variables. Clearly the various random variables x b (for different balancers b) are heavily dependent on each other, since the event that the number of tokens entering balancer b is even (resp., odd) depends upon the corresponding events for balancers at earlier layers. Recall that x b = D(n b )r b. We consider an alternate random variable W, defined as follos. W = log i=1 2i r b b v i,1 (4) Since the random variables r b for different balancers b are independent, W is easier to handle than V. We rerite Equations 3 and 4 as follos: V = {(b S) (D(n b )=1)} c b r b (5) W = {b S} c b r b (6) S is a set of balancers, c b denotes a constant specific to balancer b, and r b is the random variable corresponding to balancer b. The set S and the c b s can 9

10 be derived from Equations 3 and 4, but their exact nature is not important here. We no prove a key lemma, shoing that in order to bound the probability that V deviates significantly from its expected value, it is enough to bound the probability that W deviates from its expected value. Lemma 7 For any δ > 0, Pr { V > δ} 2Pr { W > δ}. PROOF. Consider the conditional probability α = Pr {W > δ V > δ} From Equations 5 and 6, Pr {W δ V > δ} Pr {(b S) (D(n b ) 1)} c b r b < 0 1/2 The right hand side inequality ( 1/2) is true because: for each balancer b, the random variable r b is distributed symmetrically around zero, and is chosen independent of n b. Thus, e have α 1/2. Pr {W > δ} Pr {(W > δ) (V > δ)} Pr {W > δ V > δ} Pr {V > δ} Pr {V > δ}/2 Similarly, e can sho the other direction, that Pr {V < δ} 2Pr {W < δ}, and the claim follos. We ill use the folloing lemma in further proofs. Lemma 8 [Hoeffding s bound [8]] Suppose S n = Y 0 + Y Y n 1 here the Y j s are independent random variables, and for each j = 0... n 1, Y j [a j, b j ]. Then, for any ɛ > 0, Pr {S n E[S n ] > nɛ} exp 2n 2 ɛ 2 j=0 (b j a j ) 2 n 1 10

11 Lemma 9 { } Pr V > log < 4 2 PROOF. We ill first bound the probability that W is large, and then use Lemma 7 to bound the probability that V is large. Since W is the sum of independent random variables, e use Hoeffding s inequality (stated in Lemma 8) to bound the probability that W is large. For i = 1... log, the number of balancers in node v i,1 is /2 i. Thus, W is the sum of (/2 + / ) = 1 independent random variables. The expectation of W : E[W ] = 0, since E[r b ] = 0 for every balancer b. Next, the ranges of various random variables. For i = 1... log, 2 i r b [ 2 i 1 ], +2i 1 In Lemma 8, the term (b j a j ) 2 is: 2 ( 2 )2 + 4 ( 4 ) = 2 2 Setting ɛ = 2 ln 1 in Lemma 8, e get Pr { W > 2 ln } exp { 2 ln } = 1 2 (7) Since W is the eighted sum of r b s, each of hich is symmetrically distributed around zero, W is also symmetrically distributed around zero. Hence Pr { W < 2 ln } = Pr { W > 2 ln } 1 2 (8) Equations 7 and 8 combined ith Lemma 7 yield the claim. Note that e have stated our lemma using logarithms to base 2. Theorem 10 The output sequence of Block[] ith randomized balancers is 2.36 log -smooth ith probability at least 1 4/. PROOF. Let X 1, X 2,..., X denote the output sequence of Block[] ith randomized balancers, here X 1 corresponds to the topmost output ire. 11

12 From Lemma 9, e have { } Pr X 1 M/ > log 4/ 2 The random variables X 1... X all have the same distribution, due to symmetry, and a similar equation holds for all of them. Using the union bound on probabilities, the probability that for some i the value X i M/ exceeds log is not greater than 4/. Thus, ith probability at least 1 4/, all outputs X i are ithin log of M/, and hence ithin 2.36 log of each other, as needed. 3 The Bitonic Netork MERGER[4] BITONIC[4] MERGER[4] BITONIC[4] MERGER[8] Fig. 4. Recursive Structure of a Bitonic[8] Counting Netork. The Bitonic[] smoothing netork [3] is isomorphic to the bitonic sorting netork of Batcher [4]. This netork has a simple inductive structure, shon in Figure 4. The Bitonic[2] netork is a single balancer. The Bitonic[2] netork is constructed by feeding the 2 input ires into to parallel Bitonic[] netorks, and feeding their outputs into a Merger[2] netork. The Merger[2] netork takes to input -sequences X and Y and produces an output 2-sequence Z. The Merger[] netork is also defined inductively. The Merger[2] netork is a single balancer. We construct the Merger[2] netork from to Merger[] netorks and a EvenOdd[2] netork hich as described in Section 2. Let X E denote the even subsequence x 0, x 2,..., x 2 of X, and X O denote the odd subsequence x 1, x 3,..., x 1. Similarly define Y E and Y O. The input to the first Merger[] netork is the -sequence formed by the concatenation of X E and Y O, denoted by X E Y O. Call that netork s output 12

13 sequence U. Symmetrically, the input to the second Merger[] netork is the -sequence X O Y E. Call that output sequence V. The final layer of the netork, EvenOdd[], simply joins the each output ire of the first Merger[] netork ith the corresponding output ire of the second Merger[] netork. To netorks N and M are isomorphic if the underlying directed graphs are isomorphic. Lemma 11 The Merger[] netork is isomorphic to the Block[] netork. PROOF. We argue by induction on. When is 2, both netorks consist of a single balancer. Assume that Merger[] is isomorphic to Block[], and consider the Merger[2] and Block[2] netorks. In the Block[2] netork, the final layer connects the i-th ire of one component Block[] netork ith the i-th ire of the other. Likeise, in the Merger[2] netork, the final layer connects the i-th ire of one component Merger[] netork ith the i-th ire of the other, preserving the isomorphism. Lemma 11, Theorem 10 and Corollary 4 together lead to the folloing corollary. Corollary 12 The Merger[] netork, and hence, the Bitonic[] netork are 2.36 log -smooth ith probability at least 1 4/. 4 Experiments To develop an intuition about hether our bound can be improved, e conducted a number of simple experiments testing the behavior of randomized block netorks of different sizes. These results do not prove anything, but they are suggestive. The results are shon in Figure 5. In our experiments, the number of tokens input into each ire is randomly chosen beteen 1 and , and the output smoothness is averaged over 10 runs. We simulated netorks of idth up to The results sho that hen log changes from 1 to 24, the (average) output smoothness increases very gradually, from 0.7 to 3.2. We do not expect to be able to simulate much larger netorks. 13

14 To compare ith the randomized netork, e simulated a block netork initialized in the standard state; all balancers ere initialized to up. Using the same setup as for the random netork, (the number of tokens into each ire is a random number beteen 1 and , average taken over 10 runs), the results obtained are shon belo. It can be seen that the output smoothness is very nearly equal to log 2. While e kno that the orst case smoothness of a block netork initialized by an adversary is log, these experiments suggest that the smoothness of the netork over random inputs hen initialized very regularly (not by an adversary) is no better than log /2. In other ords, these experiments suggest that the average smoothness of the block netork initialized to the standard state is O(log ), hile e have shon that the smoothness of a block netork initialized randomly is O( log ). We conclude that for applications needing approximate load balancing, a randomized block netork orks much better than a deterministic one. 14 randomized block netork deterministic block netork 12 average smoothness of output log(), =idth of netork Fig. 5. The observed smoothness of a Block[] netork ith randomized and deterministic balancers 5 Open Problems We conclude by listing some interesting open problems. 14

15 (1) Our bounds for the smoothness of the block netork does not make use of structure that may be present in the input sequence. Can e obtain better bounds if the input is already fairly smooth? (2) Can e get better bounds on the output smoothness of the randomized periodic or bitonic netorks? (3) Ho tight is the O( log ) upper bound for the block netork? Can e get a matching loer bound? References [1] W. Aiello, R. Venkatesan, and M. Yung. Coins, eights and contention in balancing netorks. In Proceedings of the annual ACM symposium on Principles of Distributed Computing, pages , August [2] M. Ajtai, J. Komlós, and E. Szemerédi. Sorting in c log n parallel steps. Combinatorica, 3:1 19, [3] J. Aspnes, M. Herlihy, and N. Shavit. Counting netorks. Journal of the ACM, 41(5): , [4] K.E. Batcher. Sorting netorks and their applications. In Proceedings of the AFIPS Spring Joint Computer Conference, volume 32, pages , [5] C. Busch and M. Herlihy. A survey on counting netorks. In Proceedings of Workshop on Distributed Data and Structures (WDAS), March [6] M. Dod, Y. Perl, L. Rudolph, and M. Saks. The periodic balanced sorting netork. Journal of the ACM, 36(4): , October [7] M. Herlihy and S. Tirthapura. Self stabilizing smoothing and counting. In Proceedings of the 23rd International Conference on Distributed Computing Systems (ICDCS), pages 4 11, [8] W. Hoeffding. Probability inequalities for sums of bounded random variables. American Statistical Association Journal, 58:13 30, [9] M. Klugerman. Small-depth Counting Netorks and Related Topics. PhD thesis, Massachusetts Institute of Technology, Department of Mathematics, Sept [10] M. Klugerman and C.G. Plaxton. Small-depth counting netorks. In Proceedings of the 24th Annual ACM Symposium on Theory of Computing, pages ,