Distributed Stochastic Optimization in Networks with Low Informational Exchange

Transcription

1 Distributed Stochastic Optimization in Networs with Low Informational Exchange Wenjie Li and Mohamad Assaad, Senior Member, IEEE arxiv:80790v [csit] 30 Jul 08 Abstract We consider a distributed stochastic optimization problem in networs with finite number of nodes Each node adjusts its action to optimize the global utility of the networ, which is defined as the sum of local utilities of all nodes Gradient descent method is a common technique to solve the optimization problem, while the computation of the gradient may require much information exchange In this paper, we consider that each node can only have a noisy numerical observation of its local utility, of which the closed-form expression is not available This assumption is quite realistic, especially when the system is too complicated or constantly changing Nodes may exchange the observation of their local utilities to estimate the global utility at each timeslot We propose stochastic perturbation based distributed algorithms under the assumptions whether each node has collected local utilities of all or only part of the other nodes We use tools from stochastic approximation to prove that both algorithms converge to the optimum The convergence rate of the algorithms is also derived Although the proposed algorithms can be applied to general optimization problems, we perform simulations considering power control in wireless networs and present numerical results to corroborate our claim Index Terms optimization, stochastic approximation, convergence analysis, distributed algorithms I INTRODUCTION Distributed optimization is a fundamental problem in networs, which helps to improve the performance of the system by maximizing some predefined objective function Significant amount of wor have been done to solve the optimization problems in various applications For example, in power control [], [3], [4] and beamforming allocation [5], [6], [7], [8] problems, transmitters need to control its transmission power or beamforming in a smart manner, in order to maximize some performance metric of the wireless communication systems, such as throughput or energy efficiency In medium access control problem [9], users set their individual channel access probability to maximize their benefit In wireless sensor networs, sensor nodes collect information to serve a fusion center, an interesting problem is to mae each node independently decide the quality of its report to maximize the average quality of information gathered by the fusion center subject to some power constraint [0], [], note that a higher level of quality requires higher power consumption This paper was presented in part at 55th Annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, United States, Oct 07 [] W Li and M Assaad are with the Laboratoire des Signaux et Systèmes LS, UMR CNRS 8506, CentraleSupélec, France wenjieli@lsscentralesupelecfr; mohamadassaad@centralesupelecfr W Li and M Assaad are also with the TCL Chair on 5G, CentraleSupélec, France This paper considers an optimization problem in a distributed networ where each node adjusts its own action to maximize the global utility of the system, which is also perturbed by a stochastic process, eg, wireless channels The global utility is the sum of the local utilities of all nodes of the networ Gradient descent method is the most common technique to deal with optimization problems In many scenarios in practice, however, the computation of gradient may require too much information exchange between the nodes, examples are provided in Section IV Furthermore, there are other contexts also where the utility function of each node does not have a closed form expression or the expression is very complex which maes it very hard to use in the optimization, eg, computation of the derivatives is very complicated or not possible In this paper, we consider therefore that a node only has a noisy numerical observation of its utility function, which is quite realistic when the system is complex and timevarying The nodes can only exchange the observation of their local utilities so that each node can have the nowledge of the whole networ However, a node may not receive all the local utilities of the other nodes due to the networ topology or other practical issues, eg, it is not possible to exchange much signaling information In this situation, a node has to approximate the global utility with only incomplete information of local utilities We have also taen in account such issue in this paper In summary, our problem is quite challenging due to the following reasons: i each node has only a numerical observation of its local utility at each time; ii each node may have incomplete information of the global utility of the networ; iii the action of each node has an impact on the utilities of the other nodes in the networ; iv the utility of each node is also influenced by some stochastic process eg, time varying channels in wireless networs and the objective function is the average global utility In this paper, we develop novel distributed algorithms to optimize the global average utility of a networ, where the nodes can only exchange the numerical observation of their local utility Different versions of algorithms are proposed depending on: i whether the value of action is constrained or unconstrained; ii whether each node has the full nowledge of local utilities of all the other nodes or only a part of the utilities of other nodes We have proved the convergence of the algorithms in all situations, using stochastic approximation tools The convergence rate of different algorithms are also derived, in order to show the convergence speed of the proposed algorithms to the optimum from a quantitative point of view and deeply investigate the impact of the parameters

2 introduced by the algorithm Our theoretical results are further justified by simulations Some preliminary results of our wor have been presented in [] This extended version provides the complete proof of all the results Moreover, the constrained optimization problem and the analysis of the convergence rate in this paper are not considered at all in [] As we will see in Section VII, the derivation of the convergence rate is especially challenging The rest of the paper is organized as follows Section II discusses some related wor and highlights our main contribution Section III describes the system model as well as some basic assumptions Section IV shows motivating examples to explain the interest of our problem Section V develops the initial version of our distributed optimization algorithm using stochastic perturbation DOSP and shows its convergence Section VI presents the first variant of the DOSP algorithm to deal with the situation where a node has incomplete information of the global utility of the networ Section V-D proposes the second variant of the DOSP algorithm to solve the constrained optimization problem Section VII focuses on the analysis of the convergence rate of the proposed algorithms Section VIII shows some numerical results as well as a comparison with an alternative algorithm and Section IX concludes this paper II RELATED WORK Most of the prior wor in the area of optimization consider that the objective function has a well nown and simple closed form expression Under this assumption, the optimization problem can be performed using gradient ascent or descent method [] This method can achieve a local optimum or global optimum in some special cases eg concavity of the utility, etc of the optimization problem A distributed asynchronous stochastic gradient optimization algorithms is presented in [3] Incremental sub-gradient methods for nondifferentialable optimization are discussed in [4] Interested readers are referred to a survey by Bertseas [5] on incremental gradient, sub-gradient, and proximal methods for convex optimization The use of gradient-based method supposes in advance that the gradient can be computed or is available at each node, which is not always possible as this would require too much information exchanges In our case, the computation of the gradient is not possible at each node since only limited control information can be exchanged in the networ This problem is nown as derivative-free optimization, see [6] and the references therein Our goal is then to develop an algorithm that requires only the nowledge of a numerical observation of the utility function The obtained algorithm should be distributed Distributed optimization has also been studied in the literature using game theoretic tools However, most of the existing wor assume that a closed form expression of the payoff is available One can refer to [7], [8] and the references therein for more details, while we do not consider non-cooperative games in this paper Stochastic approximation SA [9], [0] is an efficient method to solve the optimization problems in noisy environment Typically, the action is updated as follows a + = a + β ĝ where ĝ represents an estimation of the gradient g An important assumption is that the estimation error ε = ĝ g is seen as a zero-mean random vector with finite variance, for example, see [] If the step-size β is properly chosen, then a can tend to its optimum point asymptotically The challenge of our wor is how to propose such estimation of the gradient only with the noisy numerical observation of the utilities Most of the previous wor related to derivative-free optimization consider a control center that updates the entire action vector during the algorithm, see [6] for more details However, in our distributed setting, each node is only able to update its own action Nevertheless, a stochastic approximation method using the simultaneous perturbation gradient approximation SPGA [] can be an option to solve our distributed derivative-free optimization problem The SPGA algorithm was initially proposed to accelerate the convergence speed of the centralized multi-variate optimization problem with deterministic objective function Two measurements of the objective function are needed per update of the action The approximation of the partial derivative of an element i is given by ĝ i, = f a + γ f a γ γ i,, where γ > 0 is vanishing and = [,,, N, ] with each element i, zero mean and iid Two successive measurements of the objective function are required to perform a single estimation of the gradient The interest of the SPGA method is that each variable can be updated simultaneously and independently Spall has also proposed an one-measurement version of the SPGA algorithm in [3] with ĝ i, = f a + γ γ i, 3 Such algorithm also leads a to converge, while with a decreased speed compared with the two-measurement SPGA An essential result is that the estimation of gradient using or 3 is unbiased if γ is vanishing, as long as the objective function f is static However, if the objective function is stochastic and its observation is noisy, there would be an additional term of stochastic noise in the numerator of and 3, which may seriously affect the performance of approximation when the value γ is too small As a consequence, the SPGA algorithm cannot be used to solve our stochastic optimization problem The authors in [4] proposed a fully distributed Nash equilibrium seeing algorithm which requires only a measurement of the numerical value of the static utility function Their scheme is based on deterministic sine perturbation of the payoff function in continuous time In [5], the authors extended the wor in [4] to the case of discrete time and stochastic state-dependent utility functions, convergence to a close region of Nash equilibrium has been proved However, in a distributed setting, it is challenging to ensure that the sine perturbation of different nodes satisfy the orthogonality requirement, especially when the number of nodes is large Moreover, the continuous sine perturbation based algorithm converges slowly in a discrete-time system Stochastic pertur-

3 3 bation based algorithm has been proposed in [6] to solve an optimization problem, the algorithm is given by a + = a + βv f a + v, 4 with v the zero-mean stochastic perturbation The behavior of 4 has been analyzed in [6], however, under the assumption that the objective function is static and quadratic Our proposed algorithm is different from 4 as we use the random perturbation with vanishing amplitude In addition, the objective function is stochastic with non-specified form in our setting, which is much more challenging Furthermore, we consider a situation where nodes have to exchange their local utilities to estimate the global utility and each node may have incomplete information of the local utilities of other nodes III SYSTEM MODEL This section presents the problem formulation as well as the basic assumptions Throughout this paper, matrices and vectors are in boldface upper-case letters and in in bold-face lower-case letters respectively Calligraphic font denotes set a denotes the Euclidean norm of any vector a In order to lighten the notations, we use F i a = F a, F i,j a = F a, a i a i a j and F a = [F i a,, F N a] in the rest of the paper A Problem formulation Consider a networ consisting of a finite set of nodes N = {,, N} Each node i is able to control its own action a i, at each discrete timeslot, in order to maximize the performance of the networ Introduce the action vector a = [a,,, a N, ] T which contains the action of all nodes at timeslot Let A i denote the feasible action set of node i, ie, a i, A i Introduce A = A A N In general, the performance of a networ is not only determined by the action of nodes, but also affected by the environment state, eg, channels We assume that the environment state of the entire networ at any timeslot is described by a matrix S S, which is considered as an iid ergodic stochastic process For any realization of a and S, we are interested in the global utility f a, S of the networ, which is defined as the sum of the local utility function u i a, S of each node i, ie, f a, S = i N u i a, S The networ performance is then characterized by the average global utility F a = E S f a, S In this wor, we consider a challenging setting that nodes do not have the nowledge of S nor the closed-form expression of the utility functions Each node i only has a numerical estimation ũ i, of its local utility u i a, S at each timeslot Assume that ũ i, = u i a, S + η i,, 5 with η i, some additive noise Nodes can communicate with each other so that each node i can get a approximate value f i, of the global utility f i a, S As a summary, our aim is to propose some distributed solution of the following problem { maximize F a = E S f a, S, 6 subject to a i A i, i N An application example is introduced in Section IV to highlight the interest of this problem B Basic assumptions We present the basic assumptions considered in this paper in order to guarantee the performance of our proposed algorithm Denote a as the solution of the problem 6 Since the existence of a can be ensured by the concavity of the objective function F a, we have the following assumption A properties of objective function Both F i a and F i,j a exist continuously There exists a A such that F i a = 0 and F i,i a < 0, i N The objective function is strictly concave, ie, a a T F a 0, a A 7 Besides, for any i N and j N, there exists a constant α R + such that F i,j a α 8 We have some further assumptions on the local utility functions, which will be useful in our analysis A properties of local utility function For any i N, the function a u i a, S is Lipschitz with Lipschitz constant L S, ie, u i a, S u i ã, S L S a ã 9 Besides, E S u i a, S <, i N, so that F a is also bounded From A, we can easily deduce that a f a, S is also Lipschitz with Lipschitz constant NL S, since f a, S f ã, S = u i a, S u i ã, S i N i N u i a, S u i ã, S NL S a ã 0 In the end, we consider a common assumption on the noise term η i, introduced in 5 A3 properties of additive noise The noise η i, is zeromean, uncorrelated, and has bounded variance, ie, for any i N, we have E η i, = 0, E ηi, = α 4 <, and E η i, η j, = 0 if i j

4 4 transmitter N a, a, a N, I receiver N enb UE3 a, a 3, UE enb3 II a 4, UE UE4 a, enb Figure I A DD networ with N transmitter-receiver pairs; II Downlin multi-cell networ, each enb or base station may serve multiple UEs users IV MOTIVATING EXAMPLES the global utility function can be written as f a, S = κ a i, i N + ω a i, s ii, log + log + σ + i N j i a 3 j,s ji, It is straightforward to show that f a, S is concave with respect to a, hence the existence of the optimum a can be guaranteed In order to maximize the average global utility E S f a, S, one may apply the classical stochastic approximation method described by An essential step is that, each transmitter or receiver should be able to calculate the partial derivative, ie, ĝ i, = f a i, = ωsinr i, + r i, a i, κ This section provides some examples and shows the limit of the classical gradient-based methods We consider the power allocation problem in a general networ with N transmitter-receiver lins A lin here can be seen as a node in our system model as presented in Section III For example, in a DD networ with N transmitter-receiver pairs, each transmitter communicates with its associated receiver and different lins interfere among each other, see Figure I in a multi-cell networ, each base station may serve multiple users We focus on the downlin, ie, a user is seen as its receiver and its associated base station is seen as a transmitter, see Figure II In both models, the action a i, is in fact the transmission power of transmitter i at timeslot, of which the value cannot exceed the maximum transmission power a max Thus we have A i = [0, a max ], i N The environment state matrix S represents the time-varying channel state of the networ More precisely, S = [s ij, ] i N,j N, in which each element s ij, denotes the the channel gain between transmitter i and receiver j at timeslot The utility function can be various depending on different applications, eg, throughput and energy efficiency [7] In our first example, the local utility of each node i is given by u i a, S = ω log + r i, κa i,, where ω, κ R +, κa i, represents the energy cost of transmission, and r i, denotes the bit rate given by a i, s ii, r i, = log + SINR i, with SINR i, = σ + j i a j,s ij, Note that the maximization of log-function of the bit rate is of type proportional fairness, which is used to ensure fairness among the nodes in the networ With the above notations, ωsinr n, s ni, 4 + r n, + SINR n, a n, s nn, n N From 4, we find that the direct calculation of the partial derivative is complicated and nodes should exchange much information: Each node i should now the values of SINR n,, the cross-channel gain s ni,, as well as a n, s nn, of all nodes n N Moreover, in the situation where the channel is timevarying, it is not realistic for any receiver n to estimate the direct-channel gain s nn, and all the cross-channel gain s nj,, j N \ {n} We consider the sum-rate maximization problem as a second example, ie, to maximize y = i N r i, The challenge come from the fact that y is not concave with respect to a i, if the rate r i, is given by For this reason, we have to consider the approximation of r i, and some variable change to mae the objective function concave, which is a well nown problem [8] It is common to use change of variable ie, consider e a i, as the transmission power instead of a i, and consider the approximation r i, log SINR i, [8], so that the global utility function is written as f a, S =ω i Nlog s ii, e a i, σ + j i s ji,e a κ j, i Ne a i, 5 It is straightforward to show the concavity of f Similar to 4, we evaluate f ĝ i, = a i, = ω ω s in, e a i, σ + n N j n δ j,s jn, e a κ e a i,, j, i N 6 of which the calculation also requires much information, such as the cross-channel gain s in, n N \ {i}, and all the interference estimated by each receiver All the channel information has to be estimated and exchanged by each active node, which is a huge burden for the networ The two examples clearly shows the limit of the classical methods and motivates us to propose some novel solution

5 5 where low informational exchange are required In this paper, we consider that the nodes can only exchange their numerical approximation of local utilities It is worth mentioning that the information exchange in our setting is much less than the gradient based method We present our distributed optimization algorithms as well as their performance, in the situations where each node has the complete or incomplete nowledge of the local utilities of the other nodes, respectively It is worth mentioning that, apart from the examples presented in this section, the solution proposed in this paper can also be applied to other type of problems such beamforming, coordinated multipoint CoMP [9], and so on V DISTRIBUTED OPTIMIZATION ALGORITHM USING STOCHASTIC PERTURBATION This section presents a first version of our distributed optimization algorithm We assume that each node is always able to collect the numerical estimation of local utilities from all the other nodes In this situation, each node can evaluate the numerical value of the global utility function at each iteration or timeslot by applying f a, S = i N ũ i, = i N = f a, S + i N u i a, S + η i, η i,, 7 since each node nows the complete information of ũ i, for any i N We first consider an unconstrained optimization problem, ie, A = R N, in Section V-A The constrained optimization problem with A i = [a i,min, a i,max ], i N is then presented in Section V-D A Algorithm The distributed optimization algorithm using stochastic perturbation DOSP is presented in Algorithm Algorithm DOSP Algorithm for each node i Initialize = 0 and set the action a i,0 randomly Generate a random variable Φ i,, perform action a i, + γ Φ i, 3 Estimate ũ i,, exchange its value with the other nodes and calculate f a + γ Φ, S = j N ũi, 4 Update a i,+ according to equation 8 5 = +, go to At each iteration, an arbitrary reference node i updates its action by applying a i,+ = a i, + β Φ i, f a + γ Φ, S, 8 in which β and γ are vanishing step-sizes, Φ i, is randomly generated by each node i and Φ = [Φ,,, Φ N, ] Recall that the approximation f of the global utility is calculated by each node using 7, of which the value depends on the actual action performed by each node â i, = a i, +γ Φ i, and the environment state matrix S Note that â i, is very close to a i, when is large as γ is vanishing An example is presented in Section V-B to describe in detail the application of Algorithm in practice Obviously, 8 can be written in the general form, in which ĝ i, = Φ i, f a + γ Φ, S 9 As we will discuss in Section V-C, ĝ i, can be an asymptotically unbiased estimation of the partial derivative F/ a i and a can converge to a, under the condition that the parameters β, γ, and Φ are properly chosen We state in what follows the desirable properties of these parameters A4 properties of step-sizes Both β and γ tae real positive values with lim β = lim γ = 0, besides, β γ =, β < = = A5 properties of random perturbation The elements of Φ are iid with E Φ i, Φ j, = 0, i j There exist α > 0 and α 3 > 0 such that E Φ i, = α, Φ i, α 3 The conditions on the parameters can be easily achieved We show in Example a common setting of these parameters, which are also used to obtain the simulation results to be presented in Section VIII Example An easiest choice of the probability distribution of Φ i, is the symmetrical Bernoulli distribution with Φ i, {, } and P Φ i, = = P Φ i, = = 05, i, We can verify the conditions in A5 with α = α 3 = Let β = β 0 + ν and γ = γ 0 + ν with the constants β 0, γ 0, ν, ν R +, so that both β and γ are vanishing Since = β converges if ν > 05; = β γ diverges if ν + ν Clearly, there exist pairs of ν and ν to mae β and γ satisfy the conditions in A4 Remar The proposed algorithm has similar shape compared with the other existed methods proposed in [6], [4], [5] The difference between our solution and the sine perturbation based method [4], [5] is that, we use a random vector Φ instead of some deterministic sine functions as the perturbation term Comparing 4 and 8, we can see that the amplitude of random perturbation is vanishing in our algorithm, which is not the case in the algorithm presented in [6] B Application example This section presents the application of Algorithm to perform resource allocation in a DD networ, in order to highlight the interest of our solution We focus on the requirement arisen by Algorithm, in terms of computation, memory and informational exchange Figure briefly shows the algorithm procedure during one iteration Recall that â i, denotes the actual value of the action set by transmitter i at iteration In order to update â i,, each transmitter i needs to

6 6 transmitter N a, a, a N, I receiver N transmitter N II receiver Figure I At iteration, each transmitter i transmits to its associated receiver with transmission power â i, ; II Each receiver i sends the approximate local utility ũ i, to its transmitter, every transmitter can receive and decode this feedbac information independently and randomly generate a scalar Φ i, under the condition A5; use a pre-defined vanishing sequence γ which is common for each lin; independently update a i, by applying 8, more details will be provided soon Then â i, is given by â i, = a i, + γ Φ i, Each transmitter i transmits to its associated receiver with the transmission power of value â i, The associated receiver i should be able to approximate the numerical value of its local utility ũ i, and send this value to transmitter i as a feedbac All the transmitters can evaluate f â using 7 under the assumption that every transmitter is able to receive and decode the feedbac from all the receivers Then iteration + starts, each transmitter i needs to update its power allocation strategy, what is required is listed as follows: use a pre-defined vanishing sequence β ; reuse of the local random value Φ i,, which was generated by transmitter i at iteration It means that the value of Φ i, should be saved temporarily use the numerical value of f â, which has been explained already Each transmitter can then update a i,+ independently using 8 In the following step, each transmitter updates â i,+ in the same way as the previous iteration, ie, â i,+ = a i,+ + γ + Φ i,+, with Φ i,+ a newly generated pseudo-random value We can see that Algorithm can be easily applied in a networ of multiple lins: the algorithm itself has low complexity and each receiver only needs to feedbac one quantity ũ i, per iteration to perform the algorithm C Convergence results This section investigates the asymptotic behavior of Algorithm For any integer 0, we consider the divergence N u, u, u N, d = a a 0 to describe the difference between the actual action a and the optimal action a Our aim is to show that d 0 almost surely as In order to explain the reason behind the update rule 8, we rewrite it in the generalized Robbins-Monro form [0], ie, a + = a + β ĝ = a + β α γ F a + g α γ F a + ĝ g = a + α β γ F a + b + e, α γ where g represents the expected value of ĝ with respect to all the stochastic terms including Φ, S, and η for a given a, ie, g = E S,Φ,η ĝ, note that we prefer to highlight the stochastic terms in, an alternative way is to write as g = E ĝ a ; b denotes the estimation bias between g and the actual gradient of the average objective function F, ie, b = g α γ F a ; 3 and e can be seen as the stochastic noise, which is the difference between the value of a single realization of ĝ and its average g as defined in, ie, e = ĝ g 4 Remar The analysis presented in this wor is challenging and different from the existed results An explicit difference comes from the unique feature of the algorithm itself as discussed in Remar : we are using a different method to estimate the gradient Besides, the objective function is stochastic with general form in our problem, while it is considered as static in [], [3] and it is assumed to be static and quadratic in [6] To perform the analysis of convergence, we have to investigate the properties of b and e Lemma If all the conditions in A-A5 are satisfied, then b γ N 5 which implies that b 0 as α 3 3α α = O γ, 5 Proof: See Appendix A Lemma implies that ĝ defined in Algorithm is a reasonable estimator of the gradient with vanishing bias Lemma If all the conditions in A-A5 are satisfied and a < almost surely, then for any constant ρ > 0, we have lim P K sup K β e K K ρ = 0, ρ > 0 6 =K Proof: See Appendix B Based on the results in Lemma and in Lemma, we can build conditions under which a a almost surely Theorem If all the conditions in A-A5 are satisfied and a < almost surely, then a a as almost surely by applying Algorithm Proof: See Appendix C

7 7 D Distributed optimization under constraints In this section, we consider the constrained optimization problem, in which the action of each node taes value from an interval, ie, A i = [a i,min, a i,max ], i N We assume that a is not on the boundary of the feasible set A, ie, a i a i,min, a i,max, i N For example, in the power allocation problem, a i, presents the transmission power of transmitter i, which should be positive and not larger than a maximum value Recall that the actually performed action by nodes is â = a + γ Φ At each iteration, we need to ensure that â i, [a i,min, a i,max ] A direct solution is to introduce a projection to the algorithm, ie, 8 turns to in which for any i N, a + = Proj a + β ĝ, 7 a i,+ = min {max {a i, + β ĝ i,, a i,min + α 3 γ + }, a i,max α 3 γ + } 8 As Φ i, α 3 by A5, 8 leads to a i,+ + γ + Φ i,+ [a i,min, a i,max ] Remar 3 In order to write 8 into the form similar to, one has to re-define the bias term b i, and the stochastic noise e i, because of the operator Proj Therefore, it is not straightforward to deduce of the convergence of the constrained optimization algorithm from the results presented in Section V-C Theorem In the constrained optimization problem where A i = [a i,min, a i,max ] and a i a i,min, a i,max, i N, by applying the projection 7, we have a a as almost surely under the assumptions A-A5 Proof: See Appendix D VI OPTIMIZATION ALGORITHM WITH INCOMPLETE INFORMATION OF UTILITIES OF OTHER NODES A limit of Algorithm is that each node is required to now the local utility of all the other nodes Such issue is significant as there are many nodes in the networ It is thus important to consider a more realistic situation where a node only has has the nowledge of the local utilities of a subset I i, of nodes, with I i, N \ {i} Throughout this section, we have the following assumption: A6 at any iteration, an arbitrary node i nows the utility ũ j, of another node j with a constant probability p 0, ], ie, the elements contained in the set I i, is random, for any j i, we have P j I i, = p, P j / I i, = p 9 Notice that we do not assume any specified networ topology and each node i has a different and independent set I i, We propose a modified algorithm and then show its asymptotic performance The algorithm is described in Algorithm The main difference between Algorithm and Algorithm comes from the approximation of the objective function, ie, f I i a, S, I i, = j I i, ũ j,, if I i, 0, 0, if I i, = 0 {ũi, + N I i, Similar to 8, the algorithm is given by 30 a i,+ = a i, + β ĝ I i, = a i, + β Φ i, f I i a, S, I i, 3 The basic idea is to consider N j I i, ũ j, / I i, as a surrogate function of j N \{i} ũj,, in the case where the set I i, is non-empty Otherwise, node i does not now any utility of the other nodes, it then cannot estimate the global utility of the system As a result, node i eeps its previous action, ie, a i,+ = a i, Note that different users may have different nowledge of the global utility as I i, is independent for each node i For example, node i may now ũ j, of a different node j, whereas node j may not now ũ i, Algorithm DOSP algorithm for each node i with incomplete information of the utilities of other nodes Initialize = 0 and set the action a i,0 randomly Generate a random variable Φ i,, perform action a i, + γ Φ i, 3 Estimate ũ i,, exchange its value with the other nodes f I i and calculate utilities 4 Update a i,+ according to equation 3 5 = +, go to using 30 based on the collected local Due to the additional random term I i,, the convergence analysis of Algorithm is more complicated than that of Algorithm We start with Lemma 3, which is useful for the analysis in what follows I Lemma 3 The expected value of f i a, S, I i, over all possible sets I i, is proportional to f a, S, ie, I E Ii, f i a, S, I i, = p N f a, S 3 Proof: See Appendix E To simplify the notation, introduce q = P I i, 0 = p N 33 Similar to, we rewrite 3 as a i,+ = a i, + α qβ γ F i a + b I i, + ei i, α qγ with ĝ I i, 34 E b I i, = S,Φ,η,Ii, F i a, 35 α qγ e I i, = ĝi i, E S,Φ,η,I i, ĝ I 36 We need to investigate the property of b I i, and ei i, in order to show the convergence of Algorithm The proof is more

8 8 complicated because of the additional random term I i, in both b I i, and ei i, compared with b i, and e i, discussed in Section V Theorem 3 In the situation where nodes do not have the access to all the other nodes local utilities and the objective function is approximated by applying 30, then we still have a a as almost surely by applying Algorithm, as long as the assumptions A-A6 hold and a < almost surely Proof: See Appendix F Remar 4 Although the asymptotic convergence still holds, the convergence speed is reduced if the information of the objective function is incomplete By comparing and 34, we can see that the equivalent step size is decreased by q times Moreover, the variance of the stochastic noise e I i, is higher, as the randomness is more significant when we use n < N random symbols to represent the average of N symbols More discussion related to the convergence rate will be provided in Section VII VII CONVERGENCE RATE In this section, we study the average convergence rate of the proposed algorithm in order to investigate how fast the proposed algorithms converge to the optimum from a quantitative point of view The analysis also provides a detailed guideline of setting the parameters β and γ which determine the convergence rate We start with the analysis considering general forms of β and γ A widely used example is then considered afterwards A General analysis As a common setting in the analysis of the convergence rate [30], we have an additional assumption on the concavity of the objective function in this section, ie, A7 F a is strongly concave, there exists α 5 > 0 such that a a T F a α 5 a a, a A 37 In this section, we are interested in the evolution of the average divergence D = E a a In order to distinguish the two versions of algorithms with complete and incomplete information of local utilities, we use D C and D I to denote the divergence resulted by Algorithm and Algorithm, respectively An essential step of the analysis of the convergence rate is to investigate the relation of the divergence between two successive iterations Denote M C and M I as the upper bounds of E ĝ and of E ĝ I in Algorithm and Algorithm respectively Our result is stated in Lemma 4 Lemma 4 Assume that A7 holds, for any, D C resulted by Algorithm is such that D C + α α 5 β γ D C + N 5 α α3β 3 γ D C + M C β, 38 Similarly, D I D I resulted by Algorithm is such that + α α 5 qβ γ D I + N 5 α α3qβ 3 γ D I + M I β 39 Proof: See Appendix G We can see that both 38 and 39 can be written in the simplified general form D + Aβ γ D + Bβ γ D + Cβ, 40 with positive constants A C = α α 5, B C = N 5 α α 3 3, C C = M C 4 in 38 and in 39 A I = qα α 5, B I = qn 5 α α 3 3, C I = M I 4 Remar 5 In the constrained optimization problem, we can obtain the same result as Lemma 4 directly from 68, for any K c In fact, the unconstrained optimization problem can be seen as a special case of the constrained optimization problem with a i,min =, a i,max = +, and K c = 0 For this reason, we consider the general problem with K c taen into account, in the rest of this section Introduce K 0 = arg min, K c,β γ </A which implies that Aβ γ > 0 and K c as K 0 Lemma 4 provides us the relation between D + and D Our next goal is to search a vanishing upper bound of D using 40 In other words, we aim to search a sequence U K0,, U, such that U + U and D U, K 0 This type of analysis is usually performed by induction: consider a given expression of U and assume that D U, one needs to show that D + U + by applying 40 An important issue is then the proper choice of the form of the upper bound U Note that there exists only one stepsize β in the classical stochastic approximation algorithm described by, it is relatively simple to determine the form of U with a further setting β = β 0, see [30] Our problem is much more complicated as we use two step-sizes β and γ with general form under Assumption A4 The following lemma presents an important property of U Lemma 5 If there exists a decreasing sequence U K0,, U, such that D + U + can be deduced from D U and 40, then, U B B A γ + γ A β 43 A γ

9 9 Proof: See Appendix H Note that the lower bound of U is vanishing as γ 0 and β /γ 0 Such bound means that, using induction by developing 40, the convergence rate of D cannot be better than the decreasing speed of γ and of β /γ After the verification of the existence of bounded constant ϑ or ϱ such that D ϑ γ or D ϱ γ /β, we obtain the final results stated as follows Theorem 4 Define the following parameters: χ = ϖ = γ+ γ β γ, ɛ = max K 0 χ, ɛ = max β K 0 γ 3, 44 β +γ + β γ γ 3, ɛ 3 = max ϖ, ɛ 4 = max 45 β γ K 0 K 0 β If χ < A for any K 0, then with ϑ max D ϑ γ, K 0, 46 { DK0 γ K0, B + If ϖ < A for any K 0, then with D K0 γ K0 ϱ max, Bɛ 4 + β K0 } B + 4Cɛ A ɛ, 47 A ɛ D ϱ β γ, K 0, 48 Bɛ 4 + 4C A ɛ 3 A ɛ 3 49 Proof: See Appendix I For general forms of β and γ, Theorem 4 provides two upper bounds of the average divergence D C and D I The conditions that χ < A and ϖ < A can be checed easily considering any fixed β and γ In the situation where both conditions are satisfied, we have { D min ϑ γ, ϱ β }, K 0 γ From Theorem 4, we can see that the decreasing order of D C and D I mainly depend on the step-sizes β and γ, the incompleteness factor q only has the influence on the constant terms ϑ and ϱ, which are functions of the parameters A, B, and C defined in 4 and 4 The results of convergence rate are useful to properly choose the parameters of the algorithm Intuitively, we need to mae γ or β /γ decrease as fast as possible, having the constant ɛ, ɛ, ɛ 3, and ɛ 4 as small as possible B A special case In this section, we consider an example as mentioned in Example : β = β 0 + ν and γ = γ 0 + ν 50 Recall that 05 < ν < and 0 < ν ν in order to meet the conditions in A4 Theorem 5 Consider β and γ with given forms 50, if β 0 γ 0 max {ν, ν ν } /A, then there exists Ω <, such that D Ω + min{ν,ν ν}, K 0 5 Proof: See Appendix J We can see the explicit impact of each parameter on the upper bound of the convergence rate from Theorem 5 It is easy to show that max {ν, ν ν } 05 with the equality holds when ν = 075 and ν = 05, which corresponds to the best choice of ν and ν to optimize the decreasing order of the upper bound of D Theorem 5 also provides a sufficient condition on the parameters that should be satisfied in order to ensure the validation of the convergence rate In what follows, we present a toy example to verify Theorem 5 Example Consider N = and a simple quadratic function f a, S = s a s a +a a +a +a with a, a [0, 3] Both s and s are realizations of a uniform distributed random variable S which taes value from an interval [05, 5] Taing the average, the objective function is F a = a a + a a + a + a It is straightforward to get that F a taes its maximum value when a = a = We can also deduce that F is strongly concave with α 5 = in 37 We set the perturbation Φ as introduced in Example, with α = Thus we have A = α α 5 = by 4 Let β = β and γ = + 05 By setting β 0 {03, 08, 05}, we can verify the importance of the condition β 0 γ 0 05 = max {ν, ν ν } /A as presented in Theorem 5 Figure 3 shows the comparison results between the divergence D averaged by 000 simulations and the bound Ω + 05 from 5 Note that we are mainly interested in the asymptotic decreasing order of D, thus we set the constant term Ω = mainly to facilitate the visual comparison of different curves We find that the curves of D decrease not-less-slowly than the bound as β 0 {08, 05} When β 0 = 03 < 05, the convergence rate cannot be guaranteed, which further justify our claim in Theorem 5 Now we fix the values of β 0 and γ 0 and consider ν, ν {055, 05, 07, 05, 05, 0, 065, 035} Notice that all the four pairs of ν, ν lead to the same decreasing order of the upper bound 5, as min {ν, ν ν } = 03 The curves of D averaged

10 0 by 000 simulations are compared with the upper bound Ω + 03 in Figure 4 Clearly, when the number of iterations is large enough, all the curves decrease with the same speed or even faster, compared with the bound D = = =03 bound: iterations Figure 3 Comparison of the theoretical upper bound + 05 with the evolution of the average divergence D by 000 simulations using β = β and γ = + 05, with β 0 {03, 08, 05} D =055, =05 =07, =05 =05, =0 =065, =035 bound: iterations Figure 4 Comparison of the theoretical upper bound + 03 with the evolution of the average divergence D by 000 simulations using β = 04 + ν and γ = + ν, with ν, ν {055, 05, 07, 05, 05, 0, 065, 035} VIII SIMULATION RESULTS In this section, we apply our algorithm to a power control problem as introduced in Section IV in order to have some numerical results We consider 3 as the local utility function of each node The time varying channel h ij between node i transmitter and node j receiver is generated using a Gaussian distribution with variance σii = and σ ij = 0, i j Notice that the channel gain is s ij = h ij Besides, we set σ = 0, ω = 0 and κ = A Simulations with 4 nodes In this section, we consider N = 4 nodes In all the simulations applying different algorithms, the step size follows β = 5 075, and the initial values of a 0,i i N are generated uniformly in the interval 0, 0] In both Algorithm and Algorithm, γ = 05 and Φ i, follows the symmetrical Bernoulli distribution, ie, P Φ i, = = P Φ i, = = 05,, i We first compare our proposed algorithm with the sine perturbation based algorithm in [5], considering the situation in which every node has access to all the other nodes local utilities The sine perturbation based algorithm has a similar shape as our stochastic perturbation algorithm, with the perturbation term Φ i, replaced by a sine function λ i sin Ω i t + φ i where t = = β, Ω i Ω i and Ω i +Ω i Ω i i, i, i In the simulation, we set Ω = 63, Ω = 70, Ω 3 = 56, Ω 4 = 49, λ i = 5 and φ i = 0, i N Notice that this algorithm is not easy to implement in practice as it is hard to choose all the parameters properly, especially when N is large Furthermore, in order to show the efficiency of our algorithm, we simulate also an ideal algorithm using the exact gradient calculated by 4 which is costly to be obtained in practice as discussed in Section IV We have performed 500 independent simulations to obtain the average results shown in Figures 5 and 6 Figure 5 represents the utility function f a, S /N as a function of number of iterations We find that our algorithm converges faster than the reference algorithm proposed in [5] Figure 6 shows the evolution of the power action of the four nodes Notice that the four curves representing the action of each node are close in average in each sub-figure, since we consider the model with symmetric parameters We find the oscillation of the power action is more significant by applying the reference algorithm global utility function ideal Algo ref Algo iteration Figure 5 Evolution of the utility function, average results by 500 simulations Now we consider the situation where each node has incomplete collection of local utilities We perform 00 independent simulation with p {, 05, 05, 0} Recall that p defined in 9 represents the level of incompleteness The results are shown in Figure 7 We can see that the convergence speed decreases as the value of p goes smaller Such influence is not significant even if p = 05, ie, a node has only 50% chance to now the local utility of another user, which reduces a lot the information exchange in the networ As the value of p is very small, ie, p = 0, same trend of convergence can be observed, although the algorithm converges slowly On the The reference algorithm is quite sensitive to the parameters, the presented results are the best that we have found so far

11 power power power iteration iteration ideal Algo ref Algo iteration Figure 6 Evolution of power action of 4 nodes, average results by 500 simulations D /N p= p=05 p=05 p=0 bound iterations Figure 8 Evolution of D /N, average results by 00 simulations,with p {, 05, 05, 0} figure, we show the results obtained after up to 0 4 iterations only, which explains why for p = 0 the algorithm has not converged yet to the optimal solution D /N p= p=05 p=05 p=0 bound is available at each time and nodes need to exchange their local values to optimize the total utilities of the networ We have developed fully distributed algorithms that converge to the optimum, in the situations where each node has the nowledge of all or only a part of the local utilities of the others The convergence of our algorithms is examined by studying our algorithm using stochastic approximation technique The convergence rate of the algorithms are derived Numerical results are also provided for illustration APPENDIX iterations Figure 7 Evolution of D /N, average results by 00 simulations, with p {, 05, 05, 0} B Simulations with 0 nodes In this section, we consider a more challenging case with N = 0 We choose β = and γ = + 05 The results are presented in Figure 8 with p {, 05, 05, 0}, note that we have also plotted a line representing the optimum value of the average utility function The shape of the curves in Figure 8 is similar to those in Figure 7 The algorithm converges slower as the number of nodes increases, yet the influence of the incompleteness of the received local utilities is less important IX CONCLUSION In this paper we have a challenging distributed optimization problems, under the assumption that only a numerical value of the stochastic state-dependent local utility function of the node A Proof of Lemma In this proof, we mainly need to find the relation between g the expected estimation of gradient and F a the actual gradient of the objective function, so that the upper bound of b can be derived From the definition of g, we have g = E S,Φ,η Φ f a + γ Φ, S + i N = E Φ Φ E S f a + γ Φ, S η i, = E Φ Φ F a + γ Φ, 5 recall that the additive noise η i, is zero mean and F is the expected value of f by definition Based on Taylor s theorem and mean-valued theorem, there exists ã locating between a and a + c Φ such that F a + γ Φ = F a + j Nγ Φ j,t F j a + j,j N γ Φ j,φ j,f j,j ã 53

12 Benefit from the properties of Φ A4, we have, i N, g i, = F a E Φ Φ i, + γ F j a E Φ Φ i, Φ j, + j,j N j N γ E Φ Φi, Φ j,φ j,f j,j ã = α γ F i a + b i,, 54 from 5 and 53, with b i, = γ E Φ Φi, Φ j,φ j,f j α,j ã j,j N From assumptions A and A4, b i, can be upper bounded by b i, γ E Φ Φi, Φ j, Φ j, F j α,j ã j,j N γ E Φ α 3 α 3 α j,j N = γ N α3 3α, α 55 then we can get 5 Thus b 0 as γ is vanishing, which concludes the proof B Proof of Lemma The proof of Lemma is mainly by the application of Doob s martingale inequality [3] { We first need to show that the sequence K =K β e is martingale, which is straightforward }K K as ĝ and ĝ are independent if and E S,Φ,η e = E S,Φ,η ĝ E S,Φ,η ĝ = 0 Then, by Doob s martingale inequality, for any positive constant ρ, we have K P sup β e K K =K ρ ρ E K S,Φ,η β e =K = ρ E K K S,Φ,η β β e T e =K =K a = ρ E K S,Φ,η β e ρ =K = ρ ρ b M ρ =K =K =K =K E S,Φ,η β ĝ E S,Φ,η ĝ β E S,Φ,η ĝ E S,Φ,η ĝ β E S,Φ,η ĝ β 56 where a holds as E e T e = 0 for any and b is by Lemma 6 as stated and proved in the end of this section Since lim K =K β = 0 by Assumption A4 and M is bounded, we have M ρ =K β vanishing for any bounded constant ρ Therefore, the probability that K =K β e ρ is also vanishing, which concludes the proof Lemma 6 If all the conditions in A-A5 are satisfied and a < almost surely, then there exists a bounded constant M > 0, such that E S,Φ,η ĝ < M almost surely Proof: For any i N, we evaluate ĝ E S,Φ,η i, = E S,Φ,η Φ i, f a + γ Φ, S + Φ i, j N a = E S,Φ Φ i, f a + γ Φ, S + Nα α 4 b α 3E S,Φ f a + γ Φ, S + Nα α 4 η j, c α3e S,Φ f 0, S +NL S a + γ Φ + Nα α 4 d α3e S µ S + N L S e = α 3 a + γ N α3 + Nα α 4 µ + N L a + γ N α3 + Nα α 4 < 57 where a is due to E Φ,η Φ i, j N η j, = Nα α 4 and b is by Assumption A4 From 0, we have f a, S f 0, S f a, S f 0, S NL S a, so f a, S f 0, S + NL S a, c can be obtained We denote µ S = f 0, S in d and the inequality is because of x+y/ x + y /, x, y R In e, we introduce µ = E S µ S and L = ES L S Based on 57, we see that E ĝ = i N E ĝi, is also bounded, which concludes the proof C Proof of Theorem In this proof, we start with the evolution of the divergence d as defined in 0, then we show that d should be vanishing almost surely by applying Lemma and Lemma By definition, we have d + = a + a = a + β ĝ a = d + β ĝ + β a a T ĝ 58 We sum both sides of 58 to obtain d K+ = d 0 + K =0 β ĝ + β a a T ĝ 59