Fast parameter estimation in loss tomography for networks of general topology

Transcription

1 Fast parameter estmaton n loss tomography for networks of general topology The Harvard communty has made ths artcle openly avalable. Please share how ths access benefts you. Your story matters Ctaton Deng, Ke, Yang L, Wepng Zhu, and Jun S. Lu Fast Parameter Estmaton n Loss Tomography for Networks of General Topology. The Annals of Appled Statstcs 10 (1) (March): do: /15-aoas883. Publshed Verson do: /15-aoas883 Ctable lnk Terms of Use Ths artcle was downloaded from Harvard Unversty s DASH repostory, and s made avalable under the terms and condtons applcable to Open Access Polcy Artcles, as set forth at nrs.harvard.edu/urn-3:hul.instrepos:dash.current.terms-ofuse#oap

2 Submtted to the Annals of Appled Statstcs FAST PARAMETER ESTIMATION IN LOSS TOMOGRAPHY FOR NETWORKS OF GENERAL TOPOLOGY arxv: v1 [stat.me] 24 Oct 2015 By Ke Deng, Yang L, Wepng Zhu and Jun S. Lu Tsnghua Unversty, Unversty of New South Wales and Harvard Unversty As a technque to nvestgate lnk-level loss rates of a computer network wth low operatonal cost, loss tomography has receved consderable attentons n recent years. A number of parameter estmaton methods have been proposed for loss tomography of networks wth a tree structure as well as a general topologcal structure. However, these methods suffer from ether hgh computatonal cost or nsuffcent use of nformaton n the data. In ths paper, we provde both theoretcal results and practcal algorthms for parameter estmaton n loss tomography. By ntroducng a group of novel statstcs and alternatve parameter systems, we fnd that the lkelhood functon of the observed data from loss tomography keeps exactly the same mathematcal formulaton for tree and general topologes, revealng that networks wth dfferent topologes share the same mathematcal nature for loss tomography. More mportantly, we dscover that a re-parametrzaton of the lkelhood functon belongs to the standard exponental famly, whch s convex and has a unque mode under regularty condtons. Based on these theoretcal results, novel algorthms to fnd the MLE are developed. Compared to exstng methods n the lterature, the proposed methods enjoy great computatonal advantages. 1. Introducton. Network characterstcs such as loss rate, delay, avalable bandwdth, and ther dstrbutons are crtcal to varous network operatons and mportant for understandng network behavors. Although consderable attenton has been gven to network measurements, due to varous reasons (e.g., securty, commercal nterest and admnstratve boundary) some characterstcs of the network cannot be obtaned drectly from a large network. To overcome ths dffculty, network tomography was proposed n [1], suggestng the use of end-to-end measurement and statstcal nference to estmate characterstcs of a large network. In an end-to-end measurement, a Ths study s partally supported by Natonal Scence Foundaton of USA grant DMS , Natonal Scence Foundaton of Chna grant , Shenzhen Fund for Basc Scence grant JC A and grant JC A. Keywords and phrases: network tomography, loss tomography, general topology, lkelhood equaton, pattern-collapsed EM algorthm 1

3 2 K. DENG ET AL. number of sources are attached to the network of nterest to send probes to recevers attached to the other sde of the network, and paths from sources to recevers cover lnks of nterest. Arrval orders and arrval tmes of the probes carry the nformaton of the network, from whch many network characterstcs can be nferred statstcally. Characterstcs that have been estmated n ths manner nclude lnk-level loss rates [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], delay dstrbutons [14, 15, 16, 17, 18, 19, 20, 21, 22, 23], orgn-destnaton traffc [1, 16, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38], loss patterns [39], and the network topology [40]. In ths paper, we focus on the problem of estmatng loss rates. Network topologes connectng sources to recevers can be dvded nto two classes: tree and general. A tree topology, as named, has a sngle source attached to the root of a multcast tree to send probes to recevers attached to the leaf nodes of the tree. A network wth a general topology, however, requres a number of trees to cover all lnks of the network. Each tree has one source sendng probes to recevers n t. Because of the use of multple sources to send probes n a network wth general topology, those nodes and recevers located at ntersectons of multple trees can receve probes from multple sources. In ths case, we must consder mpacts of probes sent by all sources smultaneously n order to get a good estmate. Ths task s much more challengng than the tree topology case. Numerous methods have been proposed for loss tomography n a tree topology. Cáceres et al. [2] used a Bernoull model to descrbe the loss behavor of a lnk, and derved the MLE for the pass rate of a path connectng the source to a node, whch was expressed as the soluton of a polynomal equaton [2, 3, 4]. To ease the concern of usng numercal methods to solve a hgh degree polynomal, several papers have been publshed to accelerate the calculaton at the prce of a lttle accuracy loss: Zhu and Geng proposed a recursvely defned estmator based on a bottom-up strategy n [11, 12]; Duffeld et al. proposed a closed-form estmator n [13], whch has the same asymptotc varance as the MLE to the frst order. Consderng the unavalablty of multcast n some networks, Harfoush et al. [5] and Coates et al. [6] ndependently proposed the use of the uncast-based multcast to send probes to recevers, where Coates et al. also suggested the use of the EM algorthm [41] to estmate lnk-level loss rates. For networks beyond a tree, however, lttle research has been done, although a majorty of networks n practce fall nto ths category. Conceptually, the topology of a general network can be arbtrarly complcated. However, no matter how complcated a general topology s, t can always be covered by a group of carefully selected trees, n each of whch an end-

4 LOSS TOMOGRAPHY FOR NETWORK OF GENERAL TOPOLOGY 3 to-end experment can be carred out ndependently to study the propertes of the subnetwork covered by the tree. If these trees do not overlap wth each other, the problem of studyng the whole general network can be decomposed nto a group of smaller subproblems, each for one tree. However, due to the complexty of the network topology and practcal constrants, t s more than often that the selected trees overlaps sgnfcantly,.e., some lnks are shared by two or more trees (see Fgure 2 for example). The shared lnks of two selected trees nduce dependence between the two trees. Smply gnorng the dependence leads to a loss of nformaton. How to effectvely ntegrate the nformaton from multple trees to acheve a jont analyss s a major challenge n network tomography of general topology. The frst effort on network tomography of general topology s due to Bu et al. [8], who attempted to extend the method n [2] to networks wth general topology. Unfortunately, the authors faled to derve an explct expresson for the MLE lke the one presented n [2] for ths more general case. They then resorted to an teratve procedure (.e., the EM algorthm) to search for the MLE. In addton, a heurstc method, called mnmum varance weghted average (MVWA) algorthm, was also proposed n [8], whch deals wth each tree n a general topology separately and averages the results. The MVWA algorthm s less effcent than the EM algorthm, especally when the sample sze s small. Rabbat et al. n [40] consdered the tomography problem for networks wth an unknown but general topology, manly focusng on network topology dentfcaton, whch s beyond the scope of our current paper. In ths paper, we provde a new perspectve for the study of loss tomography, whch s applcable to both tree and general topologes. Our theoretcal contrbutons are: 1) ntroducng a set of novel statstcs, whch are complete and mnmal suffcent; 2) dervng two alternatve parameter systems and the correspondng re-parameterzed lkelhood functons, whch beneft us both theoretcally and computatonally; 3) dscoverng that the loss tomography for a general topology shares the same mathematcal formulaton as that of a tree topology; and, 4) showng that the lkelhood functon belongs to the exponental famly and has a unque mode (whch s the MLE) under regularty condtons. Based on these theoretcal results, we propose two new algorthms (a lkelhood-equaton-based algorthm called LE-ξ and an EM-based algorthm called PCEM) to fnd MLE. Compared to exstng methods n the lterature, the proposed methods are computatonally much more effcent. The rest of the paper s organzed as follows. Secton 2 ntroduces notatons for tree topologes and the stochastc model for loss rate nference. Secton 3 descrbes a set of novel statstcs and two alternatve parameter

5 4 K. DENG ET AL. θ θ θ4 θ θ 5 6 θ Fg 1. A multcast tree. 1 θ systems for loss rate analyss n tree topologes. New forms of the lkelhood functon are establshed based on the novel statstcs and re-parametrzaton. Secton 4 extends the above results to general topologes. Secton 5 and 6 propose two new algorthms for fndng MLE of θ, whch enjoy great computatonal advantages over exstng methods. Secton 7 evaluates performances of the proposed methods by smulatons. We conclude the artcle n Secton Notatons and Assumptons Notatons for Tree Topologes. We use T = (V, E) to denote the multcast tree of nterest, where V = {v 0, v 1,...v m } s a set of nodes representng the routers and swtches n the network, and E = {e 1,..., e m } s a set of drected lnks connectng the nodes. Two nodes connected by a drected lnk are called the parent node and the chld node, respectvely, and the drecton of the arrow ndcates that the parent forwards receved probes to the chld. Fgure 1 shows a typcal multcast tree. Note that the root node of a multcast tree has only one chld, whch s slghtly dfferent from an ordnary tree, and each non-root node has exactly one parent. Each lnk s assgned a unque ID number rangng from 1 to m, based on whch each node obtans ts unque ID number rangng from 0 to m correspondngly so that lnk connects node wth ts parent node. Number 0 s reserved for the source node. In contrast to [2] and [13], whose methods are node-centrc, our methods here focus on lnks nstead. For a network wth tree topology, the two reference systems are equvalent as there exsts an one-to-one mappng between nodes and lnks: every node n the network (except for the source) has a unque parent lnk. For a network wth general topology, however, the lnk-centrc system s more convenent, as a node n the network may have multple parent lnks. We let f denote the unque parent lnk, B the brother lnks, and C the

6 LOSS TOMOGRAPHY FOR NETWORK OF GENERAL TOPOLOGY 5 chld lnks, respectvely, of lnk. To be precse, B contans all lnks that share the same parent node wth lnk, ncludng lnk tself. A subtree T = {V, E } s defned as the subnetwork composed of lnk and all ts descendant lnks. We let R and R denote the set of recevers (.e., leaf nodes) n T and T, respectvely. Sometmes, we also use R or R to denote the leaf lnks of T or T. The concrete meanng of R and R can be determned based on the context. Takng the multcast tree dsplayed n Fgure 1 as an example, we have V = {0, 1,, 7}, E = {1, 2,, 7} and R = {4, 5, 6, 7}. For the subtree T 2, however, we have V 2 = {1, 2, 4, 5}, E 2 = {2, 4, 5} and R 2 = {4, 5}. For lnk 2, ts parent lnk s f 2 = {1}, ts brother lnks are B 2 = {2, 3}, and ts chld lnks are C 2 = {4, 5} Stochastc Model. In a multcast experment, n probe packages are sent from the root node 0 to all the recevers. Let X (t) = 1 f the t-th probe package reached node, and 0 otherwse. The status of probes at recevers {X r (t) } r R,1 t n can be drectly observed from the multcast experment; but, the status of probes at nternal lnks cannot be drectly observed. In the followng, we use X R = {X (t) R } 1 t n, where X (t) R = {X(t) r } r R, to denote the data collected n a multcast experment n tree T. In ths paper, we model the loss behavor of lnks by the Bernoull dstrbuton and assume the spatal-temporal ndependence and temporal homogenety for the network,.e. the event {X (t) = 0 X (t) f = 1} are ndependent across and t, and P (X (t) = 0 X (t) f = 0) = 1, P (X (t) = 0 X (t) f = 1) = θ. We call θ = {θ } E the lnk-level loss rates, and the goal of loss tomography s to estmate θ from X R Parameter Space. In prncple, θ can be any value n [0, 1]. Thus, the natural parameter space of θ s Θ = [0, 1] m, an m-dmensonal closed unt cube. In ths paper, however, we assume that θ (0, 1) for every E, and thus constran the parameter space to Θ = (0, 1) m, to smplfy the problem. If θ = 1 for some E, then the subtree T s actually dsconnected from the other part of the network, snce no probes can go through lnk. In ths case, the loss rate of other lnks n T are not estmable due to the lack of nformaton. On the other hand, f θ = 0 for some E, the orgnal network of nterest degenerates to an equvalent network where node s removed and all ts chld nodes are connected drectly to node s parent node. By constranng the parameter space of θ nto Θ, we exclude these degenerate cases from consderaton.

7 6 K. DENG ET AL. 3. Statstcs and Lkelhood Functon The lkelhood functon and the MLE. Gven the loss model for each lnk, we can wrte down the lkelhood functon and use the maxmum lkelhood prncple to determne unknown parameters. That s, we am to fnd the parameter value that maxmzes the log-lkelhood functon: (3.1) arg max θ Θ L(θ) = arg max θ Θ n(x) log P (x; θ), where x stands for the observaton at recevers (.e., a realzaton of X R ), Ω = {0, 1} R s the space of all possble observatons wth R denotng the sze of set R, n(x) s the number of occurrences of observaton x, and P (x; θ) s the probablty of observng x gven parameter value θ. However, the log-lkelhood functon (3.1) s more symbolc than practcal because of the followng reasons: (1) evaluatng the log-lkelhood functon (3.1) s an expensve operaton as t needs to scan through all possble x Ω; and, (2) the lkelhood equaton derved from (3.1) cannot be solved analytcally, and t s often computatonally expensve to pursue a numercal soluton Internal State and Internal Vew. Instead of usng the log-lkelhood functon (3.1) drectly, we consder to rewrte t n a dfferent form. Under the posted probablstc model, the overall lkelhood P (X R θ) s the product of the lkelhood from each probe,.e., P (X R θ) = x Ω n t=1 P (X (t) R θ). Thus, an alternatve form of the overall lkelhood can be obtaned by explcatng the lkelhood of each sngle probe and accumulatng them. Two concepts called nternal state and nternal vew can be generated from ths process Internal State. For a lnk E, gven the observaton of probe t at R and R f, we are able to partally confrm whether the probe passes lnk. Formally, for observaton {X (t) j } j R, we defne Y (t) = max j R X (t) j as the nternal state of lnk for probe t. If Y (t) = 1, probe t reaches at least one recever attached to T, whch mples that the probe passes lnk. Furthermore, by consderng Y (t) f and Y (t) smultaneously, we have three possble scenaros for each nternal node : Y (t) f = Y (t) = 1,.e., we observed that probe t passed lnk ; or

8 LOSS TOMOGRAPHY FOR NETWORK OF GENERAL TOPOLOGY 7 Y (t) f = 1 and Y (t) = 0,.e., we observed that probe t reached node f, but we dd not know whether t reached node or not; or Y (t) f = Y (t) = 0,.e., we dd not know whether probe t reached node f or not at all; as Y (t) f = 0 and Y (t) = 1 can never happen by defnton of Y. The three scenaros have dfferent mpacts on the lkelhood functon. Formally, defne We have (3.2) P (X (t) R where E t,1 = { E : Y (t) = 1}, E t,2 = { E : Y (t) f E t,3 = { E : Y (t) f θ) = E t,1 (1 θ ) = 1, Y (t) = 0}, = Y (t) = 0}. E t,2 ξ (θ), ξ (θ) = P (X j = 0, j R X f = 1; θ) represents the probablty that a probe sendng out from the root node of T fals to reach any leaf node n T Internal Vew. Accumulatng the nternal states of each lnk n the experment, we have (3.3) n (1) = n t=1 Y (t), whch counts the number of probes whose pass through lnk can be confrmed from observatons. Specfcally, we defne n 0 (1) = n. Moreover, defne (3.4) n (0) = n f (1) n (1), for E. We call the statstcs {n (1), n (0)} the nternal vew of lnk. Based on nternal vews, we can wrte the log-lkelhood of X R n a more convenent form: L(θ) = [ ] n (1) log(1 θ ) + n (0) log ξ (θ). E

9 8 K. DENG ET AL Re-parametrzaton. Two alternatve parameter systems can be ntroduced to re-parameterze the above log-lkelhood functon. Frst, based on the defnton of ξ (θ), we have (3.5) ξ (θ) = θ + (1 θ ) j C ξ j (θ), E. Note that f R, we have C =, and (3.5) degenerates to ξ (θ) = θ. Let ξ ξ (θ) for E. Equaton (3.5) defnes a one-to-one mappng between two parameter systems θ = {θ } E and ξ = {ξ } E,.e., Γ : Θ Ξ, ξ = Γ(θ) ( ξ 1 (θ),, ξ m (θ) ), where Θ and Ξ are the doman and mage, respectvely. The nverse mappng of Γ s Γ 1 : Ξ Θ, θ = Γ 1 (ξ) ( θ 1 (ξ),, θ m (ξ) ), where (3.6) θ (ξ) = ξ j C ξ j 1 j C ξ j, E. Usng ξ to replace θ n L(θ), we have the followng alternatve log-lkelhood functon wth ξ as parameters: L(ξ) = [ 1 ξ ] n (1) log( 1 ) + n (0) log ξ. E j C ξ j Second, L(ξ) can be further re-organzed nto L(ξ) = E n (1) log( 1 θ (ξ) ) + n f (1) log ξ ξ E = n log ξ 1 + E n (1) log ψ (ξ), where ψ (ξ) s defned as (3.7) ψ (ξ) = { log 1 θ (ξ) ξ, R, log ξ θ (ξ) ξ, / R. Smlarly, let ψ ψ(ξ) for E. Equaton (3.7) defnes a one-to-one mappng between parameter system ξ = {ξ } E and ψ = {ψ } E,.e., Λ : Ξ Ψ, ψ = Λ(ξ) ( ψ 1 (ξ),, ψ m (ξ) ),

10 LOSS TOMOGRAPHY FOR NETWORK OF GENERAL TOPOLOGY 9 where Ψ Λ(Ξ) s the mage of ψ. The nverse mappng of Λ s It can be shown that Λ 1 : Ψ Ξ, ξ = Λ 1 (ψ) ( ξ 1 (ψ),, ξ m (ψ) ). ψ = log P ( X = 1 X f = 1; X j = 0, j R ) for / R, from whch the physcal meanng of ψ can be better understood. Usng ψ to replace ξ n L(ξ), we have the followng log-lkelhood functon wth ψ as parameters: L(ψ) = n log ξ 1 (ψ) + E n (1)ψ. To llustrate the relatons of θ, ξ and ψ, let s consder the toy network n Fgure 1 wth θ = 0.1 for = 1,, 7. It s easy to check that: ξ 4 = ξ 5 = ξ 6 = ξ 7 = 0.1, ξ 2 = ξ 3 = (1 0.1) = 0.109, ξ 1 = (1 0.1) ; ψ 4 = ψ 5 = ψ 6 = ψ 7 = log , 0.1 ψ 2 = ψ 3 = log , ψ log We lst these concrete values n Table 1 for comparson purpose. The notatons, statstcs and parameter systems are summarzed nto Table 2 for easy reference. Table 1 Comparng dfferent parameter systems for the toy network n Fgure 1 Lnk θ ξ ψ Lkelhood Functon for General Networks. In ths secton, we wll extend the concept of nternal vew and alternatve parameter systems to general networks contanng multple trees. Formally, we use N = {T 1,, T K } to denote a general network covered by K multcast trees,

11 10 K. DENG ET AL. Table 2 Collecton of symbols Symbol Meanng T the multcast tree of nterest V the node set of T E the lnk set of T R the set of leaf nodes (recevers) or leaf lnks of T T the subtree wth lnk as the root lnk V the node set of subtree T R the set of leaf nodes (recevers) or leaf lnks of T f the parent lnk of lnk B the brother lnks of lnk C the chld lnks of lnk X (t) X (t) = 1 f probe t reached node, and 0 otherwse Y (t) max j R X (t) j n n (1) t=1 Y (t), number of probes that passed lnk for sure n (0) n f (1) n (1) θ P (X = 0 X f = 1), the loss rate of lnk ξ P (X j = 0, j R X f = 1), the loss rate of T ψ log P (X = 1 X f = 1; X j = 0, j R ) where each multcast tree T k covers a subnetwork wth S k as the root lnk. For example, Fgure 2 llustrates a network wth K = 2, S 1 = 0 and S 2 = 32. Let S = {S 1,, S K } be the set of root lnks n N. For each tree T k, let {n k, (1), n k, (0)} be the nternal vew of lnk E k n T k based on the n k probes sent out from S k. Specally, defne n k, (1) = 0 for lnk / E k. The experment on T k results n the followng log-lkelhood functons wth θ, ξ and ψ as parameters, respectvely: L k (θ) = [ ] n k, (1) log(1 θ ) + n k, (0) log ξ (θ), E L k (ξ) = [ 1 ξ ] n k, (1) log( 1 ) + n k, (0) log ξ, E j C ξ j L k (ψ) = n k log ξ Sk (ψ) + E n k, (1)ψ. Consderng that a multcast experment n a general network N s a pool of K ndependent experments n the K trees of N, the log-lkelhood functon of the whole experment s just the summaton of K components,

12 LOSS TOMOGRAPHY FOR NETWORK OF GENERAL TOPOLOGY Fg 2. A 5-layer network covered by two trees..e., L(θ) = K L k (θ), L(ξ) = k=1 K L k (ξ) and L(ψ) = k=1 Defne the nternal vew of lnk n a general network as: (4.1) n (1) = K n k, (1), n (0) = k=1 K n k, (0). k=1 It s straghtforward to see that L(θ) = [ ] (4.2) n (1) log(1 θ ) + n (0) log ξ (θ), E K L k (ψ). k=1 (4.3) L(ξ) = [ 1 ξ ] n (1) log( 1 ) + n (0) log ξ, E j C ξ j K (4.4) L(ψ) = n k log ξ Sk (ψ) + n (1)ψ. k=1 E Note that n ths general case, we have: for S, n Sk (1) = n k ; and for / S, n (0) + n (1) = n j (1). j F

13 12 K. DENG ET AL. Here, F stands for the unque or multple parent lnks of lnk n the general network N. 5. Lkelhood Equaton. The bjectons among the the three parameter systems mean that we can swtch among them freely wthout changng the result of parameter estmaton: Proposton 1. The results of lkelhood nference based on the three parameter systems are equvalent,.e., Γ ( arg max θ Θ L(θ)) = arg max ξ Ξ L(ξ) = Λ 1( arg max ψ Ψ L(ψ)) ; and the lkelhood equatons under dfferent parameters share the same soluton,.e., L(θ) θ = 0 L(ξ) θ=θ ξ = 0 L(ψ) ξ=ξ ψ = 0, ψ=ψ f θ Θ, or ξ Ξ, or ψ Ψ, where Γ(θ ) = ξ = Λ 1 (ψ ). Ths flexblty provdes us great theoretcal and computatonal advantages. On the the theoretcal aspect, as {n k } K k=1 are known constants, L(ψ) falls nto the standard exponental famly wth ψ as the natural parameters. Thus, based on the propertes of the exponental famly [42], we have the followng results mmedately: Proposton 2. The followng results hold for loss tomography: 1. Statstcs {n (1)} E are complete and mnmal suffcent; 2. The lkelhood equaton L(ψ) ψ = 0 has at most one soluton ψ Ψ; 3. If ψ exsts, ψ (or θ = (Λ Γ) 1 (ψ )) s the MLE. On the computatonal aspect, the parameter system ξ plays a central role. Dfferent from the lkelhood equaton wth θ as parameters, whch s ntractable, the lkelhood equaton wth ξ as parameters enjoys unque computatonal advantages. Let r = n (1) n (1) + n (0). It can be shown that the lkelhood equaton wth ξ as parameters s: (5.1) ξ = (1 r ) + r j B ξ j I( / S ),

14 LOSS TOMOGRAPHY FOR NETWORK OF GENERAL TOPOLOGY 13 for any lnk E. For lnk S, (5.1) degenerates to ξ = 1 r. For lnk / S, defne π = j B ξ j. We note that solvng ξ from (5.1) s equvalent to solvng the followng equaton about π : (5.2) π = [ ] (1 rj ) + r j π, j B whch can be solved analytcally when B = 2, and by numercal approaches [43] when B > 2. From (5.1) and (5.2), t s transparent that only local statstcs {r j } j B are nvolved n estmatng ξ. Ths observaton leads to the followng fact mmedately: f a processor keeps the values of {r j } j B n ts memory, {ξ j } j B can be effectvely estmated by the processor ndependent of estmatng the other parameters. Based on ths unque property of the lkelhood equaton wth ξ as parameter, we propose a parallel procedure called LE-ξ algorthm to estmate θ (see Algorthm 1). Algorthm 1. The LE-ξ Algorthm Hardwre requrement: {P } : a collecton of processors ndexed by node ID V ; Input: {r j} j E wth dstrbuted storage where P keeps {r j} j C ; Output: ˆθ = {ˆθ j} j E. Procedure: Operate n parallel for IDs of all non-leaf nodes { V : / R} If S, Get ˆξ j = 1 r j for the unque chld lnk j of node wth P ; If / S, Get ˆx wth P from {r j} j C by solvng equaton below about x x = [ j C (1 rj) + r ] jx, Get ˆξ j = (1 r j) + r j ˆx for every j C wth P ; End parallel operaton Return ˆθ = Γ 1 (ˆξ). The valdty of the LE-ξ algorthm s guaranteed by the followng theorem: Theorem 1. When n k for all 1 k K, wth probablty one, the LE-ξ algorthm has a unque soluton n Θ = (0, 1) m that s the MLE.

15 14 K. DENG ET AL. Proof of Theorem 1. Frst, we wll show that under the followng regularty condtons: (5.3) n (1) > 0, n (0) > 0 and j F n j (1) < j B n j (1) for E, the lkelhood equaton wth ξ as parameters has a unque soluton n (0, 1) m. Note that when (5.3) holds, we have 0 < r < 1 and j B r j > 1 for E. Because the Lemma 1 n [2] has shown that: f 0 < c j < 1 and j c j > 1, equaton for x x = [ (1 cj ) + c j x ] j has a unque soluton n (0, 1). It s transparent that under the regularty condtons n (5.3), equaton (5.2) has a unque soluton ˆπ (0, 1) for every non-root lnk. Consderng that for every lnk E, ξ = (1 r ) + r π I( / S ), t s straghtforward to see that the lkelhood equaton wth ξ as parameters has a unque soluton ˆξ (0, 1) m. Moreover, f (5.4) ˆξ Ξ, we have ˆψ = Λ(ˆξ) Ψ. Thus, based on Proposton 1 and Proposton 2, we know that f both (5.3) and (5.4) are satsfed, ˆψ, whch s the soluton of the lkelhood equaton wth ψ as parameter, s the MLE. Consderng the bjectons among θ, ξ and ψ, ths also means that ˆθ = Γ 1 (ˆξ) s the MLE. Note that the probablty that (5.3) or (5.4) fals goes to zero when n k for k = 1,, K, we complete the proof. The large sample propertes of MLE have been well studed by [2] for tree topology, where the asymptotc normalty, asymptotc varance and confdence nterval are establshed. Consderng that the lkelhood functon keeps exactly the same form for both tree and general topology, t s natural to extend these theoretcal results for MLE establshed n [2] to a network of general topology.

16 LOSS TOMOGRAPHY FOR NETWORK OF GENERAL TOPOLOGY 15 Wth a fnte sample, there s a chance that (5.3) or (5.4) fals. In ths case, the MLE of θ falls out of Θ, and may be mssed by the LE-ξ algorthm. For example, 1. If n (1) = 0 and n (0) > 0, we have ˆξ = 1; 2. If n (1) > 0 and n (0) = 0, we have ˆξ = 0; 3. If j F n j (1) = j B n j (1) > 0, we have ˆθ = 0; 4. If ˆξ ˆπ = j B ˆξj (.e., ˆξ / Ξ), we have ˆθ 0. Moreover, f n (1) = n (0) = 0, the parameters n subtree T (.e., {θ j } j T ) are not estmable due to lack of nformaton. In all these cases, we cannot guarantee that the estmate from the LE-ξ algorthm ˆθ s the global maxmum n Θ = [0, 1] m. In practce, we can ncrease sample sze by sendng addtonal probes to avod ths dlemma. In case that t s not realstc to send addtonal probes, we can smply replace ˆθ by the pont n Θ that s the closest to ˆθ, or search the boundary of Θ to maxmze L(θ). 6. Impacts on the EM Algorthm. The above theoretcal results have two major mpacts to the EM algorthm wdely used n loss tomography. Frst, as we have shown n Theorem 1, as long as regularty condtons (5.3) and (5.4) hold, lkelhood functon L(θ) has a unque mode n Θ. In ths case, the EM algorthm always converges to the MLE for any ntal value θ (0) Θ. Consderng that the chance of volatng (5.3) or (5.4) goes to zero wth the ncrease of sample sze, we have the followng corollary for the EM algorthm mmedately: Corollary 1. When n k for all 1 k K, the EM algorthm converges to the MLE wth probablty one for any ntal value θ (0) Θ. Second, the formulaton of the new statstcs {n (1)} E naturally leads to a pattern-collapsed mplementaton of the EM algorthm, whch s computatonally much more effcent than the wdely used nave mplementaton where the samples are processed one by one separately. In the nave mplementaton of the EM algorthm, one enumerates all possble confguratons of the nternal lnks compatble wth each sample carryng the nformaton on whether the correspondng probe reached the leaf nodes or not. The complexty of the nave mplementaton n each E-step can be O(n2 m ) n the worst case. Wth the pattern-collapsed mplementaton, however, the complexty of an E-step can be dramatcally reduced to O(m). The frst pattern collapsed EM algorthm s proposed by Deng et al. [23] n the context of delay tomography. The basc dea s to reorganze the observed

17 16 K. DENG ET AL. data at recevers n delay tomography nto delay patterns, and make use of a delay pattern database to greatly reduce the computatonal cost n the E-step. We wll show below that the sprt of ths method can be naturally extended to the study of loss tomography. Defne event {Y = 0 Y f = 1}, or equvalently {X j = 0, j R X f = 1}, as the loss event n subtree T, denoted as L. Let L(X V ; θ) be the loglkelhood of the complete data where the status of probes at nternal nodes are also observed, θ (r) be the estmaton obtaned at the r-th teraton, the objectve functon to be maxmzed n the (r + 1)-th teraton of the EM algorthm s: (6.1) Q(θ, θ (r) ( ) n ) = E θ (r) L(XV ; θ) X R = t=1 E θ (r) ( L(X (t) E = [ ] n (1) log(1 θ ) + n (0)Q(θ T, θ (r) ), E ; θ) X(t) R ) where Q L (θ T, θ (r) ), whch s called the localzed Q-functon of loss event L, s defned as: ] Q L (θ T, θ (r) ) = E θ (r) [logp (X T ; θ T ) L. Smlar to the results for delay patterns shown n the Proposton 1 of [23], Q L (θ T, θ (r) ) has the followng decomposton: (6.2) Q L (θ T, θ (r) ) = ( 1 ψ (θ (r) ) ) [ log(θ )+ψ (θ (r) ) log(1 θ )+ ] Q Lj (θ Tj, θ (r) ). j C Integratng (6.1) and (6.2), we have Q(θ, θ (r) ) = [ ] ω (1) log(1 θ ) + ω (0) log(θ ), E where {ω (1), ω (0)} E are defned recursvely as follows: for {S 1,, S K },.e., beng one of the root lnks, for / {S 1,, S K }, ω (1) = n (1) + n (0) ψ (θ (r) ), ω (0) = n (0) (1 ψ (θ (r) ) ) ; ω (1) = n (1) + j F ω j (0) ψ j (θ (r) ), ω (0) = j F ω j (0) (1 ψ j (θ (r) ) ).

18 LOSS TOMOGRAPHY FOR NETWORK OF GENERAL TOPOLOGY 17 And, the M-step s: θ (r+1) = ω (0) ω (0) + ω (1), E. In ths paper, we refer to the EM wth the pattern-collapsed mplementaton as PCEM to dstngush from the EM wth the nave mplementaton, whch s referred to as NEM. We have shown n [23] by smulaton and theoretcal analyss that PCEM s computatonally much more effcent than the nave EM n the context of delay tomography. In context of loss tomography, t s easy to see from the above analyss that PCEM enjoys a complexty of O(m) for each E-step, whch s much more effcent than the nave EM, whose complexty can be O(n2 m ) for each E-step n the worst case. 7. Smulaton Studes. We verfy the followng facts va smulatons: 1. PCEM obtans exactly same results as the nave EM for both tree topology and general topology, but s much faster; 2. LE-ξ obtans exactly same results as the EM algorthm when regularty condtons (5.3) and (5.4) hold, but s much faster when the speedup from parallel computaton s consdered. We carred out smulatons on a 5-layer network covered by two trees as showed n Fgure 2. The network has 49 nodes labeled from Node 0 to Node 48, and two sources Node 0 and Node 32. We conduct two set of smulatons. One s based on the deal model, and the other s by usng network smulator 2 (ns-2, [44]) Smulaton study wth the deal model. In the frst set of smulatons, the data s generated from the deal model where each lnk has a pre-defned constant loss rate. To test the performance of methods on dfferent magntude of loss rates, we draw the lnk-level loss rates from Beta dstrbutons wth dfferent parameters. The lnk-level loss rates θ = {θ } are randomly sampled from Beta(1, 99), Beta(5, 995), Beta(2, 998) or Beta(1, 999). For example, the mean of Beta(1, 999) s 0.001, so f we draw loss rates from Beta(1, 999), then on average we expect a 0.1% lnk-level loss rate n the network. Gven the loss rate vector θ for each network, we generated 100 ndependent datasets wth sample szes 50, 100, 200 and 500, respectvely. Thus, a total of = 1600 datasets were smulated. The sample sze s evenly dstrbuted nto the two trees, e.g., the source of each tree wll send out 100 probes when sample sze n = 200.

19 18 K. DENG ET AL. Table 3 Performance of dfferent methods on smulated data from deal model on the 5-layer network n Fgure 2. Beta(1,100) Beta(5,1000) Beta(2,1000) Beta(1,1000) n Method Tme (ms) MSE (1e-5) Tme (ms) MSE (1e-5) Tme (ms) MSE (1e-5) Tme (ms) MSE (1e-5) NEM PCEM LE-ξ MVWA NEM PCEM LE-ξ MVWA NEM PCEM LE-ξ MVWA NEM PCEM LE-ξ MVWA To each of the 1600 smulated data sets, we appled NEM, PCEM, LEξ and MVWA, respectvely. The stoppng rule of the EM algorthms s max θ (t+1) θ (t) 10 6, the ntal values are θ (0) = 0.03 for all E. We mplement MVWA algorthm by obtanng MLE estmate n each tree wth LE-ξ algorthm. Table 3 summarze the average runnng tme and MSEs of these methods under dfferent settngs. From the tables, we can see that: 1. The performances of all four methods n term of MSE mprove wth the ncrease of sample sze n n both networks. 2. NEM and PCEM always get exactly same results. 3. The result from LE-ξ s slghtly dfferent from these of the EM algorthms when sample sze n s as small as 100 or 200 due to the volaton of the regularty condtons (5.3) and (5.4). When n = 500, the results of LE-ξ and EM algorthms become dentcal. 4. LE-ξ, PCEM and MVWA algorthm mplemented wth LE-ξ are dramatcally faster than NEM. For example, n smulaton confguraton wth Beta(1, 1000) and sample sze n = 500, the runnng tme of NEM s almost 2,000 tmes of LE-ξ and PCEM. 5. The MSE of MVWA s always larger than the MSE of the other three methods, whch capture the MLE. Ths s consstent wth the result n [8] Smulaton study by network smulator 2. We also conduct the smulaton study usng ns-2. We use the network topology shown n Fgure 2, where the two sources located at Node 0 and Node 32 multcast probes to the attached recevers. Besdes the network traffc created by multcast

20 LOSS TOMOGRAPHY FOR NETWORK OF GENERAL TOPOLOGY 19 Table 4 Performance of dfferent methods on smulated data from network smulator 2. Sm tme (n) Method Tme (ms) MSE (1e-5) NEM s (528) PCEM LE-ξ MVWA NEM s (2038) PCEM LE-ξ MVWA NEM s (5648) PCEM LE-ξ MVWA NEM s (13355) PCEM LE-ξ MVWA probng, a number of TCP sources wth varous wndow szes and a number of UDP sources wth dfferent burst rates and perods are added at several nodes to produce cross-traffc. The TCP and UDP cross-traffc takes about 80% of the total network traffc. We generated 100 ndependent datasets by runnng the ns-2 smulaton for 1, 2, 5 or 10 smulaton seconds. The longer experments generate more multcast probes samples. We record all pass and loss events for each lnk durng the smulaton, and the actual loss rates are calculated and consdered as the true loss rates for calculatng MSEs. Table 4 shows the average number of packets (sample sze), runnng tme of methods and MSEs for dfferent ns-2 smulaton tmes. In Table 4, we observe smlar results as shown n Table 3. The performances of all four methods n term of MSE mprove wth the ncrease of sample sze n n both networks. The result from LE-ξ s slghtly dfferent from these of the EM algorthms when sample sze n s as small. More mportantly, LE-ξ and PCEM are dramatcally faster than NEM. 8. Concluson. We proposed a set of suffcent statstcs called nternal vew and two alternatve parameter systems ξ and ψ for loss tomography. We found that the lkelhood functon keeps the exactly same formulaton for both tree and general topologes under all three types of parametrzaton (θ, ξ and ψ), and we can swtch among the three parameter systems freely wthout changng the result of parameter estmaton. We also dscovered that the parameterzaton of the lkelhood functon based on ψ falls nto the standard exponental famly, whch has a unque mode n parameter space Ψ under regularty condtons, and the parametrzaton based on ξ leads to an effcent algorthm called LE-ξ to calculate the MLE, whch can be carred out n a parallel fashon. These results ndcate that loss tomography for general topologes enjoys the same mathematcal nature

21 20 K. DENG ET AL. as that for tree topologes, and can be resolved effectvely. The proposed statstcs and alternatve parameter systems also lead to a more effcent pattern-collapsed mplementaton of the EM algorthm for fndng MLE, and a theoretcal promse that the EM algorthm converges to the MLE wth probablty one when sample sze s large enough. Smulaton studes confrmed our theoretcal analyss as well as the superorty of the proposed methods over exstng methods. REFERENCES [1] Y. Vard. Network Tomography: Estmatng Source-Destnaton Traffc Intenstes from Lnk Data. Journal of the Amercan Statstcal Assocaton, 91(433): , [2] R. Cáceres, N.G. Duffeld, J. Horowtz, and D. Towsley. Multcastbased nference of network-nternal loss characterstcs. IEEE Transactons on Informaton Theory, 45(7): , [3] R. Cáceres, N.G. Duffeld, S.B. Moon, and D. Towsley. Inference of Internal Loss Rates n the MBone. In IEEE/ISOC Global Internet 99, Vol. 3, , [4] R. Cáceres, N.G. Duffeld, S.B. Moon, and D. Towsley. Inferrng lnklevel performance from end-to-end multcast measurements. Techncal report, Unversty of Massachusetts, [5] K. Harfoush, A. Bestavros, and J. Byers. Robust dentfcaton of shared losses usng end-to-end uncast probes. In Techncal Report BUCS , Boston Unversty, [6] M. Coates and R. Nowak. Uncast network tomography usng EM algorthms. Techncal Report TR-0004, Rce Unversty, September [7] M. Coates and R. Nowak. Network loss nference usng uncast end-to-end measurement. In the ITC Specalst Semnar on IP Traffc Measurement, Modelng and Management, [8] T. Bu, N. Duffeld, F.L. Prest, and D. Towsley. Network Tomography on General Topologes. In Proc. of ACM SIGMETRICS vol. 30, no. 1, 21-30, [9] F. LoPrest, D.Towsley, N.G. Duffeld, and J. Horowtz. Multcast topology nference from measured end-to-end loss. IEEE Transactons on Informaton Theory, 48(1): 26-45, [10] N. Duffeld, J. Horowtz, D. Towsley, W. We, and T. Fredman. Multcast-based loss nference wth mssng data. IEEE Journal of Selected Areas of Communcaton, 20(4): [11] W. Zhu and Z. Geng. A fast method to estmate loss rates. Informaton Networkng. Networkng Technologes for Broadband and Moble Networks, , [12] W. Zhu and Z. Geng. A bottom up nference of loss rate. Computer Communcatons, 28: , [13] N. Duffeld, J. Horowtz, F. Prest, and D. Towsley. Explct loss nference n multcast tomography. IEEE Transactons on Informaton Theory, 52(8): , [14] N.G. Duffeld, J. Horowtz, F.L. Prest, and D. Towsley. Network delay tomography from end-to-end uncast measurements. Lecture Notes n Computer Scence, 2170: , [15] F.L. Prest, N.G. Duffeld, J. Horwtz, and D. Towsley. Multcast-based nference of network-nternal delay dstrbuton. IEEE/ACM Transactons on Networkng, 10(6): , 2002.

22 LOSS TOMOGRAPHY FOR NETWORK OF GENERAL TOPOLOGY 21 [16] G. Lang and B. Yu. Maxmum pseudo lkelhood estmaton n network tomography. IEEE Transactonson on Sgnal Processng, 51(8): , [17] Y. Tsang, M. Coates, and R. Nowak. Network delay tomography. IEEE Transactonson on Sgnal Processng, 51(8): , [18] M.F. Shh and A.O. Hero III. Uncast-based nference of network lnk delay dstrbutons wth fnte mxture models. IEEE Transactonson on Sgnal Processng, 51(8): , [19] E. Lawrence, G. Mchalds, and V. Nar. Network delay tomography usng flexcast experments. Journal of the Royal Statstcal Socety, Seres B, 68(5): , [20] V. Arya, N.G. Duffeld, and D. Vetch. Temporal delay tomography. The 27th Conference on Computer Communcatons. IEEE. 2008: , [21] I.H. Dnwoode and E.A. Vance. Moment estmaton n delay tomography wth spatal dependence. Performance Evaluaton Archve, Volume 64, Issue 7-8, , [22] A. Chen, J. Cao, and T. Bu. Network Tomography: Identfablty and Fourer Doman Estmaton. IEEE Transcaton on Sgnal Processng, 58(12): , [23] K. Deng, Y. L, W. Zhu, Z. Geng, and J.S. Lu. On Delay Tomography: Fast Algorthms and Spatally Dependent Models. IEEE Transactons on Sgnal Processng, 60(11): , [24] M.G.H. Bell. The estmaton of orgn-destnaton matrces by constraned generalzed least squares. Transportaton Research, Seres B 25B(1): 13-22, [25] D.D Ortuzar and L.G. Wllumsen. Modellng transport. John Wley and Sons, [26] E. Cascetta and S. Nguyen. A unfed framework for estmatng or updatng orgn/destnaton matrces from traffc counts. Transportaton Research Part B: Methodologcal, 22(6): , [27] H. Yang, T. Sasak, Y. Ida and Y. Asakura. Estmaton of orgn-destnaton matrces from lnk traffc counts on congested networks. Transportaton Research Part B: Methodologcal, 26(6): , [28] H.P. Lo, N. Zhang and W.H.K. Lam. Estmaton of an orgn-destnaton matrx wth random lnk choce proportons: a statstcal approach. Transportaton Research Part B: Methodologcal, 30(4): , [29] L. Banco, G. Confessore, and P. Reverber. A network based model for traffc sensor locaton wth mplcatons on O/D matrx estmates. Transportaton Scence, 35(1): 50-60, [30] C. Tebald and M. West. Bayesan nference on network traffc usng lnk count data. Journal of the Amercan Statstcal Assocaton, 93(442): , [31] J. Cao, D. Davs, S. Van Der Vel, B. Yu, and Z. Zu. A scalable method for estmatng network traffc matrces from lnk counts. Techncal report, Bell Labs, 41, [32] J. Cao, W. S. Cleveland, D. Ln, and D. X. Sun. The effect of statstcal multplexng on the long range dependence of nternet packet traffc. Techncal report, Bell Labs [33] A. Medna, N. Taft, K. Salamatan, S. Bhattacharyya, and C. Dot. Traffc matrx estmaton: exstng technques and new drectons. SIGCOMM Computer Communcaton Revew, 32(4): , [34] Y. Zhang, M. Roughan, C. Lund, and D. Donoho. An nformaton-theoretc approach to traffc matrx estmaton. In Proceedngs of SIGCOMM, , [35] R. Castro, M. Coates, G. Lang, R. Nowak, and B. Yu. Network tomography: Recent developments. Statstcal Scence, 19(3): , [36] G. Lang, N. Taft, and B. Yu. A fast lghtweght approach to orgn-destnaton IP traffc estmaton usng partal measurements. IEEE Transactons on Informaton Theory, 52(6): , 2006.

23 22 K. DENG ET AL. [37] J. Fang, Y. Vard, and C. Zhang. An teratve tomogravty algorthm for the estmaton of network traffc. In R. Lu, W. Strawderman, and C. Zhang (Eds.), Complex Datasets and Inverse Problems: Tomography, Networks and Beyond, Volume 54 of Lecture Notes-Monograph Seres. IMS , [38] E. M. Arold, and A. W. Blocker, Estmatng latent processes on a network from ndrect measurements. Journal of the Amercan Statstcal Assocaton, 108(501): , [39] V. Arya, N.G. Duffeld, and D. Vetch. Multcast nference of temporal loss characterstcs. Performance Evaluaton, 64(9): , [40] M. Rabbat, R. Nowak, and M. Coates. Multple Source, Multple Destnaton Network Tomography. In Proc. of IEEE Infocom 04, , [41] A. Dempster, N. Lard, and D. Rubn. Maxmum lkelhood from ncomplete data va the EM algorthm. Journal of the Royal Statstcal Socety, Seres B, Volume 34, 1-38, [42] E.L. Lehmann and G Casella. Theory of Pont Estmaton. Sprnger, 2nd edton, [43] M.J. Todd. The computaton of fxed ponts and applcatons. Sprnger-Verlag, New York, [44] The network smulator 2 (ns-2). Address of the Frst author Yau Mathematcal Scences Center & Center for Statstcal Scence, Tsnghua Unversty, Bejng , Chna E-mal: kdeng@math.tsnghua.edu.cn Address of the Second author Department of Statstcs, Harvard Unversty, Cambrdge, MA 02138, USA E-mal: yl01@fas.harvard.edu Address of the Thrd author Department of ITEE, ADFA@Unversty of New South Wales, Canberra ACT 2600, Australa E-mal: w.zhu@adfa.edu.au Address of the Fourth author Department of Statstcs, Harvard Unversty, Cambrdge, MA 02138, USA E-mal: jlu@stat.harvard.edu