Adaptive Matching for Expert Systems with Uncertain Task Types

Transcription

1 Adaptive Mathing for Expert Sytem with Unertain Tak Type VIRAG SHAH, LENNART GULIKERS, and LAURENT MASSOULIÉ, Mirooft Reearh-INRIA Joint Centre MILAN VOJNOVIĆ, London Shool of Eonomi Abtrat Online two-ided mathing market uh a Q&A forum (e.g. StakOverflow, Quora) and online labour platform (e.g. Upwork) ritially rely on the ability to propoe adequate mathe baed on imperfet knowledge of the two partie to be mathed. Thi prompt the following quetion: Whih mathing reommendation algorithm an, in the preene of uh unertainty, lead to effiient platform operation? To anwer thi quetion, we develop a model of a tak / erver mathing ytem. For thi model, we give a neeary and uffiient ondition for an inoming tream of tak to be manageable by the ytem. We further identify a o-alled bak-preure poliy under whih the throughput that the ytem an handle i optimized. We how that thi poliy ahieve tritly larger throughput than a natural greedy poliy. Finally, we validate our model and onfirm our theoretial finding with experiment baed on log of Math.StakExhange, a StakOverflow forum dediated to mathemati. INTRODUCTION Online platform that enable mathe between trading partner in two-ided market have reently bloomed in many area: LinkedIn and Upwork failitate mathe between employer and employee; Uber allow mathe between paenger and ar driver; Airbnb and Booking.om onnet traveler and houing failitie; Quora and Stak Exhange failitate mathe between quetion and either anwer, or expert able to provide them. All thee ytem ruially rely on the ability to propoe adequate mathe baed on imperfet knowledge of the harateriti of the two partie to be mathed. For example, in the ontext of online labour platform, there i unertainty about both the kill et of andidate employee and the job requirement. Similarly, in the ontext of online Q&A platform, there i unertainty about both quetion type and uer ability to provide anwer. Thi naturally lead to the following quetion: whih mathing reommendation algorithm an, in the preene of uh unertainty, lead to effiient platform operation? A natural meaure of effiieny i the throughput that the platform ahieve, i.e. the rate of ueful mathe it allow. To addre thi quetion, one thu need firt to haraterize fundamental limit on the ahievable throughput. In thi paper, we progre toward anwering thee quetion a follow. Firt, we propoe a imple model of uh platform, whih feature a tati olletion of erver, or expert on the one hand, and a ontinuou tream of arrival of tak, or job, on the other hand. In our model, the platform operation onit of erver iteratively attempting to olve tak. After being proeed by ome erver, a tak leave the ytem if olved; otherwie it remain till uefully treated by ome erver. To model unertainty about tak type, we aume that for eah inoming tak we are given the prior ditribution of thi tak true type. Server abilitie are then repreented via the probability that eah erver ha to olve a tak of given type after one attempt at it. In a Q&A platform enario, tak are quetion, and erver are expert; a erver proeing a tak orrepond to an expert providing an anwer to a quetion. A tak being olved orrepond to an anwer being aepted. In an online labour platform, tak ould be job offer, and a erver may be a pool of worker with imilar abilitie. A erver proeing a tak then orrepond to a worker being interviewed for a job, and the tak i olved if the interview lead to a hire. We ould alo onider the dual interpretation when the labour market i ontrained

2 :2 Virag Shah, Lennart Guliker, Laurent Maoulié, and Milan Vojnović by worker rather than job offer. Then a tak i a worker eeking work, while a erver i a pool of employer looking for hire. An important feature of our model onit in the fat that when a tak proeing doe not lead to failure, it doe however affet unertainty about the tak type. Indeed, the a poteriori ditribution of the tak type after a failed attempt on it by ome erver differ from it prior ditribution. For intane in a Q&A enario, a quetion whih an expert in Calulu failed to anwer either i not about Calulu, or i very hard. For our model, we then determine neeary and uffiient ondition for an inoming tream of tak arrival to be manageable by the erver, or in other word, determine ahievable throughput of the ytem. In the proe we introdue andidate poliie, in partiular the greedy poliy aording to whih a erver hoe to erve tak for whih it hane of ue i highet. Thi heduling trategy i both eay to implement and i baed on a natural motivation. Surpriingly perhap, we how that it i not optimal in the throughput it an handle. In ontrat, we introdue a o-alled bakpreure poliy inpired from the wirele networking literature [33], whih we prove to be throughput-optimal. To validate our modeling and theoretial reult, we analyze log from Math.StakExhange, a StakOverflow forum peializing on mathemati. Our data analyi orroborate everal apet of our model, notably the modeling of unertainty about tak type a a prior on tak type, together with the modeling of erver ability through their ue probability on eah tak type. Numerial experiment baed on Math.StakExhange data analyi onfirm the theoretial reult, namely that our bakpreure poliy an ahieve ignifiantly higher throughput than other baeline poliie. We ummarize ontribution of thi paper a follow: We propoe a new model of a generi tak-expert ytem that allow for unertainty of tak type, heterogeneity of kill, and reurring attempt of expert in olving tak. We provide a full haraterization of the tability region, or utainable throughput, of the tak-expert ytem under onideration. We etablih that a partiular bakpreure poliy i throughput-optimal, in the ene that it upport maximum tak arrival rate under whih the ytem i table. We how that there exit intane of tak-expert ytem under whih imple mathing poliie uh a a natural greedy poliy and a random poliy an only upport a muh maller maximum tak arrival rate, than the bakpreure poliy. We report the reult of empirial analyi of the popular Math.StakExhange Q&A platform whih etablih heterogeneity of kill of expert, with expert knowledgeable aro different type of tak and other peialized in partiular type of tak. We alo how numerial evaluation reult that onfirm the benefit of the bakpreure poliy on greedy and random mathmaking poliie. The remainder of the paper i trutured a follow. Setion 2 preent our ytem model. In Setion 3, we preent reult for two baeline mathmaking poliie, namely Greedy and Random, and a haraterization of ahievable throughput under Random. Setion 4 preent the haraterization of tak arrival rate that an be upported under whih the ytem i table and prove the uperiority of bakpreure poliy over Random and Greedy. In Setion 5, we preent our experimental reult. Related work i diued in Setion 6. We onlude in Setion 7. Proof of the reult are provided in the Setion 8. 2 SYSTEM MODEL We denote by C the et of type that tak an take, and aume C < +. We let C denote the et of probability ditribution on C. We aume that eah inoming tak ha an aoiated type whih i not oberved. Intead, the probability ditribution z = {z } C C of the tak type i provided. Throughout, we will refer to uh ditribution a mixed type. The rationale for auming that uh mixed type are available i that they repreent ide information revealed upon tak arrival (e.g. for a quetion in a Q&A forum, ide information ould be it

3 Adaptive Mathing for Expert Sytem with Unertain Tak Type :3 text and aoiated tag; in online labor platform thi ould be a job deription and prior knowledge about the ompany poting the offer). We let S be a et of erver (or expert) preent in the ytem. For eah S, C, p, denote the probability that erver olve a tak of type after proeing it. We aume that a given tak may be inpeted everal time by a given erver and that the outome ue / failure are independent at eah inpetion. Thi an be jutified if a label in fat repreent a olletion of expert with imilar abilitie, in whih ae multiple proeing by orrepond to proeing by ditint individual expert. We make the following tatitial aumption. The ubequent mixed type Z i of inoming tak are aumed i.i.d., taking value in a ountable ubet Z of C, and we let π z = P (Z i = z) for all z Z. Tak arrive at a rate of λ per time unit on average. Finally, the time for erver to omplete an attempt on a job take on average /µ time unit, and uh attempt duration are i.i.d.. All involved oure of randomne are independent. For impliity we aume more peifially that tak arrive at the intant of a Poion proe with intenity λ, and that the time for erver to omplete an attempt at a tak follow an Exponential ditribution with parameter µ. Thee aumption will imply that the ytem tate at any given time t an be repreented a a Markov proe, whih implifie analyi, but i not eential for the throughput optimality propertie that onern u here. We now deribe diret onequene of the aumption jut made. When erver proee a tak with mixed type z Z, then the probability that it fail i given by ψ (z) = z ( p, ). () C Moreover, onditional on uh failure, Baye formula readily implie that the mixed type of the job then beome { } z ( p, ) ϕ (z) = (2) ψ (z) Thee two funtion will be intrumental in the equel to deribe poliie of interet. They will alo be ued to haraterize arrival rate λ for whih the ytem an be tabilized, i.e. for whih there exit a heduling poliy whih indue a tationary regime of the ytem behavior. Thi i our primary onern in thi work: indeed for a table ytem the long term tak reolution rate oinide with the tak arrival rate λ, and thu throughput-optimal poliie mut make the ytem table whenever thi i poible. We loe the etion with additional aumption and notation. We aume that Z i loed under ϕ ( ), i.e., for eah z Z, ϕ (z) Z. Thi loe no generality, a the loure of a ountable et with repet to a finite number of map ϕ remain ountable. At any time t let N z (t) repreent the number of tak of mixed-type z preent in the ytem and N (t) = {N z (t)} z Z. We alo let z(, t) denote the mixed type of that tak that erver work on at time t. For trategie uh that the erver elet whih tak to handle baed uniquely on the vetor N (t), the proe {N (t)} t 0 form a ontinuou-time Markov hain (CTMC) [6]. The poliie onidered in thi paper are tudied by analyzing the aoiated CTMC. We allow a tak to be aigned to multiple expert at a given time. Further, we allow pre-emptive ervie, i.e., an expert may drop ervie of a tak hould a new tak arrive into a ytem or an exiting tak reeive a repone. 3 BASELINE POLICIES We onider a natural Greedy poliy where eah expert i aigned a tak whih bet uit it kill, i.e., among the outtanding tak, an expert i aigned a tak of a mixed-type z whih minimize ψ (z). A quetion arie: doe a myopi Greedy poliy optimize the long term throughput in a tabilizable ytem? We will how via an example that the anwer to thi quetion i negative. We alo onider a Random poliy, where eah expert i aigned a tak hoen uniformly at random and haraterize it tability region. We provide example enario where both Greedy and Random have tability region whih are equivalent, and yet tritly ub-optimal. C

4 :4 Virag Shah, Lennart Guliker, Laurent Maoulié, and Milan Vojnović 3. Greedy Poliy Definition 3. (Greedy Poliy). A poliy i Greedy if given the ytem tate eah expert i aigned an outtanding tak whih maximize it ue probability, i.e., for eah N uh that N > 0 we have z() A (N ) = arg min z:n z >0 ψ (z), where tie ould be broken arbitrarily, e.g., uniformly at random among thi et. If tie are broken uniformly at random then the tranition rate for the CTMC under greedy poliy are given a follow. Let q(n, n ) be the tranition rate from tate n to tate n. Let e z denote the vetor with all oordinate equal to 0 exept for the z-oordinate whih equal. Fix a tate n. For eah z Z we have q(n, n + e z ) = λπ z, q(n, n e z ) = :z A (N ) q(n, n e z + e ϕ (z)) = µ ( ψ (z)) A (N ), :z A (N ) µ ψ (z) A (N ). Tranition rate q(n, n ) for eah (n, n ) whih i not a given above i equal to 0. We will onider the following tak-expert ytem intane. Definition 3.2 (Two type-two expert ytem). Suppoe that there are two tak type C = {, 2 } and two expert S = {, 2 }. Eah arrival i equally likely to be of both type, i.e., π z = where z atifie z = /2 for eah C, and π z = 0 if z z. Both expert provide repone at unite rate, i.e., µ = for eah. Further, for la we have p, = for eah S, and for la 2 we have p, 2 = a <, and p 2, 2 = 0. We refer to uh a tak-expert ytem a a two type-two expert ytem with parameter a. For thi ytem, if a tak of mixed-type z reeive a failure from either of the expert, then it mixed type beome z where z = 0 and z 2 =. Thu, it i uffiient to aume that Z = {z, z } where z = 2 for eah C, and z = { = 2 }. Further, it i eay to hek that ψ (z ) = ( a)/2, ψ (z ) = a, ψ 2 (z ) = /2, and ψ 2 (z ) =. We then have the following reult (it proof, a that of the other tability reult in the artile, i etablihed through the identifiation of uitable Lyapunov funtion, and given in the Setion 8). Theorem 3.3. For a two type-two expert ytem with parameter a a defined in Definition 3.2, the ytem under Greedy i table if and only if λ < 4a/(2 + a). 3.2 Random Poliy Definition 3.4 (Random Poliy). A poliy i Random if eah expert i aigned a tak hoen uniformly at random from the pool of outtanding tak. Thi poliy i partiularly eay to implement a it doe not require knowledge of any ytem parameter. The following theorem provide a tability ondition for thi poliy. Theorem 3.5. Under Random poliy, the ytem i table if and only if the following hold: λ < z Z z π z, C S µ p, Following orollary eaily reult from the above theorem.

5 Adaptive Mathing for Expert Sytem with Unertain Tak Type :5 Corollary 3.6. Conider a two type-two expert ytem with parameter a a defined in Definition 3.2. The ytem i table under Random poliy if and only if λ < 4a/(2 + a). 3.3 Sub-optimality of Greedy and Random The reult in Corollary 3.6 implie that for the two type-two expert ytem with parameter a onidered in Theorem 3.3, Greedy poliy perform not better than Random poliy. Further, we how in the next etion, via Theorem 4. and Corollary 4.4, that there exit a poliy whih ahieve optimal tability region whih i ignifiantly larger than that ahieved by Greedy and Random. In partiular, the tability threhold for tak arrival rate under optimal poliy an be up to 5/4 time (namely, when a = /2) that under either Greedy or Random. While it i intuitive that Random may perform poorly a ompared to an optimal poliy, it i omewhat ounter intuitive that Greedy may perform a ub-optimally a Random. The reaon for the poor performane of Greedy an be explained a follow. In the two type-two expert ytem, we have a flexible expert, i.e. an expert for tak of all pure-type, and a peialized expert, i.e. an expert only for pure-type. Under Greedy poliy, all expert prioritize the newly arriving tak a it optimize the probability of ahieving ue in the hort term. However, a larger long-term throughput ould be ahieved if the flexible expert ould fou more on the lagging tak, i.e., the tak of pure-type 2. 4 OPTIMAL STABILITY Main goal of thi etion i to provide neeary and uffiient ondition for tability of the ytem. We provide a poliy, alled bakpreure poliy, whih tabilize the ytem when the uffiient ondition are atified. We obtain tability ondition via apaity ontraint and flow onervation ontraint whih apture the flow of tak from one type to another upon ervie by an expert. For intane, if ν,z repreent the flow of tak of mixed-type z erved by expert, a fration ψ (z) of it leave the ytem due to ue while the ret get onverted into a flow of type ϕ (z). The total arrival rate of flow of mixed-type z, i.e., λπ z + S,z ϕ (z) ν,z ψ (z ), mut math the total ervie rate, i.e., S ν,z. Further, the total flow ervie rate expert, i.e., z Z ν,z, mut be le than it ervie apaity µ. The following i the main reult of thi etion. For a proof ee Setion 8.3 Theorem 4.. Suppoe there exit uh that min p, > 0. If there exit non-negative real number ν,z, for S, and z Z and poitive real number δ, for S uh that the following hold: z Z, λπ z + ν,z ψ (z ) = ν,z, (3) S, S,z ϕ (z) S ν,z + δ µ, (4) z Z then there exit a poliy under whih the ytem i table. If there doe not exit non-negative real number ν,z, for S, z Z and non-negative real number δ for S uh that the above ontraint hold, then the ytem annot be table. We ue the ondition of exitene of an expert uh that min p, > 0 only for a tehnial reaon to implify our proof. We believe that the reult hold even when thi i not true. We now deribe the poliy whih ahieve optimal tability. We need ome more notation to deribe the poliy. Conider a et Y Z. Let X (t) be the number of tak in the ytem at time t whih have been of type z Z\Y. For z Y, let X z (t) and Ñ z (t) be the tak of mixed-type z whih have and have not had mixed-type in Z\Y. Alo, for onveniene, for eah z Z\Y, let X z be the tak of mixed-type z, i.e., N z = X z for eah z Z\Y. Thu, we have X = z X z. Conider the following poliy.

6 :6 Virag Shah, Lennart Guliker, Laurent Maoulié, and Milan Vojnović Definition 4.2 (Bakpreure(Y) poliy). For a given Y, let X, and {Ñ z } z Y be a defined above. For eah S, z Y let Define If w,z (Ñ, X ) = Ñ z ψ (z)ñ ϕ (z), if ϕ (z) Y Ñ z ψ (z)x, if ϕ (z) Z\Y. B (Ñ, X ) = arg max z Y:Ñ z >0 w,z (Ñ, X ). µ max w,z (Ñ, X ) X min µ p, z Y:Ñ z >0 C then eah expert hooe a tak in Ñ z where z B (Ñ, X ) Y with tie broken arbitrarily. Ele, eah expert erve a tak in X hoen uniformly at random. The following theorem follow from the proof of Theorem 4.. Theorem 4.3. Suppoe there exit uh that min p, > 0. If the uffiient ondition for tability a given by Theorem 4. are atified, then there exit a finite ubet Y of Z uh that the poliy Bakpreure(Y) tabilize the ytem. Unlike bakpreure poliy propoed in [33] under a different etting, whih wa agnoti to ytem arrival rate, a et Y uh that Bakpreure(Y) poliy tabilize the ytem may depend on the value of λ. We now ue Theorem 4. to provide uffiient ondition for tability for a enario onidered in Setion 3.. For proof of the orollary below, ee Setion 8.5. Corollary 4.4. For a two type-two expert ytem with parameter a a defined in Definition 3.2, the Bakpreure(Z) poliy tabilize the ytem if λ < min {3a/(a + ), 2a}. Further, the ytem i untable if the trit inequality i revered. Compare thi with the tability region ahieved by greedy and random poliie, a given by Theorem 3.3 and Corollary 3.6, repetively. A we argued in Setion 3.3, the bakpreure(y) poliy may ignifiantly outperform greedy and random poliie. 5 EXPERIMENTAL RESULTS In thi etion, we preent our empirial reult obtained by uing data from Math.Stak-Exhange Q&A platform. In thi platform, uer pot tagged quetion that are anwered by other uer. The aker may elet one of the ubmitted anwer a the bet anwer for the given quetion, whih i made publi information in the platform. We will ue thi data to etimate the ue probabilitie of expert in anwering quetion, and ue thi parameter in imulation to ompare the throughput that an be ahieved by our two baeline poliie and the tability-optimal bakpreure poliy. Thi will how that a ubtantially larger throughput an be ahieved by the tability-optimal bakpreure poliy relative to the baeline poliie. Dataet. The dataet onit of around 800, 000 quetion. It wa retrieved on February 2nd, 207. The top mot ommon tag are given in Table in dereaing order of popularity. Among thee tag, the mot ommon i alulu whih over 6, 84 quetion, and the leat ommon i omplex analyi whih over 22, 83 quetion. In our analyi, we ued only quetion that are tagged with at leat one of the mot popular tag, whih amount to a total of 38, 239 quetion.

7 Adaptive Mathing for Expert Sytem with Unertain Tak Type :7 Table. Skill of expert etimated by uing data from the Math.Stak-Exhange Q&A platform. Cluter Tag alulu real-analyi linear-algebra probability abtrat-algebra integration equene-and-erie general-topology ombinatori matrie omplex-analyi Size Etimated kill et. The ue probabilitie of anwering quetion are etimated a follow. For a given uer-tag pair, the ue probability i etimated by the empirial frequeny of the aepted anwer by thi uer for quetion of given tag, onditional on that the uer had at leat 5 aepted anwer for quetion of the given tag, and otherwie we etimate the ue probability i et to be equal to zero. Thee ue probabilitie are etimated for 2, 000 uer with the mot aepted anwer. Among thee uer, the uer with the mot aepted anwer had 4, 665 aepted anwer, and the uer with the leat number of aepted anwer had 3 aepted anwer. 72 uer had more than 50 anwer aepted. In order to form luter of uer with imilar ue probabilitie for different tag, we ran the k-mean lutering algorithm. The etimated ue probabilitie are hown in Table. The olumn orrepond to different entroid of the luter and give average ue probabilitie for different tag. In the bottom row, we give the ize of the orreponding luter. For intane, the 65 peron in luter have on average 32% of their alulu, and 46% of their linear algebra anwer aepted. There i a pronouned eparation of uer who anwer quetion with repet to their expertie into thoe who are good at anwering quetion of any topi and thoe who are peialized in partiular topi. In Table, we highlighted the ue probabilitie of value at leat 35% from whih we oberve that (a) there i a luter, namely luter 6, whih ontain 83 very ative, experiened expert doing well on all tag; (b) there i a luter, namely luter 3, whih ontain a bulk of 33 uer with low performane over all tag; () there are luter, namely luter, 7, 8 and 9, whih ontain uer peialized in a ingle topi; and, finally, (d) the uer in remaining luter, namely luter 2, 4, 5 and 0 do well on a few related topi. There i alo a prevalene of quetion with different ombination of tag, that i, mixed type. We kept only thoe ombination of tag that our for at leat % of the total number of quetion. Thi reult in 6 tag ombination among whih are ingleton and 5 are ombination of 2 tag. For intane, % of the quetion are tagged only with alulu and 4% of the quetion are tagged with both linear-algebra and matrie. We oberved that roughly 9% of the quetion are tagged with multiple tag, howing the relevane of our model. Simulation etup. We regard 0 luter of uer a the et of expert in our ytem and onider the tag a pure type; the quetion are then either of pure or mixed type. We define the et of tak type a the et ontaining all pure type plu the 5 mot frequently een mixed type. For a mixed type we aume that the

8 :8 Virag Shah, Lennart Guliker, Laurent Maoulié, and Milan Vojnović prior are uniformly ditributed (i.e., a quetion with tag alulu and integration i with /2 hane of pure type alulu). Eah arrival tak i one of 6 type plu all vetor that an be obtained by applying a finite ompoition of funtion in {ϕ ( )} S on the arriving type. We defined the arrival frequenie of tak type in our imulation by normalizing the ourrene of the 6 type. We aumed the expert to have unit ervie rate. We make thi approximation a we do not have the information about time at whih expert begin to repond a quetion. We examined the ytem for inreaing value of tak arrival rate. We imulate our CTMC via a direte event imulator. It equentially ompute hange of the tate of the ytem a determined by the ourrene of ome event uh a a tak arrival or a repone from an expert. All our experiment were for a imulation run that onit of 3.5 million event. For the bakpreure poliy we define the et Y to onit of all pure type, the 5 mot frequently een mixed type upon arrival a deribed above, and the mixed type that reult from addreing the above mixed type by an expert exatly one. Note that pure type an be attempted multiple time without hanging it type. We thu have Y = = 66. Our hoie of Y i a reult of a ompromie between performane and omplexity. Chooing a larger et of Y may inreae the tability region by a mall fration, but may ignifiantly inreae the omplexity of the Bakpreure poliy. Performane omparion of different poliie. We evaluated the time-average number of tak in the ytem for different tak arrival rate for the three poliie under our onideration. Thee reult are hown in Figure. We oberve that the tak arrival rate at whih random, greedy, and bakpreure beome untable are, 2.2, 3.80 and 4.0, repetively. Thu, random poliy perform muh wore than any other poliy, and the bakpreure poliy ahieve throughput improvement of about 8% over the greedy poliy. We further examined the evolution of the number of tak in the ytem waiting to be erved over time for greedy and bakpreure poliy for repetive tak arrival rate, 3.78 and 3.83 (Figure 2 left) and repetive tak arrival rate 3.83 and 4.08 (Figure 2 right). We oberve that for the two omparion ae the bakpreure poliy tend to reult in maller number of tak waiting to be erved than the greedy poliy, even when operating under a larger tak arrival rate. By experimentation, we oberved that under the bakpreure poliy the ytem beome untable at an arrival rate of about 4.. We expet that extending the definition of Y would allow to ahieve even higher throughput. 6 RELATED WORK The problem tudied in thi paper i broadly related to that of multi-arm bandit, e.g., ee [, 4, 2, 20] and itation therein, in the ene of optimizing the trade-off between exploration, to learn job type, and exploitation, to optimize tak performane. It alo ha ome relation with ollaborative filtering ytem uh a thoe tudied in [8, 9, 32], whih an be interpreted a expert-tak ytem where ue probabilitie admit a low-rank matrix truture. Unlike our work, there good mathe are inferred from oberved aignment of tak to expert, whih are aording to a given tatitial model, and there are no reoure ontraint impoed on the expert. A related line of work i that on tohati online mathing, e.g., [4, 25, 26]. The tohati online mathing an be interpreted a a tak-expert ytem where eah expert i aoiated with a budget ontraint that allow to olve at mot one tak. The goal i to maximize the expeted number of ueful aignment over a given number of tak arrival. The ompetitive ratio of different aignment poliie have been etablihed under ertain retrition on the ue probabilitie, uh a auming that eah take either a ommon poitive value or zero value, or that eah take an arbitrary but mall enough value. Unlike our work where the tak type are unertain, unertainty in thee model ome from the arbitrarine of the future tak arrival and the monotonially dereaing available reoure budget. In [5], the author onidered a tak-expert ytem where tak type are of two diffiulty level (hard or eay) and expert kill are of two level (enior or junior). Senior may erve any tak, but junior may only erve eay

9 Adaptive Mathing for Expert Sytem with Unertain Tak Type :9 Number of tak in the ytem Random Greedy Bakpreure Arrival rate Fig.. Time-average number of tak in the ytem veru the tak arrival rate for three different poliie: random, greedy and bakpreure. The ytem under random poliy beome untable at tak arrival rate of about 2.2, under greedy poliy at about 3.8 and under bakpreure poliy of about 4. (not hown in the figure). Note that the bakpreure poliy till allow for the ytem to be table at the tak arrival rate of about 4.08 (the right-mot red dot in the figure). tak. The type of eah tak i aumed to be unknown upon arrival. It i hown that there exit an optimal poliy where all tak are firt routed to junior expert and then progreively moved to enior one. We allow for muh more generality with repet to the heterogeneity of kill of expert. In their model, a tak upon ervie an only beome progreively harder, whih amount to a feed-forward ytem, unlike our model. The work in [6] onider a model where the expert reoure are ontrained and hared by different tak of unertain type. They onider a etting where eah tak an be divided into a large number of ubtak of the ame type, vanihingly mall amount of whih ould be ued to aurately learn the tak type, and the ret an be erved optimally. Thi deouple the learning of eah tak type from the performane in exeution and expert utilization, and i thu different from our work. Another related literature i that of ontrained queueing ytem, where arriving tak are to be erved by heterogeneou erver ubjet to reoure ontraint, e.g., [2, 7, 8,, 5, 2, 27, 3, 35]. The goal i to effiiently utilize erver reoure while providing good performane in erviing tak, e.g., optimizing tak delay. Our mathing poliy i of a flavor imilar to the tability-optimal bakpreure poliy firt propoed in [33]. The etting loe to our i the one tudied in [3] for routing querie in peer-to-peer network. Here, the type of the querie are known but the loation of node where the querie may by uefully reolved are unertain. More tehnially, we aoiate queue with eah prior ditribution whih may be infinite in number. Thi make the tability analyi muh more hallenging. Another related work i that on heduling flexible erver [22], whih allow for tak of different type and erver of different kill. It ha been etablihed that a greedy, o alled max-weight poliy, minimize a tritly onvex ot funtion of tak delay in a heavy traffi regime. In [3] thi tritne requirement i removed for a la of ytem with homogeneou erver peed. The main differene from our work i that all thee work aume that tak type are known. Finally, we ontrat our work to the mathing problem tudied in the ontext of rowdouring ytem uh a [0, 7, 30, 36]. Thee work are onerned with laifiation tak with unknown ground truth. Thi i unlike our work, where we onider tak uh that upon eah attempt of an expert olving a tak, we oberve whether or not the tak ha been uefully olved. Another differene i that thee work typially onider a tati

10 :0 Virag Shah, Lennart Guliker, Laurent Maoulié, and Milan Vojnović Fig. 2. Total number of tak in the ytem over time for the greedy and bakpreure poliy. The blue urve orrepond to greedy poliy and red urve orrepond to bakpreure poliy. The tak arrival rate are a indiated in the figure. (Left) both poliie provide tability but bakpreure poliy ha overall better performane. (Right) greed poliy fail to provide tability, while bakpreure poliy provide tability. model where a tak aignment to a et of expert i made all at one. In [24] labeling tak arrive dynamially and their exit i tied to the expert alloation deiion, in that a tak leave one the probability of error in the label etimate fall below a threhold. 7 CONCLUSION We tudied mathing of tak and expert in a ytem with unertain tak type. We etablihed a omplete haraterization of the tability region of the ytem, i.e. the et of tak arrival rate that an be upported by a mathing poliy uh that the expeted number of tak waiting to be erved i finite. We howed that any tak arrival rate in the tability region an be upported by a bak-preure mathing poliy. We alo ompared with two baeline mathing polie, and identified intane under whih there i a ubtantial gap between the maximum tak arrival rate that an be upported by thee poliie and that of the optimum bak-preure mathing poliy. There are everal intereting diretion for future reearh. Firt, for the ae when tak type are unknown, it i of interet to onider mathing poliie that optimie different kind of performane objetive, uh a, for

11 Adaptive Mathing for Expert Sytem with Unertain Tak Type : example, minimizing the long-run average of a funtion of tak waiting time. Seond, muh remain to be aid about mathing poliie for the ae when both tak type and the kill of expert are unknown. 8 PROOFS 8. Proof of Theorem 3.3 We firt how that if λ < 4a/(2 + a) then the ytem i table. For eah t let t + τ (t) be the time at whih the firt event (arrival or ompletion of a repone) our after time t. Let τ n = E[τ (t) N (t) = n], i.e., given that N (t) = n at time t, τ n i the expeted time at whih the firt event our after time t. For example, for n = 0 we have τ n = /λ. A ommon approah to how ytem tability i to ue Lyapunov-Foter theorem, ee e.g., Propoition I.5.3 on page 2 in [3]. The idea i to ontrut a funtion L( ) uh that L(n) tend to infinity a n and that ha a tritly negative drift for all but finite value of n, i.e., there exit a ontant ϵ > 0 uh that E [ L(N (t + τ (t))) L(N (t)) N (t) = n ] ϵτ n, for all but finite value of n. Intuitively, negative drift ondition implie that a L(N (t)) beome large (i.e., a N (t) beome large) the ytem dynami i uh that L(N (t)) tend to dereae in expetation. Thi prevent the L(t) from blowing up to a t inreae and thu keep the ytem table. Roughly, the Lyapunov-Foter theorem tate that the negative drift ondition i indeed uffiient to enure that that ytem i poitive reurrent and thu table. We will ue a variant of Lyapunov-Foter theorem, provided below, whih follow from Theorem 8.3 in [28]. Theorem 8.. Conider an irreduible CTMC N (t) that take value in a ountable tate-pae. Let τ (t) and τ n be a defined above. If there exit a funtion L( ), and ontant K > 0 and ϵ > 0 uh that for L(n) > K we have E [ L(N (t + τ (t))) L(N (t)) N (t) = n ] ϵτ n, and if {n : L(n) K} i finite, then N (t) i poitive reurrent. Now uppoe that λ < 4a/(2 + a). Then, it an be heked that λ = ( + δ )a. Now, onider the following Lyapunov funtion: for eah n, we have 2 a 2(2 λ) 2 a L(n) = ( + δ ) τ n 2(2 λ) n z + n z, where δ i a ontant obtained a above. Conider tate n uh that n z > 0. For thee tate, we obtain 2 a ( ) ( L(n) = ( + δ ) λ µ µ 2 + µ ψ (z ) + µ 2 ψ 2 (z ) ) τ n 2(2 λ) 2 a a = ( + δ ) (λ 2) + + 2(2 λ) 2 2 = δ 2 a < 0. 2 Now, onider tate n uh that n z = 0 and n z > 0. For thee tate L(n) = δ 2 a τ n 2(2 λ) λ µ a = δa < 0. 2 a 2(2 λ) λ < a. Thu, there exit δ > 0 uh Thu, the ondition of Theorem 8. are atified with K = L((, ))/τ (,) and ϵ = min(δa, δ (2 a)/2). Hene, N (t) i poitive reurrent if λ < 4a/(2 + a). We now how the only if part. Suppoe that λ 4a/(2 + a). Then, the δ ued in the above argument i greater than or equal to 0. Thu, drift i non-negative for all but finite value of n. Further, ine L( ) i bounded, the

12 :2 Virag Shah, Lennart Guliker, Laurent Maoulié, and Milan Vojnović maximum hange in L( ) upon an arrival or a departure i alo bounded, uing Propoition I.5.4 on page 22 in [3], we etablih the only if part. 8.2 Proof of Theorem 3.5 Note that the ytem under random poliy i equivalent to the one where pure-type of a tak i revealed upon arrival, i.e., there i no unertainty in tak type. Thi i true ine the random poliy doe not ue the information of type (pure or mixed). We thu let that pure-type i indeed revealed upon arrival. Let X (t) be the number of tak in the ytem of pure-type. Let X (t) = {X (t)}. For eah C, the arrival rate into queue X (t) i equal to λ λz π z. z Z C λ S µ p, We firt how the if part of the reult. Suppoe that we have <. We ue the fluid limit approah developed in [9, 23, 29]. Roughly, given initial ondition X (0) = x, the fluid trajetorie of the tate proe X (t) an be obtained by aling initial ondition, peeding time, and then tudying the realed proe; i.e., letting lim k k X (0) = x, and tudying lim k k X (kt). Uing argument imilar to thoe ued in [23], the fluid limit for the number of tak in eah la an be hown to atify the following at almot all time t: for eah C and X > 0 we have d dt X = λ S µ p, X X. (5) Define a funtion L on R C a ( L(X ) = X log X γ X ), (6) where γ λ S µ p,. Further, by following the argument imilar to [23], if we have that L(X ) and d dt L(X ) a X under fluid limit then the tability of the original ytem follow. We how below that both thee limit hold. Uing (5) and (6), we obtain d dt L(X ) = ( ) ( d dt X log X γ X ), (7) ( log = X λ µ p, S X ( = µ p, λ S µ p, ) X X logγ, (8) X ) ( X log ) X X logγ whih onverge to a X grow large. Let θ = / γ and ˆγ = γ /θ for eah C. Sine γ <, we have θ >. Let D(p q) be the Kullbak- Leibler divergene between two Bernoulli ditribution with parameter p and q, i.e., D(p q) = p log( p q ) + ( p) log( p q ). Now, we an write (9)

13 Adaptive Mathing for Expert Sytem with Unertain Tak Type :3 ( ) θx L(X ) = X log ˆγ X ( = X log θ + X log X X. ˆγ = X log θ + { X D X X } ) C { ˆγ } C (0) () (2) whih onverge to a X grow large. C λ Hene, the if part of the reult follow. The ame line of argument an alo be ued to how that if S µ p, the drift i non-negative for all but finite number of tate. Further, ine L(X ) i bounded, the maximum hange in L(X ) upon an arrival or a departure i alo bounded, uing Propoition I.5.4 on page 22 in [3], we get the only if part. 8.3 Proof of Theorem 4. We firt how tability under uffiient ondition. In networked ytem, e.g. ee [, 33], a tandard approah toward proving tability of a bakpreure type poliy i to deign a tati poliy uing flow variable {ν z },z and the lak {δ } whih provide a fixed ervie rate to eah queue N z uh that it drift i uffiiently negative for eah. However, in our etup the total number of queue {N z } z Z ould be ountable, while the total available lak i finite. Thu, it i not poible to deign a tati poliy uh that the drift in eah individual queue i bounded from above by a negative ontant. Thi i unlike any finite-erver queueing ytem onidered in the previou literature. We thu take a different approah, whih an be explained roughly a follow. Sine the total exogenou arrival rate λ, and the total endogenou arrival rate, i.e. arrival into a queue due to failure at another queue, are both finite (they are bounded from above by µ ), there exit a finite et Y Z uh that the total arrival rate into Z\Y i le than min C S δ 4 p,. Eah tak whih enter a queue N z where z Z\Y i intead ent to a virtual queue X, and tay there until there i a ue. If X i large ompared to the other queue then all the erver fou on X. The finite number of remaining queue are operated via a bakpreure poliy whih aount for the expeted baklog een in thee queue. More formally, onider {ν,z },z and poitive ontant {δ } a potulated in the theorem. Without lo of generality, aume that there exit a ontant 0 < ϵ < uh that δ = ϵµ for eah S. Let Y be a finite ubet of Z uh that z Z\Y λπ z + S z ϕ ν,z ψ (z ) δ min C 4 p,. S Sine λ + S,z Z ν,z 2 µ, uh a Y exit. Let X be the number of tak in the ytem whih are or have been in pat of type z Z\Y. One a tak enter queue X it doe not leave it until ue. There may be tak in it with mixed-type in Y. Note, our poliy will depend on X and thu {z(, t)} will not be N (t) meaurable. In turn, N (t) will not be a CTMC. For z Y, let X z and Ñ z be the tak of mixed-type z whih have and have not had mixed-type in Z\Y. Alo, for onveniene for eah z Z\Y, let X z be the tak of mixed-type z, i.e., N z = X z for eah z Z\Y. We now formally define σ ( { X z } z Z, {Ñ z } z Y ) -meaurable bakpreure poliy. Thu, ( {Ñz } z Y, { X z } z Z ) i a CTMC.

14 :4 Virag Shah, Lennart Guliker, Laurent Maoulié, and Milan Vojnović We now how tability of the ytem under thi poliy for Bakpreure(Y) a given in Definition 4.2. Below we will aume that the tie in eleting z from B (Ñ, X ) are broken uniformly at random for impliity of expoition. The proof an be eaily extend to any other tie breaking approah. Conider the following Lyapunov funtion. L(Ñ, X ) = Ñz 2 + X z = Ñz 2 + X 2. z Y z Z z Y For eah t, let t + τ (t) be the time at whih the firt event (arrival or ompletion of a repone) our after time t. Clearly, τ (t) i a topping time. Further, let τñ, x (t) = E[τ (t) (Ñ, X ) = (ñ, x)]. Let D(ñ, x) E [ L(Ñ (t + τ ), X (t + τ )) L(Ñ (t), X (t)) τñ, x Ñ (t) = ñ, X (t) = x ]. D(ñ, x) i alled drift in tate n. We would like to how that there exit a poitive integer K and poitive ontant ϵ uh that Let for eah S and z Y let Then, one an hek that ν,z = x min C D(ñ, x) ϵ (ñ, x).t. max( ñ, x) K. τñ, x E[Ñ z (t +τ ) 2 Ñ z (t) 2 Ñ (t) = ñ, X (t) = x] = (2ñ z + ) Further, let Then, we have that 2 µ p, > µ max w,z (ñ, x) {z B (n)} z Y:ñ z >0 B (n). ν = x min C τñ, x E[X (t + τ ) 2 X (t) 2 Ñ (t) = ñ, X (t) = x] Thu, we get D(ñ, x) (2ñ z + ) λπ z + z Y (2x + ) S + (2x + ) Upon arranging term, we obtain µ p, > z Z\Y z ϕ z Z\Y λπ z + S λπ z + S z ϕ µ max z Y:ñ z >0 w,z (ñ, x). z ϕ ν,z ψ (z ) + ( 2ñ z + ) λπ z + S z ϕ ν,z ψ (z ) µ ν,z ν,z ψ (z ) νz ψ (z ) + ( 2ñ z + ) + ( 2x + )ν min + ( 2x + )ν min µ p, ν,z. µ p,.

15 Adaptive Mathing for Expert Sytem with Unertain Tak Type :5 D(ñ, x) 2ñ z λπ z + νz ψ (z ) ν,z z Y S z ϕ + 2x λπ z + νz ψ (z ) ν min µ p, z Z\Y S z ϕ + λ + ν,z ψ (z ) + ν,z + ν min µ p, z Z S z ϕ z Y The lat of the above three term an be bounded by a ontant, ay α = 0 µ. For eah S and z Y let ˆν,z = (µ 3δ /4)ν,z and ν z = (δ /4)ν,z. Further, let ˆν = min (µ 3δ /4)p, ν and ν = min (δ /4)p ν. Then, D(ñ, x) λπ z + ˆν z ψ (z ) ˆν,z 2ñ z z Y S z ϕ + 2x λπ z + z Z\Y S z ϕ + 2ñ z ν,zψ (z ) z Y S z ϕ Conider the following lemma. It proof i given in Setion 8.4. ˆν,z ψ (z ) ν z ˆν + α + 2x S z ϕ ν,z ψ (z ) ν Lemma 8.2. Reall the {ν,z },z a potulated by the theorem. For Θ = {θ,z } S,z Y θ, where θ and θ,z for eah, z are real, let f (Θ) = Then, z Y 2ñ z λπ z + S z ϕ θ z ψ (z ) θ,z + 2x f ( { ˆν,z } S,z Y ˆν ) f z Z\Y λπ z + S ( ) {ν,z } S,z Y {min δ /4}. z ϕ θ z ψ (z ) θ. From definition of ν,z, we get that the firt term in f ({ν,z } S,z Y {min δ /4}) i equal to 0 and that the eond term i le than or equal to 0. Thu, we obtain D(ñ, x) α + 2ñ z z Y S Rearranging, we get D(ñ, x) α 2 S z Y:ϕ (z) Y z ϕ ν,zψ (z ) ν,z ν z (ñ z ψ (z)ñ ϕ (z)) 2 + 2x S S z Y:ϕ (z) Z\Y z ϕ ν,z ψ (z ) ν. ν,z (ñ z ψ (z)x ) 2x ν. Fix ϵ > 0. We now how that there exit a poitive integer K uh that if x > K or if ñ > K then D(ñ, x) ϵ. Upon rearranging term, we obtain

16 :6 Virag Shah, Lennart Guliker, Laurent Maoulié, and Milan Vojnović D(ñ, x) α 2 S z Y:ϕ (z) Y = α 2 ν,zw,z (ñ, x) 2 ν x, S z Y = α max 2 S z Y Thu we get, ν,z (ñ z ψ (z)ñ ϕ (z)) 2 ν,zw,z (ñ, x), 2 ν x S z Y:ϕ (z) Z\Y δ D(ñ, x) α x min C 4 p,. S Hene, for any (ñ, x) uh that x > (α + ϵ) min C S δ 4 p,, we have D(ñ, x) ϵ. We alo have that Thu, D(ñ, x) α 2 S δ 4 max z Y w,z (ñ, x). ( ) δ ( ) D(ñ, x) α 2 min max S 4 w δ,z (ñ, x) α 2 min z Y S 4 S max z Y ν,z (ñ z ψ (z)x ) 2x ν w,z (ñ, x). Now uppoe that x α 2 (α + ϵ) min C S δ 4 p,. Then, if we how that max z Y S w,z (ñ, x) a ñ, then we have that D(ñ, x) ϵ a poitive integer K uh that ñ > K. We now how that S max z Y w,z (ñ, x) a ñ. Let z = arg max z Y n z. Then we have w,z (ñ, x) (n z ψ (z) min(α 2, n z ) S = S n z min(α 2, n z ) ψ (z) whih tend to infinity beaue ψ (z) = z ( p, ) = S zp, S max S C min zp, S max p, < S. z min Thu, there exit poitive ontant K and ϵ uh that if x > K or if ñ > K then D(ñ, x) ϵ. Let A {(ñ, x) : max( ñ, x) K}. Then, uing a verion of Lyapunav-Foter Theorem from [34], we have that, from any tate (ñ, x) uh that ñ + x <, the expeted time to return to A, i.e., τ A (ñ, x) i finite. Further, T up τ A (ñ, x) <. (ñ, x ) A Thu, tarting with any tate in A, we return to it in a finite expeted time. We will be done if we how that expeted time to return to tate (0, 0) i alo finite. We do thi a follow. Fix a ontant β > 0. Sine there exit S p,

17 Adaptive Mathing for Expert Sytem with Unertain Tak Type :7 uh that min p, > 0, we have that for any interval of time of ize β the probability that no arrival happen in the thi interval and that a tak leave the ytem i finite. Suppoe that ytem i in a tate (ñ, x) A at time t = 0. Now onider renewal time T i, i = 0,, 2,...,, where T 0 = 0 and for eah i > 0, T i i defined a follow: T i i equal to T i + β if indeed no arrival happen and a tak leave the ytem in the interval [T i,t i ), ele T i i the firt time of return to A after T i. Clearly E[T i ] ine T a defined above i finite. Further probability that a tak leave ytem in time T i T i i fine, ay α. Thu, time for ytem emptying after firt reahing A an be upper-bounded by um of K geometri random variable with rate α. Thu expeted time to return to tate (0, 0) i fine. Hene, the ytem i table. Now uppoe that the ytem i table. Then, the neeary ondition an be hown to hold by the ergodiity of the ytem, and letting ν,z for eah, z to be the long-term fration of time a erver attempt a tak in N z. 8.4 Proof of Lemma 8.2 Upon rearrangement of term in the expreion of f (Θ) we obtain f (Θ)/2 = θ,z (n z ψ (z )n ϕ (z)) z Y:ϕ (z) Y By uing the definition of weight w,z, we obtain Thu, f (Θ)/2 = θ,z w,z (ñ, x) xθ z Y z Y:ϕ (z) Z\Y θ,z (n z ψ (z )x) xθ. ( ) max w,z (ñ, x) θ,z xθ. z Y z Y f ({ν z } S,z Y {min δ /4})/2 ( ) max w,z (ñ, x) ν,z x min (δ /4)p z Y z Y S (µ δ /2) max w z (ñ, x) x min (δ /4)p, z Y S max w,z (ñ, x)(µ 3δ /4) x (min z Y max w,z (ñ, x)(µ 3δ /4) < x (min z Y = f ( { ˆν z } S,z Y ˆν ) /2. Hene, the lemma hold. (µ 3δ /4)p, ) (µ 3δ /4)p, ) x min max w,z (ñ, x)(µ 3δ /4) z Y (µ 3δ /4)p, 8.5 Proof of Corollary 4.4 Proof. For thi ytem we have Z = {z, z } where z = 2 onervation ontraint an be given a follow: for eah C, and z λ = ν,z, and ν,z ψ (z ) + ν,z ψ (z ) = ν,z = { = 2 }. The flow

18 :8 Virag Shah, Lennart Guliker, Laurent Maoulié, and Milan Vojnović Suppoe a 3a( ϵ ) 2. There exit an ϵ > 0 uh that λ = a+. It an be heked that {ν z },z where ν 2,z = ϵ,ν 2,z = 0,ν,z = 2a a + ( ϵ),ν,z = 2 a ( ϵ) a + and {δ } S where δ = ϵ for eah atifie uffiient ondition of Theorem 4.. Now uppoe a < 2. There exit an ϵ > 0 uh that λ = 2a( ϵ). It an be heked that {ν,z },z where ν 2,z = 2a( ϵ),ν 2,z = 0,ν,z = 0,ν,z = ( ϵ) and {δ } S where δ = ϵ for eah atifie uffiient ondition of Theorem 4.. The reult then follow from the proof of Theorem 4. by taking Y a Z. ACKNOWLEDGMENTS The author would like to thank Mar Lelarge of SAFRAN for valuable diuion. REFERENCES [] Agrawal, Shipra and Goyal, Navin. Analyi of thompon ampling for the multi-armed bandit problem. In Proeeding of the 25th Conferene on Learning Theory, 202. [2] Alreaini, M., Sathiamoorthy, M., Krihnamahari, B., and Neely, M.J. Bakpreure with adaptive redundany (bwar). In Pro. IEEE INFOCOM, pp , Marh 202. [3] Amuen, Søren. Applied probability and queue. Springer Siene & Buine Media, 2nd edition, [4] Auer, Peter, Cea-Bianhi, Niolò, and Fiher, Paul. Finite-time analyi of the multiarmed bandit problem. Mahine Learning, 47(2): , [5] Bimpiki, Kota and Markaki, Mihali G. Learning and hierarhie in ervie ytem. Unpublihed manuript, 205. [6] Brémaud, Pierre. Markov hain: Gibb field, Monte Carlo imulation, and queue, volume 3. Springer Siene & Buine Media, 203. [7] Bui, L.X., Srikant, R., and Stolyar, A. A novel arhiteture for redution of delay and queueing truture omplexity in the bak-preure algorithm. IEEE/ACM Tranation on Networking, 9(6): , De 20. [8] Cui, Y., Yeh, E.M., and Liu, R. Enhaning the delay performane of dynami bakpreure algorithm. IEEE/ACM Tranation on Networking, 205. [9] Dai, J. G. On poitive harri reurrene of multila queueing network: A unified approah via fluid limit model. The Annal of Applied Probability, 5():49 77, 995. [0] Gao, Chao, Lu, Yu, and Zhou, Dengyong. Exat exponent in optimal rate for rowdouring. In Proeeding of the 33nd International Conferene on Mahine Learning, ICML 206, New York City, NY, USA, June 9-24, 206, pp , 206. [] Georgiadi, Leonida, Neely, Mihael J., and Taiula, Leandro. Reoure alloation and ro-layer ontrol in wirele network. Foundation and Trend in Networking, (): 44, [2] Gittin, John, Glazebrook, Kevin, and Weber, Rihard. Multi-armed bandit alloation indie. John Wiley & Son, 20. [3] Gurvih, Itay and Whitt, Ward. Sheduling flexible erver with onvex delay ot in many-erver ervie ytem. Manufaturing & Servie Operation Management, (2): , [4] Ho, Chien-Ju and Vaughan, Jennifer Wortman. Online tak aignment in rowdouring market. In Proeeding of the Twenty-Sixth AAAI Conferene on Artifiial Intelligene, AAAI 2, pp. 45 5, 202. [5] Ji, Bo, Joo, Changhee, and Shroff, N.B. Delay-baed bak-preure heduling in multihop wirele network. IEEE/ACM Tranation on Networking, 2(5): , Ot 203. [6] Johari, Rameh, Kamble, Vijay, and Kanoria, Yah. Know your utomer: Multi-armed bandit with apaity ontraint. ab/ , 206. URL [7] Karger, David R., Oh, Sewoong, and Shah, Devavrat. Budget-optimal tak alloation for reliable rowdouring ytem. Operation Reearh, 62(): 24, 204. [8] Kleinberg, Jon and Sandler, Mark. Convergent algorithm for ollaborative filtering. In Proeeding of the 4th ACM Conferene on Eletroni Commere, [9] Kleinberg, Jon and Sandler, Mark. Uing mixture model for ollaborative filtering. In Proeeding of the 36th annual ACM Sympoium on Theory of Computing, [20] Lai, T.L and Robbin, Herbert. Aymptotially effiient adaptive alloation rule. Advane in Applied Mathemati, 6():4 22, 985. [2] Maguluri, Siva Theja, Burle, Sai Kiran, and Srikant, R. Optimal heavy-traffi queue length aling in an inompletely aturated with. SIGMETRICS Perform. Eval. Rev., 44():3 24, June 206.

19 Adaptive Mathing for Expert Sytem with Unertain Tak Type :9 [22] Mandelbaum, Avihai and Stolyar, Alexander L. Sheduling flexible erver with onvex delay ot: Heavy-traffi optimality of the generalized µ-rule. Operation Reearh, 52(6): , [23] Maoulié, Laurent. Strutural propertie of proportional fairne: Stability and inenitivity. Annal of Applied Probability, 7(3): , [24] Maoulié, Laurent and Xu, Kuang. On the apaity of information proeing ytem. In 29th Annual Conferene on Learning Theory, New York, United State, June 206. [25] Mehta, A. and Panigrahi, D. Online mathing with tohati reward. In IEEE 53rd Annual Sympoium on Foundation of Computer Siene, pp , Ot 202. [26] Mehta, Aranyak, Waggoner, Bo, and Zadimoghaddam, Morteza. Online Stohati Mathing with Unequal Probabilitie, pp [27] Neely, M. J., Modiano, E., and Rohr, C. E. Dynami power alloation and routing for time varying wirele network. In Pro. IEEE INFOCOM, [28] Robert, Philippe. Stohati network and queue, tohati modelling and applied probability erie, vol. 52, [29] Rybko, Alekandr Nikolaevih and Stolyar, Alexander L. Ergodiity of tohati proee deribing the operation of open queueing network. Problemy Peredahi Informatii, 28(3):3 26, 992. [30] Shah, Nihar B., Zhou, Dengyong, and Pere, Yuval. Approval voting and inentive in rowdouring. In Proeeding of the 32nd International Conferene on Mahine Learning, ICML 205, Lille, Frane, 6- July 205, pp. 0 9, 205. [3] Shah, Virag, de Veiana, Gutavo, and Keidi, George. A table approah for routing querie in untrutured p2p network. IEEE/ACM Tranation on Networking, 24(5): , Ot 206. [32] Stern, David H., Herbrih, Ralf, and Graepel, Thore. Mathbox: Large ale online bayeian reommendation. In Proeeding of the 8th International Conferene on World Wide Web, [33] Taiula, L. and Ephremide, A. Stability propertie of ontrained queueing ytem and heduling poliie for maximum throughput in multihop radio network. IEEE Tranation on Automati Control, 37: , 992. [34] Tweedie, R. L. Criteria for laifying general markov hain. Advane in Applied Probability, 8(4):737 77, 976. [35] Ying, Lei, Shakkottai, Sanjay, Reddy, Aneeh, and Liu, Shihuan. On ombining hortet-path and bak-preure routing over multihop wirele network. IEEE/ACM Tranation on Networking, 9(3):84 854, June 20. ISSN [36] Zhang, Yuhen, Chen, Xi, Zhou, Dengyong, and Jordan, Mihael I. Spetral method meet em: A provably optimal algorithm for rowdouring. J. Mah. Learn. Re., 7(): , January 206.