Randomized Accuracy-Aware Program Transformations For Efficient Approximate Computations

Transcription

1 Randozed Accuracy-Aware Progra Tranforaton For Effcent Approxate Coputaton Zeyuan Allen Zhu Saa Malovc Jonathan A. Kelner Martn Rnard MIT CSAIL Abtract Depte the fact that approxate coputaton have coe to donate any area of coputer cence, the feld of progra tranforaton ha focued alot excluvely on tradtonal eantcpreervng tranforaton that do not attept to explot the opportunty, avalable n any coputaton, to acceptably trade off accuracy for beneft uch a ncreaed perforance and reduced reource conupton. We preent a odel of coputaton for approxate coputaton and an algorth for optzng thee coputaton. The algorth work wth two clae of tranforaton: ubttuton tranforaton (whch elect one of a nuber of avalable pleentaton for a gven functon, wth each pleentaton offerng a dfferent cobnaton of accuracy and reource conupton) and aplng tranforaton (whch randoly dcard oe of the nput to a gven reducton). The algorth produce a (1 + ε) randozed approxaton to the optal randozed coputaton (whch nze reource conupton ubject to a probabltc accuracy pecfcaton n the for of a axu expected error or axu error varance). Categore and Subject Decrptor D.3.4 [Prograng Language]: Proceor optzaton; G.3 [Probablty and Stattc]: Probabltc Algorth; F.2.1 [Analy of Algorth and Proble Coplexty]: Nuercal Algorth and Proble General Ter Algorth, Degn, Perforance, Theory Keyword Optzaton, Error-Te Tradeoff, Dcretzaton, Probabltc 1. Introducton Coputer cence wa founded on exact coputaton wth dcrete logcal correctne requreent (exaple nclude copler and tradtonal relatonal databae). But over the lat decade, approxate coputaton have coe to donate any feld. In contrat to exact coputaton, approxate coputaton apre only to produce an acceptably accurate approxaton to an exact (but n any cae nherently unrealzable) output. Exaple nclude achne learnng, untructured nforaton analy and retreval, and loy vdeo, audo and age proceng. Peron to ake dgtal or hard cope of all or part of th work for peronal or claroo ue granted wthout fee provded that cope are not ade or dtrbuted for proft or coercal advantage and that cope bear th notce and the full ctaton on the frt page. To copy otherwe, to republh, to pot on erver or to redtrbute to lt, requre pror pecfc peron and/or a fee. POPL 12, January 25 27, 2012, Phladelpha, PA, USA. Copyrght c 2012 ACM /12/01... $10.00 Depte the pronence of approxate coputaton, the feld of progra tranforaton ha reaned focued on technque that are guaranteed not to change the output (and therefore do not affect the accuracy of the approxaton). Th tuaton leave the developer olely reponble for anagng the approxaton. The reult nflexble coputaton wth hard-coded approxaton choce drectly ebedded n the pleentaton. 1.1 Accuracy-Aware Tranforaton We nvetgate a new cla of tranforaton, accuracy-aware tranforaton, for approxate coputaton. Gven a coputaton and a probabltc accuracy pecfcaton, our tranforaton change the coputaton o that t operate ore effcently whle atfyng the pecfcaton. Becaue accuracy-aware tranforaton have the freedo to change the output (wthn the bound of the accuracy pecfcaton), they have a uch broader cope and are therefore able to delver a uch broader range of beneft. The feld of accuracy-aware tranforaton today n t nfancy. Only very recently have reearcher developed general tranforaton that are degned to anpulate the accuracy of the coputaton. Exaple nclude tak kppng [27, 28], loop perforaton [14, 23, 24, 31], approxate functon eozaton [6], and ubttuton of ultple alternate pleentaton [2, 3, 12, 33]. When ucceful, thee tranforaton delver progra that can operate at ultple pont n an underlyng accuracy-reource conupton tradeoff pace. Uer ay elect pont that nze reource conupton whle atfyng the pecfed accuracy contrant, axze accuracy whle atfyng pecfed reource conupton contrant, or dynacally change the coputaton to adapt to change (uch a load or clock rate) n the underlyng coputatonal platfor [12, 14]. Standard approache to undertandng the tructure of the tradeoff pace that accuracy-aware tranforaton nduce ue tranng executon to derve eprcal odel [2, 3, 14, 24, 27, 28, 31, 33]. Potental ptfall nclude odel that ay not accurately capture the charactertc of the tranfored coputaton, poor correlaton between the behavor of the coputaton on tranng and producton nput, a reultng nablty to fnd optal pont n the tradeoff pace for producton nput, and an abence of guaranteed bound on the agntude of potental accuracy loe. 1.2 Our Reult We preent a novel analy and optzaton algorth for a cla of approxate coputaton. Thee coputaton are expreed a a tree of coputaton node and reducton node. Each coputaton node a drected acyclc graph of neted functon node, each of whch apple an arbtrary functon to t nput. A reducton node apple an aggregaton functon (uch a n, ax, or ean) to t nput.

2 We conder two clae of accuracy-aware tranforaton. Subttuton tranforaton replace one pleentaton of a functon node wth another pleentaton. Each functon ha a propagaton pecfcaton that characterze the entvty of the functon to perturbaton n t nput. Each pleentaton ha reource conupton and accuracy pecfcaton. Reource conupton pecfcaton characterze the reource (uch a te, energy, or cot) each pleentaton conue to copute the functon. Accuracy pecfcaton characterze the error that the pleentaton ntroduce. Saplng tranforaton caue the tranfored reducton node to operate on a randoly elected ubet of t nput, ultaneouly elnatng the coputaton that produce the dcarded nput. Each aplng tranforaton ha a aplng rate, whch the rato between the ze of the elected ubet of t nput and the orgnal nuber of nput. Together, thee tranforaton nduce a pace of progra confguraton. Each confguraton dentfe an pleentaton for every functon node and a aplng rate for every reducton node. In th paper we work wth randozed tranforaton that pecfy a probabltc choce over confguraton. Our approach focue on undertandng the followng techncal queton: What the optal accuracy-reource conupton tradeoff curve avalable va our randozed tranforaton? Undertandng th queton ake t poble to realze a varety of optzaton goal, for exaple nzng reource conupton ubject to an accuracy pecfcaton or axzng accuracy ubject to a reource conupton pecfcaton. The prary techncal reult n th paper an optzaton algorth that produce a (1 + ε)-approxaton to the optal randozed coputaton (whch nze reource conupton ubject to a probabltc accuracy pecfcaton n the for of a axu expected error or axu error varance). We alo dcu how to realze a varety of other optzaton goal. 1.3 Challenge and Soluton Fndng optal progra confguraton preent everal algorthc challenge. In partcular: Exponental Confguraton: The nuber of progra confguraton exponental n the ze of the coputaton graph, o a brute-force earch for the bet confguraton coputatonally ntractable. Randozed Cobnaton of Confguraton: A tranfored progra that randoze over ultple confguraton ay ubtantally outperfor one that chooe any ngle fxed confguraton. We thu optze over an even larger pace the pace of probablty dtrbuton over the confguraton pace. Global Error Propagaton Effect: Local error allocaton decon propagate globally throughout the progra. The optzaton algorth ut therefore work wth global accuracy effect and nteracton between error ntroduced at the node of the coputaton graph. Nonlnear, Nonconvex Optzaton Proble: The runnng te and accuracy of the progra depend nonlnearly on the optzaton varable. The reultng optzaton proble nonlnear and nonconvex. We how that, n the abence of reducton node, one can forulate the optzaton proble a a lnear progra, whch allow u to obtan an exact optzaton over the pace of probablty dtrbuton of confguraton n polynoal te. The queton becoe uch ore nvolved when reducton node coe to the pcture. In th cae, we approxate the optal tradeoff curve, but to a (1 + ε) precon for an arbtrarly all contant ε > 0. Our algorth ha a runnng te that polynoally dependent on 1. It therefore a fully polynoal-te ε approxaton chee (FPTAS). Our algorth tackle reducton node one by one. For each reducton node, t dcretze the tradeoff curve acheved by the ubprogra that generate the nput to the reducton node. Th dcretzaton ue a pecal b-denonal dcretzaton technque that pecfcally degned for uch tradeoff proble. We next how how to extend th dcretzaton to obtan a correpondng dcretzed tradeoff curve that nclude the reducton node. The fnal tep to recurvely cobne the dcretzaton to obtan a dynac prograng algorth that approxate the optal tradeoff curve for the entre progra. We note that the optzaton algorth produce a weghted cobnaton of progra confguraton. We call uch a weghted cobnaton a randozed confguraton. Each executon of the fnal randozed progra chooe one of thee confguraton wth probablty proportonal to t weght. Randozng the tranfored progra provde everal beneft. In coparon wth a deterntc progra, the randozed progra ay be able to delver ubtantally reduced reource conupton for the ae accuracy pecfcaton. Furtherore, randozaton alo plfe the optzaton proble by replacng the dcrete earch pace wth a contnuou earch pace. We can therefore ue lnear progra (whch can be olved effcently) to odel regon of the optzaton pace ntead of nteger progra (whch are, n general, ntractable). 1.4 Potental Applcaton A prece undertandng of the conequence of accuracy-aware tranforaton wll enable the feld to ature beyond t current focu on tranforaton that do not change the output. Th ncreaed cope wll enable reearcher n the feld to attack a uch broader range of proble. Soe potental exaple nclude: Sublnear Coputaton On Bg Data: Saplng tranforaton enable the optzaton algorth to autoatcally fnd ublnear coputaton that proce only a ubet of the nput to provde an acceptably accurate output. Over the pat decade, reearcher have developed any ublnear algorth [29]. Accuracy-aware tranforaton hold out the proe of autoatng the developent of any of thee algorth. Increentalzed and Onlne Coputaton: Many algorth can be vewed a convergng toward an optal exact oluton a they proce ore nput. Becaue our odel of coputaton upport uch coputaton, our technque ake t poble to characterze the accuracy of the current reult a the coputaton ncreentally procee nput. Th capablty open the door to the autoatc developent of ncreentalzed coputaton (whch ncreentally aple avalable nput untl the coputaton produce an acceptably accurate reult) and onlne coputaton (whch characterze the accuracy of the current reult a the coputaton ncreentally procee dynacally arrvng nput). Senor Selecton: Senor network requre low power, low cot enor [32]. Accuracy-aware tranforaton ay allow developer to pecfy a enor network coputaton wth dealzed lole enor a the ntal functon node n the coputaton. An optzaton algorth can then elect enor that nze power conupton or cot whle tll provdng acceptable accuracy. Data Repreentaton Choce: Data repreentaton choce can have draatc conequence on the aount of reource (te, lcon area, power) requred to anpulate that data [10]. Gvng an optzaton algorth the freedo to adjut the ac-

3 curacy (wthn pecfed bound) ay enable an nfored autoatc electon of le accurate but ore approprate data repreentaton. For exaple, a copler ay autoatcally replace an expenve floatng pont repreentaton wth a ore effcent but le accurate fxed pont repreentaton. We antcpate the applcaton of th technology n both tandard copler for croproceor a well a hardware ynthe yte. Dynac Adaptaton In Large Data Center: The aount of coputng power that a large data center able to delver to ndvdual hoted coputaton can vary dynacally dependng on factor uch a load, avalable power, and the operatng teperature wthn the data center (a re n teperature ay force reducton n power conupton va clock rate drop). By delverng coputaton that can operate at ultple pont n the underlyng accuracy-reource conupton tradeoff pace, accuracy-aware tranforaton open up new tratege for adaptng to fluctuaton. For exaple, a data center ay repond to load or teperature pke by runnng applcaton at le accurate but ore effcent operatng pont [12]. Succeful Ue of Motly Correct Coponent: Many faulty coponent operate correctly for alot all nput. By perturbng nput and coputaton wth all aount of rando noe, t poble to enure that, wth very hgh probablty, no two executon of the coputaton operate on the ae value. Gven a way to check f a fault occurred durng the executon, t poble to rerun the coputaton untl all coponent happen to operate on value that elct no fault. Undertandng the accuracy conequence of thee perturbaton can ake t poble to eploy th approach uccefully. 1 The cope of tradtonal progra tranforaton ha been largely confned to tandard copler optzaton. A the above exaple llutrate, approprately abtou accuracy-aware tranforaton that explot the opportunty to anpulate accuracy wthn pecfed bound can draatcally ncreae the pact and relevance of the feld of progra analy and tranforaton. 1.5 Contrbuton Th paper ake the followng contrbuton: Model of Coputaton: We preent a odel of coputaton for approxate coputaton. Th odel upport arbtrary copoton of ndvdual functon node nto coputaton node and coputaton node and reducton node nto coputaton tree. Th odel expoe enough coputatonal tructure to enable approxate optzaton va our two tranforaton. Accuracy-Aware Tranforaton: We conder two clae of accuracy-aware tranforaton: functon ubttuton and reducton aplng. Together, thee tranforaton nduce a pace of tranfored progra that provde dfferent cobnaton of accuracy and reource conupton. Tradeoff Curve: It how how to ue lnear prograng, dynac prograng, and a pecal b-denonal dcretzaton technque to obtan a (1 + ɛ)-approxaton to the underlyng optal accuracy-reource conupton tradeoff curve avalable va the accuracy-aware tranforaton. If the progra contan no reducton node, the tradeoff curve exact. Optzaton Algorth: It preent an optzaton algorth that ue the tradeoff curve to produce randozed progra that atfy pecfed probabltc accuracy and reource conupton contrant. In coparon wth approache that attept to delver a deterntc progra, randozaton en- 1 The lat author would lke to thank Pat Lncoln for an nteretng dcuon on th topc. log n log n average Output log Fgure 1: A nuercal ntegraton progra. n able our optzaton algorth to 1) delver progra wth better cobnaton of accuracy and reource conupton, and 2) avod a varety of ntractablty ue. Accuracy Bound: We how how to obtan tatcally guaranteed probabltc accuracy bound for a general cla of approxate coputaton. The only prevou tatc accuracy bound for accuracy-aware tranforaton exploted the tructure preent n a et of coputatonal pattern [6, 22, 23]. 2. Exaple We next preent an exaple coputaton that nuercally ntegrate a unvarate functon f(x) over a fxed nterval [a, b]. The coputaton dvde [a, b] nto n equal-zed ubnterval, each of length x = b a x. Let x = (x1,... xn), where x = a +. n 2 The value of the nuercal ntegral I equal to n I = x f(x ) = 1 n (b a) f(x ). n =1 Say, for ntance, f(x) = x n ( log(x) ) the functon that we want to ntegrate and [a, b] = [1, 11]. Our Model of Coputaton. A llutrated n Fgure 1, n our odel of coputaton, we have n nput edge that carry the value of x nto the coputaton and an addtonal edge that carre the value of b a. For each x, a coputaton node calculate the value of (b a) f(x ). The output edge of thee node are connected to a reducton node that copute the average of thee value (we call uch a node an averagng node), a the fnal ntegral I. Progra Tranforaton. The above nuercal ntegraton progra preent ultple opportunte to trade end-to-end accuracy of the reult I n return for ncreaed perforance. Specfcally, we dentfy the followng two tranforaton that ay prove the perforance: Subttuton. It poble to ubttute the orgnal pleentaton of the n( ) and log( ) functon that copre f(x) wth alternate pleentaton that ay copute a le accurate output n le te. Saplng. It poble to dcard oe of the n nput of the averagng node (and the coputaton that produce thee nput) by takng a rando aple of n nput (here we call the reducton factor). Roughly peakng, th tranforaton ntroduce an error proportonal to 1, but decreae the runnng te of the progra proportonally to. n Tradeoff Space. In th nuercal ntegraton proble, a progra confguraton pecfe whch pleentaton to pck for each of the functon n( ), log( ), and (n prncple, although we do not =1

4 Confguraton. Weght x log,0 x log,1 x log,2 x n,0 x n,1 x n,2 /n Error Speedup C % C % Table 1: The (1 + ε)-optal randozed progra confguraton for = 0.05 and ε = do o n th exaple). The confguraton alo pecfe the reducton factor for the averagng node. If we aue that we have two alternate pleentaton of n( ) and log( ), each progra confguraton provde the followng nforaton: 1) x u, {0, 1} ndcatng whether we chooe the -th pleentaton of the functon u {log, n}, and {0, 1, 2}, and 2) ndcatng the reducton factor for the averagng node we chooe. A randozed progra confguraton a probabltc choce over progra confguraton. Functon Specfcaton. We poe two bac requreent on the pleentaton of all functon that copre f(x). The frt requreent that we have an error bound and te coplexty pecfcaton for each pleentaton of each functon. In th exaple we wll ue the followng odel: the orgnal pleentaton of log( ) execute n te T log,0 wth error E log,0 = 0; the orgnal pleentaton of n( ) execute n te T n,0 wth error E n,0 = 0. We have two alternate pleentaton of log( ) and n( ), where the -th pleentaton of a gven functon u {log( ), n( )} run n te T u, = ( 1 ) Tu,0 5 wth error E log, = 0.008, and E n, = ( {1, 2}). The econd requreent that the error propagaton of the entre coputaton bounded by a lnear functon. Th requreent atfed f the functon that copre the coputaton are Lpchtz contnuou 2. In our exaple, the functon n( ) 1-Lpchtz contnuou, nce t dervatve bounded by 1. The functon log(x) alo Lpchtz contnuou, when x 1. Fnally, the product functon Lpchtz contnuou, when the two nput are bounded. We reark here that th econd requreent enure that an error ntroduced by an approxate pleentaton propagate to caue at ot a lnear change n the fnal output. Fndng the (1 + ε)-optal Progra Confguraton. Gven perforance and accuracy pecfcaton for each functon, we can run our optzaton algorth to (1 + ε)-approxately calculate the optal accuracy-perforance tradeoff curve. For each pont on the curve our algorth can alo produce a randozed progra confguraton that acheve th tradeoff. Gven a target expected error bound, we ue the tradeoff curve to fnd a randozed progra confguraton that execute n expected te τ. The (1 + ε)-approxaton enure that th expected runnng te τ at ot (1 + ε) te the optal expected runnng te for the expected error bound. In th exaple we ue ε = 0.01 o that our optzed progra wll produce a 1.01-approxaton. In addton, we defne: the nuber of nput n = 10000, the overall expected error tolerance = 0.05, and the runnng te T n,0 = 0.08 µ and T log,0 = 0.07 µ. For th exaple our optzaton algorth dentfe the pont (, T 0/1.71) on the tradeoff curve, where T 0 the runnng te of the orgnal progra. Th ndcate that the optzed progra acheve a peedup of 1.71 over the orgnal progra whle keepng the expected value below the bound. Table 1 preent the randozed progra confguraton that acheve th tradeoff. Th 2 A unvarate functon α-lpchtz contnuou f for any δ > 0, t follow that f(x) f(x + δ) < αδ. A a pecal cae, a dfferentable functon Lpchtz contnuou f f (x) α. Th defnton extend to ultvarate functon. randozed progra confguraton cont of two progra confguraton C 1 and C 2. Each confguraton ha an aocated weght whch the probablty wth whch the randozed progra wll execute that confguraton. The table alo preent the error and peedup that each confguraton produce. The confguraton C 1 elect the le accurate approxate veron of the functon log( ) and n( ), and ue all nput to the averagng reducton node. The confguraton C 2, on the other hand, elect ore accurate approxate veron of the functon log( ) and n( ), and at the ae te aple 4750 of the 10,000 orgnal nput. Note that ndvdually nether C 1 nor C 2 can acheve the dered tradeoff. The confguraton C 1 produce a ore accurate output but alo execute gnfcantly lower than the optal progra. The confguraton C 2 execute uch fater than the optal progra, but wth expected error greater than the dered bound. The randozed progra elect confguraton C 1 wth probablty 60.8% and C 2 wth probablty 39.2%. The randozed progra ha expected error and expected runnng te T 0/1.71. We can ue the ae tradeoff curve to obtan a randozed progra that nze the expected error ubject to the executon te contrant τ. In our exaple, f the te bound τ = T 0/1.71 the optzaton algorth wll produce the progra confguraton fro Table 1 wth expected error =. More generally, our optzaton algorth wll produce an effcent repreentaton of a probablty dtrbuton over progra confguraton along wth an effcent procedure to aple th dtrbuton to obtan a progra confguraton for each executon. 3. Model of Approxate Coputaton We next defne the graph odel of coputaton, ncludng the error-propagaton contrant for functon node, and preent the accuracy-aware ubttuton and aplng tranforaton. 3.1 Defnton Progra. In our odel of coputaton, progra cont of a drected tree of coputaton node and reducton node. Each edge n the tree trant a trea of value. The ze of each edge ndcate the nuber of trantted value. The ultple value trantted along an edge can often be undertood a a trea of nuber wth the ae purpoe for exaple, a llon pxel fro an age or a thouand aple fro a enor. Fgure 2 preent an exaple of a progra under our defnton. Reducton Node. Each reducton node ha a ngle nput edge and a ngle output edge. It reduce the ze of t nput by oe ultplcatve factor, whch we call t reducton factor. A node wth reducton factor S ha an nput edge of ze R S and an output edge of ze R. The node dvde the R S nput nto block of ze S. It produce R output by applyng an S-to-1 aggregaton functon (uch a n, ax, or ean) to each of the R block. For clarty of expoton and to avod a prolferaton of notaton, we prarly focu on one pecfc type of reducton node, whch we call an averagng node. An averagng node wth reducton factor S wll output the average of the frt S value a the frt output, the average of the next S value a the econd output, and o on. The technque that we preent are qute general and apply to any reducton operaton that can be approxated well by aplng. Secton 8 decrbe how to extend our algorth to work wth other reducton operaton.

5 edge of ze coputaton node reducton node 1000 n log n 10 1 Output u 10 (a) (b) average Output (c) Fgure 2: An exaple progra n our odel of coputaton. Fgure 3: (a)(b): A cloer look at two coputaton node, and (c): Nuercal ntegraton exaple, revted. Coputaton Node. A coputaton node ha potentally ultple nput edge and a ngle output edge. A coputaton node of ze R ha: a ngle output edge of ze R; a non-negatve nuber of nput edge, each of ze ether 1 (whch we call a control-nput edge), or oe ultple tr of R (whch we call a data-nput edge). Each control-nput edge carre a ngle global contant. Datanput edge carry a trea of value whch the coputaton node partton nto R chunk. The coputaton node execute R te to produce R output, wth each executon proceng the value fro each control-nput edge and a block of t value fro each data-nput edge. The executon are ndependent. For exaple, conder a coputaton node of ze 10 wth two nput edge: one data-nput edge of ze 1000, denoted by (a 1, a 2,..., a 1000), and one control-nput edge of ze 1, denoted by b. Then, the functon that output the vector ( 100 n(a, b), =1 200 =101 n(a, b),..., 1000 =901 ) n(a, b) a coputaton node. We reark here that a reducton node a pecal knd of coputaton node. We treat coputaton and reducton node eparately becaue we optze coputaton node wth ubttuton tranforaton and reducton node wth aplng tranforaton (ee Secton 3.2). Inner Structure of Coputaton Node. A coputaton node can be further decopoed nto one or ore functon node, connected va a drected acyclc graph (DAG). Lke coputaton node, each functon node ha potentally ultple nput edge and a ngle output edge. The ze of each nput edge ether 1 or a ultple of the ze of the output edge. The functon can be of arbtrary coplexty and can contan language contruct uch a condtonal tateent and loop. For exaple, the coputaton node n Eq.(1) can be further decopoed a hown n Fgure 3a. Although we requre the coputaton node and edge n each progra to for a tree, the functon node and edge n each coputaton node can for a DAG (ee, for exaple, Fgure 3b). In prncple, any coputaton node can be repreented a a ngle functon node, but t decopoton nto ultple functon node allow for fner granularty and ore tranforaton choce when optzng entre progra. (1) Exaple. Secton 2 preented a nuercal ntegraton progra exaple. Fgure 3c preent th exaple n our odel of coputaton (copare wth Fgure 1). Note that the ultplcty of coputaton node n Fgure 1 correpond to the edge ze n Fgure 3c. The log functon node wth nput and output edge of ze n run n te. Each run conue a ngle nput and produce a ngle output. The functon node wth nput edge of ze n and 1 run n te. Each executon produce a output the product of an x wth the coon value b a fro the control edge. 3.2 Tranforaton In a progra confguraton, we pecfy the followng two knd of tranforaton at functon and reducton node. Subttuton. For each functon node f u of ze R, we have a polynoal nuber of pleentaton f u,1,..., f u,k. The functon run R te. We requre each pleentaton to have the followng properte: each run of f u, n expected te T u,, gvng a total expected runnng te of R T u,, and each run of f u, produce an expected abolute addtve error of at ot E u,,.e., x, E[ f u(x) f u,(x) ] E u,. (The expectaton over the the randone of f u, and f u.) We aue that all (T u,, E u,) par are known n advance (they are contant or depend only on control nput). Saplng. For each reducton node r wth reducton factor S r, we can decreae th factor S r to a aller factor r {1,..., S r} at the expene of ntroducng oe addtve aplng error E r( r). For exaple, for an averagng node, ntead of averagng all S r nput, we would randoly elect r nput (wthout replaceent) and output the average of the choen aple. For convenence, we denote the aplng rate of node r a η r = r S r. If the output edge of ze R, the coputaton elect r R nput, ntead of all S r R nput. The value for the reducton node nput whch are not elected need not be coputed. Dcardng the coputaton that would otherwe produce thee dcarded nput produce a peed-up factor of η r = r S r for all node above r n the coputaton tree. The followng lea provde a bound on the aplng error S E r( r) = (B A) r r r(s r 1) for an averagng node. The proof avalable n the full veron of the paper.

6 Lea 3.1. Gven nuber x 1, x 2,..., x [A, B], randoly aplng of the nuber x 1,..., x (wthout replaceent) and coputng the aple average gve an approxaton to x 1+ +x wth the followng expected error guarantee: [ x x x1 + + x E 1,..., ] (B A) ( 1). 3.3 Error Propagaton The error that the tranforaton nduce n one part of the coputaton propagate through the ret of the coputaton and can be aplfed or attenuated n the proce. We next provde contrant on the for of functon that characterze th error propagaton. Thee contrant hold for all functon n our odel of coputaton (regardle of whether they have alternate pleentaton or not). We aue that for each functon node f u(x 1,..., x ) wth nput, f each nput x j replaced by oe approxate nput ˆx j uch that E[ x j ˆx j ] δ j, the propagaton error bounded by a lnear error propagaton functon E u: E [ f u(x 1,..., x ) f u(ˆx 1,..., ˆx ) ] E u ( δ1,..., δ ). (2) We aue that all of the error propagaton functon E u for the functon f u are known a pror: E [ f u(x 1,..., x ) f u(ˆx 1,..., ˆx ] ) ( ) α jδ j. (3) Th condton atfed f all functon f u are Lpchtzcontnuou wth paraeter α. Furtherore, f f u(x 1,..., x ) dfferentable, we can let α = ax x f u(x 1,...,x ). x If fu telf probabltc, we can take the expected value of uch α. Subttute Ipleentaton. For functon wth ultple pleentaton, the overall error when we chooe the -th pleentaton f u, bounded by an error propagaton functon E u and the local error nduced by the -th pleentaton E u, (defned n the prevou ubecton): E [ f u(x 1,..., x ) f u,(ˆx 1,..., ˆx ) ] E u ( δ1,..., δ ) +Eu,. (4) Th bound follow edately fro the trangle nequalty. We reark here that the randone for the expectaton n Eq.(4) coe fro 1) the randone of t nput ˆx 1,..., ˆx (caued by error fro prevou part of the coputaton) and 2) rando choce n the pobly probabltc pleentaton f u,. Thee two ource of randone are utually ndependent. Averagng Reducton Node. The averagng functon a Lpchtzcontnuou functon wth all α = 1, o n addton to Lea 3.1 we have: Corollary 3.2. Conder an averagng node that elect rando aple fro t nput, where each nput ˆx j ha bounded error E[ ˆx j x j ] δ j. Then: [ ˆx ˆx x1 + + x E 1,...,,ˆx 1,...,ˆx ] 1 ( ) δ j + (B A) ( 1). j=1 If all nput value have the ae error bound E[ ˆx j x j ] δ, then 1 ( j=1 δj ) = δ. j 4. Approxaton Queton We focu on the followng queton: Queton 1. Gven a progra P n our odel of coputaton and ung randozed confguraton, what the optal error-te tradeoff curve that our approxate coputaton nduce? Here the te and error refer to the expected runnng te and error of the progra. We ay that the expected error of progra P, f for all nput x, E[ P (x) P (x) ]. The error-te tradeoff curve a par of functon (E( ), T ( )), uch that E(t) the optal expected error of the progra f the expected runnng te no ore than t, and T (e) the optal expected runnng te of the progra f the expected error no ore than e. The ubttuton and aplng tranforaton gve re to an exponentally large pace of poble progra confguraton. We optze over arbtrary probablty dtrbuton of uch confguraton. A nave optzaton algorth would therefore run n te at leat exponental n the ze of the progra. We preent an algorth that approxately olve Queton 1 wthn a factor of (1+ε) n te: 3 1) polynoal n the ze of the coputaton graph, and 2) polynoal n 1. The algorth ue lnear prograng ε and a novel technque called b-denonal dcretzaton, whch we preent n Secton 5. A ucceful anwer to the above queton lead drectly to the followng addtonal conequence: Conequence 1: Optzng Te Subject to Error Queton 2. Gven a progra P n our odel, and an overall error tolerance, what the optal (pobly randozed) progra P avalable wthn our pace of tranforaton, wth expected error no ore than? We can anwer th queton approxately ung the optzaton algorth for Queton 1. Th algorth wll produce a randozed progra wth expected runnng te no ore than (1 + ε) te the optal runnng te and expected error no ore than. The algorth can alo anwer the yetrc queton to fnd a (1 + ε)-approxaton of the optal progra that nze the expected error gven a bound on the expected runnng te. Conequence 2: Fro Error to Varance We ay that the overall varance (.e., expected quared error) of a randozed progra P 2, f for all nput x, E[ P (x) P (x) 2 ] 2. A varant of our algorth for Queton 1 (1 + ε)- approxately anwer the followng queton: Queton 3. Gven a progra P n our odel of coputaton, what the optal error-varance tradeoff curve that our approxate coputaton nduce? Queton 4. Gven a progra P n our odel, and an overall varance tolerance 2, what the optal (pobly randozed) progra P avalable wthn our pace of tranforaton, wth varance no ore than 2? Secton 7 preent the algorth for thee queton. Conequence 3: Probablte of Large Error A bound on the expected error or varance alo provde a bound on the probablty of obervng large error. In partcular, an executon 3 We ay that we approxately obtan the curve wthn a factor of (1+ε), f for any gven runnng te t, the dfference between the optal error E(t) and our Ê(t) at ot εe(t), and larly for the te functon T (e). Our algorth a fully polynoal-te approxaton chee (FPTAS). Secton 5 preent a ore prece defnton n whch the error functon Ê(t) alo ubject to an addtve error of oe arbtrarly all contant.

7 of a progra wth expected error wll produce an abolute error greater than t wth probablty at ot 1 (th bound follow t edately fro Markov nequalty). Slarly, an executon of a progra wth varance 2 wll produce an abolute error greater than t wth probablty at ot 1. t 2 5. Optzaton Algorth for Queton 1 We next decrbe a recurve, dynac prograng optzaton algorth whch explot the tree tructure of the progra. To copute the approxate optal tradeoff curve for the entre progra, the algorth copute and cobne the approxate optal tradeoff curve for the ubprogra. We tage the preentaton a follow: Coputaton Node Only: In Secton 5.1, we how how to copute the optal tradeoff curve exactly when the coputaton cont only of coputaton node and ha no reducton node. We reduce the optzaton proble to a lnear progra (whch effcently olvable). B-denonal Dcretzaton: In Secton 5.2, we ntroduce our b-denonal dcretzaton technque, whch contruct a pecewe-lnear dcretzaton of any tradeoff curve (E( ), T ( )), uch that 1) there are only O( 1 ) egent on the ε dcretzed curve, and 2) at the ae te the dcretzaton approxate (E( ), T ( )) to wthn a ultplcatve factor of (1 + ε). A Sngle Reducton Node: In Secton 5.3, we how how to copute the approxate tradeoff curve when the gven progra cont of coputaton node that produce the nput for a ngle reducton node r (ee Fgure 6). We frt work wth the curve when the reducton factor at the reducton node r contraned to be a ngle nteger value. Gven an expected error tolerance e for the entre coputaton, each randozed confguraton n the optal randozed progra allocate part of the expected error E r() to the aplng tranforaton on the reducton node and the reanng expected error e ub = e E r() to the ubttuton tranforaton on the ubprogra wth only coputaton node. One neffcent way to fnd the optal randozed confguraton for a gven expected error e to ply earch all poble nteger value of to fnd the optal allocaton that nze the runnng te. Th approach neffcent becaue the nuber of choce of ay be large. We therefore dcretze the tradeoff curve for the nput to the reducton node nto a all et of lnear pece. It traghtforward to copute the optal nteger value of wthn each lnear pece. In th way we obtan an approxate optal tradeoff curve for the output of the reducton node when the reducton factor contraned to be a ngle nteger. We next ue th curve to derve an approxate optal tradeoff curve when the reducton factor can be deterned by a probabltc choce aong ultple nteger value. Ideally, we would ue the convex envelope of the orgnal curve to obtan th new curve. But becaue the orgnal curve ha an nfnte nuber of pont, t nfeable to work wth th convex envelope drectly. We therefore perfor another dcretzaton to obtan a pecewe-lnear curve that we can repreent wth a all nuber of pont. We work wth the convex envelope of th new dcretzed curve to obtan the fnal approxaton to the optal tradeoff curve for the output of the reducton node r. Th curve ncorporate the effect of both the ubttuton tranforaton on the coputaton node and the aplng tranforaton on the reducton node Output Fgure 4: Exaple to llutrate the coputaton of te and error. The Fnal Dynac Prograng Algorth: In Secton 5.4, we provde an algorth that copute an approxate errorte tradeoff curve for an arbtrary progra n our odel of coputaton. Each tep ue the algorth fro Secton 5.3 to copute the approxate dcretzed tradeoff curve for a ubtree rooted at a topot reducton node (th ubtree nclude the coputaton node that produce the nput to the reducton node). It then ue th tradeoff curve to replace th ubtree wth a ngle functon node. It then recurvely apple the algorth to the new progra, ternatng when t copute the approxate dcretzed tradeoff curve for the output of the fnal node n the progra. 5.1 Stage 1: Coputaton Node Only We tart wth a bae cae n whch the progra cont only of coputaton node wth no reducton node. We how how to ue lnear prograng to copute the optal error-te tradeoff curve for th cae. Varable x. For each functon node f u, the varable x u, [0, 1] ndcate the probablty of runnng the -th pleentaton f u,. We alo have the contrant that xu, = 1. Runnng Te TIME(x). Snce there are no reducton node n the progra, each functon node f u wll run R u te (recall that R u the nuber of value carred on the output edge of f u). The runnng te ply the weghted u of the runnng te of the functon node (where each weght the probablty of electng each correpondng pleentaton): TIME(x) = (x u, T u, R u). (5) u Here the uaton u over all functon node and over all pleentaton of f u. Total Error ERROR(x). The total error of the progra alo adt a lnear for. For each functon node f u, the -th pleentaton f u, ncur a local error E u, on each output value. By the lnear error propagaton aupton, th E u, aplfed by a contant factor β u whch depend on the progra tructure. It poble to copute the β u wth a traveral of the progra backward agant the flow of value. Conder, for exaple, β 1 for functon node f 1 n the progra n Fgure 4. Let α 2 be the lnear error propagaton factor for the unvarate functon f 2( ). The functon f 3(,, ) trvarate wth 3 propagaton factor (α 3,1, α 3,2, α 3,3). We larly defne (α 4,1,..., α 4,4) for the quadvarate functon f 4, and (α 5,1, α 5,2, α 5,3) for f 5. Any error n an output value of f 1 wll be aplfed by a factor β 1: β 1 = ( α 2 (α 4,1 +α 4,2 +α 4,3 )+(α 3,1 +α 3,2 +α 3,3 )α 4,4 ) (α5,1 +α 5,2 ). 1

8 The total expected error of the progra : ERROR(x) = u (x u, E u, β u). (6) Optzaton Gven a fxed overall error tolerance, the followng lnear progra defne the nu expected runnng te: Varable: x Contrant: 0 x u, 1, u, xu, = 1 u ERROR(x) Mnze: TIME(x) By wappng the role of ERROR(x) and TIME(x), t poble to obtan a lnear progra that defne the nu expected error tolerance for a gven expected axu runnng te. (7) 0 1ε 1ε 1ε 1ε 1ε 1ε 1ε 1ε Fgure 5: An exaple of b-denonal dcretzaton. 5.2 Error-Te Tradeoff Curve In the prevou ecton, we ue lnear prograng to obtan the optal error-te tradeoff curve. Snce there are an nfnte nuber of pont on th curve, we defne the curve n ter of functon. To avod unneceary coplcaton when dong nveron, we defne the curve ung two related functon E( ) and T ( ): Defnton 5.1. The (error-te) tradeoff curve of a progra a par of functon (E( ), T ( )) uch that E(t) the optal expected error of the progra f the expected runnng te no ore than t and T (e) the optal expected runnng te of the progra f the expected error no ore than e. We ay that a tradeoff curve effcently coputable f both functon E and T are effcently coputable. 4 The followng property portant to keep n nd: Lea 5.2. In a tradeoff curve (E, T ), both E and T are nonncreang convex functon. Proof. T alway non-ncreang becaue when the allowed error ncreae the nu runnng te doe not ncreae, and larly for E. We prove convexty by contradcton: aue αe(t 1) + (1 α)e(t 2) < E(αt 1 + (1 α)t 2) for oe α (0, 1). Then chooe the optal progra for E(t 1) wth probablty α, and the optal progra for E(t 2) wth probablty 1 α. The reult a new progra P n our probabltc tranforaton pace. Th new progra P ha an expected runnng te le than the optal runnng te E(αt 1 + (1 α)t 2), contradctng the optalty of E. A lar proof etablhe the convexty of T. We reark here that, gven a runnng te t, one can copute E and be ure that (E(t), t) on the curve; but one cannot wrte down all of the nfnte nuber of pont on the curve concely. We therefore ntroduce a b-denonal dcretzaton technque that allow u to approxate (E, T ) wthn a factor of (1 + ε). Th technque ue a pecewe lnear functon wth roughly O( 1 ε ) egent to approxate the curve. Our b-denonal dcretzaton technque (ee Fgure 5) approxate E n the bounded range [0, E ax], where E ax an upper bound on the expected error, and approxate T n the bounded range [T (E ax), T (0)]. We aue that we are gven the axu acceptable error E ax (for exaple, by a uer of the progra). It alo poble to conervatvely copute an E ax by analyzng the leat-accurate poble executon of the progra. 4 In the reander of the paper we refer to the functon E( ) ply a E and to the functon T ( ) a T. Defnton 5.3. Gven a tradeoff curve (E, T ) where E and T are both non-ncreang, along wth contant ε (0, 1) and ε E > 0, we defne the (ε, ε E)-dcretzaton curve of (E, T ) to be the pecewe-lnear curve defned by the followng et of endpont (ee Fgure 5): the two black pont (0, T (0)), (E ax, T (E ax)), the red pont (e, T (e )) where e = ε E(1+ε) for oe 0 and ε E(1 + ε) < E ax, and the blue pont (E(t ), t ) where t = T (E ax)(1 + ε) for oe 1 and T (E ax)(1 + ε) < T (0). Note that there oe ayetry n the dcretzaton of the two axe. For the vertcal te ax we know that the nu runnng te of a progra T (E ax) > 0, whch alway greater than zero nce a progra alway run n a potve aount of te. However, we dcretze the horzontal error ax proportonal to power of (1 + ε) for value above ε E. Th becaue the error of a progra can ndeed reach zero, and we cannot dcretze forever. 5 The followng cla follow edately fro the defnton: Cla 5.4. If the orgnal curve (E, T ) non-ncreang and convex, the dcretzed curve (Ê, ˆT ) alo non-ncreang and convex Accuracy of b-denonal dcretzaton We next defne notaton for the b-denonal tradeoff curve dcretzaton: Defnton 5.5. A curve (Ê, ˆT ) an (ε, ε E)-approxaton to (E, T ) f for any error 0 e E ax, 0 ˆT (e) T (e) εt (e), and for any runnng te T (E ax) t T (0), 0 Ê(t) E(t) εe(t) + εe. We ay that uch an approxaton ha a ultplcatve error of ε and an addtve error of ε E. Lea 5.6. If (Ê, ˆT ) an (ε, ε E)-dcretzaton of (E, T ), then t an (ε, ε E)-approxaton of (E, T ). Proof Sketch. The dea of the proof that, nce we have dcretzed the vertcal te ax n an exponental anner, f we copute ˆT (e) for any value e, the reult doe not dffer fro T (e) by 5 If ntead we know that the nu expected error greater than zero (.e., E(T ax) > 0) for oe axu poble runnng te T ax, then we can defne ε E = E(T ax) jut lke our horzontal ax.

9 1 Output, exact curve for ngle choce of, exact curve for probabltc choce of, curve for each value on th edge dcretze t to, ung paraeter, olve unvarate optzaton proble to get, dcretze t to, ung paraeter, copute t convex envelope, Fgure 6: Algorth for Stage 2. Our cla:,, -appx. to, ;, 2, -appx. to, ;, 2, -appx. to,. ore than a factor of (1 + ε). Slarly, nce we have dcretzed the horzontal ax n an exponental anner, f we copute Ê(t) for any value t, the reult doe not dffer by ore than a factor of (1 + ε), except when E(t) aller than ε E (when we top the dcretzaton). But even n that cae the value Ê(t) E(t) rean aller than ε E. Becaue every pont on the new pecewe-lnear curve (Ê, ˆT ) a lnear cobnaton of oe pont on the orgnal curve (E, T ), 0 ˆT (e) T (e) and 0 Ê(t) E(t), Becaue (E, T ) convex (recall Lea 5.2), the approxaton wll alway le above the orgnal curve Coplexty of b-denonal dcretzaton The nuber of egent that the approxate tradeoff curve ha n an (ε, ε E)-dcretzaton at ot n p def = 2 + Eax log ε E log(1 + ε) + log T (0) T (E ax) log(1 + ε) ( 1 ( O log Eax + log 1 1 ) ) + log T ax + log, (8) ε ε E T n where T n a lower bound on the expected executon te and T ax an upper bound on the expected executon te. Our dcretzaton algorth only need to know E ax n advance, whle T ax and T n are value that we wll need later n the coplexty analy Dcretzaton on an approxate curve The above analy doe not rely on the fact that the orgnal tradeoff curve (E, T ) exact. In fact, f the orgnal curve (E, T ) only an (ε, ε E)-approxaton to the exact error-te tradeoff curve, and f (Ê, ˆT ) the (ε, ε E)-dcretzaton of (E, T ), then one can verfy by the trangle nequalty that (Ê, ˆT ) a pecewe lnear curve that an (ε + ε + εε, ε E + ε E) approxaton of the exact error-te tradeoff curve. 5.3 Stage 2: A Sngle Reducton Node We now conder a progra wth exactly one reducton node r, wth orgnal reducton factor S, at the end of the coputaton. The exaple n Fgure 3c uch a progra. We decrbe our optzaton algorth for th cae tep by tep a llutrated n Fgure 6. We frt defne the error-te tradeoff curve for the ubprogra wthout the reducton node r to be (E ub, T ub ) (Secton 5.1 decrbe how to copute th curve; Lea 5.2 enure that t non-ncreang and convex). In other word, for every nput value to the reducton node r, f the allowed runnng te for coputng th value t, then the optal expected error E ub (t) and larly for T ub (e). Note that when coputng (E ub, T ub ) a decrbed n Secton 5.1, the ze of the output edge R for each node ut be dvded by S, a the curve (E ub, T ub ) characterze each ngle nput value to the reducton node r. If at reducton node r we chooe an actual reducton factor {1, 2,..., S}, the total runnng te and error of th entre progra : 6 TIME = T ub (9) ERROR = E ub + E r(). Th becaue, to obtan value on the nput to r, we need to run the ubprogra te wth a total te T ub ; and by Corollary 3.2, the total error of the output of an averagng reducton node ply the u of t nput error E ub, and a local error E r() ncurred by the aplng. 7 Let (E 1, T 1) be the exact tradeoff curve (E 1, T 1) of the entre progra, aung that we can chooe only a ngle value of. We tart by decrbng how to copute th (E 1, T 1) approxately Approxatng (E 1, T 1): ngle choce of By defnton, we can wrte (E 1, T 1) n ter of the followng two optzaton proble: { T 1(e) = Tub (e ub ) } n {1,...,S} e ub +E r()=e and E 1(t) = n {1,...,S} t ub =t { Eub (t ub ) + E r() }, where the frt optzaton over varable and e ub, and the econd optzaton over varable and t ub. We ephaze here that th curve (E 1, T 1) by defnton non-ncreang (becaue (E ub, T ub ) non-ncreang), but ay not be convex. Becaue thee optzaton proble ay not be convex, they ay be dffcult to olve n general. But thank to the pecewelnear dcretzaton defned n Secton 5.2, we can approxately olve thee optzaton proble effcently. Specfcally, we produce a b-denonal dcretzaton (Êub, ˆT ub ) that (ε, ε E)- approxate (E ub, T ub ) (a llutrated n Fgure 6). We then olve the followng two optzaton proble: T 1(e) = n {1,...,S} e ub +E r()=e { ˆTub (e ub ) } and E 1(t) = n {Êub (t ub ) + E } r(). (10) {1,...,S} t ub =t We reark here that E 1 and T 1 are both non-ncreang nce Êub and ˆT ub are non-ncreang ung Cla 5.4. Each of thee two proble can be olved by 1) coputng the optal value wthn each lnear egent defned by (Êub,k, ˆT ub,k ) and (Êub,k+1, ˆT ub,k+1 ), and 2) returnng the allet optal value acro all lnear egent. Suppoe that we are coputng T 1(e) gven an error e. In the lnear pece of ˆT ub = ae ub + b (here a and b are the lope and ntercept of the lnear egent), we have e ub = e E r(). The objectve that we are nzng therefore becoe unvarate wth repect to : ˆT ub = (ae ub + b) = (a(e E r()) + b). (11) 6 Here we have gnored the runnng te for the aplng procedure n the reducton node, a t often neglgble n coparon to other coputaton n the progra. It poble to add th aplng te to the forula for TIME n a traghtforward anner. 7 We extend th analy to other type of reducton node n Secton 8.

10 The calculaton of a ple unvarate optzaton proble that we can olve quckly ung our expreon for E r(). 8 Coparng the optal anwer fro all of the lnear pece gve u an effcent algorth to deterne T 1(e), and larly for E 1(t). Th algorth run n te lnear n the nuber of pece n p (recall Eq.(8)) n our b-denonal dcretzaton. Th fnhe the coputaton of (E 1, T 1) n Fgure 6. We next how that (E 1, T 1) accurately approxate (E, T ): Cla 5.7. (E 1, T 1) an (ε, ε E)-approxaton to (E 1, T 1). Proof. Becaue ˆT ub approxate T ub, we know that for any e ub, 0 ˆT ub (e ub ) T ub (e ub ) εt ub (e ub ), and th gve: T 1(e) { = ˆTub (e ub ) } n {1,...,S} e ub +E r()=e n {1,...,S} e ub +E r()=e { (1 + ε)tub (e ub ) } = (1 + ε)t 1(e). Slarly, th alo gve that T 1(e) T 1(e) ung 0 ˆT ub (e ub ) T ub (e ub ). Ung a lar technque, we can alo prove that E 1(t) E 1(t) and E 1(t) (1 + ε)e 1(t) + ε E. Therefore, the (E 1, T 1) curve (ε, ε E)-approxate the exact tradeoff curve (E 1, T 1). We next further b-denonally dcretze the curve (E 1, T 1) that we obtaned nto (Ê 1, ˆT 1) ung the ae dcretzaton paraeter (ε, ε E). A dcretzaton of an approxate curve tll approxate (ee Secton Secton 5.2.3). We therefore conclude that Cla 5.8. (Ê 1, ˆT 1) a (2ε+ε 2, 2ε E)-approxaton to (E 1, T 1) Approxatng (E, T ): probabltc choce of Now, we defne (E, T ) to be the exact tradeoff curve (E 1, T 1) of the entre progra, aung that we can chooe probabltcally. We cla: Cla 5.9. (E, T ) the convex envelope of (E 1, T 1). Proof. We frt prove the cla that T the convex envelope of T 1. The proof for E lar. One de of the proof traghtforward: every weghted cobnaton of pont on T 1 hould le on or above T, becaue th weghted cobnaton one canddate randozed confguraton that chooe probabltcally, and T defned to be the optal curve that take nto account all uch randozed confguraton. For the other de of the proof, we need to how that every pont on T a weghted cobnaton of pont on T 1. Let u take an arbtrary pont (e, T (e)). Suppoe that T (e) acheved when the optal probabltc choce of {(, p )} 1 at reducton node r, where we chooe wth probablty p and when choen, the overall error-te ncurred (e, t ). Therefore, we have e = pe and T (e) = pt. Becaue T (e) the exact optal tradeoff curve, each t alo nzed wth repect to e and the fxed choce of. Th equvalent to ayng that (e, t ) le on the curve (E 1, T 1),.e., T 1(e ) = t. Th ple that T the convex envelope of T 1. In general, coputng the convex envelope of an arbtrary functon (E 1, T 1) ay be hard, but thank to our pecewe-lnear dcretzaton, we can copute the convex envelope of (Ê 1, ˆT 1) ealy. Let u denote the convex envelope of (Ê 1, ˆT 1) by (Ê, ˆT ). In fact, (Ê, ˆT ) can be coputed n te O(n p log n p) becaue (Ê 1, ˆT 1) contan only n p endpont. 8 Note that for each dfferent type of reducton code, the optzaton procedure for th unvarate optzaton can be hardcoded. Snce (Ê 1, ˆT 1) a (2ε + ε 2, ε E)-approxaton to (E 1, T 1) by Cla 5.8, we hould expect the ae property to hold for ther convex envelope: Cla (Ê, ˆT ) a (2ε+ε 2, 2ε E)-approxaton to (E, T ). Proof. By the defnton of convex envelope, for all te t, there ext oe α [0, 1] uch that E(t) = αe 1(t 1) + (1 α)e 1(t 2) and αt 1 + (1 α)t 2 = t. Then, Ê (t) αê 1(t 1) + (1 α)ê 1(t 2) 2ε E + (1 + 2ε + ε 2 )(αe 1(t 1) + (1 α)e 1(t 2)) = 2ε E + (1 + 2ε + ε 2 )E(t), where the frt nequalty ue the fact that Ê the convex envelope of Ê 1, and the econd ue the fact that Ê 1 approxate E 1. At the ae te, there ext oe β [0, 1] uch that Ê (t) = βê 1(t 3) + (1 β)ê 1(t 4) and βt 3 + (1 β)t 4 = t. Then, Ê (t) = βê 1(t 3) + (1 β)ê 1(t 4) βe 1(t 3) + (1 β)e 1(t 4) E(t), where the frt nequalty ue the fact that Ê 1 approxate E 1, and the econd ue the fact that E the convex envelope of E 1. We can derve the two lar nequalte for the te functon ˆT and conclude that (Ê, ˆT ) a (2ε + ε 2, 2ε E)-approxaton to (E, T ). So far, we have fnhed all tep decrbed n Fgure 6. We have ended wth a pecewe-lnear tradeoff curve (Ê, ˆT ) that (2ε + ε 2, 2ε E)-approxate the exact tradeoff curve (E, T ), takng nto account the probabltc choce at th reducton node a well. 5.4 The Fnal Dynac Prograng Algorth We next how how to copute the approxate error-te tradeoff curve for any progra n our odel of coputaton. We frt preent the algorth, then we dcu how to chooe the dcretzaton paraeter ε and ε E and how the dcretzaton error copoe. If the progra ha no reducton node, we can ply apply the analy fro Secton 5.1. Otherwe, there ut ext at leat one topot reducton node r whoe nput coputed fro coputaton node only. Aue (E, T ) the exact error-te tradeoff curve for the output of r. Applyng Stage 2 (Secton 5.3) to the ubprogra rooted at r, we can effcently fnd oe pecewe-lnear curve (Ê, ˆT ) that accurately approxate (E, T ). Recall that th pecewe-lnear curve (Ê, ˆT ) convex (nce t a convex envelope). If we pck all of t at ot n p endpont on the curve P = {(E r,, T r,)} np =1, then 1) every pont on the curve can be panned by at ot two pont n P becaue the functon pecewe lnear, and 2) all pont that can be panned by P le above the curve becaue the functon convex. The two obervaton 1) and 2) above ndcate that we can replace th reducton node r along wth all coputaton node above t, by a ngle functon node f r uch that t -th ubttute pleentaton f r, gve an error of E r, and a runnng te of T r,. Obervaton 1) ndcate that every error-te tradeoff pont on (Ê, ˆT ) can be pleented by a probabltc xture of, and obervaton 2) ndcate that every error-te tradeoff pont that can be pleented, no better than the orgnal curve (Ê, ˆT ). In u, th functon node acheve the ae errorte tradeoff a the pecewe-lnear curve (Ê, ˆT ). {f r,} np =1