Directed Regression. Benjamin Van Roy Stanford University Stanford, CA Abstract

Transcription

1 Diected Regession Yi-hao Kao Stanfod Univesity Stanfod, CA Benjamin Van Roy Stanfod Univesity Stanfod, CA Xiang Yan Stanfod Univesity Stanfod, CA Abstact When used to guide decisions, linea egession analysis typically involves estimation of egession coefficients via odinay least squaes and thei subsequent use to mae decisions. When thee ae multiple esponse vaiables and featues do not pefectly captue thei elationships, it is beneficial to account fo the decision objective when computing egession coefficients. Empiical optimization does so but sacifices pefomance when featues ae well-chosen o taining data ae insufficient. We popose diected egession, an efficient algoithm that combines meits of odinay least squaes and empiical optimization. We demonstate though a computational study that diected egession can geneate significant pefomance gains ove eithe altenative. We also develop a theoy that motivates the algoithm. Intoduction When used to guide decision-maing, linea egession analysis typically teats estimation of egession coefficients sepaately fom thei use to mae decisions. In paticula, estimation is caied out via odinay least squaes (OLS) without consideation of the decision objective. The egession coefficients ae then used to optimize decisions. When thee ae multiple esponse vaiables and featues do not pefectly captue thei elationships, it is beneficial to account fo the decision objective when computing egession coefficients. Impefections in featue selection ae common since it is difficult to identify the ight featues and the numbe of featues is typically esticted in ode to avoid ove-fitting. Empiical optimization (EO) is an altenative to OLS which selects coefficients that minimize empiical loss in the taining data. Though it accounts fo the decision objective when computing egession coefficients, EO sacifices pefomance when featues ae well-chosen o taining data is insufficient. In this pape, we popose a new algoithm diected egession (DR) which is a hybid between OLS and EO. DR selects coefficients that ae a convex combination of those that would be selected by OLS and those by EO. The weights of OLS and EO coefficients ae optimized via cossvalidation. We study DR fo the case of decision poblems with quadatic objective functions. The algoithm taes as input a taining set of data pais, each consisting of featue vectos and esponse vaiables, togethe with a quadatic loss function that depends on decision vaiables and esponse vaiables. Regession coefficients ae computed fo subsequent use in decision-maing. Each futue decision depends on newly sampled featue vectos and is made pio to obseving esponse vaiables with the goal of minimizing expected loss. We pesent computational esults demonstating that DR can substantially outpefom both OLS and EO. These esults ae fo synthetic poblems with egession models that include subsets of elevant

2 featues. In some cases, OLS and EO delive compaable pefomance while DR educes expected loss by about 0%. In none of the cases consideed does eithe OLS o EO outpefom DR. We also develop a theoy that motivates DR. This theoy is based on a model in which selected featues do not pefectly captue elationships among esponse vaiables. We pove that, fo this model, the optimal vecto of coefficients is a convex combination of those that would be geneated by OLS and EO. Linea Regession fo Decision-Maing Suppose we ae given a set of taining data pais O = {(x (), y () ),, (x (N), y (N) )}. Each nth data pai is compised of featue vectos x (n),..., x(n) K RM and a vecto y (n) R M of esponse vaiables. We would lie to compute egession coefficients R K so that given a data pai (x, y), the linea combination x of featue vectos estimates the expectation of y conditioned on x. We estict attention to cases whee M >, with special inteest in poblems whee M is lage, because it is in such situations that DR offes the lagest pefomance gains. We conside a setting whee the egession model is used to guide futue decisions. In paticula, afte computing egession coefficients, each time we obseve featue vectos x,..., x K we will have to select a decision u R L befoe obseving the esponse vecto y. The choice incus a loss l(u, y) = u G u + u G y, whee the matices G R L L and G R L M ae nown, and the fome is positive definite and symmetic. We aim to minimize expected loss, assuming that the conditional expectation of y given x is K = x. As such, given x and, we select a decision ( u (x) l u u, ) x = = G G x. The question is how best to compute the egession coefficients fo this pupose. To motivate the setting we have descibed, we offe a hypothetical application. Example. Conside an Intenet banne ad campaign that tagets M classes of customes. An aveage evenue of y m is eceived pe custome of class m that the campaign eaches. This quantity is andom and influenced by K obsevable factos x m,..., x Km. These factos may be coelated acoss customes classes; fo example, they could captue custome pefeences as they elate to ad content o how cuent economic conditions affect customes. Fo each mth class, the cost of eaching the u m th custome inceases with u m because ads ae fist tageted at customes that can be eached at lowe cost. This cost is quadatic, so that we pay γ m u m to each u m customes, whee γ m is a nown constant. The application we have descibed fits ou geneal poblem context. It is natual to pedict the esponse vecto y using a linea combination x of factos with the egession coefficients computed based on past obsevations O = {(x (), y () ),, (x (N), y (N) )}. The goal is to maximize expected evenue less advetising costs. This gives ise to a loss function that is quadatic in u and y: M l(u, y) = (γ m u m u m y m ). m= One might as why not constuct M sepaate linea egession models, one fo each esponse vaiable, each with a sepaate set of K coefficients. The eason is that this gives ise to MK coefficients; when M is lage and data is limited, this could lead to ove-fitting. Models of the sot we conside, whee egession coefficients ae shaed acoss multiple esponse vaiables, ae sometimes efeed to as geneal linea models and have seen a wide ange of applications 7, 8]. It is well-nown that the quality of esults is highly sensitive to the choice of featues, even moe so than fo models involving a single esponse vaiable 7]. =

3 3 Algoithms Odinay least squaes (OLS) is a conventional appoach to computing egession coefficients. This would poduce a coefficient vecto OLS R y(n) x (n). () K n= Note that OLS does not tae the decision objective into account when computing egession coefficients. Empiical optimization (EO), as studied fo example in, 6], offes an altenative that does so. This appoach minimizes empiical loss on the taining data: EO R K = l(u (x (n) ), y (n) ). () n= Note that EO does not explicitly aim to estimate the conditional expectation of the esponse vecto. Instead it focusses on decision loss that would be incued with the taining data. Both OLS and EO can be computed efficiently by minimizing convex quadatic functions. As we will see in ou computational and theoetical analyses, OLS and EO can be viewed as two extemes, each offeing oom fo impovement. In this pape, we popose an altenative algoithm diected egession (DR) which poduces a convex combination DR = ( λ) OLS + λ EO of coefficients computed by OLS and EO. The tem diected is chosen to indicate that DR is influenced by the decision objective though, unlie EO, it does not simply minimize empiical loss. The paamete λ 0, ] is computed via coss-validation, with an objective of minimizing aveage loss on validation data. Aveage loss is a convex quadatic function of λ, and theefoe can be easily minimized ove λ 0, ]. DR is designed to geneate decisions that ae moe obust to impefections in featue selection than OLS. As such, DR addesses issues simila to those that have motivated wo in data-diven obust optimization, as suveyed in 3]. Ou focus on maing good decisions despite modeling inaccuacies also complements ecent wo that studies how models deployed in pactice can geneate effective decisions despite thei failue to pass basic statistical tests 4]. 4 Computational Results In this section, we pesent esults fom applying OLS, EO, and DR to synthetic data. To geneate a data set, we fist sample paametes of a geneative model as follows:. Sample P matices C,..., C P R M Q, with each enty fom each matix dawn independently fom N (0, ).. Sample a vecto R P fom N (0, I). 3. Sample G a R L L and G b R L M, with each enty of each matix dawn fom N (0, ). Let G = G a G a and G = G a G b. Given geneative model paametes C,..., C P and, we sample each taining data pai (x (n), y (n) ) as follows:. Sample a vecto φ (n) R Q fom N (0, I) and a vecto w (n) R M fom N (0, σ wi).. Let y (n) = P i= ic i φ (n) + w (n). 3. Fo each =,,, K, let x (n) = C φ (n). The vecto φ (n) can be viewed as a sample fom an undelying infomation space. The matices C,..., C P extact featue vectos fom φ (n). Note that, though esponse vaiables depend on P featue vectos, only K P ae used in the egession model. Given geneative model paametes and a coefficient vecto R K, it is easy to evaluate the expected loss l() = E x,y l(u (x), y)]. It is also easy to evaluate the minimal expected loss l = 3

4 Excess Loss OLS EO DR Excess Loss OLS EO DR N (a) K (b) Figue : (a) Excess losses deliveed by OLS, EO, and DR, fo diffeent numbes N of taining samples. (b) Excess losses deliveed by OLS, EO, and DR, using diffeent numbes K of the 60 featues. min E x,y l(u (x), y)]. We will assess each algoithm in tems of the excess loss l() l deliveed by the coefficient vecto that the algoithm computes. Excess loss is nonnegative, and this allows us to mae compaisons in pecentage tems. We caied out two sets of expeiments to compae the pefomance of OLS, EO, and DR. In the fist set, we let M = 5, L = 5, P = 60, Q = 0, σ w = 5, and K = 50. Fo each N {0, 5, 0, 30, 50}, we an 00 tials, each with an independently sampled geneative model and taining data set. In each tial, each algoithm computes a coefficient vecto given the taining data and loss function. With DR, λ is selected via leave-one-out coss-validation when N 0, and via 5-fold coss-validation when N > 0. Figue (a) plots excess losses aveaged ove tials. Note that the excess loss incued by DR is neve lage than that of OLS o EO. Futhe, when N = 0, the excess loss of OLS and EO ae both aound 0% lage than that of DR. Fo small N, OLS is as effective as DR, while, EO becomes as effective as DR as N gows lage. In the second set of expeiments, we use the same paamete values as in the fist set, except we fix N = 0 and conside use of K {45, 50, 55, 58, 60} featue vectos. Again, we an 00 tials fo each K, applying the thee algoithms as in the fist set of expeiments. Figue (b) plots excess losses aveaged ove tials. Note that when K = 55, DR delives excess loss aound 0% less than EO and OLS. When K = P = 60, thee ae no missing featues and OLS matches the pefomance of DR. Figue plots the values of λ selected by coss-validation, each aveaged ove the 00 tials, as a function of N and K. As the numbe of taining samples N gows, so does λ, indicating that DR is weighted moe heavily towad EO. As the numbe of featue vectos K gows, λ diminishes, indicating that DR is weighted moe heavily towad OLS. 5 Theoetical Analysis In this section, we fomulate a geneative model fo the taining data and futue obsevations. Fo this model, optimal coefficients ae convex combinations of OLS and EO. As such, ou model and analysis motivate the use of DR. 5. Model In this section, we descibe a geneative model that samples the taining data set, as well as missing featues, and a epesentative futue obsevation. We then fomulate an optimization poblem whee the objective is to minimize expected loss on the futue obsevation conditioned on the taining data and missing featues. It may seem stange to condition on missing featues since in pactice they ae unavailable when computing egession coefficients. Howeve, we will late establish that optimal 4

5 λ 0.4 λ N (a) K (b) Figue : (a) The aveage values of selected λ, fo diffeent numbes N of taining samples. (b) The aveage values of selected λ, using diffeent numbes K of the 60 featues. coefficients ae convex combinations of OLS and EO, each of which can be computed without obseving missing featues. Since diected egession seaches ove these convex combinations, it should appoximate what would be geneated by a hypothetical algoithm that obseves missing featues. We will assume that each featue, whethe obseved o missing, is a linea function of an infomation vecto dawn fom R Q. Specifically, the N taining data samples depend on infomation vectos φ (),..., φ (N) R Q. A linea function mapping an infomation vecto to a featue vecto can be epesented by a matix in R M Q, and to descibe ou geneative model, it is useful to define an inne poduct fo such matices. In paticula, we define the inne poduct between matices A and B by A, B = (Aφ (n) ) (Bφ (n) ). N n= Ou geneative model taes seveal paametes as input. Fist, thee ae the numbe of samples N, the numbe of esponse vaiables M, and the numbe of featue vectos K. Second, a paamete µ Q specifies the expected dimension of the infomation vecto. Finally, thee ae standad deviations σ, σ ɛ, and σ w, of obseved featue coefficients, missing featue coefficients, and noise, espectively. Given paametes N, M, K, µ Q, σ, σ ɛ, and σ w, the geneative model poduces data as follows:. Sample Q fom the geometic distibution with mean µ Q.. Sample φ (),..., φ (N) R Q fom N (0, I Q ). 3. Sample C,..., C K and D,, D J R M Q with each enty i.i.d. fom N (0, ), whee K + J = MQ. 4. Apply the Gam-Schmidt algoithm with espect to the inne poduct defined above to geneate an othonomal basis C,..., C K, D,..., D J fom the sequence C,..., C K, D,..., D J. 5. Sample R K fom N (0, σ I K ) and R J fom N (0, σ ɛ I J ). 6. Fo n =,, N, sample w (n) R M fom N (0, σwi M ), and let x (n) = C φ (n) C K φ (n)], (3) z (n) = D φ (n) DJ φ (n)], (4) y (n) = = x (n) + J j= j z (n) j + w (n). (5) 5

6 7. Sample φ unifomly fom {φ (),, φ (N) } and w R M fom N (0, σ wi M ). Geneate x, z, and ỹ by the same functions in (3), (4), and (5). The samples z (),..., z (N), z epesent missing featues. The Gam-Schmidt pocedue ensues two popeties. Fist, since C, D j = 0, missing featues ae uncoelated with obseved featues. If this wee not the case, obseved featues would povide infomation about missing featues. Second, since D,..., D J ae othonomal, the distibution of missing featues is invaiant to otations in the J-dimensional subspace fom which they ae dawn. In othe wods, all diections in that space ae equally liely. We define an augmented taining set O = {(x (), z (), y () ),, (x (N), z (N), y (N) )} and conside selecting egession coefficients ˆ R K that solve min El(u ( x), ỹ) O]. R K Note that the pobability distibution hee is implicitly defined by ou geneative model, and as such, ˆ may depend on N, M, K, µ Q, σ, σ ɛ, σ w, and O. 5. Optimal Solutions Ou pimay inteest is in cases whee pio nowledge about the coefficients is wea and does not significantly influence ˆ. As such, we will fom hee on estict attention to the case whee σ is asymptotically lage. Hence, ˆ will no longe depend on σ. It is helpful to conside two special cases. One is whee σ ɛ = 0 and the othe is whee σ ɛ is asymptotically lage. We will efe to ˆ in these exteme cases as ˆ 0 and ˆ. The following theoem establishes that these extemes ae deliveed by OLS and EO. Theoem. Fo all N, M, K, µ Q, σ w, and O, ˆ 0 R K y(n) n= = x (n) and ˆ R K l(u (x (n) ), y (n) ). n= Note that σ ɛ epesents the degee of bias in a egession model that assumes thee ae no missing featues. Hence, the above theoem indicates that OLS is optimal when thee is no bias while EO is optimal as the bias becomes asymptotically lage. It is also woth noting that the coefficient vectos ˆ 0 and ˆ can be computed without obseving the missing featues, though ˆ is defined by an expectation that is conditioned on thei ealizations. Futhe, computation of ˆ 0 and ˆ does not equie nowledge of Q o σ w. Ou next theoem establishes that the coefficient vecto ˆ is always a convex combination of ˆ 0 and ˆ. Theoem. Fo all N, M, K, µ Q, σ w, σ ɛ, and O, whee λ =. + σ w Nσɛ ˆ = ( λ)ˆ 0 + λˆ, Ou two theoems togethe imply that, with an appopiately selected λ 0, ], ( λ) OLS + λ EO = ˆ. This suggests that diected egession, which optimizes λ via coss-validation to geneate a coefficient vecto DR = ( λ) OLS + λ EO, should appoximate ˆ well without obseving the missing featues o equiing nowledge of Q, σ ɛ, o σ w. 6

7 5.3 Intepetation To develop intuition fo ou esults, we conside an idealized situation whee the coefficients and ae povided to us by an oacle. Then the optimal coefficient vecto would be O R K El(u ( x), ỹ) O,, ]. It can be shown that OLS is a biased estimato of O, while EO is an unbiased one. Howeve, the vaiance of OLS is smalle than that of EO. The optimal tadeoff is indeed captued by the value of λ povided in Theoem. In paticula, as the numbe of taining samples N inceases, vaiance diminishes and λ appoaches, placing inceasing weight on EO. On the othe hand, as the numbe of obseved featues K inceases, model bias deceases and λ appoaches 0, placing inceasing weight on OLS. Ou expeimental esults demonstate that the value of λ selected by coss-validation exhibits the same behavio. 6 Extensions Though we only teated linea models and quadatic objective functions, ou wo suggests that thee can be significant gains in boade poblem settings fom a tighte coupling between machine leaning and decision-maing. In paticula, machine leaning algoithms should facto decision objectives into the leaning pocess. It will be inteesting to exploe how to do this with othe classes of models and objectives. One might ague that featue mis-specification is not a citical issue in light of effective methods fo subset selection. In paticula, athe than selecting a few featues and facing the consequences of model bias, one might select an enomous set of featues and apply a method lie the lasso 0] to identify a small subset. Ou view is that even this enomous set will esult in model biases that might be amelioated by genealizations of DR. Thee is also the concen that data equiements gow with the size of the lage featue set, albeit slowly. Undestanding how to synthesize DR with subset selection methods is an inteesting diection fo futue eseach. Anothe issue that should be exploed is the effectiveness of coss-validation in optimizing λ. In paticula, it would be helpful to undestand how the estimate elates to the ideal value of λ identified by Theoem. Moe geneal wo on the selection of convex combinations of models (e.g.,, 5]) may lend insights to ou setting. Let us close by mentioning that the ideas behind DR ought to play a ole in einfocement leaning (RL) as pesented in 9]. RL algoithms lean fom expeience to pedict a sum of futue ewads as a function of a state, typically by fitting a linea combination of featues of the state. This socalled appoximate value function is then used to guide sequential decision-maing. The poblem we addessed in this pape can be viewed as a single-peiod vesion of RL, in the sense that each decision incus an immediate cost but beas no futhe consequences. It would be inteesting to extend ou idea to the multi-peiod case. Acnowledgments We than James Robins fo helpful comments and suggestions. The fist autho is suppoted by a Stanfod Gaduate Fellowship. This eseach was suppoted in pat by the National Science Foundation though gant CMMI Appendix ] ] Poof of Theoem. Fo each n, let x (n) = x (n) x (n), z (n) = K z (n) z (n). J Let X = x () x ] (N), Z = z () z ] (N), Y = y () y ] (N), = E O], = E O]. Fo any matix V, let V denote (V V ) V. Recall that C, D j = 0,, j implies that each column of X is othogonal to 7

8 each column of Z. Because,, O ae jointly Gaussian, as σ, we have ] J (, ) σw n= y(n) x (n) j z (n) j + J = j= σɛ j j= ] σ w Y σ w X ] ] σ w Z (X X) X Y (, ) 0 0 σ ɛ I J = (Z Z + σ w σ I) Z Y ɛ ]. Let a (n) = G G x (n), b (n) = G G z (n), A = a () a ] (N), B = b () b ] (N). We have ˆ El(u ( x), ỹ) O] N Eỹl(u ( x), ỹ) x = x (n), O] n= u (x (n) ) G u (x (n) ) + u (x (n) ) G Eỹ x = x (n), O] n= n= 4 a (n) a (n) a (n) (a (n) + b (n) ) = + A B = X Y + A B(Z Z + σ w I) Z Y. (6) Taing σ ɛ 0 and σ ɛ yields ˆ 0 = X Y, (7) ˆ = X Y + A BZ Y. (8) The fist pat of the theoem then follows because ˆ 0 = X Y Y X We now pove the second pat. Note that agmin l(u (x (n) ), y (n) ) n= A A N n= h = σ ɛ y(n) n= = x (n). u (x (n) ) G u (x (n) ) + u (x (n) ) G y (n) n= h (n) y (n) = (A A) H Y, whee h (n) = G G G x (n) and H = h () h ] (N). Each th column of H G G G C φ (). G G G C φ (N) is in span{col X, col Z} because G G G C R M Q = span{c,, C K, D,, D J }. Since the esidual Y = Y XX Y ZZ Y upon pojecting Y onto span {col X, col Z} is othogonal to the subspace, we have h Y = 0, and hence H Y = 0. This implies H Y = H XX Y + H ZZ Y. Futhe, since a (n) a (n) = h (n) x (n), a (n) b (n) = h (n) z (n), n, we have ˆ = X Y + A BZ Y = (A A) ( A AX Y + A BZ Y ) = (A A) ( H XX Y + H ZZ Y ) = (A A) H Y. Poof of Theoem. Because D i, D j = {i = j}, we have Z Z = NI. Plugging this into (6) and compaing the esultant expession with (7) and (8) yield the desied esult. 8

9 Refeences ] J.-Y. Audibet. Aggegated estimatos and empiical complexity fo least squae egession. Annales de l Institut Heni Poincae Pobability and Statistics, 40(6): , 004. ] P. L. Batlett and S. Mendelson. Empiical minimization. Pobability Theoy and Related Fields, 35(3):3 334, ] D. Betsimas and A. Thiele. Robust and data-diven optimization: Moden decision-maing unde uncetainty. In Tutoials on Opeations Reseach. INFORMS, ] O. Besbes, R. Philips, and A. Zeevi. Testing the validity of a demand model: An opeations pespective ] F. Bunea, A. B. Tsybaov, and M. H. Wegamp. Aggegation fo Gaussian egession. The Annals of Statistics, 35(4): , ] D. Haussle. Decision theoetic genealizations of the PAC model fo neual net and othe leaning applications. Infomation and Computation, 00:78 50, 99. 7] K. Kim and N. Timm. Univaiate and Multivaiate Geneal Linea Models: Theoy and Applications with SAS. Chapman & Hall/CRC, ] K. E. Mulle and P. W. Stewat. Linea Model Theoy: Univaiate, Multivaiate, and Mixed Models. Wiley, ] R. S. Sutton and A. G. Bato. Reinfocement Leaning: An Intoduction. MIT Pess, Cambidge, MA, ] R. Tibshiani. Regession shinage and selection via the lasso. Jounal of Royal Statistical Society,