Prediction Uncertainty of Density Functional Approximations for Properties of Crystals with Cubic Symmetry

Transcription

1 pub.ac.org/jpca Downloaded by UIV PIERRE ET MARIE CURIE on September 7, 05 Prediction Uncertainty of Denity Functional Approximation for Propertie of Crytal with Cubic Symmetry Pacal Pernot,*,, Bartolomeo Civalleri, Davide Preti, and Andrea Savin,# Laboratoire de Chimie Phyique, UMR8000, CRS, F-9405 Oray, France Laboratoire de Chimie Phyique, UMR8000, Univ. Pari-Sud, F-9405 Oray, France Department of Chemitry and IS Center, Univerity of Torino, Via P. Giuria 7, I-05 Torino, Italy Department of Chemical and Geological Science, Univerity of Modena and Reggio-Emilia, Via Campi 83, I-45 Modena, Italy Laboratoire de Chimie Theórique, UMR766, CRS, F Pari, France # Laboratoire de Chimie Theórique, UMR766, UPMC Univ Pari 06, F Pari, France *S Supporting Information ABSTRACT: The performance of a method i generally meaured by an aement of the error between the method reult and a et of reference data. The prediction uncertainty i a meaure of the confidence that can be attached to a method prediction. It etimation i baed on the random part of the error not explained by reference data uncertainty, which implie an evaluation of the ytematic component() of the error. A the prediction of mot denity functional approximation (DFA) preent ytematic error, the tandard performance tatitic, uch a the mean of the abolute error (MAE or MUE), cannot be directly ued to infer prediction uncertainty. We invetigate here an a poteriori calibration method to etimate the prediction uncertainty of DFA for propertie of olid. A linear model i hown to be adequate to addre the ytematic trend in the error. The applicability of thi approach to modet-ize reference et (8 ytem) i evaluated for the prediction of band gap, bulk moduli, and lattice contant with a wide panel of DFA.. ITRODUCTIO The ucce of denity functional theory, of modern algorithm and computer ha produced not only a large amount of numerical reult but alo a large number of denity functional approximation (DFA). To chooe among thoe, benchmark data et are increaingly ued. Although thi hould be een a a quantification of experience, one hould be alo warned that uing tatitical tool to quantify DFA performance ha it pitfall, and care i needed. If ranking i a concern for DFA deigner to ae the overall performance of new development, it i le practically ueful to end uer, who need to elect a method with criteria uch a code availability, computing performance, and mot important, prediction uncertainty. The latter provide a confidence meaure on the reult of a DFA for a given property. If, in addition to performance tatitic, uer are informed of the prediction uncertainty of DFA, they might have a better rationale to elect a method atifying their pecific requirement. The definition of prediction uncertainty for computational chemitry method ha been formalized by Irikura et al. in the virtual meaurement (VM) framework. The interet of VM i to define a tatitical approach in agreement with international tandard for the evaluation of meaurement uncertainty, a recommended by the Guide to the Expreion of Uncertainty in Meaurement (GUM). 3 The VM approach ha been adopted by the ational Intitute of Standard and Technology (IST), notably for it Computational Chemitry Comparion and Benchmark Databae (CCCBDB). The VM framework ha been reported in the computational chemitry literature motly to etimate prediction uncertainty for caled harmonic and anharmonic vibrational frequencie and zero-point energie.,4 0 Recently, Rucic trongly recommended it ue to improve the uncertainty evaluation of predicted thermochemical quantitie. The interet of thi approach ha alo been demontrated for molecular imulation. 4 In the GUM approach to uncertainty etimation, it i aumed that the reult of a meaurement ha been corrected for all recognized ignificant ytematic effect and that every effort ha been made to identify uch effect. 3 Thi i a key point that i challenging for computational chemitry, where mot error ource are known to be ytematic, due to the variou approximation in the chemitry model. The correction of ytematic error can only be achieved by comparion with reference data. The aement of a prediction uncertainty require therefore either an internal calibration (adjutment of parameter) of a method againt a reference data et or an a poteriori calibration of the reult of thi method. We addre the latter approach in thi article. Special Iue: Jacopo Tomai Fetchrift Received: October, 04 Revied: January 3, 05 Publihed: January 7, American Chemical Society 588 DOI: 0.0/jp509980w

2 Downloaded by UIV PIERRE ET MARIE CURIE on September 7, 05 The internal calibration of emiempirical DFA followed by propagation of the uncertainty on calibrated parameter to prediction ha been parely reported, 5 7 and recently applied to computational catalyi. 8,9 Few tudie along imilar line have alo been reported for molecular imulation force field. 0 In the a poteriori approach, calibration i ued to remove the predictable part of the error (ytematic error). Prediction uncertainty for a method i then derived from the remaining, unpredictable part of the error (random error). 3 In the aforementioned vibrational frequency application, correction of ytematic error i done through the caling of the calculated data, and the root-mean-quare of the error (RMSE) of caled vibrational frequencie ha been hown to provide, under mild condition, a good approximation of prediction uncertainty. 8,9 Thi caling approach ha been ued recently by Lejaeghere et al. 3 to etimate the prediction error of olid tate DFA for elemental crytal. A will be hown below, two point need to be addreed to complement thee caling tudie for other ytem and propertie: (i) the calibration model cannot alway be reduced to a imple caling, and (ii) the reference data uncertaintie are not alway mall enough to be neglected in the tatitical analyi. We preent therefore a detailed derivation of prediction uncertainty of computational method by a poteriori calibration in a more general framework than for a ingle caling factor, moreover taking into account the uncertainty on the reference data. The method i applied to the calculation of lattice contant, bulk moduli, and band gap for a et of 8 crytal (emiconductor and inulator) with cubic ymmetry, by 8 different DFA (local, emilocal, and hybrid). The paper conit of four main ection. In the firt part, we preent the difficulty of deriving prediction uncertainty from common performance tatitic provided in the benchmark literature and the need to deign a pecific approach. In the econd part, we illutrate a typical ditribution of error oberved in the application cae and derive an adequate tochatic calibration/prediction model. In the third ection, we apply the calibration/prediction model to the reference data and tudy it validity. Dicuion of the advantage and limitation of the VM approach in the context of thi tudy i the object of the fourth ection.. METHOD PERFORMACE EVALUATIO Performance evaluation of computational chemitry method relie on two ingredient: a benchmark data et ued a a reference to ae calculation accuracy, and performance tatitic on the difference between calculation and reference data (ee, e.g., Peverati and Truhlar 4 for a recent review). Both ingredient play a crucial role in performance aement... Definition. We thereafter call error the difference between the value of a property, c m,, calculated for a ytem by a method (e.g., DFA) m, and the correponding reference value, o (oberved or calculated): em, = cm, o () For performance aement of a method, one ue tatitic ummarizing the error et containing the error value of all ytem for a given method, E m ={e m, ; =, }, where i the number of ytem in the reference et. In the following, we conider determinitic method and aume that all ource of code uncertainty are controlled at a negligible level (numerical error, convergence threhold effect, etc. ). In thi cae, the error can be attributed (i) to reference data uncertainty, u, and if thi ource alone cannot explain the amplitude of the error, (ii) to method inadequacy error, characterizing the inability of a method to predict the reference data within their error bar. The uncertainty of the reference data i therefore a key piece of information to properly ae method inadequacy error... Performance Etimator: MAD v MAD. Several performance tatitic are commonly ued in the benchmark literature to rank method. We review thee etimator to appreciate their uability, or lack thereof, in the etimation of prediction uncertainty. Firt, there i ome confuion in the computational chemitry literature about the nomenclature of the performance tatitic. In particular, the ue of ome acronym conflict with the tandard ue in the tatitical literature. The main example i the mean abolute deviation (MAD), which i commonly ued in the community to refer to the mean of the abolute error (MAE) MAE = em, = wherea for tatitician 5 MAD i a meaure of diperion around a reference point, either the mean abolute deviation (from the arithmetic mean E m), M[ean]AD = em, Em = or the median abolute deviation (from the median med(e m )) M[edian]AD = med( Em med( Em ) ) (4) Synonym of MAE in the computational chemitry literature are the mean unigned error/deviation (MUE/D) and the average abolute error/deviation (AAE/D). The occaional occurrence in thi corpu of a meaningle definition of MAD a mean average deviation i even more confuing. 6 The arithmetic mean i often referred to a mean igned error (MSE) MSE = Em = em, = Uncertainty i defined in the International Vocabulary of Metrology 7 a a non-negative parameter characterizing the diperion of the quantity value being attributed to a meaurand. With regard to thi definition, it i important to acknowledge that MAE and MeanAD are different tatitic: MeanAD i a meaure of diperion (for a normal ditribution of tandard deviation σ, one ha MeanAD = /πσ /. For E m = 0, MAE and MeanAD are identical, but when E m 0, MAE i a non-invertible mixture of diperion and location tatitic. In the extreme cae where all error are poitive, MAE i equal to MSE, a meaure of location. In a recent paper intended on clarifying the difference between MAE and prediction uncertainty, Rucic addree MAE (called MAD in the paper, but unambiguouly ynonymized with MUE) a a diperion meaure, which it i () (3) (5) 589 DOI: 0.0/jp509980w

3 Downloaded by UIV PIERRE ET MARIE CURIE on September 7, 05 not for non-zero-centered error ample, the tandard cae in computational chemitry. The MAE can be ued, among many other criteria, to rank method but hould not be ued to ae the uncertainty aociated with a given method. The ame remark apply to the root-mean-quare error (RMSE) RMSE = em, = which i commonly ued alongide the MAE in the benchmark literature. The correponding meaure of diperion i the rootmean-quare deviation (RMSD) RMSD = ( em, Em ) The equality = RMSE = RMSD + MSE (8) clearly how the RMSE a a mixture of location and diperion meaure. The interet of RMSE in the context of performance meaure i alo diputed, becaue of the better robutne of MAE to outlier, but the debate i ongoing. 8,9.3. From Performance Etimator to Prediction Uncertainty. If an error et i affected by a contant (i.e., ytem-independent) ytematic contribution, then the MSE etimate the mean value of the ytematic error, and the RMSD provide the tandard deviation of the remaining (random) part of the error. In the cae of a negligible contribution of the reference data uncertainty, the RMSD and the uncertainty on the MSE could then be combined to etimate a prediction uncertainty. A will be illutrated in the next ection, DFA do not generally produce only contant ytematic error. 3 Additional correction are neceary to acce the random contribution of the error. Moreover, diperion tatitic (MeanAD and RMSD) are not alway provided in the benchmark literature, preventing the etimation of prediction uncertainty from exiting benchmark. An exception i for caled harmonic frequencie, where the RMSD of caled frequencie i generally available. 9,30 An additional iue with the MAE and RMSE etimator i that they do aggregate reference data uncertainty with the error due to the method. If the reference data uncertainty i not negligible before method inadequacy, thee etimator, a they do not average out the random reference data error, underetimate method performance. The applicability of MAE and RMSE require therefore the ue of high-accuracy reference data,,3 which might be a evere retriction for ome propertie. We how in the following how to circumvent thee difficultie in a practical way and etimate prediction uncertainty by tatitical modeling of the error..4. Some Iue in the Ue of Benchmark Data Set. Beide the requirement for high-quality data in the reference et, one can be confronted with more challenging iue: the experimental data do not necearily reflect the bet, exact reference to be ued. There are everal reaon for thi:. The calculated quantitie do not necearily correpond to the experimental data. Thi can be the cae for the fundamental band gap, ee, e.g., Civalleri et al. (6) (7). The theoretical method i not necearily uppoed to provide the quantity analyzed (Kohn Sham orbital energie, even exact, are not uppoed to provide fundamental band gap 3 34 ). 3. The experimental data are ubject to factor that are not properly taken into account (e.g., temperature, in particular for bulk modulu). 4. The incluion of the ytem into the benchmark data et i conditioned to data availability, which introduce a bia in the repreentativity of the data et. 3. PREDICTIO UCERTAITY ESTIMATIO To etimate a prediction uncertainty, a four-tep procedure i ued:. build and validate a tatitical model of the error from the benchmark et (calibration model),. evaluate the uncertaintie of the parameter involved in thi model, 3. propagate the uncertaintie of the parameter in the calibration model to the prediction model, and 4. validate the prediction model. Validation in tep and 4 i neceary to enure that calculated value and reference data agree within the error bar defined by the calibration or prediction model. One generally face the cae where reference data uncertainty alone cannot explain the error amplitude, and correction to the calibration model have to be done, either by updating it determinitic part (repreenting the ytematic error) or it tochatic part (repreenting the random error). 3.. Ditribution of Error. Deigning a tatitical model require u to examine the data and their ditribution. We illutrate the proce on the cae of the B3LYP DFA for lattice contant (LC), extracted from the full application et decribed in ection 4.. Thi example i well repreentative of the other cae conidered in the preent article Sytematic and Random Error. Figure a diplay a catter plot of the reference data v the calculated data. One oberve that the point are grouped along a line that i not the identity line. Thi i evidence of the preence of ytematic error and of a trend in the ytematic error, which in thi cae increae a a function of the calculated property value. Plotting the error (et E LC,B3LYP ) againt the calculated value (Figure b) reveal more clearly the trend: the error increae more or le linearly with the value of the lattice contant and have a non-null mean value. The orange line repreent the leat-quare linear fit of the error. The MSE and MSE ± RMSD value of thee data are repreented by horizontal full and dahed line, repectively. The trend line repreent a ytematic effect (a determinitic contribution) in the error, which ha to be corrected before we can etimate the random contribution on which the uncertainty etimation i baed. 3 The corrected error, obtained by ubtraction of the leat-quare regreion line are hown in Figure b (triangle) and preent a zero-centered, more or le ymmetrical ditribution. One can ee that they do not preent any obviou trend. Moreover, their RMSD i maller than for the uncorrected error. Hitogram of the error before and after linear correction can be compared in Figure. It i intereting to contrat the width of the corrected ditribution (about 0.0 Å) with the typical meaurement error on lattice contant, which are conidered to be an order of magnitude maller (about 0.00 Å, 590 DOI: 0.0/jp509980w

4 Downloaded by UIV PIERRE ET MARIE CURIE on September 7, 05 Figure. Structure of the error for lattice contant calculated by the B3LYP method: (a) Reference v calculated data (bullet), the leatquare regreion line through the point i the orange/olid line, and the black/dahed line repreent the identity line. (b) Error v calculated value, before and after linear correction. A linear trend (orange line) can be aigned to a ytematic (predictable) component of the initial error. The horizontal line in (b) repreent the MSE (full line) and MSE ± RMSD (dahed line) of the correponding data et. Figure. Ditribution of error obtained uing the B3LYP functional to predict lattice contant, before and after linear correction of ytematic error. The hitogram are produced by ditributing the data into bin of 0.0 Å. ee ection 4.). One ha thu to face the fact that, even after a linear correction, the B3LYP DFA cannot predict the reference data within their uncertainty range From Determinitic Calculation to Random Error. Conidering the very mall uncertainty on lattice contant, the error in the E LC,B3LY P et can be mainly attributed to the method inability to reproduce reference data and can be decompoed in Figure into predictable/ytematic and unpredictable/random contribution. In the following, we will refer to the random part of method inadequacy a method inadequacy error, the ytematic part being addreed through correction. The method inadequacy error ha a random-like trace a a function of lattice contant value (Figure b), depite the fact that the model chemitry (i.e., method and bai et) calculation are determinitic. 3 Thi i not truly a random proce, in the ene that repeated calculation with a model chemitry for the ame ytem would provide the ame value, but it variation with the lattice contant value i practically unpredictable without doing the calculation. Moreover, for a given bai et, thi random contribution i irreducible without changing the DFA or plitting the reference data et (if it appear heterogeneou). The method inadequacy error repreent therefore our lack of knowledge on the prediction of propertie of new ytem, due to the ue of an approximate method, and to ome extent, of a limited reference data et. Thi pattern can be exploited to define and etimate prediction uncertainty by modeling method inadequacy error by a random variable, a detailed in the next ection. 3.. Calibration/Prediction Statitical Modeling. In thi ection we preent the implementation of the VM framework by an a poteriori calibration model, enabling u () to correct for the ytematic error of a method, () to evaluate the method inadequacy uncertainty, and (3) to etimate the prediction uncertainty of the calibrated method Calibration. Let u tart with the implet tatitical model linking the calculated value (c m, ) and the uncertain reference value (o ± u ) o = cm, + ϵ ( =, ) (9) where the ϵ are independent random variable of mean 0 and known, finite, tandard deviation u. Thi model i a generalization of eq : it ue random variable ϵ to decribe tochatic procee from which one aume that the actual error e m, are realization. In mot cae of interet in the preent tudy and many other, thi model i invalid, in the ene that the value calculated by a given DFA are not compatible with the reference value within their uncertainty range. 35 To get a valid calibration model, one ha to account for the tructure of the error et. A ytematic trend oberved in the error et of the benchmark data can be corrected by a tranformation of the calculated value c m,, providing a new (calibration) model o = f ( c ; ϑ ) + ϵ m m, m (0) where ϑ m repreent the et of parameter defining f m. The functional form and parameter value of f m are methoddependent. It i important for the prediction ability of the model to chooe a functional form that doe not overfit the data. One can alway find a high-degree polynomial fitting 59 DOI: 0.0/jp509980w

5 Downloaded by UIV PIERRE ET MARIE CURIE on September 7, 05 exactly all point in the error et. However, thi kind of correction ha no generalizability; i.e., it perform poorly at the prediction tage. Low-order polynomial or function with few parameter (compared to ) hould be preferred. After optimization of the parameter ϑ m (ϑ m repreent the et of optimal parameter), the validity of the model depend on the comparion between the reidual error r = o f ( c ; ϑ m, ) m m m () and the reference data uncertaintie u. In the leat-quare optimization framework, one compare the χ value χ = =, r m, u () to the number of degree of freedom n df = ϑ, where ϑ i the number of free parameter in f m. 36,37 If χ n df, the corrected model can be conidered a valid, and the prediction uncertainty will be limited to the parametric uncertainty of the correction function, a defined below (eq 6). In mot practical cae, however, the reference data uncertaintie are mall compared to the reidual error, which invalidate thi calibration model (χ n df ). If the reidual error till preent dicernible trend, the correction function f m ha to be updated. If the reidual error r m, preent a random-like pattern, for which no further determinitic correction appear uitable, one introduce a new tochatic term δ m, to decribe the diperion of the error in exce of the reference data uncertainty, which we attribute to method inadequacy o = f ( c ; ϑ ) + ϵ + δ m m, m m (3) where δ m i a random, unpredictable, variable of mean 0 (ytematic error are corrected by f m ) and unknown, finite, tandard deviation d m. The value of d m i etimated to enure the tatitical validity of eq 3. Practically, d m can be choen a the difference between the variance of the reidual error r m, and the mean variance of the reference data (Appendix). With thi choice, the corrected calculated value and the reference data are compatible within the combination of their repective error bar Prediction. For the etimation of a new value of a property knowing a calculated value c* (i.e., for a ytem not in the benchmark et), the prediction model and prediction variance are 3 p ( c* ) = f ( c* ; ϑ ) + δ m m m m (4) u * = * ϑ p ( c ) uf ( c ; m) + dm m m (5) where δ m 0 ha been left in the prediction model a a reminder of the occurrence of d m in the prediction variance. The term u f m (c*;ϑ m) repreent the parametric uncertainty on the value of the function f m at c*. Thi contribution reult from the uncertainty in the optimal parameter et due to the tochatic term in eq 3. For functional form of f m linear in ϑ m or howing weak nonlinearity on the variation domain of ϑ m, it can be etimated by combination of variance 3,38 T u * ϑ f ( c ; m) = J Σ ϑ J m m (6) where J i a vector of enitivity coefficient evaluated at ϑ m = ϑ m, ϑ = f ( c * ; ) m m Ji ϑ mi, ϑ m (7) and Σ ϑm i the variance covariance matrix of the parameter. For highly nonlinear function, Monte Carlo uncertainty propagation can be ued Linear Cae. A linear tranformation function, f m (x;a m,b m )=a m + b m (x), will be ued in thi tudy, leading to the calibration model o = am + bmcm, + ϵ + δm (8) Weighted leat-quare regreion can be ued to etimate the optimal value of all parameter, a m, b m, d m, and the uncertaintie and covariance of the line parameter u(a m ), u(b m ), and u(a m,b m ). The detail are provided in the Appendix. The prediction model and prediction variance are p ( c* ) = a + b c* m m m (9) u ( c* ) = u ( c* ; a, b ) + d p f m m m m m (0) f m m m m m m m u ( c* ; a, b ) = u ( a ) + c* u ( b ) + c* u( a, b ) () The prediction uncertainty u pm depend on the calculated value c*. However, if the benchmark et i large enough and if c* lie within the range covered by the benchmark et (no extrapolation), the uncertainty on the calibration model can become negligible before d m, 9 and eq 0 reduce to up ( c* ) dm m () Thi convenient approximation will be teted in the next ection. We init on the fact that the prediction uncertainty u pm ha two contribution: the method inadequacy error d m and the correction model uncertainty u f m. An example of the relative contribution of thee quantitie i hown in Figure 3, where the major contribution of d m can be appreciated. In term of Figure 3. Prediction uncertainty uing the B3LYP functional for lattice parameter: the dahed line repreent the contribution of the calibration model uncertainty, ±u f ; the dotted line repreent the method inadequacy error contribution, ±d m ; the full line are the total prediction uncertainty, ±u p ; the blue bullet are the reidual error for the reference et ued for calibration, and the red triangle are the reidual error for the data in the validation et. 59 DOI: 0.0/jp509980w

6 Table. Lit of the DFT Method Aeed in the Preent Work a Downloaded by UIV PIERRE ET MARIE CURIE on September 7, 05 method name exchange c SR c MR c LR ω SR ω LR correlation ref HF HF HF LDA SVW S VW 43, 44 GGA PBE PBE PBE 45 PBEol PBEol PBEol 46 mgga M06-L M06-L M06-L 47 GH-GGA B3LYP B LYP 43, 44, B97 B B97 5, 5 PBE0 PBE PBE 45, 53, 54 PBEol0 PBEol PBEol 46 GH-mGGA M06 M M06 55 SC-RSH HSE06 PBE PBE 45, 56 HSEol PBEol PBEol 46, 57 MC-RSH HISS PBE PBE 45, 58, 59 LC-RSH LC-ωPBE PBE PBE 45, 60 LC-ωPBEol PBEol PBEol 46, 60 RSHXLDA S VW 44, 6 64 ωb97 B B97 5, 65 ωb97-x B B97 5, 65 a Parameter are alo reported for global (GH) and range-eparated hybrid (RSH) exchange functional. variance, the contribution of u f m to u pm i about 0% at the extremitie of the plotted lattice contant range, which correpond to a value of u pm larger than d m by about 0%. A hown by Pernot and Cailliez, 8,9 ignoring method inadequacy error lead to unreliable prediction uncertainty etimation. 4. APPLICATIO 4.. Benchmark and Validation Data. We analyze the lattice contant, bulk modulu, and band gap for a et of 8 crytal with cubic ymmetry (emiconductor and inulator) and compare 8 different denity functional approximation (local, emilocal, and hybrid functional). All calculation have been carried out with the CRYSTAL4 code. 40,4 All-electron and effective-core potential calculation have been done by uing atom-centerd Gauian-type bai et. The latter have been taken from ref 4, except for alkali halide and SrTiO 3 for which a triple-ζ quality bai et ha been employed. The full et of data i reported in the Supporting Information Choice of Reference Data. Reference data were collected for the following crytal (Strukturbericht deignation in parenthee; taken from the Crytal Lattice Structure web page: emiconductor, alo preent in the SC40 data et, 4 namely, C(A4), Si(A4), Ge(A4), SiC(B3), B(B3), BP(B3), BA(B3), AlP(B3), AlA(B3), AlSb(B3), Ga(B3), GaP(B3), GaA(B3), GaSb((B3), InP(B3), InA(B3), InSb(B3), ZnS- (B3), ZnSe(B3), ZnTe(B3), CdTe(B3), and MgS(B); 4 alkali halide LiF(B), LiCl(B), af(b), and acl(b); and two oxide MgO(B) and SrTiO 3 (E ). The reference data et include () experimental lattice contant value corrected for the zero-point anharmonic expanion, a reported in ref 66, () experimental bulk modulu value, taken from ref 57 and 67 69, and (3) lowtemperature (below 77 K) experimental (fundamental) band gap value. 4,68,70,7 For bulk modulu, we referred to low-temperature data, 57,67,68 if available, and, when poible, the zero-point anharmonic expanion correction ha been included from ref 57. The band gap conidered cover order of magnitude, between 0. and ev Validation Data. A et of nine ytem ha been et aide for validation purpoe. Thee are ytem for which we did not find bulk modulu reference data: Al(B3), CdS(B3), CdSe(B3), MgSe(B), MgTe(B), BaS(B), BaSe(B), BaTe- (B), and LiH(B) Reference Data Uncertaintie. Concerning the error bar for lattice contant, the uncertainty from X-ray diffraction experiment depend on the ample (i.e., powder or ingle crytal) and on the intrument/detector. It i claimed that the uncertainty can reach Å or even maller. 7,73 However, due to the procedure adopted to obtain the reference ZPAEcorrected data, which mixe experimental lattice contant and computed ZPAE correction, we aume that an uncertainty of 0.00 Å i more repreentative. For band gap, mot of the reference data correpond to lowtemperature (LT) value, but ome of them have been meaured at room temperature (RT). When LT and RT data are compared, a reported by Lucero et al., 74 the former are ytematically larger than the latter by 0.0 ev on average (3 ytem), with a maximum difference of 0.30 ev. However, from ref 70 and reference therein, the error bar for the band gap range from 0.00 ev, or le, up to 0.0 ev. Thi depend on the experimental technique adopted to meaure it (e.g., diffue reflectance, photoluminecence pectrocopy,...) but i more or le independent from the temperature. Therefore, we conider 0.0 ev a the uncertainty for reference experimental band gap. The experimental uncertaintie for bulk modulu range from a few tenth of GPa up to 4 5 GPa. Again, it depend on the meaurement approach: either from equation of tate data by mean of X-ray diffraction meaurement, uually for a given hydrotatic path, or through the knowledge of elatic contant. For the latter, variou technique can be employed (e.g., Brillouin cattering, ultraonic reonance,...) and meaurement can be carried out at different temperature, thu allowing extrapolation at the tatic limit. Here, we refer to an average etimated experimental uncertainty of GPa. 593 DOI: 0.0/jp509980w

7 Downloaded by UIV PIERRE ET MARIE CURIE on September 7, 05 One hould conider thee global etimation of reference data uncertaintie a optimitic. They often reult from the imple trancription of experimental repeatability tatitic, 3,7 without conidering additional uncertaintie reulting from ample preparation, material impuritie, uncertainty in variou correction, apparatu calibration, etc. A more peimitic cenario will be explored in ection Choice of Denity Functional Approximation. The DFA ued in the preent work can be claified into the following group: local and emilocal denity functional (i.e., LDA, GGA, and mgga), linear global hybrid (GH, where the denity functional exchange i mixed up linearly with the Hartree Fock exchange), and range-eparated hybrid (RSH). In the latter cla of functional, the amount of HF exchange included depend on the ditance between electron. They are obtained from the eparation of the Coulomb operator in different range (three range in the current implementation) by mean of the error function a erfc( ωsrr) erfc( ωsrr) erf( ωlrr) erf( ωlrr) = + + r r r r SR (3) where ω i the length cale of eparation. Range-eparated hybrid can be ubdivided in long-range-corrected (LC-RSH), middle-range hybrid (MC-RSH), and hort-range-corrected (SC-RSH) functional, alo known a creened Coulomb. In thee approximation, the long-, middle-, and hort-range part of the exchange, repectively, i decribed by Hartree Fock. The general form of a range-eparated hybrid i xc RSH xc DFA SR HF x,sr HF DFA LR x,lr x,lr E = E + c ( E E ) + c ( E E ) + c ( E E ) MR DFA x,sr MR HF x,mr LR DFA x,mr (4) According to the value of c SR, c MR, c LR, ω SR, and ω LR, hort-, middle-, and long-range-corrected RSH functional can be defined. When ω = 0 and c SR = c MR = c LR, range-eparated hybrid reduce to linear global hybrid. Mixture of RSH and linear hybrid functional have alo been conidered, a for the ωb97-x. The lit of DFT method conidered in the preent work i ummarized in Table, including HF. ote that in contrat to the generally ued B3LYP, we ued the variant implemented in the CRYSTAL code, where the local functional i fitted to the accurate correlation energy of the uniform electron ga, i.e., VW5, 44 and not to the VW3 random phae approximation. The tatitical calculation preented in thi paper have been done in the R environment, 75 either with core function, additional package a mentioned in the text, or pecifically developed routine. 4.. Benchmark Statitic Error and Deviation Value. The etimation of MAE, RMSE, MSE, and RMSD for all method and propertie are reported in Table 4. The comparion of MSE and RMSE value tell u that mot method preent ignificant ytematic error ( MSE RMSE). ote that becaue of the preence of trend in the ytematic error, a mall abolute value of the MSE i not an indicator of the abence of ytematic error. Table. Sample Statitic for the Method on the Benchmark Set for Band Gap (ev): Mean Abolute Error (MAE), Root-Mean-Square Error (RMSE), Mean Signed Error (MSE), and Root-Mean-Square Deviation (RMSD) a MAE RMSE MSE RMSD HF LDA PBE PBEol B B3LYP PBE PBEol HSE HSEol HISS RSHXLDA ωb ωb97x LC-ωPBE LC-ωPBEol M06-L M a The minimal abolute value in each column are in bold type. Table 3. Same a Table for Bulk Modulu (GPa) MAE RMSE MSE RMSD HF LDA PBE PBEol B B3LYP PBE PBEol HSE HSEol HISS RSHXLDA ωb ωb97x LC-ωPBE LC-ωPBEol M06-L M Reult in Table 4 agree with previou benchmark tudie. 4,57,66,67,74,76 Hybrid method are by far uperior to emilocal functional in the prediction of the band gap of olid a expected becaue of the incluion of ome HF exchange within the generalized Kohn Sham formalim. In thi repect, reult for global, hort and middle range-eparated hybrid are not far from each other. Surpriingly, long-range-corrected hybrid tend to ytematically overetimate band gap although HF exchange i included at long-range to recover the correct decay of the exchange potential. For lattice parameter and bulk moduli, GGA and related hybrid functional for olid (e.g., PBEol family) give improved reult with repect to common functional devied for molecule (e.g., PBE family). Interetingly, we confirm reult by Lucero et al. 74 for the HISS functional, which give overall good reult when compared to 594 DOI: 0.0/jp509980w

8 Downloaded by UIV PIERRE ET MARIE CURIE on September 7, 05 Table 4. Same a Table for Lattice Contant (Å) MAE RMSE MSE RMSD HF LDA PBE PBEol B B3LYP PBE PBEol HSE HSEol HISS RSHXLDA ωb ωb97x LC-ωPBE LC-ωPBEol M06-L M reult for other hybrid. Highly parametrized mgga functional uch a M06-L and M06 do not ignificantly improve reult with repect to other examined functional. A expected, incluion of HF exchange in the M06 hybrid functional lead to a better prediction of band gap than the emilocal counterpart. Overall, computed MAE and MSE for LDA, PBE, PBEol, M06-L, HSE06, HSEol, and HISS on a imilar et of olid agree with the one reported in other work. 4,57,66,67, DFA Ranking. One could attempt a ranking baed on MSE and RMSD tatitic. At thi level, the bet performing method are characterized by two criteria: () the error are nearly centered on zero; and () they have a mall diperion. With uch criteria, there i no important ditinction for band gap between HSEol and other hybrid, like PBE0, PBEol0, and even HSE06, ωb97, M06, and B3LYP. The ituation i le clear for bulk moduli, with a light advantage to PBE0 and HSE06, and, for lattice parameter, PBEol0 and HSEol are the bet contender. Whatever the performance tatitic, none of the method eem optimal for all the propertie Statitical Modeling. Figure 4 6 how the probability denitie of the E m error et for the three propertie, before and after linear calibration (eq 8). The error ditribution for the raw data (before calibration) confirm or reveal a few feature relevant for the following development: mot method provide biaed etimate for ome or all propertie, the hape of the ditribution varie coniderably between method and propertie, ome ditribution are trongly aymmetric wherea other are bimodal, and ome point eem to lie far of the main batch (outlier) and many ditribution preent a long tail Calibration. To determine the polynomial degree of the trend in ytematic error, we ued Bayeian Model Selection (BMS) 77 for all error et. BMS calculate the poterior probability ditribution over a et of model, combining a parimony criterion (Occam razor) with a goodne-of-fit criterion. It avoid to overfit the data with overly complex model. We ued the algorithm decribed in Mana et al., 78 over polynomial degree from to 3. BMS how that the linear Figure 4. Probability denity of the E p,m error et for band gap: for each method, the upper denity repreent the error for the raw calculation data and the lower denity the reidual error for the calibrated data. Figure 5. Same a Figure 4 for bulk modulu. model i the mot probable, except for RSHXLDA, ωb97 and ωb97-x in the prediction of band gap, where econd order polynomial have lightly higher poterior probability. A the linear model i not fully rejected for thee cae, we conidered a linear correction for all cae. A can be een in Figure 4 6, linear correction, beide eliminating prediction bia, contribute often to produce more ymmetrical ditribution (e.g., HF for band gap), albeit without alway reulting in normal ditribution (e.g., ωb97 for band gap). In many cae, the diperion of the error i notably reduced (e.g., B3LYP for lattice contant), along with the ditribution tail (e.g., PBE for band gap). For ome method, one oberve a mere hift of the ditribution due to bia correction, a for ωb97 for band gap; thi correpond to 595 DOI: 0.0/jp509980w

9 Downloaded by UIV PIERRE ET MARIE CURIE on September 7, 05 Figure 6. Same a Figure 4 for lattice contant. method for which the point were originally ditributed along a line nearly parallel to the identity line. Depite the linear correction, there might remain a variation of the diperion of the error a a function of the property value. In Figure b, the corrected error diplay no ignificant trend but get larger in abolute value for ytem with increaing lattice parameter. Capturing thi behavior in the method inadequacy model might contribute to improve the quality of prediction uncertainty. Thi can, however, be conidered a a econd order effect and it correction ha not been attempted in the preent tudy, where, conidering the mall ize of our calibration ample, we aimed at teting the implet correction etup Internal Validation of the Calibration Model. To validate the linear correction, we calculated the q tatitic through the Leave-One-Out cro-validation method 79 q ( ) ϑ = ( o f ( c ; )) = m m m ( o o ), = (5) where one perform linear regreion for et of data without one of the point (regreion parameter noted ϑ m ( ) ), and compare the um-of-quare of the prediction error of thee regreion for the left-out data (numerator) to the umof-quare of the deviation of the ample point from their mean (denominator). The q tatitic goe from 0 to, the larger the better. The q tatitic for all method/propertie pair are reported in Figure 7. Value of q above 0.95 can be conidered a excellent. There i therefore no evidence at thi tage againt the choice of a linear correction function Prediction Uncertainty Analyi. We calculated the contribution of the calibration model uncertainty u f to the total variance in eq 0. Thi uncertainty i typically larger at the extreme of the calibration range, and minimal around the mean value of the calibration data (Figure 3). It relative contribution for any calculated value x of a property i (ignoring the method index) Figure 7. Validation of linear calibration by leave-one-out (q ) tatitic. uf r() x = u p () x () x (6) The maximal and minimal value of r(x) over the calibration range have been found to be very weakly dependent on the DFA for a given property, but trongly dependent on the property. The DFA-averaged maximal value of r(x) i about 5% for band gap, 30% for bulk moduli and 5% for lattice contant, and it minimal value i about 5% for all propertie. The approximation of the prediction uncertainty by method inadequacy uncertainty d m alone (eq ) i therefore too optimitic, notably for band gap and bulk moduli, in the ene that it underetimate uncertainty, notably at the extremitie of the calibration range. In term of uncertainty, uing d m repreent a maximal underetimation of prediction uncertainty of about 8% for lattice contant, 5% for band gap and 0% for bulk moduli. Thee value are to be compared with the relative uncertainty on the tandard deviation of a normalditributed ample of ize, Δu/u /(( )) /, which for = 8 i about 5%. Therefore, except maybe for lattice contant, one cannot conider thee difference a negligible. One ha alo to keep in mind that they are obtained for value of the reference data uncertainty which are plauibly underetimated. Increaing the reference data uncertaintie can only reduce the method dicrepancy uncertainty d m (eq 3), and increae the relative contribution of the correction model uncertainty, u f, to the prediction uncertainty, u p. The non-negligible contribution of the correction model to the total uncertainty i mainly due to the mall ize of the benchmark et (8 value). For intance,the calibration uncertainty contribution ha been hown by Pernot and Cailliez 8 to be negligible for calibration et with 500 and 500 harmonic vibrational frequencie, and around % for et of 39 zero point vibrational energie. A thorough way to improve the contant uncertainty approximation (eq ) i therefore to complement the benchmark et by new reference data, which i not alway poible at hort-term. Another olution i to provide uer with all the parameter required to ue eq 0 (u(a), u(b), and u(a,b); ee Supporting Information). Although accurate, thi olution doe not enable a quick aement of a et of DFA. To provide a reliable uncertainty etimate while maintaining implicity, we ue the mean value of the prediction uncertainty 596 DOI: 0.0/jp509980w

10 Downloaded by UIV PIERRE ET MARIE CURIE on September 7, 05 up = n n i= u p ( x) i (7) calculated on a regular grid of value of the property x covering it calibration range. We reported in Table 5 7 the linear correction factor to apply to the different DFA of thi tudy and their approximate Table 5. Linear Correction Factor a and b and Approximate Prediction Uncertaintie d and u p for All Method on Band Gap (ev) a b a d u p HF LDA PBE PBEol B B3LYP PBE PBEol HSE HSEol HISS RSHXLDA ωb ωb97-x LC-ωPBE LC-ωPBEol M06-L M a The lope parameter b i dimenionle. Table 6. Same a Table 5 for Bulk Modulu (GPa) a b a d u p HF LDA PBE PBEol B B3LYP PBE PBEol HSE HSEol HISS RSHXLDA ωb ωb97-x LC-ωPBE LC-ωPBEol M06-L M a The lope parameter b i dimenionle. prediction uncertaintie d m (eq ) and u p (eq 7; n = 000). Uncertaintie are conventionally reported with two ignificant digit, 3 but conidering the mall ample ize and the approximation involved, one hould not attach too much credit to the lat digit. Comparion with Table 4 how that correction of the trend in ytematic error enable a trong reduction of the Table 7. Same a Table 5 for Lattice Contant (Å) a b a d u p HF LDA PBE PBEol B B3LYP PBE PBEol HSE HSEol HISS RSHXLDA ωb ωb97-x LC-ωPBE LC-ωPBEol M06-L M a The lope parameter b i dimenionle. higher value of the tandard deviation of reidual error: the method have prediction uncertaintie between 0.4 and 0.8 ev for band gap (RMSD between 0.4 and. ev), 7. and 0 GPa for bulk moduli (RMSD between 7.4 and 7 GPa), 0.03 and Å for lattice contant (RMSD between 0.05 and Å) DFA Ranking. The bet preforming DFA do not ee their diperion notably improved, but Table 5 7 offer a new landcape for method election: it can be ued to elect method according to uncertainty requirement. For intance, auming that one want to be able to etimate the lattice contant for a new compound with an uncertainty maller than 0.0 Å, the lat column of Table 7 tell u that one can chooe among five (corrected) method: PBEol, PBE0, PBEol0, HSE06, and HSEol. If our requirement i an uncertainty below 0.05 Å, baically all method hould be able to comply Prediction Uncertainty Etimation. Table 5 7 can alo be ued to etimate the uncertainty of a calculated value and to elaborate an uncertainty budget, for intance in the comparion of the propertie of variou compound for creening tudie. A an example of uncertainty evaluation, let u aume that we calculated a band gap of 0 ev for a new compound uing the PBE method. The prediction of the true value for thi property i calculated from Table 5 a = 4.35 ev (eq 0), with an uncertainty of 0.60 ev (the value of the uncertainty uing the exact expreion eq 0 i only lightly larger: 0.67 ev; the full et of coefficient for eq 0 are provided a Supporting Information) External Validation of Prediction Model. To validate the prediction uncertainty model derived in the previou ection, we ue a validation et of nine ytem not included in the calibration et. For thi ytem, we have data only for band gap and lattice contant. o external validation i done on bulk modulu. The principle of the validation i, for each DFA and property to. correct the calculated value by the appropriate linear factor (Table 5 7 and eq 9),. calculate the reidual error with the validation reference value (eq ), and 597 DOI: 0.0/jp509980w

11 Downloaded by UIV PIERRE ET MARIE CURIE on September 7, calculate the number of error falling within a prediction confidence interval extended to account for reference data uncertainty (eq 40). Here, we count the error within a σ confidence interval (about 95% coverage for a normal ditribution). Two difficultie are to be expected: () with mall validation ample, the tatitic might be far from their aymptotic value (i.e., one hould not expect that 95% of the point fall within a 95% confidence interval), and () the definition of confidence interval require to know the hape of the error ditribution. Conidering the latter point, the part of the prediction uncertainty due to the correction function (u f ) can be aumed to have a normal ditribution: the optimal regreion coefficient being the combination of many uncertain contribution (o ) with finite variance (eq 8 and 9), the Central Limit Theorem enure that they are normally ditributed. Thi i not the cae for the method inadequacy term d m, repreenting eentially the reidual error ditribution, which are often non normal (Figure 4 6). It i therefore to be expected that confidence interval built on a normality hypothei will not be fully conitent with the validation data. The reult of the validation tet are reported in Table 8. For band gap, 9 point fall within the predicted interval in cae, Table 8. umber of Point of the Validation Set within the Predicted σ Error Range after Linear Correction a band gap lattice contant HF 9 7 LDA 8 8 PBE 9 7 PBEol 8 6 B B3LYP 9 7 PBE0 9 7 PBEol0 8 8 HSE HSEol 7 7 HISS 7 7 RSHXLDA 9 5 ωb ωb97-x 9 5 LC-ωPBE 9 6 LC-ωPBEol 9 5 M06-L 8 9 M a The validation et contain 9 point. 8 point in 4 cae, and 7 in cae. We can conider that the predicted uncertaintie are atifying, even if they are probably lightly overetimated. Thi i on the afe ide: uer have a very mall rik to overetimate the accuracy of their calculation. For lattice contant, there are 7 cae where only 6 point or le are correctly predicted. For two DFA (RSHXLDA and ωb97), the problem perit if one ue a 3σ confidence interval. Referring to the reult on Bayeian Model Selection (ection 4.3.), thi could ugget that the linear correction model i inufficient. However, invetigation of the error ditribution for thee cae how that there i a mall overlap between the calibration and validation error et. The linear correction doe not contribute to hift thoe point toward the center of the ditribution (Figure 8). For thee DFA, it eem that the calibration et i not fully repreentative of the pecie in the validation et. Figure 8. Error reditribution by linear correction, for calibration (blue dot) and validation (red triangle) et of lattice contant etimated by the RSHXLDA method. The gray area repreent the σ interval ued for validation. Globally, we checked that, depite the caveat of mall ample ize and non-normal ditribution, the prediction model provide reaonable confidence interval, except for a few DFA (RSHXLDA and ωb97), for which the lattice contant calibration et i poorly repreentative of the data in the validation et. Ever after linear correction, thee DFA hould not be recommended to predict lattice contant Senitivity to Reference Data Uncertainty. Up to thi point, all evaluation have been done with reference data uncertainty value u which are plauibly underetimated (ection 4..). To ae the impact of u on the prediction uncertainty u p, we reevaluate u p (eq 7) with value of u more akin to account for variou perturbation uch a temperature effect, correction uncertainty... We conider indeed a wort cae cenario, i.e., the larget value of u that do not compromie the leat-quare regreion validity. A overetimated value of u would produce unlikely mall value of χ (eq ), we requet χ to be above the 5% quantile of the tandard chi-quared ditribution with degree of freedom (χ min 5.4). The correponding value are u = 0.3 ev for band gap, 7 GPa for bulk moduli, and 0.05 Å for lattice contant. 80 The new value of u p are hown in Table 9 alongide thoe iued from Table 5 7. For all propertie, the effect i more viible for method which had a prediction uncertainty cloe or below the wort cae value of u. One reached a point where ome method (mot of them for bulk modulu, indicating that the wort cae value of u = 7 GPa might be too large) have their prediction uncertainty maller than reference data uncertainty. In thi cenario, uch method, after a poteriori correction, could be elected to replace advantageouly cotly and/or difficult meaurement, with the ame level of confidence. Globally, the prediction uncertainty preent a low enitivity to the reference data uncertainty: for band gap and lattice contant, an increae of more than order of magnitude in u reult at mot in a reduction by a factor (band gap) or 4 (lattice contant) of u p. Thee factor are, however, till much larger than the 5% of relative uncertainty on a tandard deviation one can expect for ample of thi ize (ee ection 4.4). Thi analyi how that for a reliable etimation of method prediction uncertainty one need an adequate evaluation of 598 DOI: 0.0/jp509980w

12 Downloaded by UIV PIERRE ET MARIE CURIE on September 7, 05 Table 9. Effect of Reference Data Uncertainty (u )on Prediction Uncertainty u p a band gap (ev) bulk modulu (GPa) lattice contant (Å) u HF LDA PBE PBEol B B3LYP PBE PBEol HSE HSEol HISS RSHXLDA ωb ωb97x LC-ωPBE LC-ωPBEol M06-L M a The value in boldface are the one where u p u. reference data uncertainty, which i probably the mot enitive iue in the implementation of the VM framework Looking Back at the Reference Data. Figure 9 how the ditribution of error per ytem, E p, ={E p,m, ;m DFA}, before and after the linear correction applied in the previou ection. At the ytem level, the reduction of diperion i remarquable, confirming that calibrated DFA produce more conitent reult. After correction, ome ytem preent a Figure 0. Effect of linear calibration on error ditribution per ytem for bulk modulu. Above/blue: before. Below/red: after. Figure. Effect of linear calibration on error ditribution per ytem for lattice contant. Above/blue: before. Below/red: after. Figure 9. Effect of linear calibration on error ditribution per ytem for band gap. Above/blue: before. Below/red: after. ignificant bia; i.e., their error ditribution doe not overlap the zero axi. Outtanding example are the bulk moduli of Ga, MgS, and SrTiO 3, or the band gap for LiF, af and acl. Thee ytem are contributing to the outlier in the error ditribution per method in Figure 4 6, and the fact that all method are unable to predict thee ytem propertie deerve further attention. For the bulk modulu, the reaon i probably the experimental temperature. In fact, data for Ga, MgS, and 599 DOI: 0.0/jp509980w