Evaluation of Validation Metrics. O. Polach Final Meeting Frankfurt am Main, September 27, 2013

Transcription

1 Evaluaton of Valdaton Metrcs O. Polach Fnal Meetng Frankfurt am Man, September 7, 013

2 Contents What s Valdaton Metrcs? Valdaton Metrcs evaluated n DynoTRAIN WP5 Drawbacks of Valdaton Metrcs Conclusons

3 Introducton Valdaton of tme hstores Valdaton usng comparson of tme hstores s tradtonally carred out by subjectve engneerng judgment Recently, there has been an nterest n quantfyng such comparsons wth the ntent of mnmsng the subjectvty, whle stll mantanng a correlaton wth experts opnon The computable measures that quantfy these comparsons are usually referred to as Valdaton Metrcs Durng the IAVSD Symposum n Manchester 011, Ncholas Wlson mentoned an FRA research project about ths topc, carred out by Transportaton Technology Center n Pueblo, USA Robert Fres from Transportaton Technology Center kndly provded us a lst of references related to ths topc Dynotran WP5 partners decded to test the Valdaton Metrcs wth the am to replace the subjectve valdaton assessment by quanttatve crtera 3

4 Overvew of Valdaton Metrcs Hstory Perhaps the earlest reference to a metrc for comparng tme hstores of wave forms s the 1984 publcaton of Geers [1] Integrals of the two wave forms to be compared are computed and used to evaluate the dfference n the magntude and phase of the wave forms The magntude as well as phase form of error factors was adapted by Russell [] The new phase form by Russell was combned wth the 1984 Geers metrc by Sprague and Geers n 004 [3] Several varatons and comparsons of dfferent metrc forms has been publshed References: [1] Geers, T.L.: An objectve error measure for the comparson of calculated and measured transent response hstores. The Shock and Vbraton Bulletn 54 (1984), pp [] Russell, D. M.: Error measures for comparng transent data: Part I: Development of a comprehensve error measure, Part II: Error measures case study. Proceedngs of the 68th Shock and Vbraton Symposum, Hunt Valley, Maryland, 1997 [3] Sprague, M.A., Geers, T.L.: A Spectral-element method for modellng cavtatonn transent fludstructure nteracton. Internatonal Journal for Numercal Methods n Engneerng 60 (004), pp

5 Contents What s valdaton metrcs? Valdaton Metrcs evaluated n DynoTRAIN WP5 Drawbacks of Valdaton Metrcs Conclusons 5

6 Defnton of Valdaton Metrcs s Assessment of comparson of tme hstores s carred out applyng error factors: Magntude error factor Assessment of ampltude devaton Phase error factor Assessment of phase devaton and frequency content Comprehensve error factor Combnaton of Magntude and Phase factor to one assessment value The error factors are defned by ntegrals The followng sldes dsplay the formulas for numercal calculaton, consderng dentcal tme step (samplng frequency) of both sgnals (measurement and calculaton) 6

7 Magntude error factor by Russell Magntude s The error factor by Russell can be nterpreted as the dfference n magntude between two functons n term of orders of magntude wth m measured tme hstory value c ( M ) log ( M ) M R = sgn calculated tme hstory value M = ( Sc - Sm ) Sc Sm Magntude error factor by Sprague and Geers Dfference n magntude M SG = Sc Sm -1 7

8 Phase s Phase error factor (Russell as well as Sprague and Geers use the same defnton) Measure of the temporal phase correlaton between two vectors P wth m measured tme hstory value c R = P SG 1 Ê = arccosá p Á Ë Sc Sc calculated tme hstory value m Sm ˆ Ranges from 0 to 1.0 Phase error factor 0.0 no phase error Phase error factor 1.0 completely out of phase 8

9 Comprehensve s Comprehensve error factor by Russell Combnaton of magntude an phase error to one factor ( ) M P C R = p R + 4 R Absolute error values < 0. are proposed n references to be consdered as good agreement Comprehensve error factor by Sprague and Geers Combnaton of magntude an phase error to one factor SG SG C = M + P SG 9

10 Applcaton of error factors Examples Very good agreement Exercse: TUB41 Magntude Russell Sprague and Geers Phase Comprehensve Dfference n quas-statc value (magntude dsagreement) Exercse: BT0016 Magntude Russell Sprague and Geers Phase Comprehensve

11 Applcaton of error factors Examples Moderate agreement Exercse: TUB413 Magntude Phase Comprehensve Russell Sprague and Geers Dsagreement n frequency content Exercse: BT0056 Magntude Phase Comprehensve Russell Sprague and Geers

12 Contents What s valdaton metrcs? Valdaton Metrcs evaluated n DynoTRAIN WP5 Drawbacks of Valdaton Metrcs Conclusons 1

13 Drawbacks of Valdaton Metrcs Magntude s a relatve measure, whch at low measured values often contradcts the experts assessments Example 1: Postve subjectve assessment, but hgh Magntude Rato Y/Q [-] Measurement Smulaton Bm passenger coach Dstance [km] Subjectve postve assessment (partners) 100% Valdaton Metrcs M SG = P SG = 0.46 C SG =

14 Drawbacks of Valdaton Metrcs Small frequency devaton between smulaton and measurement or a poor sgnal synchronsaton strongly nfluences the Phase, whle subjectve experts assessments often dsregard effects lke ths Example : Postve subjectve assessment, but hgh Phase Car body Rato acceleraton Y/Q z * [-] [m/s²] Bm passenger coach Measurement Measurement Smulaton Bm passenger coach Smulaton Dstance [km] [km] Subjectve postve assessment (partners) 100% Valdaton Metrcs M SG = P SG = C SG =

15 Drawbacks of Valdaton Metrcs Magntude s domnated by the sgnal s quas-statc porton. Thus, the bgger s the sgnal s quas-statc porton, the lower s the nfluence of the same ampltude devaton to the Magntude Example 3: Negatve subjectve assessment but low Magntude as well as Phase Car body Rato acceleraton Y/Q z * [-] Vertcal wheel load [m/s²] Q [kn] Measurement Laas freght vehcle Smulaton Bm passenger coach Measurement Measurement Smulaton Bm passenger coach 0 5 Smulaton Tme [s] Dstance [km] [km] Subjectve postve assessment (partners) Subjectve postve 0% assessment (partners) 100% Valdaton Metrcs M SG = P SG = C SG =

16 Contents What s valdaton metrcs? Valdaton Metrcs evaluated n DynoTRAIN WP5 Drawbacks of Valdaton Metrcs Conclusons 16

17 Conclusons Valdaton usng comparson of tme hstores or dstance plots, respectvely, s tradtonally carred out by subjectve engneerng judgment A subjectve engneerng judgment by an ndependent revewer s also the state-ofthe-art n the standards for valdaton of mult-body vehcle models used to replace the physcal tests n vehcle acceptance process To acheve an objectve valdaton of vehcle models n the acceptance process, a quantfed comparson of transent and stochastc tme hstores s requred Valdaton metrc provdes a potental for quantfyng comparsons between smulaton and measurement wth the ntent of mnmzng the subjectvty, whle stll mantanng a correlaton wth experts opnon Two valdaton metrcs (by Russell and by Sprague and Geers) were tested durng the valdaton exercses carred out n DynoTRAIN WP5 Applcaton of Valdaton Metrcs s not proposed for valdaton of vehcle models ntended for smulatons of on-track tests accordng to EN because of Mssng correlaton between subjectve assessments and nvestgated Valdaton Metrcs Identfed drawbacks of Valdaton Metrcs n regard to valdaton of vehcle models ntended for smulatons n context of authorsaton of ralway vehcles 17