Certification of Derivatives Computed by Automatic Differentiation

Transcription

1 Certification of Derivatives Computed by Automatic Differentiation Mauricio Araya Polo & Laurent Hascoët Project TROPCS WSEAS, Cancún, México, May 13,

2 Plan ntroduction (Background) Automatic Differentiation Direct Mode Reverse Mode The Problem Example Our Approach Description Numerical Result Conclusions Future Work 2

3 ntroduction (Background) Automatic Differentiation (AD) : Given program that evaluates function F, builds new program that evaluates derivatives of F Scientific Applications : Derivatives are useful in optimization, sensitivity analysis and inverse problems Non-differentiability : ntroduced in programs by conditional statements (tests) May produced wrong derivatives Lack of Validation : AD models (neither AD Tools) include verification of the differentiability of the functions Novel AD model with Validation : We evaluate interval around input data where no non-differentiability problem arises, this information propagated through conditional statements 3

4 Automatic Differentiation Programs Structure: set of concatenated sequence of instructions i P = 1 ; 2 ; ; p 1; p but control flow (flowgraph): 3 depending on the inputs the example program might be: P = 1 ; T 1 ; 2 ; 4 1 T 1 4 or P = 1 ; T 1 ; 3 ; 4 2 instruction T 1 represents the conditional statement (test) Mathematical Models: composition of elementary functions f i Y = F(X) = f p f p 1 f 2 f 1 Program P evaluates the model F, for every function f i we have a computational representation i, in right order 4

5 Automatic Differentiation (2) Direct Mode: directional derivatives Y = F (X) dx = f p(x p 1 ) f p 1(x p 2 ) f 1(x 0 ) dx with x i = f i f 1, and f i () jacobians then the new program P, with i corresponding to f i () P = 1; 1 ; 2; 2 ; ; p 1; p 1 ; p flowgraph again: 3 ; 3 depending on the inputs the differentiated example program might be: P = 1 ; 1 ; T 1 ; 2 ; 2 ; 4 ; 4 ; 1 1 T 1 ; 4 4 or P = 1 ; 1 ; T 1 ; 3 ; 3 ; 4 ; 4 the differentiated example pro- ; 2 2 gram retains the control flow structure of the original program 5

6 Automatic Differentiation (3) 1 2 Original Code subroutine sub1(x,y,o1) x = y x o1 = x x + y y T 1 if ( o1 > 190) then 3 o1 = o1 o1/2 else 4 o1 = o1 o1 20 endif end Direct Differentiated Code subroutine sub1 d(x, xd, y, yd, o1, o1d) xd = yd x + y xd x = y x o1d = 2 x xd + 2 y yd o1 = x x + y y T 1 if (o1 > 190) then 3 o1d = (o1d o1) 3 o1 = (o1 o1/2) else 4 o1d = 40 o1d o1 4 o1 = o1 o1 20 endif end Table 1: Example of Direct Mode of AD 6

7 Automatic Differentiation (3) Reverse Mode: adjoints, gradients X = F (X) Ȳ = f 1 (x 0 ) f 2 (x 1 ) f p (x p 1 ) Ȳ then the new program P, P = 1 ; 2 ; ; p 1 ; p ; p ; Ī p 1 ; ; Ī 2 ; Ī 1 or P = P ; P with Īi corresponding to f t i () Remark: The reverse sweep ( P ) eventually needs some values of the forward sweep ( P ), but x 0 and others x i might be modified by the forward sweep, thus we have to store them, which for some programs leads to important memory consumption 7

8 Automatic Differentiation (4) 1 2 Original Code subroutine sub1(x,y,o1) x = y x o1 = x x + y y T 1 if ( o1 > 190) then 3 o1 = o1 o1/2 else 4 o1 = o1 o1 20 endif end 1 2 T Reverse Differentiated Code subroutine sub1 b(x, xb, y, yb, o1, o1b) PUSH(x) x = y x o1 = x x + y y if (o1 > 190) then o1b = (o1 o1b) else o1b = 40 o1 o1b endif 8 < : xb = xb + 2 x o1b yb = yb + 2 y o1b 1 8POP(x) < yb = yb + x xb : xb = y xb end Table 2: Example of Reverse Mode of AD 8

9 9 The Problem Motivation: The question of derivatives being valid only in a certain domain is a crucial problem of AD f derivatives returned by AD are used outside their domain of validity, this can result in errors that are very hard to detect Description: Programs have control flow structure, including conditional statements (tests) Some of the test are introduced by intrinsic functions like abs, min, max, etc Differentiated program keeps the control flow structure of given program Sometimes the derivatives depends in the control flow structure When some input is too close to a switch of the control flow, the resulting derivative may be very different or wrong, to the point of be useless

10 The Problem (2) o1 Evaluation of program P 15e+06 Evaluation of program P, xd,yd = 1,1 1e+06-1e e+06-3e+06-4e+06-5e+06-6e+06-7e+06-8e+06-9e+06 o1d 1e x e+06 y -15e x Plot of left shows the evaluation of program example with discontinuity problem Plot of right shows the evaluation of differentiated program example with input space direction (1,1) (x=364,o1d= ) and (x=365,o1d= )!!! 10

11 The Problem (3) Main cases of problems introduced by conditional statements (from B Kearfott paper) 11

12 Our Approach every test (t) is analyzed, under small change in the input the test must remain in the same side of the inequality variables used for by example instructions if t i 0 then t i + t i 0 (1) needed to built the current test the variation of t ( t i ) have to be expressed in terms of the intermediates variables (B i ) t i = J(T i ) B i and the variation of the intermediates variables is B i = J(B i ; ; B 0 ) X = J(B i ) J(B 0 ) X where X represents the variation of the inputs values re-composing the expression t i + t i 0 from (1), < J(T i ) J(B i ) J(B 0 ) X e j > < t i e j > (2) 12

13 Our Approach (2) we want isolate X, a good way to do that is transpose the jacobians in (2) < X J(B 0 ) J(B i ) J(T i ) e j > < t i e j > (3) we can use the reverse mode of AD to compute J(B 0 ) J(B i ) J(T i ) e j in (3) unfortunately, in real situations the number of tests is so large that the computation of this approach is not practical Solutions: combine constraints to propagate just one half-spaces reduce the size of the problem less tests or less inputs, or both 13

14 Our Approach (3) we analyze one test (t 0 ), under small change in the input the test must remain in the same side of the inequality if t 0 0 then t 0 + t 0 0 (4) the variation of t ( t 0 ) have to be expressed in terms of the intermediates variables (B 0 ) t 0 = J(T 0 ) B 0 and the variation of the intermediates variables is B 0 = J(B 0 ) β Ẋ where β Ẋ represents the variation of the inputs values β the magnitude and Ẋ the direction of the variation re-composing the expression (4), β J(T 0 ) J(B 0 ) Ẋ t 0 14

15 Our Approach (4) the following expression give us the magnitude of change of the input values, without change the sign of the test t 0 β (5) J(T 0 ) J(B 0 ) Ẋ to compute expression (5) we introduced a function call that propagate the effect of every test trough the program, resulting in a interval of validity, as follows: Direct Differentiated Code subroutine sub1 d(x,xd,y,yd,o1,o1d) Direct Differentiated Code with Validation subroutine sub1 dva(x,xd,y,yd,o1,o1d) xd = yd x + y xd x = y x o1d = 2 x xd + 2 y yd o1 = x x + y y xd = yd x + y xd x = y x o1d = 2 x xd + 2 y yd o1 = x x + y y T if (o1 > 190) then o1d = (o1d o1) o1 = (o1 o1/2) else o1d = 40 o1d o1 4 4 o1 = o1 o1 20 endif end V 1 T CALL VALDTY TEST(o1-190, o1d) if (o1 > 190) then o1d = (o1d o1) o1 = (o1 o1/2) else o1d = 40 o1d o1 4 4 o1 = o1 o1 20 endif end 15

16 Our Approach (5) n the example, the β magnitude is: β (o1 190) o1d = 190 (x 2 + y 2 ) 2 x (yd x + y xd) + 2 y yd We can access global variables gmin and gmax, which hold the upper and lower bounds of the validity interval The numerical results of the example are: 2 Evaluation of program P validated, xd,yd = 1,1 gmin gmax gmin-gmax 15 gmin = ndp gmax = ndp x x o1d gmin gmax ndp ndp ndp ndp ndp ndp ndp ndp ndp ndp 16

17 Our Approach (6) The following numerical result was obtained using a CFD solver STCS, loc, the differentiated version has loc, 542 validated tests from 2582 total tests STCS output adens Norg STCS with input=(norg,adens,+) and output=(azomes,qnplante,resmes,+) 17

18 Our Approach (7) Preliminary results of validation gmin-gmax STCS adens STCS with input=(norg=4,adens,+) and output=gmin 18

19 Conclusions Users overlook the problem of wrong derivatives due changes in control-flow AD tools must be able to detect this kind of situation and provide warning Project Our model Tropics, givenra the information about valid domains of input for direct differentiated programs The proposed model was developed inside the AD Tool Tapenade The overhead of use our method is only 3% over plain direct mode, this figure was obtain testing the model in several real-life codes 19

20 Future Work ntegrate the approach to real-life algorithms, applications (underway) Promising adaptation to Non-Smooth Optimization Extend the approach (or propose a new one) for the reverse mode of AD 20

21 Bibliography Araya-Polo M, Hascoët L, Domain of Validity of Derivatives Computed by Automatic Differentiation, Rapport de Recherche RR-5237, NRA Sophia-Antipolis, 2004 Hascoët, L, Pascual, V, TAPENADE 21 User s guide Technical report #224 NRA, Berz, M, Bischof, G, Corliss, G, and Griewank, editors Computational Differentiation: Techniques, Applications, and Tools SAM, Philadelphia, PA, 1996 Corliss, G, Faure, Ch, Griewank, A, Hascoët, L, and Naumann, U Automatic Differentiation of Algorithms, from Simulation to Optimization, Springer, Selected papers from AD2000, 2001 Kearfott, R B, Treating Non-Smooth Functions as Smooth Functions in Global Optimization and Nonlinear Systems Solvers, Scientific Computing and Validated Numerics, ed G Alefeld a nd A Frommer, Akademie Verlag, pp ,

22 TROPCS Project, NRA Sophia-Antipolis Team Laurent Hascoet (leader) Valérie Pascual Benjamin Dauvergne Stephen Wornom and Nathalie Bellesso Alain Dervieux Christophe Massol Bruno KOOBUS Mauricio Araya Theme Scientific Computing and Optimisation Computer Science for analysis and transformation of scientific programs (Parallelization and Differentiation) Tool TAPENADE: analysis and AD of source programs Applications Sensitivity Analysis Optimum Design (Aeronautics) nverse Problems & Data Assimilation (Weather forecast) 22

23 Questions? 23