Computation of Information Value for Credit Scoring Models

Size: px

Start display at page:

Download "Computation of Information Value for Credit Scoring Models"

Austin Moore
5 years ago
Views:

1 Jedovnice 20 Computation of Information Value for Credit Scoring Models Martin Řezáč, Jan Koláček Dept. of Mathematics and Statistics, Faculty of Science, Masaryk University

2 Information value The special case of Kullback-Leibler divergence given by: where are densities of scores of bad and good clients. Finanční matematika v praxi I, Jedovnice 20 /30

3 Kernel estimate The kernel density estimates are defined by where and K is the Epanechnikov kernel. Finanční matematika v praxi I, Jedovnice 20 2/30

4 Empirical estimate of I val Finanční matematika v praxi I, Jedovnice 20 3/30

5 Empirical estimate of I val However in practice, there could occur computational problems. The Information value index becomes infinite in cases when some of n 0 j or n j are equal to 0. Choosing of the number of bins is also very important. In the literature and also in many applications in credit scoring, the value r=0 is preferred. Finanční matematika v praxi I, Jedovnice 20 4/30

6 Empirical estimate with supervised interval selection (ESIS), We want to avoid zero values of n 0 j or n j. We propose to require to have at least k, where k is a positive integer, observations of scores of both good and bad client in each interval. This is the basic idea of all proposed algorithms. Finanční matematika v praxi I, Jedovnice 20 5/30

7 Empirical estimate with supervised interval selection, the first ESIS: Set where is the empirical quantile function appropriate to the empirical cumulative distribution function of scores of bad clients. Finanční matematika v praxi I, Jedovnice 20 6/30

8 Empirical estimate with supervised interval selection Usage of quantile function of scores of bad clients is motivated by the assumption, that number of bad clients is less than number of good clients. If n 0 is not divisible by k, it is necessary to adjust our intervals, because we obtain number of scores of bad clients in the last interval, which is less than k. In this case, we have to merge the last two intervals. Furthermore we need to ensure, that the number of scores of good clients is as required in each interval. To do so, we compute n j for all actual intervals. If we obtain n j < k for jth interval, we merge this interval with its neighbor on the right side. This can be done for all intervals except the last one. If we have n < k for the j last interval, than we have to merge it with its neighbor on the left side, i.e. we merge the last two intervals. Finanční matematika v praxi I, Jedovnice 20 7/30

9 and Empirical estimate with supervised interval selection Very important is the choice of k. If we choose too small value, we get overestimated value of the Information value, and vice versa. As a reasonable compromise seems to be adjusted square root of number of bad clients given by The estimate of the Information value is given by where n and n 0 correspond to observed counts of good and bad clients j j in intervals created according to the described procedure. Finanční matematika v praxi I, Jedovnice 20 8/30

10 Simulation results Consider n clients, 00p B % of bad clients with and 00(-p B )% of good clients with f N(, ). : 0 Because of normality we know I val. Consider following values of parameters: n = , n = 000 μ 0 = 0 σ 0 = σ = μ = 0.5,,.5 p B = 0.02, 0.05, 0., f : N( 0, 0) 0 Finanční matematika v praxi I, Jedovnice 20 9/30

11 Simulation results ) Scores of bad and good clients were generated according to given parameters. 2) Estimates ˆ, Iˆ, Iˆ were computed. 3) Square errors were computed. 4) Steps )-3) were repeated one thousand times. 5) was computed. I val, DEC val, KERN val, ESIS Finanční matematika v praxi I, Jedovnice 20 0/30

12 Simulation results n=00000, = IV_decil 0, , , ,00068 IV_kern 0, , ,0003 0, IV_esis 0, , , ,00027 n=00000, =.0 0 IV_decil 0, , , , IV_kern 0, , , , IV_esis 0, , , , n=00000, =.5 0 IV_decil 0, , , ,02066 IV_kern 0,0956 0, , , IV_esis 0, , , , n=000, = IV_decil 0, , , , IV_kern 0, , , , IV_esis 0, , , , n=000, =.0 0 IV_decil 0, , , , IV_kern 0,7382 0, , ,0323 IV_esis 0,5088 0, , , n=000, =.5 0 IV_decil,663859, , , IV_kern 0, , , ,96856 IV_esis 0, ,3525 0,7293 0,94676 worst average best performance Finanční matematika v praxi I, Jedovnice 20 /30

13 and Adjusted empirical estimate with supervised interval selection (AESIS) It is obvious that the choice of parameter k is crucial. So the question is: Is the choice optimal (according to )? What effect has n 0 on the optimal k? And what effect, if any, has the difference of means? 0 Finanční matematika v praxi I, Jedovnice 20 2/30

14 Simulation results Consider 0000 clients, 00p B % of bad clients with 00(-p B )% of good clients with f : N(,). Set 0 0 and consider 0.5,and.5,. E(( Iˆ val I val ) 2 ) f : N( 0,) 0 and k p B p B 0 0 3/30

15 Simulation results Dependence of on k,. 0 The highlighted circles correspond to values of k, where minimal value of the is obtained. The diamonds correspond to values of k given by. 0 p B k 0 p B ˆ I val, AESIS 4/30

16 Simulation results n=000, = IV_decil 0, , , , IV_kern 0, , , , IV_esis 0, , , , IV_aesis 0, , , , n=000, =.0 0 IV_decil 0, , , , IV_kern 0,7382 0, , ,0323 IV_esis 0,5088 0, , , IV_aesis 0,2568 0, , , n=000, =.5 0 The classical estimate had the lowest in case of weak model and extremely low number (namely 20) of bad clients. If the number of bad client is slightly higher, had the best performance. Considering models with higher predictive power, and had the best performance. IV_decil,663859, , , IV_kern 0, , , ,96856 IV_esis 0, ,3525 0,7293 0,94676 IV_aesis 0, , ,3587 0, Finanční matematika v praxi I, Jedovnice 20 5/30

17 Simulation results n=00000, = IV_decil 0, , , ,00068 IV_kern 0, , ,0003 0, IV_esis 0, , , ,00027 IV_aesis 0, , , , n=00000, =.0 0 IV_decil 0, , , , IV_kern 0, , , , IV_esis 0, , , , IV_aesis 0, ,0057 0, ,0003 n=00000, =.5 0 IV_decil 0, , , ,02066 IV_kern 0,0956 0, , , IV_esis 0, , , , IV_aesis 0, , , ,0472 We can see that the classical estimate was outperformed by all other estimates of the Information value in all considered cases when the number of clients was high (00000 in our case). The best performance was achieved by in case of weak scoring models, by in case of models with high performance and by for models with very high performance. Finanční matematika v praxi I, Jedovnice 20 5/30

18 Algorithm for the modified ESIS: ) 2) 3) q [] k q j F n smax max( q j, q j 0 ) q j 0 F 0 k n ESIS. 0, ˆ I val, ESIS where s max 4) Add to the sequence, i.e. q [ q, smax ] 5) Erase all scores s max 6) While n 0 and n are greater than 2*k, repeat step 2) 5) 7) q [min( score ), q] Finanční matematika v praxi I, Jedovnice 20 7/30

19 and AESIS. Simulation results Consider 000 and 0000 clients, 00p B % of bad clients with and 00(-p B )% of good clients with f : N(,). Set 0 0 and consider 0.5,and.5,. E(( Iˆ val I val ) 2 ) f : N( 0,) 0 n 000 n 0000 k p B k pb Finanční matematika v praxi I, Jedovnice 20 8/30

20 Simulation results Dependence of on k. n 000, p 0.2 B n 0000, p 0.2 B k pb n 3. 4 ˆ ˆ0 n 000 pb n k 3. 4 ˆ ˆ0 0 n 0000 p B k 0 p B ˆ I val, AESIS pb n k 3. 4 ˆ ˆ0 0 p B k 0 p B /30

21 Simulation results n=00000, = IV_decil 0, , , ,00068 IV_kern 0, , ,0003 0, IV_esis 0, , , ,00027 IV_aesis 0, , , , IV_esis 0, , , ,00079 IV_aesis 0, , , ,00023 n=00000, =.0 0 IV_decil 0, , , , IV_kern 0, , , , IV_esis 0, , , , IV_aesis 0, ,0057 0, ,0003 IV_esis 0, , , ,034 IV_aesis 0, , , ,00534 n=00000, =.5 0 IV_decil 0, , , ,02066 IV_kern 0,0956 0, , , IV_esis 0, , , , IV_aesis 0, , , ,0472 IV_esis 0, , ,297 0,6950 IV_aesis 0, , , , Finanční matematika v praxi I, Jedovnice 20 n=000, = IV_decil 0, , , , IV_kern 0, , , , IV_esis 0, , , , IV_aesis 0, , , , IV_esis 0,0279 0, , , IV_aesis 0, , , ,00903 n=000, =.0 0 IV_decil 0, , , , IV_kern 0,7382 0, , ,0323 IV_esis 0,5088 0, , , IV_aesis 0,2568 0, , , IV_esis 0, ,4470 0,5949 0, IV_aesis 0, , , , n=000, =.5 0 IV_decil,663859, , , IV_kern 0, , , ,96856 IV_esis 0, ,3525 0,7293 0,94676 IV_aesis 0, , ,3587 0, IV_esis,50276, , ,86092 IV_aesis 0, , ,2057 0, /30

22 . Simulation results The new algorithm (ESIS.) ended disastrously in most cases. The only exception was a situation where n=000 a μ -μ 0 =0.5. This corresponds to a scoring model with very poor discriminatory power and a very small number of observed bad clients (20-200). AESIS. ranked among the average. Finanční matematika v praxi I, Jedovnice 20 2/30

23 ESIS.2 U původního ESIS často dochází ke slučování vypočtených intervalů ve druhé fázi algoritmu. Pro výpočet se používá jen F0.. Aby byla splněna podmínka n >k, je zřejmě nutné, aby hranice prvního intervalu byla větší než k. F n To vede k myšlence použít ke konstrukci intervalů nejprve F. a následně, od nějaké hodnoty skóre F0.. Jako vhodná hodnota skóre pro tento účel se jeví hodnota s 0, ve které se protínají hustoty skóre, rozdíl distribučních funkcí skóre nabývá své maximální hodnoty a také platí, že funkce f IV nabývá nulové hodnoty., s 0 Point of intersection of densities = = Point of maximal difference of CDFs Point of zero value of f IV Finanční matematika v praxi I, Jedovnice 20 22/30

24 ESIS.2, Algorithm for the modified ESIS: ) 2) 3) 4) s0 arg max F q j j k n F, j,, F ( 0 ) n k j k n0 n F0 j F0 ( s0),, n0 k k s q j s F 0 0 0, q [min( score ), q, q0, max( score ) ] 5) Merge intervals given by q where number of bads is less than k. 6) Merge intervals given by q 0 where number of goods is less than k. Finanční matematika v praxi I, Jedovnice 20 ˆ I val, ESIS 2 where 23/30

25 and AESIS.2 Simulation results Consider 000, 0000 and clients, 00p B % of bad clients with f0 : N( 0,) and 00(-p B )% of good clients with f : N(,). Set 0 0, 5 0.5,and. and consider. E(( Iˆ val I val ) 2 ) n k p B n n k pb Finanční matematika v praxi I, Jedovnice 20 0 k p B 24/30

26 Simulation results Dependence of on k. n 000, p 0.2 B n 00000, p 0.05 B n 0000, p 0.2 B pb n k ˆ ˆ0 ˆ I val, AESIS 2 2 n 0000 pb n k ˆ ˆ0 0 2 p B k 0 p B /30

Simulation results n=000, n=000, =.0 0 = 0.

27 Simulation results n=000, n=000, =.0 0 = IV_decil 0, , , , IV_kern 0, , , , IV_esis 0, , , , IV_aesis 0, , , , IV_esis 0,0279 0, , , IV_aesis 0, , , ,00903 IV_esis2 0,0382 0, , , IV_esis2a 0, , ,044 0, IV_esis2b 0, , , ,04985 IV_aesis2 0,0570 0, ,043 0, esis2 esis2a eis2b aesis2 algoritmus ESIS.2 ESIS.2 ESIS.2 ESIS.2 Finanční matematika v praxi I, Jedovnice 20 k k pb n 3. ˆ ˆ0 4 pb n k ˆ ˆ0 2 IV_decil 0, , , , IV_kern 0,7382 0, , ,0323 IV_esis 0,5088 0, , , IV_aesis 0,2568 0, , , IV_esis 0, ,4470 0,5949 0, IV_aesis 0, , , , IV_esis2 0,7946 0, , ,02286 IV_esis2a 0,2349 0, , ,03476 IV_esis2b 0, , , , IV_aesis2 0, ,0964 0, , n=000, =.5 0 worst average best performance IV_decil,663859, , , IV_kern 0, , , ,96856 IV_esis 0, ,3525 0,7293 0,94676 IV_aesis 0, , ,3587 0, IV_esis,50276, , ,86092 IV_aesis 0, , ,2057 0,3756 IV_esis2 0, , , ,6534 IV_esis2a 0, ,8388 0, , IV_esis2b 0, ,260 0, ,09247 IV_aesis2 0, , , , /30

28 Simulation results n=00000, = 0.5 n=00000, =.0 0 IV_decil 0, , , ,00068 IV_kern 0, , ,0003 0, IV_esis 0, , , ,00027 IV_aesis 0, , , , IV_esis 0, , , ,00079 IV_aesis 0, , , ,00023 IV_esis2 0, , , ,0003 IV_esis2a 0, , , , IV_esis2b 0,0052 0, , ,00039 IV_aesis2 0, ,0002 0,0009 0, IV_decil 0, , , , IV_kern 0, , , , IV_esis 0, , , , IV_aesis 0, ,0057 0, ,0003 IV_esis 0, , , ,034 IV_aesis 0, , , ,00534 IV_esis2 0, , , , IV_esis2a 0, ,004 0, ,00035 IV_esis2b 0, , , , IV_aesis2 0, , , , worst average best performance Finanční matematika v praxi I, Jedovnice 20 n=00000, =.5 0 IV_decil 0, , , ,02066 IV_kern 0,0956 0, , , IV_esis 0, , , , IV_aesis 0, , , ,0472 IV_esis 0, , ,297 0,6950 IV_aesis 0, , , , IV_esis2 0,0258 0, , , IV_esis2a 0, , , , IV_esis2b 0,0927 0, , ,00373 IV_aesis2 0, , , , /30

29 . Simulation results n=000, = r IV_kern 0, , , ,08946 IV_esis 0, , , , IV_aesis2 0, , , ,03350 n=000, =.0 0 r IV_kern 0,7382 0, , ,0323 IV_esis 0,5088 0, , , IV_esis2 0,7946 0, , ,02286 n=000, =.5 0 r IV_kern 0, , ,8628 0, IV_esis2a 0, ,0847 0, , IV_aesis2 0, ,0825 0, ,0328 n=00000, = r IV_kern 0, , , , IV_esis 0, ,0009 0,0099 0,00075 IV_aesis2 0, , , , n=00000 =.0 0 r IV_kern 0, , , , IV_esis 0, , , , IV_esis2 0, , , , n=00000 =.5 0 r IV_kern 0, , , ,0026 IV_esis2a 0, , , , IV_aesis2 0, , , , Relativní (r) jde o z předchozích tabulek vydělené příslušnou teoretickou hodnotou IV. Umožní lepší porovnání eliminuje vliv absolutní výše IV. Finanční matematika v praxi I, Jedovnice 20 28/30

30 Conclusions The classical way of computation of the information value, i.e. empirical estimate using deciles of scores, may lead to strongly biased results. We conclude that kernel estimates and empirical estimates with supervised interval selection (ESIS, AESIS, ESIS., ESIS.2 and AESIS.2) are much more appropriate to use. Finanční matematika v praxi I, Jedovnice 20 29/30

31 Děkuji za pozornost. Finanční matematika v praxi I, Jedovnice 20 30/30

INFORMATION VALUE ESTIMATOR FOR CREDIT SCORING MODELS

ECDM Lisbon INFORMATION VALUE ESTIMATOR FOR CREDIT SCORING MODELS Martin Řezáč Dept. of Mathematics and Statistics, Faculty of Science, Masaryk University Introduction Information value is widely used