SUPPORT VECTOR MACHINES FOR BANKRUPTCY ANALYSIS

Transcription

1 1 SUPPORT VECTOR MACHINES FOR BANKRUPTCY ANALYSIS Wlfgang HÄRDLE 2 Ruslan MORO 1,2 Drthea SCHÄFER 1 1 Deutsches Institut für Wirtschaftsfrschung (DIW) 2 Center fr Applied Statistics and Ecnmics (CASE), Humbldt-Universität zu Berlin Crprate Bankruptcy Predictin with SVMs

2 Mtivatin 2 Linear Discriminant Analysis Fisher (1936); cmpany scring: Beaver (1966), Altman (1968) Z-scre: Z i = a 1 i1 + a 2 i a d id = a i, where i = ( i1,..., id ) R d are financial ratis fr the i-th cmpany. The classificatin rule: successful cmpany: failure: Z i z Z i < z Crprate Bankruptcy Predictin with SVMs

3 Mtivatin 3 Linear Discriminant Analysis X 2 Failing cmpanies? Surviving cmpanies X 1 Crprate Bankruptcy Predictin with SVMs

4 Mtivatin 4 Linear Discriminant Analysis Failing cmpanies Surviving cmpanies Distributin density Z Scre Crprate Bankruptcy Predictin with SVMs

5 Mtivatin 5 Cmpany Data: Prbability f Default Surce: Falkenstein et al. (2000) Crprate Bankruptcy Predictin with SVMs

6 Mtivatin 6 RiskCalc Private Mdel Mdy s default mdel fr private firms A semi-parametric mdel based n the prbit regressin d E[y i i ] = Φ{a 0 + a j f j ( ij )} j=1 f j are estimated nn-parametrically n univariate mdels Crprate Bankruptcy Predictin with SVMs

7 Mtivatin 7 Linearly Nn-separable Classificatin Prblem X 2 Failing cmpanies Surviving cmpanies X 1 Crprate Bankruptcy Predictin with SVMs

8 Outline f the Talk 8 Outline 1. Mtivatin 2. Supprt Vectr Machines and their Prperties 3. Epected Risk vs. Empirical Risk Minimizatin 4. Realizatin f an SVM 5. Nn-linear Case 6. Cmpany Classificatin and Rating with SVMs Crprate Bankruptcy Predictin with SVMs

9 Supprt Vectr Machines and Their Prperties 9 Supprt Vectr Machines (SVMs) SVMs are a grup f methds fr classificatin (and regressin) that make use f classifiers prviding high margin. SVMs pssess a fleible structure which is nt chsen a priri The prperties f SVMs can be derived frm statistical learning thery SVMs d nt rely n asympttic prperties; they are especially useful when d/n is big, i.e. in mst practically significant cases SVMs give a unique slutin and utperfrm Neural Netwrks Crprate Bankruptcy Predictin with SVMs

10 Supprt Vectr Machines and Their Prperties 10 Classificatin Prblem Training set: {( i, y i )} n i=1 with the distributin P( i, y i ). Find the class y f a new bject using the classifier f : R d {+1; 1}, such that the epected risk R(f) is minimal. i R d is the vectr f the i-th bject characteristics; y i { 1; +1} r {0; 1} is the class f the i-th bject. Regressin Prblem Setup as fr the classificatin prblem but: y R Crprate Bankruptcy Predictin with SVMs

11 Epected Risk vs. Empirical Risk Minimizatin 11 Epected Risk Minimizatin Epected risk R(f) = is minimized wrt f: 1 2 f() y dp(, y) = E P(,y)[L(, y)] f pt = arg min R(f) f F L(, y) = 1 0, if classificatin is crrect, f() y = 2 1, if classificatin is wrng. F is an a priri defined set f (nn)linear classifier functins Crprate Bankruptcy Predictin with SVMs

12 Epected Risk vs. Empirical Risk Minimizatin 12 Empirical Risk Minimizatin In practice P(, y) is usually unknwn: use Empirical Risk ˆR(f) = 1 n n i=1 1 2 f( i) y i Minimizatin (ERM) ver the training set {( i, y i )} n i=1 ˆf n = arg min f F ˆR(f) Crprate Bankruptcy Predictin with SVMs

13 Epected Risk vs. Empirical Risk Minimizatin 13 Empirical Risk vs. Epected Risk Risk R R ˆ R ˆ (f) R (f) f f pt f ˆ n Functin class Crprate Bankruptcy Predictin with SVMs

14 Epected Risk vs. Empirical Risk Minimizatin 14 Cnvergence Frm the law f large numbers lim n ˆR(f) = R(f) In additin ERM satisfies lim n min f F ˆR(f) = min f F R(f) if F is nt t big. Crprate Bankruptcy Predictin with SVMs

15 Epected Risk vs. Empirical Risk Minimizatin 15 Vapnik-Chervnenkis (VC) Bund Basic result f Statistical Learning Thery (fr linear classifiers): ( R(f) ˆR(f) h + φ n, ln(η) ) n where the bund hlds with prbability 1 η and φ ( h n, ln(η) ) n = h(ln 2n h + 1) ln( η 4 ) n Crprate Bankruptcy Predictin with SVMs

16 Epected Risk vs. Empirical Risk Minimizatin 16 Structural Risk Minimizatin Structural Risk Minimizatin search fr the mdel structure S h, S h1 S h2... S h... S hk F, such that f S h minimizes the epected risk upper bund. h is VC dimensin. S h is a set f classifier functins with the same cmpleity described by h, e.g. P(1) P(2) P(3)... F, where P(i) are plynmials f degree i. The functinal class F is given a priri Crprate Bankruptcy Predictin with SVMs

17 Epected Risk vs. Empirical Risk Minimizatin 17 Vapnik-Chervnenkis (VC) Dimensin Definitin. h is VC dimensin f a set f functins if there eists a set f pints { i } h i=1 such that these pints can be separated in all 2h pssible cnfiguratins, and n set { i } q i=1 eists where q > h satisfies this prperty. Eample 1. The functins f = A sinθ have an infinite VC dimensin. Eample 2. Three pints n a plane can be shattered by a set f linear indicatr functins in 2 h = 2 3 = 8 ways (whereas 4 pints cannt be shattered in 2 q = 2 4 = 16 ways). The VC dimensin equals h = 3. Eample 3. The VC dimensin f f = {Hyperplane R d } is h = d + 1. Crprate Bankruptcy Predictin with SVMs

18 Epected Risk vs. Empirical Risk Minimizatin 18 VC Dimensin (d=2, h=3) Crprate Bankruptcy Predictin with SVMs

19 Realizatin f the SVM 19 Linearly Separable Case The training set: {( i, y i )} n i=1, y i = {+1; 1}, i R d. Find the classifier with the highest margin the gap between parallel hyperplanes separating tw classes where the vectrs f neither class can lie. Margin maimizatin minimizes the VC dimensin. Crprate Bankruptcy Predictin with SVMs

20 Realizatin f the SVM 20 Linear SVMs. Separable Case The margin is d + + d = 2/ w. T maimize it minimize the Euclidean nrm w subject t the cnstraint (1). 2 T w+b=0 margin T w+b=-1 -b w d -- d + w T w+b=1 0 1 Crprate Bankruptcy Predictin with SVMs

21 Realizatin f the SVM 21 Let w + b = 0 be a separating hyperplane. Then d + (d ) will be the shrtest distance t the clsest bjects frm the classes +1 ( 1). i w + b +1 fr y i = +1 i w + b 1 fr y i = 1 cmbine them int ne cnstraint y i ( i w + b) 1 0 i = 1, 2,..., n (1) The cannical hyperplanes i w + b = ±1 are parallel and the distance between each f them and the separating hyperplane is d + = d = 1/ w. Crprate Bankruptcy Predictin with SVMs

22 Realizatin f the SVM 22 The Lagrangian Frmulatin The Lagrangian fr the primal prblem L P = 1 2 w 2 n α i {y i ( i w + b) 1} i=1 The Karush-Kuhn-Tucker (KKT) Cnditins L P w k = 0 n i=1 α iy i ik = 0 k = 1,..., d L P b = 0 n i=1 α iy i = 0 y i ( i w + b) 1 0 α i 0 i = 1,..., n α i {y i ( i w + b) 1} = 0 Crprate Bankruptcy Predictin with SVMs

23 Realizatin f the SVM 23 Substitute the KKT cnditins int L P and btain the Lagrangian fr the dual prblem L D = n α i 1 2 i=1 n i=1 n α i α j y i y j i j j=1 The primal and dual prblems are min ma L P w k,b α i s.t. α i 0 ma α i L D n α i y i = 0 i=1 Since the ptimizatin prblem is cnve the dual and primal frmulatins give the same slutin. Crprate Bankruptcy Predictin with SVMs

24 Realizatin f the SVM 24 The Classificatin Stage The classificatin rule is: where w = n i=1 α iy i i b = 1 2 ( + + ) w g() = sign( w + b) + and are any supprt vectrs frm each class α i = arg ma α i L D subject t the cnstraint y i ( i w + b) 1 0 i = 1, 2,..., n. Crprate Bankruptcy Predictin with SVMs

25 Realizatin f the SVM 25 Linear SVMs. Nn-separable Case In the nn-separable case it is impssible t separate the data pints with hyperplanes withut an errr. Crprate Bankruptcy Predictin with SVMs

26 Realizatin f the SVM 26 Linear SVM. Nn-separable Case Crprate Bankruptcy Predictin with SVMs

27 Realizatin f the SVM 27 The prblem can be slved by intrducing psitive slack variables {ξ i } n i=1 int the cnstraints i w + b 1 ξ i fr y i = 1 i w + b 1 + ξ i fr y i = 1 ξ i 0 i If an errr ccurs, ξ i > 1. The bjective functin: 1 2 w 2 + C n i=1 ξ i where C ( capacity ) cntrls the tlerance t errrs n the training set. Under such a frmulatin the prblem is cnve Crprate Bankruptcy Predictin with SVMs

28 Realizatin f the SVM 28 The Lagrangian Frmulatin The Lagrangian fr the primal prblem fr ν = 1: L P = 1 2 w 2 + C n ξ i n α i {y i ( i w + b) 1 + ξ i } n ξ i µ i i=1 i=1 i=1 The primal prblem: min ma L P w k,b,ξ i α i,µ i Crprate Bankruptcy Predictin with SVMs

29 Realizatin f the SVM 29 The KKT Cnditins L P w k = 0 w k = n i=1 α iy i ik k = 1,..., d L P b = 0 n i=1 α iy i = 0 L P ξ i = 0 C α i µ i = 0 y i ( i w + b) 1 + ξ i 0 ξ i 0 α i 0 µ i 0 α i {y i ( i w + b) 1 + ξ i} = 0 µ i ξ i = 0 Crprate Bankruptcy Predictin with SVMs

30 Realizatin f the SVM 30 The dual Lagrangian des nt cntain ξ i r their Lagrange multipliers L D = n α i 1 2 n n α i α j y i y j i j (2) i=1 i=1 j=1 The dual prblem is subject t ma α i L D 0 α i C n α i y i = 0 i=1 Crprate Bankruptcy Predictin with SVMs

31 Nn-linear Case 31 Nn-linear SVMs Map the data t a Hilbert space H and perfrm classificatin there Ψ : R d H Nte, that in the Lagrangian frmulatin (2) the training data appear nly in the frm f dt prducts i j, which can be mapped t Ψ( i ) Ψ( j ). If a kernel functin K eists such that K( i, j ) = Ψ( i ) Ψ( j ), then we can use K withut knwing Ψ eplicitly Crprate Bankruptcy Predictin with SVMs

32 Nn-linear Case 32 Mapping int the Feature Space. Eample R 2 R 3, Ψ( 1, 2 ) = ( 2 1, 2 1 2, 2 2), K( i, j ) = ( i j) 2 Data Space Feature Space Crprate Bankruptcy Predictin with SVMs

33 Nn-linear Case 33 Mercer s Cnditin (1909) A necessary and sufficient cnditin fr a symmetric functin K( i, j ) t be a kernel is that it must be psitive definite, i.e. fr any 1,..., n R d and any λ 1,..., λ n R the functin K must satisfy: n i=1 n λ i λ j K( i, j ) 0 j=1 Eamples f kernel functins: K( i, j ) = e ( i j ) Σ 1 ( i j )/2 K( i, j ) = ( i j + 1) p K( i, j ) = tanh(k i j δ) anistrpic Gaussian kernel plynmial kernel hyperblic tangent kernel Crprate Bankruptcy Predictin with SVMs

34 Nn-linear Case 34 Classes f Kernels Statinary kernel is a kernel which is translatin invariant: K( i, j ) = K S ( i j ) Istrpic (hmgeneus) kernel is ne which depends nly n the distance between tw data pints: K( i, j ) = K I ( i j ) Lcal statinary kernel is a kernel f the frm: K( i, j ) = K 1 ( i + j 2 )K 2 ( i j ) where K 1 is a nn-negative functin, K 2 is a statinary kernel. Crprate Bankruptcy Predictin with SVMs

35 Nn-linear Case 35 Scres X X1 spr04c1p2.pl Figure 1: SVM classificatin results fr slightly nisy spiral data (RB = 0.4ˆΣ 1/2, C = 1.2/n). The spirals spread ver 3π radian; the distance between the spirals equals 1. The nise was injected with the parameters ε i N(0, I). The separatin is perfect. Crprate Bankruptcy Predictin with SVMs

36 Nn-linear Case 36 Scres X X1 pr2c1.pl Figure 2: SVM classificatin results fr the range peel data (RB = 2ˆΣ 1/2, C = 1/n). d = 2, n 1 = n +1 = 100, +1,i N(0, 2 2 I), 1,i N(0, I). Accuracy: 84% crrectly crss-validated bservatins Crprate Bankruptcy Predictin with SVMs

37 Nn-linear Case 37 Scres X X1 qur2c2.pl Figure 3: SVM classificatin results (RB = 2ˆΣ 1/2, C = 2/n). d = 2, n 1 = n +1 = 200; +1,i 1 2 N((2, 2), I) N(( 2, 2), I), 1,i 1 2 N((2, 2), I) N(( 2, 2), I). Accuracy: 96.3% crrectly crssvalidated bservatins Crprate Bankruptcy Predictin with SVMs

38 Cmpany Classificatin 38 Eample: Cmpany Classificatin Surce: annual reprts frm , Securities and Echange Cmmissin (SEC) ( 48 cmpanies with TA>$1 bln went bankrupt in f them with reliable data were selected they were matched with 42 surviving cmpanies f similar size and industry (standard industrial classificatin cde, SIC) bankrupt cmpanies filed Chapter 11 f the US Bankruptcy Cde Crprate Bankruptcy Predictin with SVMs

39 Cmpany Classificatin 39 The cmpanies were characterized by 14 variables frm which the fllwing financial ratis were calculated: 1. Prfit measures: EBIT/TA, NI/TA, EBIT/Sales; 2. Leverage ratis: EBIT/Interest, TD/TA, TL/TA; 3. Liquidity ratis: QA/CL, Cash/TA, WC/TA, CA/CL, STD/TD; 4. Activity r turnver ratis: Sales/TA, Inventries/COGS. SIZE= lnta. The average capitalizatin f a cmpany: $8.12 bln; d = 14, n = 84 Crprate Bankruptcy Predictin with SVMs

40 Cmpany Classificatin 40 Cluster Analysis f the Cmpanies, cmpany clusters Operating Bankrupt EBIT/TA NI/TA EBIT/Sales EBIT/INT TD/TA TL/TA SIZE Crprate Bankruptcy Predictin with SVMs

41 Cmpany Classificatin 41 Operating Bankrupt QA/CL CASH/TA WC/TA CA/CL STD/TD Sales/TA INV/COGS There are 19 members in the cluster f surviving cmpanies and 65 members in cluster f failed cmpanies. The result significantly changes in the presence f utliers. Crprate Bankruptcy Predictin with SVMs

42 Cmpany Classificatin 42 Prbability f Default Leverage (TL/TA) Prfitability (NI/TA) mr5c1.pl Figure 4: Lw cmpleity f classifier functins (the radial basis is 5ˆΣ 1/2 ). The capacity is fied at C = 1/n. Accuracy: 60.7% crrectly crssvalidated ut-f-sample bservatins Crprate Bankruptcy Predictin with SVMs

43 Cmpany Classificatin 43 Prbability f Default Leverage (TL/TA) Prfitability (NI/TA) mr1p2c1.pl Figure 5: Optimal cmpleity f classifier functins (the radial basis is 1.2ˆΣ 1/2 ). The capacity is C = 1/n. Accuracy: 75.0% crrectly crssvalidated ut-f-sample bservatins (maimum) Crprate Bankruptcy Predictin with SVMs

44 Cmpany Classificatin 44 Prbability f Default Leverage (TL/TA) Prfitability (NI/TA) mr05c1.pl Figure 6: Ecessively cmple classificatin functins (the radial basis is 0.5ˆΣ 1/2 ). The capacity is fied at C = 1/n. Accuracy: 69.0% crrectly crss-validated ut-f-sample bservatins Crprate Bankruptcy Predictin with SVMs

45 Cmpany Classificatin 45 Prbability f Default Leverage (TL/TA) Prfitability (NI/TA) mr1p2c03.pl Figure 7: Lw capacity (C = 0.3/n). The radial basis is fied at 1.2ˆΣ 1/2. Accuracy: 71.4% crrectly crss-validated ut-f-sample bservatins Crprate Bankruptcy Predictin with SVMs

46 Cmpany Classificatin 46 Prbability f Default Leverage (TL/TA) Prfitability (NI/TA) mr1p2c1.pl Figure 8: Optimal capacity (C = 1/n). The radial basis is fied at 1.2ˆΣ 1/2. Accuracy: 75.0% crrectly crss-validated ut-f-sample bservatins (maimum) Crprate Bankruptcy Predictin with SVMs

47 Cmpany Classificatin 47 Prbability f Default Leverage (TL/TA) Prfitability (NI/TA) mr1p2c10.pl Figure 9: High capacity (C = 10/n). The radial basis is fied at 1.2ˆΣ 1/2. Accuracy: 63.1% crrectly crss-validated ut-f-sample bservatins. Crprate Bankruptcy Predictin with SVMs

48 Cmpany Classificatin 48 Adaptin f an SVM t Cmpany Rating The scre values f = w + b estimated by an SVM crrespnd t default prbabilities: select a sliding windw f ± f f PD cunt the bankrupt and all cmpanies inside the windw if the data is representative f the whle ppulatin, PD(f) = # bankrupt /# repeat the prcedure fr anther value f f Crprate Bankruptcy Predictin with SVMs

49 Cmpany Classificatin 49 Rating Grades and Prbabilities f Default One-year PD AAA AA A+ A A- BBB BB B+ B B- Rating Grades (S&P) Crprate Bankruptcy Predictin with SVMs

50 Cmpany Classificatin 50 Leverage (TL/TA) Prfitability (NI/TA) Figure 10: Bundaries crrespnd t f = ± If the data were representative f the whle cmpany ppulatin, PD green = 0.24, PD yellw = 0.50 and PD red = The radial basis is 2ˆΣ 1/2, C = 1/n. Crprate Bankruptcy Predictin with SVMs

51 Accuracy Measures 51 Out-f-Sample Accuracy Measures Percentage f crrectly crss-validated ut-f-sample bservatins Pwer curve (PC) aka Lrenz curve r cumulative accuracy prfile Accuracy rati (AR) Crprate Bankruptcy Predictin with SVMs

52 Accuracy Measures 52 Cumulative Accuracy Prfile Curve Mdel being evaluated Cumulative default rate 1 Mdel with zer predictive pwer Perfect mdel 0 0 number f successful cmpanies number f bankrupt cmpanies Number f cmpanies, rdered by their scre Crprate Bankruptcy Predictin with SVMs

53 Accuracy Measures 53 Accuracy Rati Mdel being evaluated 1 1 Cumulative default rate Mdel with zer predictive pwer A Cumulative default rate Mdel with zer predictive pwer B Perfect mdel 0 0 number f all cmpanies 0 0 number f successful cmpanies number f bankrupt cmpanies Number f cmpanies, rdered by their scre Number f cmpanies, rdered by their scre Accuracy Rati (AR) = A/B Crprate Bankruptcy Predictin with SVMs

54 Accuracy Measures 54 Cmparisn f Different Methds Crrectly crss-validated bservatins, % Data Sets DA Lgit CART SVM spirals range peel diagnal bankruptcy: w/ variable selectin 0.65 with variable selectin Crprate Bankruptcy Predictin with SVMs

55 Accuracy Measures 55 References Altman, E. (1968). Financial Ratis, Discriminant Analysis and the Predictin f Crprate Bankruptcy, The Jurnal f Finance, September: Basel Cmmittee n Banking Supervisin (2003). The New Basel Capital Accrd, third cnsultative paper, Beaver, W. (1966). Financial Ratis as Predictrs f Failures. Empirical Research in Accunting: Selected Studies, Jurnal f Accunting Research, supplement t vl. 5: Falkenstein, E. (2000). RiskCalc fr Private Cmpanies: Mdy s Default Mdel, Mdy s Investrs Service. Crprate Bankruptcy Predictin with SVMs

56 Accuracy Measures 56 Füser, K. (2002). Basel II was muß der Mittelstand tun?, Mittelstandsrating/$file/Mittelstandsrating.pdf. Härdle, W. and Simar, L. (2003). Applied Multivariate Statistical Analysis, Springer Verlag. Mertn, R. (1974). On the Pricing f Crprate Debt: The Risk Structure f Interest Rates, The Jurnal f Finance, 29: Ohlsn, J. (1980). Financial Ratis and the Prbabilistic Predictin f Bankruptcy, Jurnal f Accunting Research, Spring: Platt, J.C. (1998). Sequential Minimal Optimizatin: A Fast Algrithm fr Training Supprt Vectr Machines, Technical Reprt MSR-TR-98-14, April. Crprate Bankruptcy Predictin with SVMs

57 Accuracy Measures 57 Divisin f Crprate Finance f the Securities and Echange Cmmissin (2004). Standard industrial classificatin (SIC) cde list, Securities and Echange Cmmissin (2004). Archive f Histrical Dcuments, Tikhnv, A.N. and Arsenin, V.Y. (1977). Slutin f Ill-psed Prblems, W.H. Winstn, Washingtn, DC. Vapnik, V. (1995). The Nature f Statistical Learning Thery, Springer Verlag, New Yrk, NY. Crprate Bankruptcy Predictin with SVMs