Predicting Faults Using the Complexity of Code Change

Transcription

1 Predicting Faults Using the Complexity of Code Change International Conference on Software Engineering (2009) Ahmed E. Hassan Yoo Jin Lim

2 Introduction Background Code Change Models Case Study Related Work Conclusion Discussion Appendix Contents 2/24

3 Fault Prediction Introduction (1/2) Predict the incidence of faults in code Early warning for developers and managers Affects test efforts, costs, and product quality Commonly associated with code complexity Ex) More LOC More complex code Process metrics outperform code metrics Then, consider code change process complexity 3/22

4 Motivation Introduction (2/2) To prevent fault occurrences, predict faults. To manage negative effects of complexity, measure it. Goal Suggest a predictor of future faults based on code change complexity models 4/22

5 Code change process Background (1/3) Pattern of source code modifications Recorded by source control systems Repository has all files change history Modification Record Date: --/--/---- Name: --- Date: Line # s: -- ~ -- Date: Name: Lines of code: Name: Line # s: Line Lines # s: Message: of code: Lines of code: Message: Message: Fault Repairing modifications (FR) General Maintenance modifications (GM) Feature Introduction modifications (FI) 5/22

6 Shannon Entropy Background (2/3) Measure entropy (amount of uncertainty) in a distribution n H n P = p k log 2 p k, p k 0 and k=1 p k = probability of occurence for element k n k=1 p k = 1 Minimal vs. Maximum Entropy Ex) Output distribution of a system with 4 symbols H 4 P = 0 H 4 P = 2 6/22

7 Background (3/3) Quantifying the code change complexity Highly scattered changes Highly complex project A B C D Change patterns with high entropy are harder to track! 7/22

8 Basic Code Change Model(1/2) Complexity of a change period Distribution P Calculation of H 4 (P) = 4 k=1 p k log 2 p k = p file = Change count of a file Total change count Measurement approach Use the file as unit of code Only use the FI modifications Quantify for several changes within a period 8/22

9 Basic Code Change Model(2/2) Evolution of code change entropy A B C D Fixed number of files Fixed number of periods 9/22

10 Extended Code Change Model(1/2) Evolution Periods Time based periods Modification limit based periods Burst based periods Burst = many code modifications followed by none (+) The most general method # Modifications Time 10/22

11 Extended Code Change Model(2/2) Adaptive System Sizing Normalized Static Entropy, H = [0,1] H P = 1 Max Entropy for Distribution H n P = 1 log 2 n H n(p) n = number of all files entropy of distributions of different sizes can be compared Adaptive Sizing Entropy, H n = number of recently modified files Using time: Set of files modified in the preceding x months Using previous periods: those modified in the preceding x periods 11/22

12 File Code Change Model(1/4) Complexity of a file History Complexity Period Factor (HCPF) HCPF i A For a file j during period i, a set of files F i is modified: HCPF i B H i period i, HCPF i j = c ij H i, j F i 0, otherwise c ij = contribution of entropy H i to file j HCPF 1 with c ij = 1 : All modified files are affected by full entropy. HCPF 2 with c ij = p j : The more a file is modified, more it is affected. HCPF 3 with c ij = 1 F i : The more modified files, less each is affected. 12/22

13 File Code Change Model(2/4) Example of HCPF HCPF i j = c ij H i, j F i 0, otherwise Calculate HCPF for file A during this period file HCPF 1 (A) HCPF 2 (A) 1 1 = = p file H P = 1 HCPF 3 (A) = /22

14 File Code Change Model(3/4) History Complexity Metric (HCM) Simple HCM for a file For a file j over a set of evolution periods {a,,b}: HCM a,,b j = i {a,,b} HCPF i (j) HCPF 1 HCPF 2 HCPF 3 HCM for a subsystem For all files in subsystem S HCM a,,b S = j S HCM a,,b (j) HCM 1s HCM 2s HCM 3s 14/22

15 File Code Change Model(4/4) History Complexity Metric (HCM) Decay model of HCM Earlier modifications have less contribution to complexity HCM a,,b j = e φ (T i Current Time) HCPF i 1 (j) i {a,,b} T i = end time of period i φ = decay factor a x b y = e Four metrics in total HCM 1s, HCM 2s, HCM 3s, and HCM 1d 15/22

16 Case Study (1/4) Summary of the studied systems App. Name Type Start Date Subsystem Count Prog. Lang. NetBSD OS Mar C FreeBSD OS Jun C OpenBSD OS Oct C Postgres DBMS Jul C KDE Windowing System Apr C++ KOffice Productivity Suite Apr C++ Study approach Build Statistical Linear Regression (SLR Model) To predict faults in subsystems during the 4 th and 5 th years Measure & compare the error between models Modifications vs. Faults vs. Entropy (4) Determine statistical significance of the difference in error 16/22

17 Case Study (2/4) Linear Regression Models y = β 0 + β 1 x y = number of faults in a subsystem (FR modifications) x = specific metrics for each subsystem SLR Model Model m Model f Model HCM1s Model HCM2s Model HCM3s Model HCM1d Value of x Number of modifications Number of prior faults HCM 1s value HCM 2s value HCM 3s value HCM 1d value The SLR Model HCM 1d has the best fit for all. 17/22

18 Case Study (3/4) Prediction Error for the SLR Models y i = β 0 + β 1 x i y i = Number of predicted faults in the subsystem in 4 th and 5 th years Absolute prediction error: e i = y i y i Total prediction error: n E = 2 e i i=1 Statistical Significance of Differences b/t Models Paired t-test is used P-value < 0.05: we can with high probability reject H 0 : H 0 : μ e A, i e B, i = 0 18/22

19 Case Study (4/4) Statistical significance of differences in error Significant results (P-value < 0.05) highlighted Modifications vs. Faults Modifications vs. Entropy Faults vs. Entropy App. Name E m E f (%) E HCM3s E m (%) E HCM1d E m (%) E HCM3s E f (%) E HCM1d E f (%) NetBSD (+04%) (-14%) (-36%) (-10%) (-34%) FreeBSD (+48%) (-22%) (-33%) (+16%) (-01%) OpenBSD (+02%) (-18%) (-23%) (-16%) (-22%) Postgres (+49%) (-37%) (-40%) (-06%) (-10%) KDE (+07%) (-13%) (-42%) (-07%) (-38%) KOffice (+04%) (-01%) (-18%) (+05%) (-15%) E m E f E HCM in most cases Predictors based on faults are better than modifications, and complexity models are better than faults or modifications. 19/22

20 Barry et al. (2003) Related Work Studied a retail software s code modification records Identified evolution patterns in the software system Adb-El-Hafiz (2001) Quantified the complexity of the source code to measure entropy. This paper s model uses code modification records to quantify the complexity of the code change process. 20/22

21 Conclusion Verified the conjectures: Complex code change process negatively affects the software system. The more complex changes to a file, the higher the chance the file will contain faults. Contributions Computed the complexity of the code change process instead of just that of the source code Presented a better predictor of future faults Help managers plan ahead and be ready for future 21/22

22 Limitations Discussion Did not consider unrepaired faults Generalization limited to large open source systems Results do not show a causality relation Future work Study commercial systems Build a richer and detailed metric for complexity 22/22

23

24 Appendix (1/6) Code Change Process (cont d) Modification Types Fault Repairing modifications (FR) Fixing a fault Not used in calculating the complexity of the change process Used to count fault in case study for validation General Maintenance modifications (GM) Bookkeeping modifications such as copyright update Not used in analysis Feature Introduction modifications (FI) Adding or enhancing features Used to calculating the code change process complexity 24

25 Faults Appendix (2/6) SLR example for x = number of modifications β 0 and β 1 estimated using fault data from 2 nd and 3 rd years Modification Record Date: --/--/---- Date: Date: Name: --- Name: Name: Line # s: -- ~ -- Line # s: Line Lines # s: Lines of code: of code: Lines of code: Message: FR Message: Message: Automatic Lexical Analysis x y Modification Count Actual Fault Count Subsystem #1 Subsystem # App #1 Subsystem #n 7 10 y = βb 0 + βm 1 x R 2 value Modifications 25

26 Appendix (3/6) Prediction error example Predict faults in the application in the 4 th and 5 th years App #1 Modification Record Date: --/--/---- Date: Date: Name: --- Name: Name: Line # s: -- ~ -- Line # s: Line Lines # s: Lines of code: of code: Lines of code: Message: FR Automatic Lexical Analysis x i Modification Count Subsystem # Subsystem #2 Subsystem #n 7 Message: Message: y Actual Fault Count y = β 0 + β 1 x i y Predicted Fault Count e 1 e 2 e n E = n 2 e i i=1 26

27 Modification vs. Faults Appendix (4/6) Prior faults should be used to predict faults instead of prior modifications. 27

28 Appendix (5/6) Modifications vs. Entropy Both HCM based models are likely to outperform prior modifications 28

29 Faults vs. Entropy Appendix (6/6) Models based on entropy are as good as (or even better) predictors of faults in comparison to prior faults for most studied software systems. 29