MTAT Software Engineering Management

Transcription

1 MTAT Software Engineering Management Lecture 07: SPI & Measurement Part A Dietmar Pfahl Spring dietmar.pfahl@ut.ee

2 Announcement ATI Career Day Friday

3 Announcement Industry Guest Lecture Monday 23 March Increasing the predictability of software delivery with lean processes by Marek Laasik (VP Engineering at Fortumo)

4 Project Short Presentation on March 25 Purpose: to present initial ideas about the improvement project you intend to plan, and to get feedback regarding suitability Duration: 3-5 min / max. 3 slides Content: (1) Context of the proposed improvement project, (2) Issues to be addressed/resolved and corresponding improvement goals, (3) Optional: Sketch of the process changes you suggest to make in order to achieve the improvement goals

5 Structure of Lecture 7 Motivation and Definitions (Measure, Measurement) Example Measures (Process, Product, Resource) Subjectve Measurement

6 Project Planning and Control All activities aim at matching the current course of the sw project with the planned course of the SW Project

7 Processes Types of Process Models Product-Engineering Processes Engineering Processes Process-Engineering Processes Non-Engineering Processes Business Processes Social Processes Process Modeling Processes Software Knowledge Development Processes Maintenance Processes Project Mgmt Processes Quality Mgmt Processes Measurement Processes Improvement Processes Product Line Processes Conf Mgmt Processes Product Models... Quality Models Process Models Process Taxonomy PROFES Business Process Models... Social Process Models Engineering Process Models Life Cycle Models Technical Process Models Managerial Process Models Process Engineering Proc. Models

8 SPI Planning and Control Process Improvement Model SPIP Planning SPIP Control All activities aim at matching the current course of the SPI project with the planned course of the SPI Project (SPIP) Context SPIP Steering SPIP Start SPIP Enactment SPIP End SPIP = SPI Project

9 Measurement in PROFES

10 Definitions: Measurement and Measure Measurement: Measurement is the process through which values (e.g., numbers) are assigned to attributes of entities of the real world. Measure: A measure is the result of the measurement process, so it is the assignment of a value to an entity with the goal of characterizing a specified attribute. Source: Sandro Morasca, Software Measurement, in Handbook of Software Engineering and Knowledge Engineering - Volume 1: Fundamentals (refereed book), pp , Knowledge Systems Institute, Skokie, IL, USA, 2001, ISBN: X. A Entity: Program Attribute: Size Size Measure Scale & Unit 4 e * 3 d * 2 c * 1 b * 0 a * B LOC (lines of code)

11 Software Measurement Challenge Measuring physical properties (attributes): entity attribute unit* scale (type) value range* Human Height cm ratio 178 (1, 300) Human Temperature C interval 37 (30, 45) Measuring non-physical properties (attributes): entity attribute unit* scale (type) value range* Human Intelligence/IQ index ordinal 135 [0, 200] Program Modifiability???? Software properties are usually non-physical: size, complexity, functionality, reliability, maturity, portability, flexibility, understandability, maintainability, correctness, testability, coupling, coherence, interoperability, unit and range are sometimes used synonymously with scale

12 Measurement What is meaningful? Some statements: 1. I am twice as tall as you! 2. In Madrid it s twice as hot (on average) as in Tartu during summer! 3. Software X is more complex than software Y! 4. Software X is twice as complex as software Y! 5. On average, our software has complexity 3.45! 6. On average, our software has high complexity! Which statements are meaningful? What statistics (e.g., mode, median, mean) and what statistical tests could be applied (e.g., parametric vs. non-parametric)?

13 Measurement Scale Types

14 Measurement Scale Types cont d The classification of scales has an important impact on their practical use, in particular on the statistical techniques and indices that can be used. Example: Indicator of central tendency of a distribution of values ( Location ). Mode = most frequent value of distribution Median = the value such that not more than 50% of the values of the distribution are less than the median and not more than 50% of the values of the distribution are greater than the median

15 Scale Types and Meaningful Measurement Scales are defined through their admissible transformations Scales (and their admissible transformations) help us decide whether a statement involving measures is meaningful what type of statistical analyses we can apply Definition of Meaningfulness: A statement S with measurement values (i.e., measures m 1,, m n ) is meaningful iff its truth or falsity value is invariant under admissible transformations Tr. iff: if and only if Tr(S[m 1,, m n ]) is true iff S[Tr(m 1 ),, Tr(m n )] is true

16 Meaningfulness of Measurement-Based Statements Definition: A statement involving measures is meaningful, if its truth value remains unchanged under any admissible transformation of its scale type Example: In Madrid, during summer, it s on average twice as hot as in Tartu (measured on the Celsius scale: e.g., 40 C vs. 20 C) -> Meaningful?

17 Meaningfulness of Measurement-Based Statements Procedure to check for meaningfulness: 1. Apply the admissible transformation to measures in a statement S and obtain a transformed statement S. 2. If S can be shown to be equivalent to S, then the statement S is meaningful for the scale associated with the admissible transformation.

18 Meaningfulness of Measurement-Based Statements Example: --- In Madrid, during summer, it s on average twice as hot as in Tartu (measured on the Celsius scale: e.g., 40 C vs. 20 C) -> Meaningful? Statement: TM = 2*TT The Celsius scale is of type interval (m =a*m + b, a>0) To check: TM = 2*TT (?) under assumption that TM = 2*TT is true Proof: (1) TM = a*tm+b = a*(2*tt)+b (2) 2*TT = 2*(a*TT+b) = a*(2*tt)+2*b We see: (1) = (2) only if b=0 -> easy to construct counter-example with b<>0 Thus: Statement is not meaningful

19 Example Interval Scales: Fahrenheit & Celsius

20 Meaningfulness Example 1 Is statement (1) on the right meaningful, if X is measured on a ratio scale? (1) (2) x1 x 2 2 a x a 2 1 x 2 Ratio Scale

21 Meaningfulness Example 1 Is statement (1) on the right meaningful, if X is measured on a ratio scale? Apply any admissible transformation M =am (a>0) for ratio scales: (1) (2) x1 x 2 2 ( a x1 x2 ) ( a 2 ) Ratio Scale

22 Meaningfulness Example 1 Is statement (1) on the right meaningful, if X is measured on a ratio scale? (1) x 1 2 x 2 m Ratio Scale Apply any admissible transformation M =am (a>0) for ratio scales: By arithmetic manipulation, (2) can always be made equivalent to Tr(1) using any admissible transformation Tr. Therefore, the first statement is meaningful for a ratio scale. (2) a x 1 a 2 x 2 a m

23 Meaningfulness Example 2 Is statement (1) on the right meaningful, if X is measured on an interval scale? (1) x 1 2 x 2 m Interval Scale

24 Meaningfulness Example 2 Is statement (1) on the right meaningful, if X is measured on an interval scale? Apply any admissible transformation M =am+b (a>0) for interval scales: (1) (2) x 1 a x 2 1 x 2 b a x 2 m 2 b a m b Interval Scale By arithmetic manipulation, (2) can always be made equivalent to Tr(1). Therefore, the first statement is meaningful for an interval scale.

25 Meaningfulness Example 3 Ordinal Scale Is statement (1) on the right meaningful, if X is measured on an ordinal scale? Apply an admissible transformation for ordinal scales, e.g., x =x 3 : For any pair of measurements x 1 and x 2, there exists always one admissible transformation such that statement (2) is false when (1) is true. Therefore, statement (1) is not meaningful for an ordinal scale. (1) (2) x x x 2 x m m 3 x 1 x 2 2 3

26 Meaningfulness Geometric Mean The geometric mean of a data set [a 1, a 2,..., a n ] is given by Scale Type? On which scale type is the geometric mean meaningful?

27 Structure of Lecture 7 Motivation and Definitions (Measure, Measurement) Example Measures (Process, Product, Resource) Subjective Measurement

28 Measurable Entities in a SW Process (Model) Ressource tool An entity can represent any of the following: Process/Activity: any activity (or set of activities) related to software development and/or maintenance (e.g., requirements analysis, design, testing) these can be defined at different levels of granularity Product/Artifact: any artifact produced or changed during software development and/or maintenance (e.g., source code, software design documents) Resources: people, time, money, hardware or software needed to perform the processes Ressource role Product in Activity Product out MTAT / Lecture 13 / Dietmar Pfahl 2014

29 Examples of Software Product Attributes Size Length, Complexity, Functionality Modularity Cohesion Coupling Quality Value (Price)... Quality (-> ISO 9126) Functionality Reliability Usability Efficiency Maintainability Portability

30 Product Measure Ex. 1: Code Size Entity Attribute Unit Scale Type Range Who collects/reports the data? When (how often) is the data collected? How is the data collected? Who is responsible for data validity? Code module Size (or better: Length) Netto Lines of Code (NLOC) Ratio (0, ) Developer Once, at end of week Using tool CoMeas Project Manager

31 Product Measure Ex. 2: Code Quality 1 Entity Attribute Unit Scale Type Range Who collects/reports the data? When (how often) is the data collected? How is the data collected? Who is responsible for data validity? Code module (class file) Quality (or better: Correctness) Defects (Def) Ratio [0, ) Developer Continuously during unit testing Using defect reporting tool TRep Project Manager

32 Product Measure Ex. 3: Code Quality 2 Entity Attribute Unit Scale Type Range Who collects/reports the data? When (how often) is the data collected? How is the data collected? Who is responsible for data validity? Code module (file) Quality (or better: Defect Density) Def / NLOC Ratio [0, ) Developer Continuously during unit testing Using tools TRep and CoMeas Project Manager

33 Common OO Code Measures Measure Coupling Cohesion Cyclomatic Complexity Method Hiding Factor Attribute Hiding Factor Depth of Inheritance Tree Number of Children Weighted Methods Per Class Number of Classes Lines of Code (net and total; comment) Churn (new + changed LoC) Desirable Value Lower Higher Lower Higher Higher Low (tradeoff) Low (tradeoff) Low (tradeoff) Higher (with ident functionality) Lower (with ident functionality) Lower (with ident functionality) MTAT / Lecture 13 / Dietmar Pfahl 2014

34 Complexity McCabe Measure Cyclomatic Complexity (CC) Desirable Value Lower Description Defines the number of independent (simple) paths in a Control Flow Graph (CFG). Draw CFG, then calculate CC as follows: CC = #(edges) #(nodes) + 2 CC = #(decisions) + 1 CC = = 6 MTAT / Lecture 13 / Dietmar Pfahl 2014

35 Direct vs. Indirect (Derived) Measures Direct measure: a measure that directly characterizes an empirical property and does not require the prior measurement of some other property Indirect measure: uses one or more (direct or indirect) measures of one or more attributes in order to measure, indirectly, another supposedly related attribute. Requires first the measurement of two or more attributes, then it combines them using a mathematical model. speed = distance / time [km/h] accuracy = ( actual estimate / estimate ) * 100% [Percentage] Is estimate a measure?

36 Indirect Measures Examples: Defect Density (DD) Reliability (Rel) Productivity (Prod) Scale type of an indirect measure M will generally be the weakest of the scale types of the direct measures M 1,, M n

37 Indirect Measures Examples: DD = Quality 1 / Size [Unit: #Def/NLOC] Reliability = Quality 1 / Time [#Def/hour] Productivity 1 = Size / Time [NLOC/hour] Productivity 2 = Size / Effort [NLOC/person-hour]...

38 Subjective Objective Quantitative Qualitative Subjective Objective Qualitative (nominal, ordinal)?? Quantitative (interval, ratio)??

39 Subjective Objective Quantitative Qualitative Assume you measure 8 times the same attribute of the same entity (A: size [LOC] B: complexity [?]) 1. A: 120 A: 120 A: B: 4 B: 4 B: high 2. A: 124 A: 120 A: B: 4 B: 4 B: high 3. A: 120 A: 120 A: B: 4 B: 5 B: very high 4. A: 120 A: 120 A: B: 4 B: 4 B: high 5. A: 124 A: 120 A: B: 4 B: 3 B: medium 6. A: 120 A: 120 A: B: 4 B: 4 B: high 7. A: 124 A: 120 A: B: 4 B: 4 B: high 8. A: 124 A: 120 A: B: 4 B: 4 B: high Six different Measurement Series

40 Subjective Objective Quantitative Qualitative Assume you measure 8 times the same attribute of the same entity (A: size [LOC] B: complexity [?]) 1. A: 120 A: 120 A: B: 4 B: 4 B: high 2. A: 124 A: 120 A: B: 4 B: 4 B: high 3. A: 120 A: 120 A: B: 4 B: 5 B: very high 4. A: 120 A: 120 A: B: 4 B: 4 B: high 5. A: 124 A: 120 A: B: 4 B: 3 B: medium 6. A: 120 A: 120 A: B: 4 B: 4 B: high 7. A: 124 A: 120 A: B: 4 B: 4 B: high 8. A: 124 A: 120 A: B: 4 B: 4 B: high Guess: Columns 1 to 5 are Quantitative BUT: Columns 4&5 Might be Labels (not Numbers)

41 Subjective Objective Quantitative Qualitative Assume you measure 8 times the same attribute of the same entity (A: size [LOC] B: complexity [?]) 1. A: 120 A: 120 A: B: 4 B: 4 B: high 2. A: 124 A: 120 A: B: 4 B: 4 B: high 3. A: 120 A: 120 A: B: 4 B: 5 B: very high 4. A: 120 A: 120 A: B: 4 B: 4 B: high 5. A: 124 A: 120 A: B: 4 B: 3 B: medium 6. A: 120 A: 120 A: B: 4 B: 4 B: high 7. A: 124 A: 120 A: B: 4 B: 4 B: high 8. A: 124 A: 120 A: B: 4 B: 4 B: high Guess: Columns 2 and 4 are Objective BUT: What if Column 4 Had value high?

42 Types and Uses of Measures Types of Measures Direct vs. Indirect Subjective vs. Objective Has to do with measurement process (human involvement) (reliability) Qualitative vs. Quantitative Has to do with scale type Uses of Measures Assessment vs. Prediction NB: Measurement for prediction requires a prediction model

43 Measurable Entities in a SW Process (Model) Ressource tool An entity can represent any of the following: Process/Activity: any activity (or set of activities) related to software development and/or maintenance (e.g., requirements analysis, design, testing) these can be defined at different levels of granularity Product/Artifact: any artifact produced or changed during software development and/or maintenance (e.g., source code, software design documents) Resources: people, time, money, hardware or software needed to perform the processes Ressource role Product in Activity Product out

44 Examples of Software Process and Resource Attributes that can be measured Process-related: Efficiency: How fast (time, duration), how much effort (effort, cost), how much quantity/quality per time or effort unit (velocity, productivity)? Effectiveness: Do we get the results (quantity/quality) we want? e.g., test coverage Capability: CMMI level Resource-related: People: Skill, knowledge, experience, learning, motivation, personality Organisation: Maturity Method/Technique/Tool: Effectiveness, efficiency, learnability, cost

45 Process Measure Ex. 1: Acceptance Test Time Entity Attribute Unit Scale Type Range Who collects/reports the data? When (how often) is the data collected? How is the data collected? Who is responsible for data validity? Acceptance Test Time (or Duration ) Calendar Day Interval or Ratio [0, ) Customer XYZ At end of every test day Using reporting template RT Product Owner

46 Process Measure Ex. 2: Coding Effort Entity Attribute Unit Scale Type Range Who collects/reports the data? When (how often) is the data collected? How is the data collected? Who is responsible for data validity? Coding Effort Person-hour Ratio [0, ) Developer At end of every work day Using reporting template RE Project Manager

47 Time versus Effort Time: Entity: Some Activity (e.g., Test) Attribute: Time (or Duration) Unit: Year, Month, Week, (Work) Day, Hour, Minute, Second,... Range: [0, ) Scale type: ratio Characterisation: Direct Quantitative Objective/Subjective??? Effort: Entity: Some Activity (e.g., Test) Attribute: Effort Unit: Person-Year,, Person- Day, Person-Hour, Range: [0, ) Scale type: ratio Characterisation: Direct Quantitative Objective/Subjective???

48 Effort vs. Time Trade-Off Person Effort = 4 person-days (pd) Day Person Effort = 4 pd What does it mean when I say: This task takes 4 days This task needs 4 person-days Person Day Effort = 4 pd Day

49 Agile Measurement: Sprint Burndown Chart Example Sprint Backlog (Task List)

50 Agile Measurement: Burn-Down & Burn-Up Both can be used to calculate (average) team velocity = Story Points (or: Storys) per Team per Sprint

51 Agile Measurement: Velocity [Story Points / Sprint] Story Point (or: Task) Solid agile teams have consistent velocity (+/- 20%) Fluctuations? -> Look to stabilize team / environment Velocity trending up/down? -> Look at technical debt handling (rework) and team dynamics...

52 Resource Measure Ex. 1: Programming Skill Entity Attribute Unit Scale Type Range Who collects/reports the data? When (how often) is the data collected? How is the data collected? Who is responsible for data validity? Team Member Programming Skill Programming Test Score (PTS) Ordinal [0, 1, 2, 3, 4, 5] NB: Each number needs explanation! Skill Test Agency Whenever a test is conducted Programming Test Certification Agency

53 Resource Measure Ex. 2: Personality Entity Attribute Unit Scale Type Range Who collects/reports the data? When (how often) is the data collected? How is the data collected? Who is responsible for data validity? Team Member Personality Myer Briggs Type Indicator (MBTI) Nominal Set of 16 Types: ISTJ, ISFJ, INFJ,..., ENTJ Test Agency Whenever a test is conducted MBTI Instrument Certification Agency

54 Structure of Lecture 7 Motivation and Definitions (Measure, Measurement) Example Measures (Process, Product, Resource) Subjective Measurement

55 Objective vs. Subjective Measurement Objective Measurement Usually, the measurement process can be automated (Almost) no random measurement error, i.e., the process is perfectly reliable Rule of Thumb: Subjective measures have proven to be useful but if an objective measure is available, then it is (usually) preferable Subjective Measurement Human involvement in the measurement process If we repeat the measurement of the same object(s) several times, we might not get exactly the same measured value every time, i.e., the measurement process is not perfectly reliable

56 Procedures for Subjective Measurement Subjective Measures usually entail a well-defined Measurement Procedure that precisely describes: How to collect the data (usually via questionnaires on paper or online) How to conduct interviews How to review documents (software artifacts) In which order to assess the dimensions/items of the data collection instrument, etc. Examples: ISO9000 Audit, CMMI/SPICE Assessment, Function Points

57 Objective vs. Subjective Measurement Examples: Subjective Measurement Classification of defects into severity classes Function Points (when counted manually) Software Process Assessments Objective Measurement Lines of Code Cyclomatic Complexity Memory Size Test Coverage

58 Basic Concepts in Subjective Measurement Construct Item 1. Item n Measurement Instrument Construct: A conceptual object that cannot be directly observed and therefore cannot be directly measured (i.e., we estimate the quantity we are interested in rather than directly measure it); for example: User Satisfaction Competence of a Software Engineer Efficiency of a Process Maturity of an Organization Item: A subjective measurement scale that is used to measure a construct A question on a questionnaire is an item

59 Dimensionality of Constructs Constructs can be one-dimensional or multi-dimensional If a construct is multidimensional, then each dimension covers a different and distinct aspect of the construct e.g., the different dimensions of customer satisfaction Item 1 Construct. Item n One-Dimensional

60 Likert Type Scales Evaluation-type Example: Familiarity with and comprehension of the software development environment Frequency-type Example: Customers provide information to the project team about the requirements Agreement-type Example: The tasks supported by the software at the customer site change frequently Little Unsatisfactory Satisfactory Excellent Never Rarely Occasionally Most of the time Strongly Agree Agree Disagree Strongly Disagree

61 Semantic Differential Scale Items which include semantic opposites Example: Processing of change requests to existing systems or services: the time that MIS staff takes until responding to change requests received from users of existing computer-based information systems or services. Slow Fast Timely Untimely

62 Assigning numbers to scale responses Likert-Type Scales: Strongly Agree -> 1 Agree -> 2 Disagree -> 3 Strongly Disagree -> 4 Ordinal Scale But: Often the distances between the four response categories are approximately (conceptually) equidistant and thus are treated like approximate interval scales. Semantic Differential Scale: Slow Fast Ordinal scale, but again, often treated as interval scales

63 Reliability versus Validity Assume you measure several times the same attribute of an entity (say, complexity of a code module) and the centre point is the true (but unknown) value.

64 Reliability versus Validity Not reliable: too much random bias (noise) Not valid: too much systematic bias Assume you measure several times the same attribute of an entity (say, complexity of a code module) and the centre point is the true (but unknown) value.

65 Reliability Estimation Techniques Classes Number of administrations is the number of times that the same object is measured (per observer) Number of instruments is the number of different but equivalent instruments that would need to be administered Number of Administrations (per Observer / Rater) One Number of Instruments One Inter-Rater Internal Consistency Two Parallel Forms (immediate) Two Test-Retest Parallel Forms (delayed)

66 Inter-Rater Agreement vs. Internal Consistency Example Book 1 Book 2 R1 Quality Readability bad good Suspense little much Book 3 Book 4 R2 Length Weight long heavy short light 4 Books 2 Reviewers 1 Instrument 4 Items

67 Inter-Rater Agreement vs. Internal Consistency Example Data R1: Book 1: Q: - R: 2 - S: 3 - L: 3 - W: 3 Book 2: Q: - R: 4 - S: 3 - L: 2 - W: 2 Book 3: Q: - R: 2 - S: 3 - L: 1 - W: 2 Book 4: Q: - R: 4 - S: 5 - L: 4 - W: 3 Average Inter-Item Correlation R: S: L: W: R: 1 S: L: W: Avg = 0.55 R2: Inter-rater Agreement (Readability): Book 1: Q: - R: 3 - S: 3 - L: 3 - W: 3 Book 2: Q: - R: 3 - S: 4 - L: 3 - W: 2 Book 3: Q: - R: 2 - S: 1 - L: 2 - W: 2 Book 4: Q: - R: 4 - S: 4 - L: 3 - W: 3 R1: R: R2: R: Fleiss Kappa = 0.33 (fair agreement) Quality rating: Book 1: Book 2: Book 3: Book 4:

68 Next Lecture Topic: SPI & Measurement Part B For you to do: Have a look at the PROFES Quick Reference and Manual -> What does it say about measurement? Finish and submit Homework 2 Deadline: March 16, 20:00 (sharp!) Prepare your short presentation (March 25) Submit slides (max 3) at the latest by March 24 (23:59)