Measuring lateness. Definition of Measurement (Fenton)

Transcription

1 When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre kind. Lord Kelvin 1 Measuring lateness To measure time on an absolute scale we can measure the elapsed minutes since the start of the lecture. If a student arrives after more than 5 minutes have elapsed then their picture is taken and added to the slides The next few slides are late comers from this lecture. 2

2 Measuring lateness JUST JOKING! 3 Definition of Measurement (Fenton) Measurement is the process of empirical objective assignment of numbers to entities, in order to characterise a specific attribute. Entity: an object or event Attribute: a feature or property of an entity Objective: the measurement process must be based on a well-defined rule whose results are repeatable 4

3 Example Measures ENTITY ATTRIBUTE MEASURE Person Age Years at last birthday Person Age Months since birth Source code Length # Lines of Code (LOC) Source code Length # Executable statements Testing process duration Time in hours from start to finish Tester efficiency Number of faults found per KLOC Testing process fault frequency Number of faults found per KLOC Source code quality Number of faults found per KLOC Operating system reliability Mean Time to failure rate of occurrence of failures 5 Avoiding Mistakes in Measurement Common mistakes in software measurement can be avoided simply by adhering to the definition of measurement. In particular: You must specify both entity and attribute The entity must be defined precisely You must have a reasonable intuitive understanding of the attribute before you propose a measure The theory of measurement formalises these ideas 6

4 Don t be Wilde s Cynic Estimation based on measurement is often used to work out project effort Do not confuse this with the price you should charge to the customer There is a difference between cost and value A cynic is one who knows the cost of everything and the value of nothing [Oscar Wilde] 7 Cost and Value Build cost: The cost to you This can be assessed by estimation.. Based on measurements Price: The value to the customer is different Price (p) has to reflect value not build cost (b) This can be a (huge) advantage to you p>>b or a warning p<b 8

5 Example Use of Measurement Suppose you set up a maintenance project You goal is to ensure nothing goes wrong How do you know that you are saving money? How do you fight for more resources for your project? How do you avoid being the first to be chopped or outsourced? 9 Be Clear of Your Attribute It is a mistake to propose a measure if there is no consensus on what attribute it characterises. o Results of an IQ test intelligence? or verbal ability? or problem solving skills? o # defects found / KLOC quality of code? quality of testing? 10

6 A Cautionary Note We must not re-define an attribute to fit in with an existing measure. His IQ rating is zero - he didn t manage a single answer Well I know he can t write yet, but I ve always regarded him as a rather intelligent dog 11 Types and uses of measurement Two distinct types of measurement: direct measurement indirect measurement Two distinct uses of measurement: for assessment for prediction 12

7 Some Direct Software Measures Length of source code (measured by LOC) Duration of testing process (measured by elapsed time in hours) Number of defects discovered during the testing process (measured by counting defects) Effort of a programmer on a project (measured by person months worked) 13 Some Indirect Software Measures Programmer productivity Module defect density Defect detection efficiency Requirements stability Test effectiveness ratio System spoilage LOC produced person months of effort number of defects module size number of defects detected total number of defects numb of initial requirements total number of requirements number of items covered total number of items effort spent fixing faults total project effort 14

8 Measurement Theory Objectives Measurement theory is the scientific basis for all types of measurement. It is used to determine formally: When we have really defined a measure Which statements involving measurement are meaningful What the appropriate scale type is What types of statistical operations can be applied to measurement data 15 Measurement Theory: Key Components Empirical relation system the relations which are observed on entities in the real world which characterise our understanding of the attribute in question, e.g. Fred taller than Joe (for height of people) Representation condition real world entities are mapped to number (the measurement mapping) in such a way that all empirical relations are preserved in numerical relations and no new relations are created e.g. M(Fred) > M(Joe) precisely when Fred is taller than Joe 16

9 Representation Condition Real World Number System Joe M Fred Joe taller than Fred M(Joe) > M(Fred) Empirical relation preserved under M as Numerical relation 17 Meaningfulness in Measurement Some statements involving measurement appear more meaningful than others: Fred is twice as tall as Jane The temperature in Tokyo today is twice that in London The difference in temperature between Tokyo and London today is twice what it was yesterday Formally a statement involving measurement is meaningful if its truth value is invariant of transformations of allowable scales 18

10 Measurement Scale Types Some measures seem to be of a different type to others, depending on what kind of statements are meaningful. The 5 most important scale types of measurement are: Nominal Ordinal Interval Ratio Absolute Increasing order of sophistication 19 Scales Nominal scale: (classification) eg. blood groups, program language, colour Ordinal scale: (ordering) eg. {Excellent, Very Good, Good, Satisfactory} Interval Scale: (quantifying differences) eg. Date Ratio scale: (ratios and zero are meaningful) eg. Length Absolute scale: (counting) 20

11 Nominal Scales Egs. A classification No Order, No Size Blood Groups: O+, A, AB, AB- Multiple choice answers Which of the following is makes Cornflakes the best for you each morning? A. High in Vitamins B. Delicious Hot or Cold C. Full of Natural Sunshine Note: ordering of letters is not important. What programming language is this program written in? What is the project name to which piece of code belongs? 21 Ordinal Scales as above plus: order meaningful No size, no comparison of differences This country would be better off without the monarchy. Do you: A. Strongly Agree B. Agree C. Neither Agree nor disagree D. Disagree E. Strongly disagree This program is more cohesive than that one 22

12 A-level results A, B, C, D, E Ordinal Scales (II) Order important, but can you compare difference A-B with B-C? Assigning points to grades and totalling is trying to do this. A= 12, B= 9, C=5 etc. This is a common mistake in use of measurement in computing (as elsewhere) 23 Interval Scale Measurement Powerful, but rare in practice Distances between entities matter, but not ratios Mapping must preserve order and intervals Examples: Timing of events occurrence, e.g. could measure these in units of years, days, hours etc, all relative to different fixed events. Thus it is meaningless to say Project X started twice as early as project Y, but meaningful to say the time between project X starting and now is twice the time between project Y starting and now Heat measured on Fahrenheit or Centigrade scale 24

13 Ratio Scales As above plus: zero meaningful; ratios meaningful length, mass, temperature (but in Kelvin not centigrade) price: Two for the price of one. 25 Absolute Scales Used for counting Number of students in class Number of lines of code Number of faults Number of programmers Number of 26

14 Scale Types Summary Scale Types Nominal Ordinal Interval Ratio Absolute Characteristics Entities are classified. No arithmetic meaningful. Entities are classified and ordered. Cannot use + or -. Entities classified, ordered, and differences between them understood ( units ). No zero, but can use ordinary arithmetic on intervals. Zeros, units, ratios between entities. All arithmetic. Counting; only one possible measure. All arithmetic. 27 Natural Evolution of Measures As our understanding of an attribute grows, it is possible to define more sophisticated measures; e.g. temperature : 200BC - rankings, hotter than first thermometer preserving hotter than Fahrenheit scale Centigrade scale Absolute zero, Kelvin scale 28

15 Example: The Mean Suppose we have a set of values {a 1,a 2,...,a n } and wish to compute the average The mean is a 1 +a a n n The mean is not a meaningful average for a set of ordinal scale data 29 Alternative Measures of Average Median: The midpoint of the data when it is arranged in increasing order. It divides the data into two equal parts Suitable for ordinal data. Not suitable for nominal data since it relies on order having meaning. Mode: The commonest value Suitable for nominal data 30

16 Summary of Meaningful Statistics Scale Type Nominal Ordinal Interval Ratio Absolute Average Mode Median Arithmetic mean Geometric mean Any Spread Frequency Percentile Standard deviation Coefficient of variation Any 31 Arithmetic mean? Geometic mean? Standard deviation? Definition: The coefficient of variation is an attribute of a distribution: its standard deviation divided by its mean. 32

17 Permissible changes of Units Nominal scale: Any 1-1 mapping from M to M (a renaming) M (a) = M (b) iff M(a)=M(b) Ordinal scale: Any monotonic mapping from M to M M (a) > M (b) if M(a) > M(b) 33 Permissible changes of Units (II) Interval Scale: Any affine transformation M (v) = a.m(v) + b (a>0) Ratio scale: Any linear transformation M (v) = a.m(v) (a>0) Absolute scale No transformations 34

18 Measuring Software 3 Dimensions Length Functionality Complexity Industrial Practise # lines for code # pages for specifications 35 Horses for Courses usefulness of a measure depends on the purpose for which it is used! Roughly speaking: Length and complexity are most useful as a measure of cost of code Functionality and complexity are most useful as a measure of cost of (implementing) a spec. 36

19 The LOC Measure is Widely used LOC: Number of Lines Of Code The simplest and most widely used measure of program size. Easy to compute and automate Used (as normalising measure) for effort/cost estimation (Effort = f(loc)) quality assessment/estimation (defects/loc)) productivity assessment (LOC/effort) Alternative (similar) measures KLOC: Thousands of Lines Of Code KDSI: Thousands of Delivered Source Instructions NCLOC: Non-Comment Lines of Code Number of Characters or Number of Bytes 37 how many LOC here? with TEXT_IO; use TEXT_IO; procedure Main is --This program copies characters from an input --file to an output file. Termination occurs --either when all characters are copied or --when a NULL character is input Nullchar, Eof: exception ; Char: CHARACTER; Input_file, Output_file, Console: FILE_TYPE; Begin loop Open (FILE => Input_file, MODE => IN_FILE, NAME => CharsIn ); Open (FILE => Output_file, MODE =>OUT_FILE, NAME => CharOut ); Get (Input_file, Char); if END_OF_FILE (Input_file) then raise Eof; elseif Char = ASCII.NUL then raise Nullchar; else Put(Output_file, Char); end if ; end loop ; exception when Eof => Put (Console, no null characters ); when Nullchar => Put (Console, null terminator ); end Main 38

20 Problems with LOC No standard definition Measures length of programs rather than size Wrongly used as a surrogate for: effort complexity functionality Fails to take account of redundancy and reuse Cannot be used comparatively for different types of programming languages Only available at the end of the development lifecycle 39 Problems with What counts Do declarations count? Do comments count? Do import lists count? Do compiler directives count? What about side effects? if (x++, y--, z=k=0, j-- == 0) Blank lines? Procedure headings, debugging code, keywords on their own? 40

21 Structural measures Can define a measure on code by its structure control flow structure data flow structure data structure These can then be the basis of measures of length complexity test coverage Control Flow Structure We will consider a (standard) simple language of: Atomic statements (assignment) (A) conditional (C) loops (L) sequencing (S) 42

22 Control Flowgraphs Atomic Loop Conditional Sequence Start or Stop node 43 Start Node In degree = 0 Stop Node Out degree = 0 Procedure nodes Out degree = 1 Predicate Nodes Out Degree = 2 Four Types of Node 44

23 Nesting If P then A else (while P do A) ; A Replace a procedure node and its edge by a flowgraph S C A A L A 45 Structural Measures Define measure in terms of four functions A nullary function for atomic statements M(A) = FA Two binary function for conds and seqs M(C(P1,P2)) = FC(M(P1),M(P2)) M(S(P1,P2)) = Fs(M(P1),M(P2)) A unary function for loops M(L(P1)) = FL(M(P1)) 46

24 Example Hierarchical Measures Number of Nodes: FA() = 2 Fs(m1,m2) = m1 + m2-1 FC(m1,m2) = m1 + m2 FL(m1) = m Example Hierarchical Measures Number of Edges: FA() = 1 Fs(m1,m2) = m1 + m2 FC(m1,m2) = m1 + m2 + 2 FL(m1) = m

25 Example Hierarchical Measures Number of Assignments: FA() = 1 Fs(m1,m2) = m1 + m2 FC(m1,m2) = m1 + m2 FL(m1) = m1 49