Week 1 Basic Statistical Concepts, Part I

Transcription

1 Week 1 Basic Statistical Concepts, Part I

2 Week 1 Objectives This week we will give an introduction to the statistical package R, and to statistics. R will be used throughout this class and new features will be introduced as needed. The introduction here will cover arithmetic operations, generating sequences of numbers and storing them in data objects, performing operations on data objects, sampling and importing data sets. After motivating the need for statistics in engineering and scientific research, we will introduce fundamental notions that forms the foundation for the material in later weeks. In particular we will introduce the notions of population, census, sampling, and sampling variability, and will define such basic quantities as the proportion, mean, variance and percentiles.

3 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles

4 R is a free download statistical software. In addition to its statistical functions and graphic capabilities, R is a powerful programming language with similar structure as MATLAB. We will mainly use the R statistical functions and graphics. To find out about R go to To download R go to For a more detailed introduction see r-project.org/doc/manuals/r-intro.pdf.

5 Commands can be given in separate lines or in the same line. If in the same line they need to be separated by ; Use # for inserting comments to be ignored by R. See below for examples. R as a calculator: 2*4; 2**3; (10+11)/7; sqrt(16); sin(pi); cos(2*pi); log(exp(1)); log(10,base=10) # Angles for trigonometric functions should be given in radians, not degrees Generate sequences of numbers: 1:10; seq(1,10); seq(1,10,1); seq(2,10, 2) rep(1,5); rep(seq(1,4),2); rep(1:4,2); rep( a,5) Combine sequences: c(2,7,4,6); c(rep(0,5),rep(1,7))

6 Store sequences of numbers in objects : x=c(rep(0,5),rep(1,7)); y=seq(2,10,2) x; y x[2]; x[2:5]; y[3]; x[2]*y[3] Arithmetic on objects: sum(x); mean(x); sum(x)/length(x) x/2; 2*x; x**2; sqrt(x); x+10 c(1,2,3)*c(4,5,6); sum(c(1,2,3)*c(4,5,6)) sort(c(2,7,4,6)) Sampling from an object x: sample(x,n) # for n<length(x) gives a simple random sample of size n from x sample(x,n,replace = T) # gives a sample of size n with replacement from x Try also: x=c( Alex, B., Elela, H., Chris, A., John, D., Tania, E. ); sample(x,3); sample(x,3,replace=t)

7 Define functions: f=function(x){x**2}. Try f(2); f(c(2,3)) Integrate a function: integrate(f, 0, 3). Try also g=function(x){x**(-2)}; integrate(g, 1, Inf) Import and access data: The data we will be using are available in http: //media.pearsoncmg.com/cmg/pmmg_mml_shared/ mathstatsresources/akritas/datasets.html Go to the above site, click on BearsData.txt and copy the url. The following command imports this data set into the R data frame br : br = read.table( paste the BearsData.txt url here, header =T). That is, type: br = read.table( mml shared/ mathstatsresources/akritas/bearsdata.txt, header =T)

8 The object br is a data frame br[1:3,] # shows the first three lines of the data frame br$head.l[1:3] attach(br); Head.L[1:3]; detach(br) Controlling decimal places: options(digits = 3); 2/3 Removing all objects from memory: rm(list=ls(all=true)) # R still remembers the commands you have issued

9 Example (Examples of Engineering/Scientific Studies) Comparing the compressive strength of two or more cement mixtures. Comparing the effectiveness of three cleaning products in removing four different types of stains. Predicting failure time on the basis of stress applied. Assessing the effectiveness of a new traffic regulatory measure in reducing the weekly rate of accidents. Testing a manufacturer s claim regarding a product s quality. Studying the relation between salary increases and employee productivity in a large corporation.

10 These studies require Statistics due to the intrinsic variability: The compressive strength of different preparations of the same cement mixture will differ. The figure in HistComprStrCement.pdf shows 32 compressive strength measurements (MegaPascal units), of test cylinders (6 in. diameter, 12 in. high), using water/cement ratio of 0.4, measured on the 28th day after they are made. Under the same stress, two beams fail at different times. The proportion of defective items of a certain product will differ from batch to batch. Intrinsic variability renders the objectives of the case studies, as stated, ambiguous.

11 The objectives of the case studies can be made precise if stated in terms of averages or means. Comparing the average hardness of two different cement mixtures. Predicting the average failure time on the basis of stress applied. Estimation of the average coefficient of thermal expansion. Estimation of the average proportion of defective items. Moreover, because of variability, the words average and mean have a technical meaning which can be made clear through the concepts of population and sample.

12 Definition Population is a well-defined collection of objects or subjects, of relevance to a particular study, which are exposed to the same treatment or method. Population members are called units. The objective of a study is to investigate certain characteristic(s) of the units of the population(s) of interest.

13 Example (Populations and Unit Characteristics) All water samples taken from a lake. Characteristics: Mercury concentration; Concentration of other pollutants. All items of a certain manufactured product (that have, or will be produced). Characteristic: Proportion of defective items. All students enrolled in Big Ten universities during the academic year. Characteristics: Favorite type of music; Political affiliation. Two types of cleaning products. Characteristic: cleaning effectiveness.

14 Populations consisting of the same type of units but differ in the treatment, or method, applied to them are called treatment populations. Example (Treatment Populations) The concentration of pollutants in water samples is analyzed by two different labs. Water samples sent to Lab 1 constitute population 1, and those sent to Lab 2 constitute population 2. The time to failure of beams is studied under different stress conditions. The beams subjected to each stress condition constitute different populations.

15 Census and Samples Full (i.e., population-level) understanding of a characteristic can only be achieved by examining all population units. This is called census. However, taking a census can be time consuming and expensive: The 2000 U.S. Census costed $6.5 billion, while the 2010 Census costed $13 billion. Moreover, census is not feasible if the population is hypothetical or conceptual, i.e., not all members are available for examination. Because of the above, we typically settle for examining all units in a sample, which is a subset of the population.

16 Due to the intrinsic variability, the sample properties/attributes of the characteristic of interest will differ from the population-level properties/attributes. For example: The average mercury concentration in 25 water samples will differ from the overall mercury concentration in the lake. The proportion in a sample of 100 PSU students who favor expanding the use of solar energy will differ from the corresponding proportion of all PSU students. The relation between bear s chest girth and weight in a sample of 10 bears, will differ from the corresponding relation in the entire population of 50 bears in a forested region.

17 The GOOD NEWS is that, if the sample is suitably drawn, then sample properties approximate the population properties.

18 Weight Chest Girth Figure: Population and sample relationships between chest girth and weight of black bears.

19 Sampling Variability Sample properties of the characteristic of interest also differ from sample to sample. For example: 1 The number of US citizens, in a sample of size 20, who favor expanding solar energy, will (most likely) be different from the corresponding number in a different sample of 20 US citizens. 2 The average mercury concentration in two sets of 25 water samples drawn from a lake will differ. The term sampling variability is used to describe such differences in the properties of the characteristic of interest from sample to sample.

20 Weight Chest Girth Figure: Illustration of Sampling Variability.

21 Parameters and Statistics Population level properties/attributes of characteristic(s) of interest are called (population) parameters. Examples of parameters include averages, proportions, percentiles, and the correlation coefficient. The corresponding sample properties/attributes of characteristics are called statistics. Sample statistics approximate the corresponding population parameters but are not equal to them. Statistical inference deals with the uncertainty issues which arise in approximating parameters by statistics. The tools of statistical inference include point and interval estimation, hypothesis testing and prediction.

22 Outline Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles

23 For valid statistical inference the sample must be representative of the population. For example, a sample of PSU basketball players is not representative of PSU students, if the characteristic of interest is height. Typically it is hard to tell whether a sample is representative of the population. So, we define a sample to be representative if... (cyclical definition!!) it allows for valid statistical inference. The only guarantee for that comes from the sampling method, i.e., the method used to collect the sample. The good news is that there are several sampling methods which guarantee representativeness.

24 Definition A sample of size n is a simple random sample (s.r.s.) if the selection process ensures that every sample of size n has equal chance of being selected. In simple random sampling every member of the population has the same chance of being included in the sample. Example A class consists of 35 male and 15 female students. To obtain a sample of 2, one flips a coin and if H selects a female, if T selects a male student. Does every student have the same chance of being included in the sample? Is this s.r.s?

25 The fact that every member of the population has the same chance of being included in the sample does not, by itself, imply that the sample is s.r.s. Example To select a sample of 2 students from a population of 20 male and 20 female students, one selects at random one male and one female student. Does every student have the same chance of being included in the sample? Is this s.r.s?

26 Another sampling method for obtaining a representative sample is called stratified sampling. Definition A stratified sample consists of simple random samples from each of a number of groups (which are non-overlapping and make up the entire population) called strata. Examples of strata include: ethnic groups, age groups, and production facilities.

27 A common method of choosing the within-strata sample sizes assures that the sample representation of each stratum equals to its population representation (proportional allocation). The stratified sample in the last example used proportional allocation. If the units in the different strata differ in terms of the characteristic under study, stratified sampling is preferable to s.r.s. If different production facilities differ in terms of the proportion of defective products, a stratified sample is preferable for estimating the overall proportion of defective items produced.

28 Conceptually, selecting a s.r.s. of size n from a population of N units corresponds to the following steps: STEP 1: Assign to each unit a number from 1 to N. STEP 2: Write each number on a slips of paper, place the N slips of paper in an urn, and shuffle them. STEP 3: Select n slips of paper at random, one at a time and without replacement. Because of STEP 3, simple random sampling is also referred to as sampling without replacement.

29 If the selection of slips of paper in STEP 3 of the previous slide is done with replacement, we say that we have sampled with replacement. For example, rolling a die 10 times gives a sample with replacement from the numbers 1, 2,..., 6. Sampling with replacement is easier to work with from a mathematical point of view. When the population is very large, sampling with and without replacement are practically equivalent.

30 Non-representative Sampling Non-representative samples arise whenever there is selection bias. That is, the sampling process excludes, or leaves under-represented, parts of the population. Two examples of non-representative samples are self-selected and convenience samples. Self-selected samples often occur in opinion surveys or questionnaires. For example, in political surveys, those who feel that things are running smoothly, or without strong feelings, are less likely to respond than activists.

31 Convenience samples are made up from the most easily accessed units. For example, randomly selecting students from your classes will not result in a sample that is representative of all PSU students since your classes are mostly comprised of students with the same major as you. The Literary Digest poll of 1936 is often used to illustrate the misleading potential of self-selected and convenience samples.

32 Example (The Literary Digest poll of 1936) The magazine had been extremely successful in predicting the results in US presidential elections, but in 1936 it predicted a 3-to-2 victory for Republican Alfred Landon over the Democratic incumbent Franklin Delano Roosevelt. This prediction was based on 2.3 million responses to a questionnaire sent to 10 million voters randomly selected from phone books. On the other hand Gallup correctly predicted the outcome of that election by surveying only 50,000 people.

33 Outline Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles

34 Variable = a Numerical Characteristic Numerical characteristics such as length, mercury concentration, or number of accidents are called quantitative. Non-quantitative characteristics are called qualitative or categorical. For example, gender, make of car, eye color, political affiliation, or strength of opinion. For statistical purposes, the categories in qualitative characteristics are labeled with numbers, e.g., male = 1, female = +1. For characteristics such as strength of opinion, the chosen numbers should reflect the implicit ordering. A characteristic expressed as a number is called a variable.

35 Types of Variables Quantitative variables expressing measurements on a continuous scale are called continuous. Measurements of length, strength, weight, or time to failure are examples of continuous variables. Qualitative variables, as well as quantitative variables expressing counts, e.g., number of defective items in a batch, are called discrete. When two or more characteristics are measured on each population unit, we have bivariate or multivariate variables. Example of bivariate: Salary increase and productivity. Example of multivariate: Age, income, education level.

36 When a unit is randomly sampled from a population, the value of its variable will be denoted by X (or Y, or Z, etc). Because of the intrinsic variability, X is not known a-priori and thus it is called a random variable (r.v.). The population from which a random variable is drawn is called the underlying population of the r.v. The collection of of the variable values of all population units is called the statistical population. The statistical population of a r.v. should not be confused with the set of values a variable can take.

37 Example (Examples of Statistical Populations) 1 A list of the weight of every PSU student is the statistical population of the r.v. weight. 2 A list of 1s and 0s representing every student s opinion on whether or not solar energy should be expanded is the statistical population of the r.v. expressing opinion on solar energy.

38 Sampling from the Statistical Population It should be intuitively clear that taking a sample of n units form some population and recording the variable of each sampled unit, is equivalent to taking a sample of n units from the statistical population of the random variable. Henceforth, the word sample will be mainly used to denote a sample from the statistical population. Thus, samples will be thought of as a collection of numbers. Moreover, 1 The numbers are not known a-priori, so they are r.v. s. 2 A sample of size n will be denoted by X 1, X 2,..., X n. 3 Once the sample values are recorded they will be denoted by x 1, x 2,..., x n.

39 Outline 1 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles

40 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Proportions are relevant whenever the variable of interest is categorical, or has been categorized. Definition 1 If the population has N units, and N i units are in category i, then the population proportion for category i, is p i = #{population units of category i} #{population units} = N i N. 2 If a sample of size n is taken, and n i sample units are in category i, then the sample proportion for category i, is p i = #{sample units of category i} #{sample units} = n i n.

41 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Example 1 In a sample of 1000 adults, 72% favor tougher penalties for drunk driving. Is the correct notation for 0.72 p or p? 2 In a population of 80 engineering majors taking a required statistics class, 40 are enthusiastic about having computer labs. In a s.r. sample of 20 from these students 8 are enthusiastic. What is the correct notation for 40/80 = 0.5 and for 8/20 = 2/5? Always remember that, under s.r. sampling, p approximates, but in general is different from p.

42 Outline 1 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles

43 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Consider a population of N units, and let v 1, v 2,..., v N denote the statistical population corresponding to some variable. Then the population average or population mean, denoted by µ, is the arithmetic average of all values in the statistical population. Thus, µ = 1 N N v i. i=1 If the random variable X denotes the value of the variable of a randomly selected population unit, then the population mean is also called expected value of X, or mean value of X, and is denoted by µ X or E(X).

44 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Example In a population of 500 tin plates, the number of plates with 0, 1 and 2 scratches is N 0 = 190, N 1 = 160 and N 2 = 150. Thus, in the statistical population v 1,..., v 500, 190 v i equal 0, 160 equal 1, and 150 equal 2. The population mean is µ = v i = 0 N N N = 0.92 i=1 If a tin plate is selected at random and X is the rv denoting the number of scratches, the mean value of X is 0.92 and we write µ X = 0.92, or E(X) = 0.92.

45 Two important properties of the mean Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles The mean µ satisfies: n (v i µ) = 0. i=1 Indeed, n i=1 (v i µ) = n i=1 v i nµ = nµ nµ = 0. For any number a we have n (v i µ) 2 i=1 n (v i a) 2. i=1 Setting the first derivative of n i=1 (v i a) 2 with respect to a equal to zero and solving for a gives a = µ. Also the second derivative is positive.

46 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Use the statistical population of the variable scratches for the 500 tin plates and the following R commands to illustrate the two properties of the mean: X=c(rep(0,190), rep(1,160), rep(2,150)); sum(x-mean(x)) sum((x-mean(x))**2); sum((x-1)**2)

47 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles If a sample of size n is taken, and x 1, x 2,..., x n denote the variable values of the sample units, then the sample average or sample mean, denoted by x, is x = 1 n x i n Under s.r. sampling, a sample mean approximates, but in general is different from the population mean. Example If a s.r. sample of n = 100 is taken from the 500 tin plates, it could be that there are n 0 = 40, n 1 = 34 and n 2 = 26 plates with 0, 1 and 2 scratches. In this case, x = i=1

48 Proportions are Averages! Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles A proportion is always the mean of a suitably defined random variable. To see this consider the example with the tin plates, where N 1 = 160 out of N = 500 have one scratch. Then: For the random variable X which takes the value 1 if a tin plate has one scratch and the value 0 otherwise, its statistical population, v 1,..., v 500, consists of 160 1s and 340 0s. Thus, µ X = v i = = i=1 But 0.32 = N 1 /N is the proportion of tin plates with 1 scratch.

49 Outline 1 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles

50 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Let v 1, v 2,..., v N be a statistical population with mean µ. Definition The population variance, σ 2, is defined as σ 2 = 1 N (v i µ) 2. N i=1 The standard deviation is the positive square root of the variance: σ = σ 2. If the rv X denotes a randomly selected value from the statistical population, then a synonymous terminology for the population variance is variance of X, and is denoted by σx 2, or Var(X). The standard deviation of X is σ X = σx 2.

51 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles A simpler computational formula for the variance is Ėxample σ 2 = 1 N N i=1 v 2 i µ 2 Consider the tin plate example, so the statistical population v 1,..., v 500, has 190 v i equal 0, 160 equal 1, 150 equal 2, and µ = Then, σ 2 = =

52 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles If x 1, x 2,..., x n denotes a sample from the statistical population, the sample variance and its computational formula are: S 2 = 1 n (x i x) 2 = 1 [ n xi 2 1 ( n ) 2 x i ]. n 1 n 1 n i=1 i=1 The sample standard deviation is S = S 2. i=1 Under s.r. sampling, S 2 approximates, but in general is different from σ 2. Example Consider the s.r. sample of n = 100 tin plates, which has 40, 34 and 26 plates with 0, 1 and 2 scratches. Then, S 2 = 1 [ ] =

53 Why Divide by n 1? Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Because this way the mean of the sample variances resulting from all possible samples taken with replacement is equal to the population variance. A simple demonstration of this follows: Take a s.r.s. of size two from the infinite (conceptual) population of all coin tosses, i.e., toss a coin twice, setting 0 for H and 1 for T. The possible samples are {0, 0}, {0, 1}, {1, 0}, {1, 1}. Verify that the four sample variances average to In Chapter 3, we will see that the variance of this infinite population of all coin tosses equals the population variance of {0, 1}, which is (check!) 0.25.

54 Outline 1 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles

55 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Roughly speaking, the (1 α)100th sample percentile separates the part having the (1 α)100% smaller values, from that which has the α100% larger values. Thus: The 90th sample percentile separates the largest 10% from the lower 90% values in the data set. The 50th sample percentile is also called the sample median. The 25th, the 50th and the 75th sample percentiles are also called sample quartiles. The 25th and 75th percentiles are the lower quartile and upper quartile, respectively. The distance between the lower and upper quartiles is called the interquartile range or IQR.

56 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Order Statistics as Sample Percentiles Let X 1,..., X n be a s.r. sample from a continuous population. The ordered sample values are denoted X (1), X (2),..., X (n). Thus, X (1) < X (2) < < X (n). X (i), the ith smallest sample value, is defined to be the 100 i 0.5 n -th sample percentile.

57 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Example A s.r.s. of 10 black bears weights is: Give the order statistics, and state the sample percentiles they correspond to. Solution: The R command sort( c(154, 158, 356, 446, 40, 154, 90, 94, 150, 142) ) returns the order statistics: 40, 90, 94, 142, 150, 154, 154, 158, 356, 446. These order statistics are the 5th, 15th, 25th, 35th, 45th, 55th, 65th, 75th, 85th and 95th sample percentiles. For example, X (3) = 94 is the 100(3 0.5)/10 = 25-th sample percentile. In R these percentiles are obtained with: 100*(1:10-0.5)/10.]

58 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles In the above example none of the order statistics corresponds to the median or the 90th percentile. In general, if n is even, none of the order statistics corresponds to the median. For example, If n = 4 then X (2) is the = 37.5th sample percentile, while X (3) is the = 62.5th sample percentile. Depending on n, the above definition may not identify other percentiles of interest. In such cases, percentiles are defined by the use of interpolations.

59 Hand Calculation of Sample Median Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Definition Let X (1), X (2),..., X (n) denote the ordered sample values in a sample of size n. The sample median is defined as X ( n+1 2 ), if n is odd X = X ( n 2) + X ( n +1) 2, if n is even 2

60 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Example (Relation Between X and X) Find the sample median of X 1 = 2.3, X 2 = 3.2, X 3 = 1.8, X 4 = 2.5, X 5 = 2.7. Solution. Here, X (1) = 1.8, X (2) = 2.3, X (3) = 2.5, X (4) = 2.7, X (5) = 3.2. Since sample size is odd, X = X ( n+1 2 ) = X (3) = 2.5. For this data, X = X = 2.5. If X (5) is changed to 4.2, then X = 2.7 but X = 2.5. Thus X is affected by outliers, where as X is not. In general, if the histogram of the data is positively skewed X > X, and if it is negatively skewed X < X.

61 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Hand Calculation of Sample Quartiles and Sample IQR Definition The sample lower quartile or SLQ is defined as the median of the smallest n/2 values, if n is even the median the smallest (n + 1)/2 values, if n is odd The sample upper quartile or SUQ is defined as the median of the largest n/2 values, if n is even the median the largest (n + 1)/2 values, if n is odd The sample interquartile range, or SIQR, is defined as SIQR = SUQ SLQ,

62 Population Proportions and Sample Proportions Population Averages and Sample Averages Population Variance and Sample Variance Sample Percentiles Example Find the lower and upper quartiles of the n = 9 observations 9.39, 7.04, 7.17, 13.28, 9.00, 7.46, 21.06, 15.19, Solution. Since n is odd, the SLQ is the median of the Smallest 5(= (n + 1)/2) values: 7.04, 7.17, 7.46, 7.50, 9.00 and the SUQ is the median of the Largest 5(= (n + 1)/2) values: 9.00, 9.39, 13.28, 15.19, Thus SLQ = 7.46, and SUQ =