Interactietechnologie

Transcription

1 Interactietechnologie Statistical Evaluation Remco Veltkamp

2 learning.com Select " SIGN IN OR SIGN UP" Select "Use your School/University Account" search "Utrecht" and select "Utrecht University" choose Utrecht University login with solis id select in menu My Courses, "Join a new course (student)" Enter enrollment key BEPQS watch the videos in the three modules: Descriptive statistics Inferential statistics Reliability

3 Types of Data Quantitative Continuous e.g. task time, completion ratio Discrete e.g. rating scale 1,, 5 Binomial: 0 or 1 e.g. completion y/n Categorial Gender Operating system Color names Music notes Questionnaire data can be converted from discrete to continuous by averaging

4 Statistics Three kinds of lies: lies, damn lies, and statistics, probably not by Mark Twain Descriptive statistics: the statistical procedures for describing, synthesizing, analyzing, and interpreting quantitative data Inferential statistics: the statistical procedures for generalizing to a population from a sample

5 Estimation Sample from population Random Representativeness often more important Population statistics in Greek letters: μ (mu) for mean, Ϭ (sigma) for standard deviation Sample statistics in Latin lettter: for average, s for standard deviation

6 Central Tendency (Arithmetic) Mean, average: / Another name for average If describing a population, denoted as μ If describing a sample, denoted as, called x bar Seriously affected by unusual values called outliers Median, middle, central sample value Sensitive to small samples Geometric mean: / Mode, modal: most frequent value in the data 84, 85, 86, 103, 111, 1, 180, 183, 35, 78

7 Variance Spread around mean Average squared distance a data point is from the mean n i1 ( x i ) n n Note: will not work i1 ( xi ) n

8 Standard Deviation Square root of the variance The typical (standard) difference (deviation) of an observation from the mean i1 x n n ( i )

9 Skewness When the mean is greater than the median, the data distribution is skewed to the right When the median is greater than the mean, the data distribution is skewed to the left When mean and median are very close to each other, the data distribution is approximately symmetric

10 Skewness / ) ( 3 ) / ( 1) ( 3/ 1 3/ 1 3 median mean mode mean n n x x x x n i i n i i

11 Distribution vs Probability Probability is area under curve of probability distribution function (pdf) f(x) c P( c xd) f ( x) dx d c d x

12 Distribution of research data User research data typically far from normal E.g. 15 SUS scores from some usability test: mean 80, sd 4 not symmetric, not bell shaped These 15 are from a larger sample of 311 users: 1000 means of 15 random samples from 311:

13 Central Limit Theorem As sample size approaches infinity, the distribution of sample means will be normal distribution, regardless distribution of parent population Example: population completion rate of 57%, 1000 random samples of size 50 looks like normal distrubtion around 0,57 fail pass

14 Normal Distribution A.k.a. Gaussian distribution A.k.a. z distribution Bell shaped & symmetrical Mean, median, mode are equal Random variable has infinite range Mean Median Mode

15 Normal Distribution f(x) = frequency of random variable x = standard deviation of population = ; e =.7188 x = value of random variable ( < x < ) = mean of population 1 1 ) ( x e x f

16 Why? Expected value: because symmetric around Area under function: dx xe dx e x x x dx e dx e dx e x x x 1 1 x e

17 Emperical Rule For the normal distribution: 68.% of values fall within 1 standard deviation of the mean 95.4% of values fall within standard deviations of the mean 99.7% of values fall within 3 standard deviations of the mean Example: weights of 000 Euro coins average 7.35 gr, st dev gr

18 Tchebichev s Rule For arbitrary distribution (not per se normal distribution): At least 75% of the measurements differ from the mean less than twice the standard deviation At least 89% of the measurements differ from the mean less than three times the standard deviation This is Tchebichev s Rule: At least 1 1/k of the observation falls within k standard deviations from the mean

19 Example Suppose that, for handwashing duration: Mean = 0 s Standard deviation =3 washing technique/ Then: A least 75% of the measurements are between 14 and 6 At least 89% of the measurements are between 11 and 9

20 z Score Given a weight x, the number of standard deviations this value is away from the mean is called the z score

21 Standard error In the special case of distribution of mean values, the standard deviation is called standard error Larger sample size gives more accurate standard deviation, closer to standard error E.g. 15 SUS scores from some usability test: mean 80, sd 4 Estimate of the standard error: s n

22 t Distribution A.s.a. student distribution Often population mean μ and standard deviation Ϭ are not known Then z score cannot be used t Distribution, depends on sample size t Score, depends on degrees of freedom, which depends on sample size

23 Use of t distribution Signifinance testing: Compare two samples, test if they are significantly different df=n Confidence interval: Test one sample, determine range of values that have specified level of confidence (e.g. 95%, alpha = 5%) df=n 1 In Excel: formula =TINV(alpha,df) see Sauro&Lewis

24 Significance testing Is a new design significantly better? Or could new measurements come from same distribution? Task completion times from 14 users: mean 33, std Task completion times from 13 users: mean 18, std 10 Task completion times from many users: mean 6, std 13 Is a difference of 15 plausible?

25 Significance testing Task completion times from many users: mean 6, std two sample difference in mean value: Is a difference of 15 plausible?

26 Two sample standard error Instead of s n Take weighted average as the estimate of the standard error

27 Significance testing, t score Instead of z score for known z distribution, use: Two sample t score

28 Significance testing, t score Example Means 33 and 18, difference 15 Standard error of 6.5 t Score of.3: difference of 15 seconds is within.3 standard errors from the mean

29 p Value Area under curve below and above t score Example: t score of.3 give area under curve of 0.04 (with n=15) p Value of 0.04 means that 4% of differences in completion time would be greater than 15s if both measurements come from same distribution.3 standard errors account for 96% of the area under distribution 0.04) t:.3.3

30 Confidence interval Often population mean μ and standard deviation Ϭ are not known Confidence interval: range of values we assume have specified chance containing unknown population parameters Twice margin of error Example Estimate of completion rate 57% 95% confident that margin of error is /+ 3% 95% confidence interval [54%, 60%]

31 Confidence interval Confidende level Typically 95%, α = 0.5 Variability Standard deviation Sample size To halve margin of error, quadruple sample size s n

32 Confidence interval For completion rate Binomial data, 0 or 1 For rating scales Numerical data Task time data Continuous data

33 Confidence interval for completion rate Wald interval First invented by Laplace Abraham Wald Pierre Simon Laplace

34 Wald interval For example, 7 out 10 complete task, 95% confidence interval (meaning 1.96 critical value) However, inaccurate for small samples

35 Adjusted Wald Interval For 95% confidence intervals Add two successes (1.96) and two failures to total (3.84) Adjusted total number: n adj = n + z = n + (1.96) = n is ca. n + 4 Adjusted critical value:

36 Adjusted Wald Interval Standard Wald interval update with new adjusted values of p and n :

37 Adjusted Wald interval For example, 7 out 10 complete task, 95% confidence interval (meaning 1.96 critical value) Substitute in the Wald equation

38 Confidence Interval for Rating Scales Numerical data t confidence interval:

39 Confidence Interval for Task Time Continuous data Task time positively skewed Median has higher variability than mean Median is biased, mean not But median is less sensitive to outliers For small sample size, geometric mean completion time is better than mean completion time For large sample size, median completion time is better than mean completion time Completion time of 19 users

40 Confidence Interval for Geometric Mean Take the log of the values Then: Then transform back to original scale

41 Confidence Interval for Median Median is 50 percentile

42 Further Reading Appendix and Chapters and 3 of J. Sauro and J.R. Lewis, Quantifying the User experience: Practical statistics for user research (01)