Instance Selection. Motivation. Sample Selection (1) Sample Selection (2) Sample Selection (3) Sample Size (1)

Transcription

1 Instance Selection Andrew Kusiak 2139 Seamans Center Iowa City, Iowa Motivation The need for preprocessing of data for effective data mining is important. Tel: Fax: Sample Selection (1) M i = data mining algorithm i W = entire data set S = data sample selected from W = performance of the data mining algorithm Sample Selection (2) For a given data mining algorithm Mi, its performance on the sample S selected from W should be roughly the same as that on the entire data set W, i.e., (M i, S) (M i, W) Sample Selection (3) An ideal outcome of sample selection should be model independent. By model independence we mean that the performance difference with respect to using data S vs data W is Sample Size (1) Generally a sample size is estimated so that the estimate ε and the true value ε 0 of a population do not differ by more than a standard error in more than δ of the cases. (M i, S) (M j, W), i j 1

2 Sample Size (2) By setting up a probability inequality P( ε ε 0 ) <= δ (1) we solve for the sample size n for a given value of ε and δ, where the estimate ε, called a confidence limit, is usually a function of the sample size n, and 1 δ is confidence level. However, ε 0 is usually not known. Sample Size Selection Procedure Step 1. Select a small preliminary sample size m. Step 2. Estimate ε 0 based on the sample size m and substitute it to the equation (1). P( ε ε 0 ) <= δ (1) Step 3. Solve the equation (1) for n. If n >= m, then n m additional instances are selected for the final sample, otherwise no more additional instances are selected and the preliminary sample is retained as the final sample. Sample Size Selection Approximate formula n n 0 h 02 /h 2 h = t n-1, 1 - α/2 s sqrn where: - n 0 = initial sample size - n = new sample size - h 0 = initial width of the confidence interval - h = new width of the confidence interval h = t n-1, 1 - α/2 s sqrn Goodness of an Estimate The ultimate objective of any sampling method is to make inferences about a population of interest. Unbiased Estimator An estimator is e is unbiased for an unknown value of e 0, if the expected value of e is equal e 0, i.e., E(e) = e 0. Properties of a god estimator: G Unbiasedness G Small sampling error The unbiasedness of an estimator ensures that on average e will take the value equal to e 0. 2

3 Sampling Error Sampling variance measures the difference of an estimator e from its expected value e 0, i.e., V(e) = E[e E(e)] 2. Types of Errors Sampling errors Due to lack of full representation of all instances in the sample. To enhance the usefulness of estimation, we can compute a specific region (e 0 ε, e 0 + ε), known as confidence interval, in which the true value of the parameter of interest e 0 lies with a probability of 1 δ (i.e., the confidence level), where δ satisfies P( e e 0 >= ε) = δ. Non-sampling errors Due to shortcomings of the sampling procedures, ambiguity of definitions, faulty measurement techniques, and so on. Sampling Sampling Cost reduction Scope enlargement Analysis speed improvement Increased accuracy Risk of reaching incorrect conclusions: The estimates resulting from samples might not necessarily well represent the population. Sampling Methods General purpose vs specific purpose Equal probability vs varying probability Sampling with replacement vs sampling without replacement One stage vs muli-stage Adaptive vs non-adaptive General Purpose Sampling Methods (1) Random sampling: Every instance has the same chance of being selected. Stratified sampling: Applies to a population involving a number of approximately homogenous sub-populations (strata). An independent sample is formed from each stratum and combined into a singe sample. Cluster sampling: All instances of the selected clusters form a sample. 3

4 General Purpose Sampling Methods (2) Systematic sampling: n instances are selected from the population of N according to a randomly selected number from the interval [1,, k], where k = [N/n], where ] is the closest integer, [ and every k th number afterwards. Multi-phase sampling: Often implemented in two phases, where the second sample is usually selected as a sub-sample of the first one. General Purpose Sampling Methods (3) Network sampling: A sample and units related to the units in the sample make up a sample. Inverse sampling: Used for populations with rare events. Sampling continues until some specified conditions are met. Methods (1) Distance sampling: Use to estimate density or abundance of biological populations by establishing randomly paced lines. Spatial sampling: Used in geostatistics. Methods (2) Capture-recapture sampling: Used to estimate the number of individuals in the population by recapturing the previously captured and marked individuals. Line-intercept sampling: Used in ecological studies. Panel sampling: Used in social surveys. Methods (3) Monte Carlo strategies: A family of methods based on generating samples form a given probability distribution P(x) or/and estimating expectations of functions under this distribution. Shannon sampling: Based on Shannon sampling theory. Non-probability Sampling Methods G Accidental sampling G Purposive sampling G Modal instance sampling G Expert sampling GQuota sampling G Heterogeneity sampling G Snowball sampling 4

5 Instance Selection Filter approach Wrapper approach Filter Approach Data evaluation without data mining. Example metrics: variance, distribution, joint distribution. Wrapper Approach Use of a data mining algorithm to evaluate the results. Example metrics: execution time, accuracy, complexity (of data mining results), and memory requirements. References Liu, H. and H. Metoda (Eds), Instance Selection and Constructive Data Mining, Kluwer, Boston, MA, Singh, R. and N. Mangat, Elements of Survey Sampling, Kluwer, Boston, MA,