CHAPTER 4: DATASETS AND CRITERIA FOR ALGORITHM EVALUATION

Transcription

1 CHAPTER 4: DATASETS AND CRITERIA FOR ALGORITHM EVALUATION 4.1 Overview This chapter contains the description about the data that is used in this research. In this research time series data is used. A time series is a collection of observations made sequentially through time. Many time series are routinely recorded in economics and finance. It is a set of values, usually collected at regular intervals daily, weekly, monthly, quarterly, or yearly[4.1]. Time series analysis is based on the fundamental assumption that future estimates are based on prior, historical value, of the same variable. This implies that the historical pattern exhibited by the variable to be forecasted will extend into future. In addition it is implicitly assumed that historical data are available. In time series forecasting, past observations are collected and analyzed to develop a suitable mathematical model which captures the underlying data generating process for the series [4.2]. This chapter tells us about various variables that comprises of the data sets belonging to business field. The performance evaluation criteria for the forecasting model are also the part of this chapter. The forecast error is defined as the algebraic difference between the actual realized value for a particular time period and the forecast. It is desirable that the forecast error for a series of forecasts is as close to zero as possible. Based on the forecast and actual values nine parameters are computed for each technique used in this context. These parameters are helpful in comparative analysis. These are Mean Forecast Error (MFE), Sum of Squared Error (SSE), Mean Absolute Error (MAE) Root Mean Squared Error (RMSE), Mean Absolute Percentage Error (MAPE), Normalized Root Mean Squared Error (NRMSE), Mean Percentage Error (MPE), Mean Square Error (MSE), and Normalized Mean Square Error (NMSE) [3.8][4.3]. 101

2 4.2 Data and Data Sets The word data is originally a Latin noun meaning something given. It is a plural of datum. Information in raw or unorganized form (such as alphabets, numbers, or symbols) that refer to, or represent, conditions, ideas, or objects. Data is limitless and present everywhere in the universe. Data can exist in a variety of forms -- as numbers or text on pieces of paper, as bits and bytes stored in electronic memory, or as facts stored in a person's mind. Data are Facts that can be analyzed or used in an effort to gain knowledge or make decisions. Data is a collection of facts, such as numbers, words, measurements, observations or even just descriptions of things. Quantitative and Qualitative data may be found all over in business, economic and numerous other areas. Quantitative data are anything that can be expressed as a number, or quantified. Examples of quantitative data are scores on achievement tests, number of hours of study, or weight of a subject. These data may be represented by ordinal, interval or ratio scales and lends themselves to most statistical manipulation. Quantitative data sets have numbers associated with them. Quantitative data always are associated with a scale measure. Probably the most common scale type is the ratio-scale. Qualitative data cannot be expressed as a number. Data that represent nominal scales such as gender, socio economic status, and religious preference are usually considered to be qualitative data. Qualitative data can be arranged into categories that are not numerical. These categories can be physical traits, gender, colors or anything that does not have a number associated to it. Qualitative data is sometimes referred to as categorical data [4.4]. 102

3 Dataset is a collection of data, usually presented in tabular form. Each column represents a particular variable. Each row corresponds to a given member of the data set in question. It lists values for each of the variables, such as height and weight of an object or values of random numbers. Each value is known as a datum. The data set may comprise data for one or more members, corresponding to the number of rows. The concept of a dataset is common to almost every scientific discipline where data provide the empirical basis for research activities [4.5]. 4.3 Features of Data Set A data set (or dataset) is a collection of data. Most commonly a data set corresponds to the contents of a single database table, where every column of the table represents a particular variable, and each row corresponds to a given member of the data set in question. The data set lists values for each of the variables, such as height and weight of an object, for each member of the data set. Each value is known as a datum. The data set may comprise data for one or more members, corresponding to the number of rows. Most definitions of dataset have four features: Grouping, Content, Relatedness, and Purpose [4.6]. The term data set may also be used more loosely, to refer to the data in a collection of closely related tables, corresponding to a particular experiment or event. Grouping Grouping terms like set, aggregation, container, and collection are routinely used to indicate that datasets are data treated collectively, as a unit. These terms often occur as the nouns in categorical expressions that suggest that this feature identifies the fundamental kind of thing a dataset is Content: Although the term data is sometimes used without any additional qualification to indicate the contents of datasets, most definitions imply that the constituents of a dataset are things of some 103

4 particular kind. The data in datasets are variously described with terms such as observations, facts, values, and records of values. Relatedness It is evident from these definitions that datasets are thought of as grouping together constituents (data) that are related to each other in some way that goes beyond both the grouping itself, and the identification of the grouped things as all being of the same general kind of entity. Purpose Beyond the immediate objective of recording information datasets have a larger distinctive intended application as well. They are clearly created in order to contribute in some way to scientific activity. This might be by providing evidence to be analyzed, suggesting new hypotheses, providing refutation or confirmation of existing hypotheses, or supplying new phenomena to be explained. 4.4 Time Series Data Forecasting is done by monitoring changes that occur over time and projecting into the future. There are two common approaches to forecasting: qualitative and quantitative. Qualitative forecasting methods are especially important when historical data are unavailable. Qualitative forecasting methods are considered to be highly subjective and judgmental. Quantitative forecasting methods make use of historical data.the goal of these methods is to use past data to predict future values. Quantitative forecasting methods are subdivided into two types: time series and causal. Time-series forecasting methods involve forecasting future values based entirely on the past and present values of a variable. Time-series forecasting assumes that the factors that have influenced activities in the past and present will continue to do so in approximately the same way in the future. Time-series 104

5 forecasting seeks to identify and isolate these component factors in order to make predictions. Typically, the following four factors are examined in time-series models: Trend Cyclical effect Irregular or random effect Seasonal effect A trend is an overall long-term upward or downward movement in a time series. Trend is not the only component factor that can influence data in a time series. The cyclical effect depicts the upand-down swings or movements through the series. Cyclical movements vary in length, usually lasting from 2 to 10 years. They differ in intensity and are often correlated with a business cycle. In some time periods, the values are higher than would be predicted by a trend line (i.e., they are at or near the peak of a cycle). In other time periods, the values are lower than would be predicted by a trend line (i.e., they are at or near the bottom of a cycle). Any data that do not follow the trend modified by the cyclical component are considered part of the irregular effect, or random effect. When you have monthly or quarterly data, an additional component, the seasonal effect, is considered, along with the trend, cyclical, and irregular effects [4.7]. 4.5 Data Sets used in this Research Following are various business related data sets used in this research are as follows: 1. Foreign Exchange Rate. 2. Cold Drink Sales Data 3. Hardware Item Sales Data. 4. Gold Price Data set 5. Grocery item sale Data. 105

6 Foreign Exchange Rate Data: The foreign exchange market is the largest and most liquid of the financial market. Foreign exchange rates are amongst the most important economic indices in the international monetary markets. Like many other economic time series, foreign exchange market has its own trend, cycle, season and irregularity. In this research, exchange rate data from 24 July 2013 to 23 Jan 2014 for training purpose. For testing purpose the data from 24 Jan 2014 to 7 March 2014 is used. In order to fit exchange rate data for Artificial neural network with back-propagation, moving average for one week, moving average for two week, moving average for one month, moving average for one quarter and moving average for half year is also calculated. The source of data is a website monitored by RBI and has web address as bi.htm. In order to use this data with artificial neural network simple moving average of one week, simple moving average of two week, simple moving average of one month, simple moving average of one quarter, simple moving average of half year and last day value of exchange rate are computed before fed into them as inputs to ANN Sale Data of Cold Drink This data is collected from registered and reputed firm (warehouse) that involved in the sale of cold drink boxes. Each box contains 9 bottles. The figure collected in this data set is sale of total number of bottles per day. Besides, this data set has seasonality effect. For this purpose season related data i.e. minimum and maximum temperature of each day is also collected. The sale of cold drink is also affected by status of the day i.e. holiday or not. This data is also collected. Thus this data set has following attributes Minimum Temperature, Maximum Temperature, Day Status, Previous Day Sale. Data related to sale of cold drink bottles taken from 1 June 2013 to

7 July 2013 used for training purpose. For testing purpose, the data from 29 July 2013 to 27 August 2013 is used. The source of the data is the cold drink warehouse for sale quantity, Meteorological Department J&K Government for minimum and maximum temperature, and calendar for holiday Sale Data of Hardware Item. This data is related to sale of tiles. It comprises of following attributes Minimum Temperature, Maximum Temperature, Day weight, Day Status, Alternate product sale, Previous Day Sale. The source of the data is the authorized hardware items distributor, Meteorological Department J&K Government for minimum and maximum temperature, and calendar for holiday. The sale of a particular company item is affected by alternative company product. The data related to alternative product is also collected. In this case data taken from 1 June 2013 to 28 July 2013 are used for training purpose. For testing purpose, the data from 29 July 2013 to 27 August 2013 is used Gold Price Data set This data set is associated with the price of gold in terms of Indian Rupee. The other factor that affect gold price is generally exchange rate. This data set thus comprises of two attributes namely value of Indian Rupee in term of US dollar, and Price of Gold in term of Indian Rupee. In this research data taken from 12-nov-2013 to 14 feb-2014 are used for training purpose. For testing purpose, the data from 14-feb-2014 to 27-march-2014 is used. The source of data for exchange rate is bi.htm (Site Maintained by RBI) and for gold prices is 107

8 Grocery item sale Data. The source of this data is showroom for sale of grocery items, Meteorological Department for minimum and maximum temperature, and calendar for holiday. This data set comprises of Minimum Temperature, Maximum Temperature and Previous Day Sale. Data taken from 2 August 2014 to 30 September 2014 is historic and used for training purpose and from 2 October 2014 to 16 October 2014 for testing purpose. 4.6 Evaluation Criteria Various performance measures are proposed in order to estimate forecast accuracy and to compare different models[4.8][4.9]. These are also known as performance metrics. Each of these measures is a function of the actual and forecasted values of the time series. Before proceeding further let us define some terminology used in this study of the performance criteria for forecasting model. y i = Actual Values f i = forecasted values ei = y i - f i ; error terms n = size of the test data Mean Forecast Error (MFE): A simple measure of forecast accuracy is the mean or average of the forecast error. Mathematically, 108

9 Mean Forecast Error (MFE) = Properties of Mean Forecast Error (i) (ii) (iii) (iv) (v) It is a calculation of the average deviation of forecasted values from actual ones. It illustrates the direction of error. For this reason it is also known as the Forecast Bias. In MFE, the effects of positive and negative errors cancel out and there is no way to know their exact amount. MFE does not panelize excessive errors. For a good forecast, i.e. to have a minimum bias, it is desirable that the MFE is as close to zero as possible. A zero MFE does not mean that forecasts are perfect, i.e. contain no error; rather it only indicates that forecasts are on proper target Mean Absolute Error (MAE): It is the arithmetic mean of the magnitudes of absolute errors in all the measurements of the quantity [4.10]. Mathematically, Mean Absolute Error (MAE) = Properties of Mean Absolute Error (i) (ii) (iii) It is a calculation of the average absolute deviation of forecasted values from actual ones. Also known as Mean Absolute Deviation(MAD) Shows the magnitude of overall error, occurred due to forecasting. 109

10 (iv) (v) Unlike MFE, it does not provide any idea about the direction of errors. For a good forecast, the obtained MAE should be as small as possible Mean Absolute Percentage Error (MAPE): It is a measure of accuracy of a method for constructing fitted time series data. Mathematically Mean Absolute Percentage Error (MAPE) = / Properties of Mean Absolute Percentage Error (i) It usually expresses accuracy as a percentage. (ii) Also referred to as Mean Absolute Percentage Error (iii) With zeros or near-zeros, MAPE can give a distorted picture of error. The error on a near-zero item can be infinitely high, causing a distortion to the overall error rate when it is averaged in. (iv) It does not show the direction of error (v) It is undefined whenever actual value is Mean Percentage Error (MPE): The mean percentage error is the computed average of percentage errors by which forecasts of a model differ from actual values of the quantity being forecast. Mathematically Mean Percentage Error (MPE)= (/ ) 110

11 Properties of Mean Percentage Error (i) It is undefined whenever actual value is 0. (ii) It shows the direction of error occurred. (iii) By obtaining a value of MPE close to zero, we cannot conclude that the corresponding model performed very well. (iv) It is desirable that for a good forecast the obtained MPE should be small Mean Square Error(MSE): The mean square error is the average or mean of the squared distance between the actual and forecasted value. Mean square error gives an overall idea of the error occurred during forecasting. Mathematically Mean Square Error = ) 2 ( Properties of Mean Square Error (i) (ii) (iii) (iv) It is a measure of average squared deviation of forecasted values. It panelizes extreme errors occurred while forecasting. Does not provide any idea about the direction of overall error. Although MSE is a good measure of overall forecast error, but it is not as intuitive and easily interpretable (v) MSE is sensitive to the change of scale and data transformations. 111

12 Sum of Squared Error (SSE): It measures the total squared deviation of forecasted observations, from the actual values. Mathematically Sum of Square Error = () 2 Properties of Sum of square Error (i) The sum of squared errors is larger for big data sets than for small data sets. (ii) Objective is to minimize the sum of the squared errors i.e. The smaller the SSE, the more accurate the predicting model Root Mean Squared Error (RMSE): The Root Mean Square Error (RMSE) (also called the root mean square deviation, RMSD) is a frequently used measure of the difference between values predicted by a model and the values actually observed from the environment that is being modeled. These individual differences are also called residuals, and the RMSE serves to aggregate them into a single measure of predictive power. RMSE is nothing but the square root of calculated MSE. Mathematically, Root Mean Squared Error = () 2 / n Properties of Sum of square Error (i) RMSE is a good measure of accuracy 112

13 Normalized Root Mean Squared Error (NRMSE): NMSE normalizes the obtained MSE after dividing it by the test variance. Mathematically, Normalized Root Mean Squared Error = () () Properties of Sum of square Error (i) (ii) It is a balanced error measure and is very effective in judging forecast accuracy of a model. The smaller the NMSE value, the better forecast Normalised Mean Square Error (NMSE) : The NMSE is an estimator of the overall deviations between predictedd and measured values. It is defined as[4.10]: Normalized Mean Square Error 113