When Do the USDA Forecasters Make Mistakes?

Transcription

1 When Do the USDA Forecasters Make Mistakes? by Olga Isengildina-Massa, Berna Karali, and Scott H. Irwin 1 May 2013 Forthcoming in Applied Economics 1 Olga Isengildina-Massa is an Associate Clinical Professor at the University of Texas at Arlington. Berna Karali is an Associate Professor in Department of Agricultural and Applied Economics at the University of Georgia. Scott H. Irwin is the Laurence J. Norton Chair of Agricultural Marketing in the Department of Agricultural and Consumer Economics at the University of Illinois at Urbana-Champaign.

2 When Do the USDA Forecasters Make Mistakes? I. Introduction The U.S. Department of Agriculture s failure to accurately predict the size of the current corn crop harvest has undermined confidence of some investors in the agency s forecasting, which has for years been held as gospel. (Wall Street Journal, October 22, 2010) The US Department of Agriculture (USDA) is widely considered unmatched in its data gathering resources The National Agricultural Statistics Service, part of the USDA, has a $156.8m budget for approximately 500 reports each year and 1,050 employees (Meyer, 2011). Due to the unmatched resources used and the generally high quality of information provided, USDA forecasts function as the benchmark to which other private and public forecasts are compared. The dominant role of USDA forecasts is not surprising given the classic public goods problem of private underinvestment in information and the critical role that public information plays in coordinating the beliefs of market participants. The data collection and forecasting role of the USDA is particularly important in an environment where G-20 countries are developing an agricultural market information system (AMIS) in response to recent food price volatility. This system will improve access to data on production and stocks of most food commodities (Boschat and Moffett, 2011). It has been widely documented that the release of many USDA reports moves the markets. For example, Isengildina-Massa et al. (2008) showed that volatility in corn and soybean futures markets increases about seven times on report release days. The release of the reports is usually widely watched by traders and other market participants. The market impact of the reports implies that the errors in USDA forecasts may have large consequences. For example, a Wall Street Journal article (Pleven and McGinty, 2011) recently indicated that government reports about the U.S. corn crop have become increasingly unreliable of late, 1

3 contributing to wild swings in corn prices Corn (CBOT futures) prices have hit the basic limit 20 times since the start of 2009, with eight of those instances, or 40% of them, coming on the day of a USDA report Between 1996 and 2008, only 20% of such limit moves came on report days. The article further stated that over the past two years, the Department of Agriculture s monthly forecasts of how much farmers will harvest have been off the mark to a greater degree than any other two consecutive years in the last 15. In addition to market effects, errors in USDA price estimates may lead to large changes in payments to producers, since some government payments are calculated based on these estimates. Evaluation of USDA forecasts in the previous literature has largely focused on production and price forecasts. For example, Bailey and Brorsen (1998) demonstrated that USDA forecasts underestimated beef and pork production in the 1980s, but the bias has disappeared since then and the variance of the forecasts has also declined. Sanders and Manfredo (2002) found that USDA forecasts of pork, beef, and broiler production were unbiased but not efficient with respect to past forecasts and they did not encompass all the information contained in simple time-series models. Sanders and Manfredo (2003) investigated one-quarter ahead forecasts of quarterly live cattle, live hog, and broiler prices and found that broiler price forecasts were biased and all forecast series had a tendency to repeat errors. While the USDA correctly identified the direction of price change in at least 70% of its forecasts, only 48% of actual prices fell inside a forecasted range for broilers and 35% for hogs. Isengildina et al. (2004) found that corn price forecast intervals contained the actual prices 36%-82% of the time, while soybean intervals did so 59%-86% of the time. Isengildina et al. (2006) argued that revisions to USDA corn and soybean production forecasts were smoothed, meaning that not all 2

4 information available at the time the forecasts were made was incorporated in the forecasts and some was carried into the next forecast, which could cause substantial loss in accuracy. The goal of this article is to analyse USDA forecast errors in an attempt to identify patterns and better understand when the USDA forecasters make mistakes. We hypothesize that errors in USDA forecasts may stem from two general sources. The first one deals with forecasters abilities and reflects behavioural characteristics and tendencies as described by Kahneman and Tversky (1973). Thus, forecasting behaviour may be different with respect to predicting positive versus negative change and extreme versus moderate rate of change (Denrell and Fang, 2010). The second source reflects data that are not (efficiently) included in the forecasts. Previous studies (Sanders and Manfredo, 2002, 2003) showed that USDA forecasts do not encompass simple time-series models. In our study, we investigate whether USDA forecasters fail to efficiently take into account macroeconomic data and tend to show mistakes during certain economic conditions, which has not been analysed in previous studies of these forecasts. Specifically, we investigate how these factors affect the size and the direction of the forecast errors. The findings of this study will help forecast providers to improve their procedures and assist forecast users with interpretation and utilization of information contained in the USDA forecasts. This study focuses on forecasts for all US corn, soybeans, and wheat categories published within the World Agricultural Supply and Demand Estimates (WASDE ) reports over the 1987/88 through 2009/10 marketing years. These forecasts are of particular interest because corn, soybeans, and wheat account for over 90% of total US grain and oilseed production. While some characteristics of corn and soybean production and price forecasts have been examined before (Isengildina et al., 2004, 2006), the accuracy of most other USDA forecasts describing 3

5 supply and demand conditions has been largely overlooked. To the best of our knowledge, only a few previous studies analyzed a broad range of WASDE production and consumption forecasts for corn and soybeans (Botto et al., 2006), cotton (Isengildina et al., 2012), and sugar (Lewis and Manfredo, 2012). Knowledge of supply and demand forecast accuracy is important because these categories serve as building blocks for price forecasts. Furthermore, supply and demand estimates are published within a set of other forecasts in WASDE reports that have been shown to affect the markets. II. WASDE Forecasts WASDE reports are typically released by the USDA between the 9 th and the 12 th of each month and contain forecasts of supply and demand for most major crops. WASDE forecasts follow a balance sheet approach, accounting for each component of supply and utilization (see Vogel and Bange, 1999 for a detailed description of the USDA crop forecast generation process). The supply side of the balance sheet consists of beginning stocks, production, and imports. Utilization includes domestic use (or consumption), exports, and ending stocks. Domestic use is often further subdivided into commodity specific use categories. The demand side of the balance sheet also includes a residual category to account for usage that cannot be measured directly. The ending stocks for year become beginning stocks for year 1. The balance sheet approach requires that total supply must equal domestic use plus exports and ending stocks. Prices tie both sides of the balance sheet together by rationing available supplies between competing uses (Vogel and Bange, 1999, p. 10). Unlike all other estimates, price forecasts are published in interval form. Therefore, to be consistent with the other forecasts, midpoints of the published intervals were considered. 4

6 Several agencies within USDA are involved in preparing these forecasts: the National Agricultural Statistics Service (NASS) collects data on US crop production and stocks; the Farm Service Agency (FSA) describes current policy environment and farmers reaction to current policies; the Agricultural Marketing Service (AMS) provides current price and marketing reports; the Foreign Agriculture Service (FAS) provides information regarding foreign production, use and trade; the Economic Research Service (ERS) identifies important economic effects and implications for prices, quantities supplied and demanded; and the World Agricultural Outlook Board (WAOB), with separate leaders for each commodity, coordinates the interagency process used to produce WASDE estimates. This process ensures that the best available data are used and the estimates are consistent across all USDA publications. All estimates are forecast on a marketing year basis (September through August for corn and soybeans and June through May for wheat), thus forecasted values represent marketing year averages. Fig. 1 demonstrates marketing years and forecast months for the three commodities studied. The first forecast for a marketing year is released in May preceding the US marketing year. Production and beginning stocks forecasts are generally finalized by October (for wheat) and January (for corn and soybeans) of the marketing year. All other estimates are typically finalized by next September (for wheat) and November (for corn and soybeans). Thus, there is a 9 (6) month forecasting cycle for production and beginning stocks forecasts, and a 19 (17) month cycle for all other categories for corn and soybeans (wheat). Within this forecasting cycle, forecasts differ substantially with respect to the amount of available information. For corn and soybeans, May through August forecasts are provided during the growing season and reflect new information about the development of the crop. September through November forecasts represent harvest-time forecasts and production 5

7 uncertainty is limited to the challenges of the harvesting process. December through August forecasts generally are considered post-harvest with uncertainty driven mainly by the speed of consumption. Post-August estimates reflect revisions of the observed marketing year data. USDA wheat forecasts combine information for both winter wheat and spring wheat. For winter wheat, the growing season is May, the harvest season is June-July, and the post-harvest season is August-May. For spring wheat, the growing season is May-July, the harvest season is August- September, and the post-harvest season is October-May. Combining this information, the forecasts may be largely differentiated into highly uncertain stage 1 forecasts during May-August (May-June), less uncertain stage 2 forecasts during September-November (July-August), stage 3 forecasts during December-August (September-May) and stage 4 estimates during September- November (June-September) for corn and soybeans (wheat), as shown in Fig. 1. [INSERT FIGURE 1 HERE] USDA s WASDE forecasts are considered fixed-event forecasts because the series of forecasts is related to the same terminal event, where is the release month of the final estimate for marketing year. The forecasted value published in month j of marketing year is denoted as j y t, where 1,,, and 19 for corn and soybeans and 17 for wheat. Thus, each subsequent forecast is an update of the previous forecast describing the same terminal event. The terminal event for each category describes a marketing year total (for production, consumption, and stocks categories) or average (for price) values. Based on our definition of a 19 (17) month forecasting cycle, WASDE generates 18 (16) updates for each US variable forecasted except production and beginning stocks 2 (eight updates for corn and soybeans and 2 Because the beginning stocks category is the same as previous year s endings stocks, it is not evaluated as a separate category in this study. 6

8 five updates for wheat) within each marketing year. The marketing years covered in this study are 1(1987/88),, (2009/10), and 23. To avoid the impacts of changing forecast levels over the study period on forecast evaluation and analysis, forecasts the previous year s values: j f t were defined as percent changes in forecasted values from 100, 1,,23; 1,, 6 for corn and soybeans; 1,,4 for wheat (1) 100, 1,,23; 7,, for corn and soybeans; 5,, for wheat (2) Note that the denominator in Equation 1 is labelled 12 rather than since, for USDA s year forecasts during 7 for corn and soybeans and 5 for wheat, are not yet known because the previous forecasting cycle is not finished by the time the new forecasting cycle starts. For example, corn and soybean forecasts for 13,, 19, 13,,17 for wheat for marketing year 1 and for 1,,7 1,, 5 for wheat for marketing year are released in the same report. Thus, a first forecast for 2009/10 marketing year would be calculated as a percent change from the 13 th forecast for 2008/09 marketing year (released in the same report), since the final value will not become known for a few months. Since forecasts are measured in this study as percent changes from the previous year to control for scale, percentage forecast error is calculated as 3 :, 1,, 1; 1,,23 (3) 3 In this notation is identical to traditional percent forecast error when the previous year s values are finalized:

9 The absolute value of this measure reflects the magnitude of the forecast error and its average describes forecast bias. The means and standard deviations of the final forecast levels ( ) shown in Table 1 demonstrate the average magnitude and variability of the forecast supply and demand categories. Since most of the analysis in this article is conducted using percent changes in variables and percent errors, unit levels are presented here as a reference point for the relative size of the various categories. On the unit level ( ), food, seed, and industrial (FSI hereafter) is a highly variable category, which likely reflects sharp changes associated with the use of corn for ethanol production, driven both by policy changes and the price of gasoline. Percent forecast changes ( ) provide another indication of relative volatility in forecasted categories, highlighting variability in year-to-year changes in all ending stocks and wheat feed and residual forecasts. Since wheat is relatively expensive, its use for feed is residual to that of corn, thus making its feed and residual use a relatively small and highly variable category. [INSERT TABLE 1 HERE] Across all commodities, ending stocks is a highly variable category. Given the balance sheet nature of WASDE forecasts, ending stocks reflect the difference between all supply and demand categories, which means that forecast errors in other categories would be carried over to ending stocks errors. Fig. 2 illustrates the correlations between errors in endings stocks and errors in supply and demand categories. Since ending stocks are calculated as supply less total use, we observe a positive correlation between ending stocks errors and production errors and mostly negative correlation between ending stocks errors and use category errors. The magnitude of the correlations suggests that across all three commodities, production forecast errors are the main sources of error in ending stocks in the beginning of the forecasting cycle. 8

10 For wheat, errors in seed and feed and residual use are also highly correlated with ending stocks forecast errors early in the forecasting cycle. As the forecasting cycle progresses and production uncertainty becomes resolved, other categories become important sources of error in ending stocks, such as exports and feed and residual for corn, exports and crushings for soybeans, and exports and feed and residual for wheat. These illustrations suggest that better production and export forecasts would help improving ending stocks forecasts for these commodities. [INSERT FIGURE 2 HERE] Unlike the above statistics that focused on the final values ( 19), error statistics shown in the last three columns of Table 1 are calculated for the first 18 forecasts ( 1,,18). Mean absolute percent errors (MAPE) illustrate the size of forecast errors, reflecting large errors for highly variable categories described above: all ending stocks, soybean seed and residual, and wheat feed and residual forecasts. Mean percent errors (MPE) illustrate the direction of errors and the presence of bias in the forecasts. Based on our forecast error calculation in Equation 3, positive error indicates underestimation and negative error reflects overestimation in the USDA forecasts. Significant underestimation is detected in corn price, soybean crushings and price, and wheat exports and price, while overestimation is present in soybean seed and residual and ending stocks forecasts and wheat seed and feed and residual forecasts. Fig. 3 illustrates how these forecasts biases changed over time. For example, overestimation in soybean ending stocks forecasts was prevalent from 1994/95 through 2004/05 marketing years and less common in previous marketing years. This figure also shows lack of trend in errors over time, suggesting that forecasts did not become better or worse. However, there is some evidence of larger price forecast errors across three commodities as well as production and export forecast errors in wheat, export and ending stocks errors in soybeans, and 9

11 ending stocks errors in corn since the mid-2000s. This pattern reflects challenges USDA had with taking into account structural changes in these markets that occurred around 2006, due to emerging linkages between crop and energy markets (e.g., Mallory, Irwin, and Hayes, 2012) that can be traced to high crude oil prices and policy incentives, such as that provided by the 2005 Energy Policy and Renewable Fuel Act (RFA) and 2007 Energy Independence and Security Act (EISA). In the remainder of this study, we attempt to explain the causes of errors in WASDE forecasts, focusing on both their size, as measured by the absolute percent error (APE), and direction, as measured by percent error (PE). [INSERT FIGURE 3 HERE] III. Sources of Error Behavioral sources of error In this study we hypothesize that errors in USDA forecasts stem from two general sources. The first source of errors deals with forecasters characteristics. Previous studies suggest that forecasting behaviour may be different with respect to predicting positive versus negative change (Dreman and Berry, 1995; Ashiya, 2003) and extreme versus moderate rate of change (Denrell and Fang, 2010). Amir and Ganzach (1998) argue that leniency, representativeness, and anchoring are the main reasons for forecasters bias. Leniency describes a tendency for overoptimistic predictions, resulting in overestimating positive and underestimating negative changes as found in earnings forecasts (e.g., Givoly and Lakonishok, 1984; Schipper, 1991; Amir and Ganzach, 1998). Amir and Ganzach (1998) suggest that one of the possible reasons for leniency in earnings forecasts is the analysts desire to maintain good relations with management as a primary source of information. In the context of public commodity forecasts, however, this 10

12 argument does not hold as the information contained in these forecasts has a more two-sided effect, i.e., producers favor forecasts that increase prices, while users favor just the opposite. Thus, an asymmetric reaction to positive versus negative change in WASDE forecasts would likely be due to mis-calibration of forecasters reaction to information and can result in optimism as well as pessimism. Given the public nature of the forecasts and their ability to affect markets, these biases should be unintended. The representativeness heuristic leads to overreaction, causing forecasters to overpredict change (both positive and negative). Furthermore, Kahneman and Tversky (1973) argue that when using this heuristic, people choose a prediction value which reflects the extremity of the predicted information, i.e. larger overestimation is associated with extreme versus moderate rate of change, as observed by Denrell and Fang (2010). Contrary to representativeness, the anchoring heuristic results in underreaction. This heuristic explains situations when forecasters anchor their predictions at a previous value and adjust based on additional information. Since adjustment is typically insufficient, forecasts based on this heuristic are often too moderate (underestimation) with respect to both positive and negative changes (Slovic and Lichtenstein, 1971; Kahneman and Tversky, 1973). These behavioural sources of forecast errors are modelled in this study with three variables: lagged forecast errors ( is the previous year s error for the same report month), percent change in forecast level ( ), as described in Equations 1 and 2, and percent change in forecast level multiplied by a negative change indicator ( 1 if 0, 0 otherwise). For the analysis of the size of errors (APE), representativeness would cause larger errors associated with larger forecasted change and negative correlation with past errors. Anchoring would result in an 11

13 opposite pattern. Leniency (and pessimism) would lead to larger errors associated with positive or negative change (significant coefficient on the negative change indicator). For the direction of error (PE) analysis, evaluation of the behavioural sources of error is consistent with traditional efficiency tests that evaluate whether forecast errors are orthogonal to the forecasts themselves as well as to prior forecast errors (Mincer and Zarnowitz, 1969; Nordhaus, 1987; Holden and Peel, 1990). If forecasts are efficient, there should be no relationship between forecast errors and these variables; i.e., the null hypothesis is all coefficients are zero. Correlation with past errors indicates a systematic component in forecast errors that can be predicted using lagged errors, i.e. tendency to repeat or overcorrect past errors. Anchoring would result in forecasters repeating previous errors while representativeness would lead to overcorrection of errors. Positive correlation with forecast level indicates that the absolute value of the forecast is smaller than the actual realization, underestimation of change in either direction, indicative of anchoring. Negative correlation means the change is overestimated, suggesting representativeness in predictions. When interacted with the negative change indicator, this variable describes the differences in forecasting positive versus negative change reflecting leniency or pessimism. Macroeconomic sources of error The second source of errors considered in this study is due to macroeconomic information that has not been fully incorporated in the forecasts. Specifically, we investigate whether USDA forecasts tend to show mistakes during certain economic conditions. Vogel and Bange (1999, p.14) note that USDA forecasting process includes information on such diverse factors as exchange rates, oil prices, the effects of domestic and foreign agricultural policy and economic growth. These macroeconomic sources of forecast errors are represented in this study within 12

14 five variables described in Table 2. Exchange rates are measured using the trade weighted exchange index (FX), which represents a weighted average of the foreign exchange value of the US dollar against the currencies of a broad group of major US trading partners. 4 A stronger dollar (as manifested by an increase in FX index) would cause a drop in exports also affecting domestic use and price. Oil prices affect multiple levels of input costs, from fertilizer (often derived from petroleum products) to transportation. Higher input costs would lead to contraction in production, which in combination with higher transportation costs would put an upward pressure on prices. Oil prices also affect corn use categories due to the use of corn for ethanol production. The Spot price of West Texas Intermediate (WTI) crude oil at Cushing, Oklahoma location is used to measure this effect. The impact of economic growth and business cycles on WASDE forecasts is measured by Industrial Production Index (IPI, 2007 base) following Ludvigson and Ng (2005). As increases in industrial production illustrate periods of economic expansion, we expect consumption or use to increase during these times putting an upward pressure on prices. 5 Inflation affects prices paid and received by farmers as well as expectations about future prices and storage decisions. Producer Price Index for farm products (PPI, 1982 base) is used to measure the impact of inflation on WASDE forecast errors. Because changes in ending stocks and prices are affected by all supply and use categories, the macroeconomic sources of errors described above may also have an effect on stocks and prices. The impact of structural changes in the grain and oilseed markets that occurred around 2006 (described in the 4 Broad currency index includes the Euro area, Canada, Japan, Mexico, China, United Kingdom, Taiwan, Korea, Singapore, Hong Kong, Malaysia, Brazil, Switzerland, Thailand, Philippines, Australia, Indonesia, India, Israel, Saudi Arabia, Russia, Sweden, Argentina, Venezuela, Chile and Colombia. For more information about tradeweighted indexes see 5 Alternative specifications also included short term interest rate, which was dropped from the final estimation due to the high correlation with Industrial Production Index. 13

15 previous section) is modelled with a post-2006 dummy. Our analysis will reveal whether USDA forecast errors tend to grow during certain economic conditions and in what direction. [INSERT TABLE 2 HERE] IV. Estimation Method Consistent with the dual objectives of this study, absolute percent forecast errors are used as dependent variables in regression analysis to measure the impact of considered factors on the size of the error and percent errors are used to investigate sources of bias or direction in forecast errors. The independent variables discussed in the previous section are included in the analysis of errors of all US corn, soybean, and wheat forecasts from 1987/88 through 2009/10 marketing years. For each crop for price and ending stocks, the following regressions are estimated: 2006 (4) and where are the forecasting cycle stages as defined in the WASDE forecasts section, (5) is the previous year s error for the same report month, is percent change in forecast level, as described in Equations 1 and 2, and is a negative change indicator ( 1 if j f t < 0, 0 otherwise). (Producer Price Index), (Industrial Production Index), (monthly average cash oil price), and (exchange rate index) are macroeconomic variables and 14

16 2006 is a structural change dummy variable described in Table 2. Regressions for other categories have a similar format, but include not all but only relevant macroeconomic variables as discussed earlier. Because commodity forecasts in this study are measured as percent changes in forecasted values from the previous year s values, macroeconomic variables are measured as percent changes from the previous year as well using: 100, 1,,23; 1,, (6) where is the value of the macro variable in WASDE forecast month 1 for marketing year and for corn and soybeans (wheat). Similarly, represents the value of that macro variable for the same calendar month but in the previous year. Since macro variables for the forecast month are not known when those forecasts are made, the values of the macro variables from the previous forecast month, 1, are used. All equations are estimated using Ordinary Least Squares (OLS) method. However, as shown by Beck and Katz (1995) the OLS standard errors of the estimated parameters are incorrect when the time-series, cross-section data are characterized by having repeated observations on fixed units. In order to correct for the correlation among the forecast months in a given marketing year and the correlation between the observations on the same calendar day, we estimate the variance-covariance matrix as in Karali and Thurman (2009). Specifically, for a given crop we first run OLS regressions for each balance sheet category and obtain residuals. For production forecast errors, we only use the first eight report months as they are completed early during the marketing year. The residuals from the regression of production forecast errors are correlated across report months due to the fact that each monthly report is forecasting the same terminal event. We compute variances of residuals for each report month as the sample mean of the squared residuals across all marketing years as: 15

17 , 23; 1,, 8 for corn and soybeans; 1,, 5 for wheat (7) where is the residual of report month in marketing year. Then, covariances between two report months are computed as the sample mean of the products of related residuals:, 23; ;, 1,, 8 for corn and soybeans;, 1,, 5 for wheat (8) For all other balance sheet categories, the correlation structure is more complicated. This is because of the overlapping forecasts released in the same report month. These forecasts are released on the same calendar day but correspond to two different marketing years since the forecasting cycle is longer than 12 months. However, because the information available is the same when these forecasts are made, these observations would be correlated. Thus, in addition to correlation between report months for a given marketing year, there also exists correlation between the forecasts of two consecutive marketing years that are released on the same calendar day. For instance, the 13 th forecast figures for the marketing year 1987/1988 are released in May In the same WASDE report, the first forecast figures for the marketing year 1988/1989 are also released. Thus, the forecast errors of report months 13 through 18 (16) for marketing year overlap with the forecast errors of report months 1 through 6 (4) for marketing year 1 for corn and soybeans (wheat). Hence there are 6 (4) overlapping forecast errors in a marketing year for corn and soybeans (wheat), resulting in a total of 132 (88) overlapping observations. 6 To correct for this contemporaneous correlation we compute the covariance between the same-day observations as:,, 1, 23; 132; 1,, 6 for corn and soybeans; 88; 1,, 4 for wheat (9) 6 For marketing year 23, forecast errors do not overlap with the forecast errors in the next marketing year. Thus, we have a total of 22x6=132 overlapping observations for corn and soybeans, and 22x4=88 for wheat. 16

18 Variances of the residuals for each report month and covariances between the residuals of two report months in a given marketing year are computed as before: 1, 1, 23; ;, 1,, 18 for corn and soybeans;, 1,,16 for wheat (10) Using these estimates we construct the variance-covariance matrix of OLS residuals and use it to compute standard errors of OLS parameter estimates. We present t-statistics computed using these standard errors along with OLS coefficient estimates in our result tables. V. Results Size of errors Factors affecting the size of errors of WASDE corn, soybean, and wheat forecasts are shown in Tables 3-5. Results are discussed in both percentage terms and physical units. For all balance sheet categories, coefficients of stage 1, 2, and 3 variables illustrate the reduction in the size of forecast errors across the forecasting cycle as more information becomes available. For example, Table 3 demonstrates that stage 1 corn production forecast errors are on average about 4% (397 million bushels) larger than stage 3 errors. Errors in all other categories are interpreted relative to stage 4 errors, thus stage 1wheat ending stocks forecast errors are about 25% (161 million bushels) larger than stage 4 errors. The largest reduction in the size of errors during the forecasting cycle is detected in corn ending stocks and exports forecasts, soybean ending stocks and seed and residual forecasts, and wheat feed and residual and ending stocks forecasts. [INSERT TABLE 3 HERE] 17

19 The impact of the behavioral sources of error is captured through the size of lagged error, forecast size and direction. The most common behavioural source affecting the size of errors in corn forecasts (Table 3) is the positive correlation between the size of the current error and last year s error, revealing a pattern of large errors being followed by large errors. This pattern is consistent with anchoring and may be indicative of challenges with forecasting structural change, particularly when anchoring one s forecast on previous information provides consistently erroneous results. This pattern is detected in corn production and price forecasts. Thus, a 1% (2.5 cents/bu.) error in the June corn price forecast in year is typically followed by a 0.16% (0.40 cents/bu.) error in the June forecast the following year. A similar persistence of errors of even larger magnitude is detected in wheat ending stocks and price forecast errors (Table 5). Correlation with previous forecast errors in soybeans (Table 4) is only found in one instance, crushings, and is negative (0.11% or 1.6 million bushels), suggesting that large errors are typically followed by smaller errors. [INSERT TABLE 4 HERE] The size of soybean forecast errors (Table 4) is most commonly associated with the size and the direction of the forecast. Thus, the magnitude of forecast error increases by 0.2% (505 thousand bushels) when soybean ending stocks are forecasted to increase and decrease by about 0.12% ( = ) or 309 thousand bushels with each percent decrease in forecasted stocks. A similar pattern is found in soybean seed and residual forecasts and wheat feed and residual forecasts (Table 5). The opposite pattern is observed in corn price and soybean production forecasts. Soybean production forecast error decreases by 0.07% (1.9 million bushels) when production is forecasted to increase and increases by 0.06% ( = 0.062) or 1.5 million bushels with each percent decrease in forecasted production. In addition, 18

20 evidence of smaller errors associated with increasing wheat production and declining ending stocks is also found. These patterns of asymmetric errors associated with positive versus negative forecasted change are consistent with both leniency and pessimism. Further investigation in the direction of errors presented later in this study is required to disentangle whether leniency or pessimism is causing these errors. [INSERT TABLE 5 HERE] Macroeconomic sources of forecast errors are reflected in the last five variables included in regression results presented in Tables 3-5. Results demonstrate that errors in price forecasts across all three commodities increase by about 0.1% (0.29 cents/bu. for corn, 0.70 cents/bu. for soybeans, and 0.40 cents/bu. for wheat) with each percent growth in inflation (PPI). Periods of economic growth (as reflected in the increase in IPI) are accompanied by larger errors in corn feed and residual (0.28% or 15 million bushels) and price (0.18% or 0.45 cents/bu.) forecasts as well as wheat feed and residual forecasts (2.08% or 4.8 million bushels). On the contrary, errors in soybean ending stocks forecasts decrease by 1.4% (3.5 million bushels), with each percent increase in IPI. Thus, wheat feed and residual and soybean ending stocks forecast errors are the most sensitive to changes in economic growth. Changes in oil prices tend to increase forecast errors mostly during periods of falling oil prices. During the period of study, each percent decline in oil price is accompanied by an increase in forecast errors of corn production (0.03% or 2.7 million bushels) and FSI use (0.01% or 304 thousand bushels) as well as wheat production (0.12% or 2.6 million bushels) and price (0.04% or 0.15 cents/bu.). Soybean price forecasts also exhibit 0.03% (0.21 cents/bu.) larger errors as oil price declines, but soybean ending stocks forecast errors actually decline by 0.14% 19

21 (347 thousand bushels) during these periods. These results demonstrate that the effects of changes in oil prices on forecast errors are mostly negative and small in magnitude. Changes in exchange rates are also mostly negatively correlated with the size of forecast errors, but the magnitude of these effects is substantially larger. The largest effects are observed in wheat, as wheat exports and feed and residual forecast errors increase by 0.21% (2.3 million thousand bushels) and 0.94% (2.2 million bushels), respectively, while seed use forecast errors decrease by 0.13% (112 thousand bushels), with each percent decrease in FX index. Errors of soybean price forecasts increase by 0.35% (2.25 cent/bu.), as US dollar weakens against other currencies. Corn FSI forecast errors also decrease by 0.13% (3 million bushels), but corn export forecast errors increase by 0.30% (5.6 million bushels) with weaker US dollar. The last macroeconomic variable, a post-2006 dummy, describes changes in forecast errors that are likely associated with challenges in forecasting structural changes in these markets that took place in the mid-2000s. The largest increase in errors in post-2006 period is observed in the residual categories across all three commodities with errors in wheat feed and residual forecasts growing by 36.3% (84 million bushels), soybean seed and residual by 16.5% (22 million bushels), and corn feed and residual by 2.5% (133 million bushels), making it the largest source of forecast errors in these categories outside of uncertainty associated with early forecasts. Wheat production and wheat and corn price forecast errors increased by 6.19% (136 million bushels), 5.67% (20.83 cents/bu.) and 4.09% (10.29 cents/bu.), respectively, and soybean exports forecast errors also grew by 2.12% (19 million bushels) during period. These results provide quantitative evidence for the trends observed in Fig. 2 and discussed in recent media as quoted in the introduction. 20

22 Direction of errors Evaluation of the impact of considered factors on the direction of forecast errors is consistent with traditional efficiency tests. In the direction of error results presented in Tables 6-8, stage coefficients are interpreted as changes in bias across the forecasting cycle and can be compared to the average bias reported in Table 1. Both Tables 1 and 6 indicate that corn forecasts are mostly bias free. While underestimation of corn price is detected on average across all stages of forecasting cycle, it is not statistically different from zero in individual stages. On the other hand, multiple evidence of bias is detected in soybean forecasts. For example, while Table 1 reports average bias for soybean price forecasts as 2.18%, Table 7 indicates that USDA forecasts underestimate soybean price by 5.8% (37 cents/bu.) in stage 1 forecasts and 3.4% (22 cents/bu.) in stage 2. Underestimation in soybean exports and crushings appears most pronounced in stage 2 and 3 forecasts, while overestimation of soybean ending stocks is present across the first three stages. According to Table 8, stage 2 wheat forecasts are most affected by bias with underestimation detected in exports and price, and overestimation in feed and residual and seed forecasts. Other stages are mostly bias free with the exception of exports that are underestimated in the first stage as well. [INSERT TABLE 6 HERE] With respect to behavioural sources, the direction of error results suggest that USDA corn and soybean analysts have a tendency to overcorrect previous year s errors as suggested by the negative correlation with past errors in corn production, ending stocks, and feed and residual forecasts and soybean exports and price forecasts. Thus, a 1% (94 million bushels) overestimation in corn production in year is followed by a 0.28% (26 million bushels) underestimation in year 1 and 1% (6.4 cents/bu.) overestimation in soybean price is followed 21

23 by a 2.4% (1.5 cents/bu.) underestimation the following year. This pattern combined with the evidence of persistent errors found in the case of corn production errors (Table 3) suggests that while large errors in corn production are followed by large errors, these errors are made in opposite directions, indicative of overreaction or representativeness heuristic. On the other hand, persistence of errors in wheat forecasts is consistent across the size of error and direction of error results (Tables 5 and 8). Positive correlation with past errors is detected in wheat production, seed, ending stocks, and price forecasts. Thus, a 1% (3.7 cents/bu.) underestimation of wheat price in year is followed by a 0.63% (2.3 cents/bu.) underestimation in year 1, suggesting a tendency for repeating past errors, and indicative of anchoring. [INSERT TABLE 7 HERE] The direction of error results allows us to discern whether asymmetric errors associated with positive versus negative forecasted change are caused by leniency or pessimism. A tendency to overestimate growth in corn FSI use by 0.41% (9.6 million bushels) and underestimate contraction by 0.34% ( = 0.342) or 7.9 million bushels is indicative of leniency. Some evidence of overestimation of growth is also detected in soybean seed and residual, ending stocks, and price forecasts and in wheat food, feed and residual, and price forecasts. On the other hand, underestimation of growth by 0.20% (3 million bushels) and overestimation of contraction by 0.11% (1.6 million bushels) in soybean crushings is indicative of pessimism. Overestimation of contraction is also found in wheat exports and ending stocks forecasts. Overall, the patterns of leniency and pessimism are mixed across different categories and do not exhibit consistent evidence of strategic behaviour that would lead to over- or underpredicting prices (i.e., optimism with respect to use and pessimism with respect to production and ending stocks would lead to over-predicting prices and vice versa). 22

24 [INSERT TABLE 8 HERE] With respect to macroeconomic sources of error, forecast rationality implies that if all relevant information is incorporated in the forecasts, forecast errors should be unpredictable using the information available at the time the forecasts are made. Thus, if USDA forecasts efficiently incorporate macroeconomic information, coefficients of the macro variables in direction of error regressions should be zero. Our analysis indicates that corn forecasts efficiently incorporate information about inflation, while soybean and wheat forecasts do not. In soybean forecasts, each percent growth in inflation is associated with overestimation of 0.42% (1 million bushels) in ending stocks, and underestimation of 0.13% (84 cents/bu.) in price forecasts. In wheat forecasts, higher inflation is associated with overestimation of 0.42% (2.8 million bushels) in ending stocks, and underestimation of 0.16% (0.57 cents/bu.) in price forecasts. Information about economic growth as reflected in changes in the IPI is associated with overestimation of 0.25% (13 million bushels) and 0.14% (3.3 million bushels) in corn feed and residual and FSI forecasts, respectively. On the other hand, each percent growth in IPI leads to underestimation of 0.79% (1 million bushels), 1.15% (2.9 million bushels) and 0.13% (1 million bushels) in soybean seed and residual, ending stocks and wheat food use, respectively. Thus, the impact of economic growth is overstated in corn and understated in soybean and wheat forecasts. Higher oil prices are associated with underestimation in corn feed and residual (0.04% or 2.2 million bushels), and overestimation in soybean price (0.05% or 0.34 cents/bu.) and wheat production (0.11% or 2.3 million bushels) forecasts. Consistent with the size of error results, the magnitude of this inefficiency is very small. 23

25 A stronger US dollar appears to cause overestimation in price forecasts of 0.31% for corn (0.77 cents/bu.) and wheat (1.15 cents/bu.) and 0.49% for soybeans (3.15 cents/bu.). Overestimation during the periods of stronger US dollar is also observed in corn FSI use and wheat seed and export use. Most other categories including soybean seed and residual, and crushings as well as wheat food use, feed and residual, and ending stocks exhibit underestimation as US dollar strengthens. Consistent with the size of error results, structural changes captured by the post-2006 dummy variable have the largest effect on forecast errors. In corn forecasts, larger errors in post period are associated with underestimation in ending stocks by 18.2% (291 million bushels), FSI by 3.2% (75 million bushels), and overestimation in feed and residual forecasts by 3% (157 million bushels). In soybeans, the post-2006 period is characterized with underestimation in exports by 6% (54.7 million bushels) and overestimation in seed and residual use forecasts by 21% (27.5 million bushels). Since 2006, the underestimation in wheat ending stocks is 9.6% (63 million bushels) and overestimation in wheat feed and residual forecasts is 41% (95.1 million bushels) are observed. VI. Summary and Conclusions This article seeks to examine USDA corn, soybean, and wheat forecast errors from 1987/88 through 2009/10 marketing years to better understand when forecasters make mistakes. We hypothesize that errors may stem from two general sources: behavioural and macroeconomic factors. We find that the largest increase in the size of USDA forecast errors is associated with structural changes in commodity markets that took place in the mid-2000s. Post-2006, corn price and feed and residual forecast errors grew by 4.1% and 2.5%, respectively; soybean seed and 24

26 residual and export forecast errors grew by 16.5% and 2.1%, respectively; and wheat price, production, and feed and residual errors grew by 5.7%, 6.2%, and 36.3%, respectively. Corn, soybean, and wheat forecast errors also grew during the periods of economic growth and with changes in exchange rates, while inflation and changes in oil price had a much smaller impact. Changes in economic growth had the largest impact on wheat feed and residual and soybean ending stocks forecasts. Wheat forecast errors were also the most sensitive to changes in exchange rates, likely due to substantial export orientation of the wheat market. With respect to behavioural sources, several forecasts in all three commodities demonstrated persistence in errors (i.e. large errors are followed by large errors) which is consistent with anchoring and may be indicative of challenges with forecasting structural change. Our investigation of the direction of forecast errors revealed that asymmetric errors associated with positive versus negative forecasted change in corn FSI use is indicative of leniency. On the other hand, underestimation of growth and overestimation of contraction in soybean crushings forecasts is indicative of pessimism. Overall, the patterns of leniency and pessimism were mixed across different categories and do not exhibit consistent evidence of strategic behaviour that would have led to over- or under- predicting prices. Multiple cases of inefficiency with respect to macroeconomic factors were also found in the direction of error results. Our analysis revealed that corn forecasts efficiently incorporated information about inflation, while soybean and wheat forecasts did not. The impact of economic growth was overstated in corn and wheat forecasts and understated in soybean forecasts. A stronger US dollar was associated with overestimation in price forecasts and both over- and underestimation in consumption categories, while changes in oil prices had a very limited impact. 25

27 Fildes and Stekler (2002, p. 454) argue that The use of rationality tests, especially those that relate the forecast errors to known information, can be viewed as important diagnostic checks to determine why the errors occurred and to improve the forecasting process and the quality of subsequent predictions. Our findings can thus be used by USDA forecast providers to improve their forecasts. For example, if our estimates were used to adjust May 2010 corn Food, Seed, and Industrial forecast for the 2010/11 marketing year, the forecast adjustment factor would be calculated by multiplying values of macro and behavioral variables known at the time by the significant inefficiency coefficients shown in Table 6. Given the values of the variables (measured as percent change from previous year s values) known in the beginning of May 2010 of for forecast size, for IPI, and for FX, the adjustment factor is * * *(-7.637) = When this adjustment factor is added to the published forecast of (5,950 million bushels), the adjusted forecast is (6,247 million bushels). Since the final estimate of corn Food, Seed, and Industrial forecast for 2010/11 marketing year was 6,428 million bushels, this adjustment reduced the May forecast error from 478 million bushels to181 million bushels. The findings of this study can also help market participants to interpret USDA information. If market participants are fully aware of the flaws and inefficiencies in USDA forecasts and adjust for them in their decision making process, limited or no economic losses would result (Orazem and Falk, 1989). The degree to which market participants anticipate and adjust information contained in USDA forecasts presents an interesting area for future research. 26