In this project I will analyze a time series of the number of deaths that occurred during the Vietnam War.

Transcription

1 Introduction[1] In this project I will analyze a time series of the number of deaths that occurred during the Vietnam War. The Vietnam War was perhaps the United States gravest mistake in foreign policy in the 20th century. The Vietnam War first began as a result of the Viet Minh s failure to capture South Vietnam in the First Indochina War in In 1954, the French, who occupied the Vietnam region, were defeated at Dienbienphu. The United States became involved in Vietnam politics during the discussion of the Geneva Agreements during which a provisional boundary was drawn to separate North Vietnam from South Vietnam until nationwide elections could be held in The United States does not accept this agreement. In 1955, president-to-be Diem rejects the Geneva Accords and refuses nationwide elections. During the same year, France withdraws from Vietnam. During the late 1950s the Communists begin an insurgency in South Vietnam and weapons begin moving along the Ho Chi Minh Trail to South Vietnam and in 1960 the Vietcong are organized. The most tragic part of the war was the Tet Offensive which occurred in U.S. forces were not prepared for the North Vietnamese and Vietcong s battle, but were able to succeed in recapturing most areas of the country. This is the first point where the U.S. military s assessment of the war is questioned by the public. After the war had ended, 58,000 American troops had died. My initial plan was to analyze a time series of deaths in the Iraq War, however, the sample size would have (fortunately) been too small, there was nothing interesting about it, and it appeared to have been fit by an MA(0) model, so I decided to use an (unfortunately) larger dataset. I was hoping to perform a statistical significance test on the trends of deaths in the Vietnam War as well as the Iraq War at its current stage to get a feel for how similar the trends are up to this point, but I do not have experience in multivariate time series analysis. That idea was motivated by the sentiment that the Iraq War is becoming like the Vietnam War. Data Collection The data are originally come from the Vietnam Veterans Memorial Fund website [2]. The method by which a researcher can extract information is based on a search engine that allows visitors to search by various fields, including date of death. The search results returned include the number of records returned for that search. Since I could not find a pre-existing dataset, I wrote a Python script to query the search engine and gather the number of war dead from the search results. I must state as a disclaimer that this database only contains soldiers killed that were identified. I am sure there are others that are not included in this 1

2 database, but I believe that if there is a trend, it will remain the same. The database begins at January 1957 and ends in December PBS[1] states that the last conflict occurred in May 1975, thus I collect data from January 1960 to December However, it also states that for the United States, the war is officially over in March 1973, thus I only consider data until August Despite the events leading up to the war beginning in the mid-1940s, I am really only interested in the war and the escalation of the war, so I start data collection at January The Python code is listed in Appendix B. Analysis The plot of the raw time series and its correlogram is exhibited in figures 1 and 2 respectively. The correlogram shows that the ACF converges to zero very, very slowly, and thus the time series is not stationary. The x axis is represented in lags, or time distance roughly speaking. x = 4 represents the fourth lag or a difference of four months. The y axis is the value of the autocorrelation function (ACF) at lag x = h, γ(h). At first glance of the original time series, there appears to be a linear trend up until about January From 1966, leading up to the Tet Offensive and after, deaths increased dramatically until June 1968 (month 101) and then declined until the end of the war. On closer look, it appears that fatalities were in an upward trend even before the Tet Offensive, and the trend became curvillinear after about December 1964 (month 60/year 5). Fitting a quadratic to the time series before 1966 yields results that indicate there are three separate phases in this time series. These results are plotted in figure 3. The quadratic does not fit well for data points from early in the time series (early 1960s). The residuals plot, which shows the signed difference between the trend curve and actual data points, clearly shows the lack of fit early in the time series. The superimposed dotted line suggests that the first phase of the time series ends after December 1961 (2 years). As was calculated earlier, the end of the second phase occurs after December 1965 (month 60). For brevity, I will refer to phase I as early war, phase II as escalation, and phase III as chaos. Before fitting a model to my time series, I must remove the trend. Figure 4 shows the original time series with dotted lines indicating the partitioning of the time series into three phases. Phase I: Early War Figure 5 shows the subset of the time series corresponding to phase I. The x axis is simply the number of months elapsed since January Figure 6 displays the correlogram for this subset. The autocorrelation 2

3 function is within the bounds of white noise for h = 1,..., 25 which implies that the series is stationary. Stationarity implies that the mean function is not dependent on time. This makes my job as the researcher easy since I have no trend or seasonality to remove. Phase II: Escalation Phase II consists of the subset of the time series containing data from January 1961 to December The plot clearly seems to follow a parabolic curve, so I fit a quadratic to the phase II subset time series. Figure 7 shows phase II and its quadratic fit as well as the residual plot for the fit. After removing the quadratic trend, the correlogram (figure 8) shows that the ACF decays to 0 very quickly (it is constantly within the white noise bounds), so the detrended series for phase II is stationary. Phase III: Chaos A plot of the phase III subset of the time series in figure 9 indicates a very interesting (and tragic) trend. It appears that the deadliest month of the war was June The hill-like structure of the trends of deaths from 1965 to 1973 appears to be symmetric about the deadliest month. If one looks closely at the scatterplot of the trend before June 1968 and the trend after June 1968, it looks fairly cubic. I could either split phase III into two parts and fit each part with a cubic, or attempt to fit an order 6 polynomial to this data. I will choose to fit the trend using an order 6 polynomial. The residuals from the order 6 polynomial do not look like white noise, and seem to possibly have some periodicity (figure 10), particularly the three high spikes in the center of the plot. There also seems to be problem with non-constant variance. The sixth order fit is the best I can do. Unfortunately, it underestimates the peak in the summer of Additionally, the correlogram in figure 11 decays slowly, although it does decay to 0 in 25 lags. In figure 12 I display the plot of the spectral density estimates. The x axis represents the number of cycles per year a peak at that point would imply of the periodicity of the time series. The y axis is the spectral density estimate of the residuals from the order 6 fit. The sharp peak at 1, which is outside of the bounds for white noise, implies that there is indeed periodicity in phase III, and that there is one cycle/period per year. The seasonal decomposition of the phase III time series by loess is displayed in figure 13. The first row of the plot is the original phase III time series which is followed by an illustration of the seasonal trend in the data. The third row displays the overall trend in the dataset, that I originally modeled with an order 6 polynomial. The final row is the resulting time series that I want to extract (the remainder). A magnification of one period is displayed in figure 14. This plot indicates that if the seasonality 3

4 component is detected correctly, what we consider to be the winter and spring months (January through June) in the United States was a deadlier time for troops in the war than was the rest of the year (July through December). There is an interesting dip in April that I cannot explain. Perhaps this is just due to random variation. Recall that the time series for phase III has been detrended, so the y axis represents the expected residual from detrending, not the raw number of deaths. The resulting time series is detrended and seasonality removed. The final correlogram in figure 15 shows that the ACF decays to zero very quickly, but there is a problem at lag 5 because the autocorrelation function drops out of the bounds for white noise, however, this only occurs once and may be related to having a strange dataset or an imperfect fit from the order 6 polynomial as well as perhaps not removing some other trend or periodicity that was present that I could not uncover. Figure 16 shows a relatively smooth and somewhat flatter periodogram which suggests that I have removed the periodicity, but also confirms that after detrending and removing the seasonality component, the remainder is not white noise. Putting it All Together Now that my entire time series has been detrended (phase II and III) and seasonality removed (phase III), I can investigate a model to fit to the time series. The correlogram in terms of the autocorrelation function clearly tails off to zero as lag in months increases (figure 17). The correlogram in terms of the partial autocorrelation function (figure 18) is somewhat more dubious. Although it appears that the PACF cuts off immediately, one can see that the variability of the PACF seems to decrease over time and seems to pop out of the bounds for white noise occasionally. From both of these revelations, I believe that I can say that the PACF tails off as the lag increases. Thus, the proper model to fit to this time series is an ARMA(p, q). After running a loop that searches for the best parameters of p and q, I found that the model that fits according to the Ljung-Box method with minimal AIC = , is ARIMA(8,0,4). Figure 19 displays the time series diagnostics for the ARMA(8,0,4) model in R. The last row of the plot shows the results of the hypothesis tests for the Ljung-Box test. Since all of the points are above the dotted line, we accept the null hypothesis H 0 that the residuals of the ARIMA fit are random[3]. The AR and MA coefficients for my ARIMA model are provided in appendix A. Since p = 8, this tells me that the value at time t depends on the values of the previous 8 time points. That is, assuming a perfect fit, the number of deaths in a particular month depends on a linear combination of number of deaths from the previous 8 months. Since q = 4, assuming a perfect fit, this means that my time series relies on the previous 4 white 4

5 noise terms. My coefficients also reveal some interesting results. Since φ 1 = is negative, this suggests that the previous time point is negatively related with the current time point which is not at all expected. This is a sign that there is possibly still some cyclic behavior in the time series. Based on magnitude alone, the time point two months before the current, x t 2, is positively related with the current time point x t. This may imply that x t 2 is a stronger predictor of x t than is x t 1 considering magnitude alone. That is, the number of deaths two months prior to the current month may be the best predictor of the death total for the current month. The closed form expression for my fit is x t = α+φ 1 x t 1 +φ 2 x t 2 +φ 3 x t 3 +φ 4 x t 4 +φ 5 x t 5 +φ 6 x t 6 +φ 7 x t 7 +φ 8 x t 8 +w t +θ 1 w t 1 +θ 2 w t 2 +θ 3 w t 3 +θ 4 w t 4 Limitations There are many limitations in this analysis, particularly in Phase III. I could not find a polynomial that best fit the time series for phase III. A loess fit produced worse results than a simple polynomial fit. The reason for this is that for a period of 2.5 years from month 90 to about month 120 (June 1967 to December 1969) there is an extreme amount of variability in the number of deaths that occurred from month to month in this period. This may indicate that the war was very volatile during this period, and it makes curve fitting incredibly difficult. A more sophisticated tool such as weighted least squares or splines may have been more appropriate, but I unfortunately do not understand the mechanics of those techniques yet. Due to this volatility, it is difficult to identify periodicity by eye, but I do suspect that it is there, and the periodogram confirmed my suspicion. I did not get good results from subtracting the seasonal component of phase III because of the volatility I believe. The result is that the residuals from removing seasonality are not white noise, and the periodogram displays spectral density estimates that are below the bounds for white noise. There may have also been other periodic trends or general trends that depended on other variables, such as the progress of the draft. Due to the problem with non-constant variance, it is probably not wise to fit an ARIMA model to this data if it is to be used for prediction. Since the dependent variable in this time series is the number of deaths of U.S. soldiers in a particular month, which is basically a count, it may be considered a Poisson random variable, so it may have been wise to transform the dependent variable by looking at the square root of the number of deaths per month before beginning this analysis. Perhaps a logarithmic transformation of the response may have also been better. 5

6 When analyzing war data, I believe that any researcher would begin by working with raw death counts as I did, perhaps because as humans we are biased against incidents that cause such tragedy. However, after I analyzed this data it occurred to me that the spikes in deaths are probably because there were more soldiers deployed in the Vietnam region during that time period, so naturally there would be more deaths assuming that the number of deaths is dependent on the number of soldiers in the region. A better method may be to use ratios. That is, divide the number of deaths in a month by the number of live soldiers deployed in the region on the first of the month. This ratio would serve as a death rate during the war and would probably yield much better results. Conclusion It would probably serve as a shock to anyone to see a visualization of this data as a time series. When I first plotted the data, I was astounded at how quickly the war took lives. The monthly trend of deaths in the Vietnam is not clear cut, but can be approximated if the war from 1960 to 1973 is considered as three separate phases. The first phase, which represents the early war, has little or no trend. The second phase is characterized by a steep quadratic trend that leads into the third phase which is chaos and can best be fitted with a sixth order polynomial. One sees that in the time vicinity of the deadliest month, June 1968, the number of deaths is very volatile which is represented in my plots as non-constant variance. One also sees that during this phase, there is a seasonal component, at least to the time vicinity of June After removing this trend and seasonality, and despite those results being less than satisfactory, I was able to fit an ARMA(8,4) model to the time series to represent it in a closed form. The number of deaths at a given time t seems to rely on the number of deaths that occurred during the previous eight months. It would be beneficial to instead look at the ratio of deaths to number of soldiers deployed, as well as some other auxiliary variables. It would also be a good idea for future researchers to delve deeper into the history of the foreign policy and military actions surrounding this tragic war, as that will perhaps shed more light on what can better explain the startling number of deaths. Let s hope that history does not repeat itself. 6

7 A Coefficients Coefficients for my ARIMA(8,0,4)/ARMA(8,4) model: Coefficient Estimate Standard Error ar1 φ ar2 φ ar3 φ ar4 φ ar5 φ ar6 φ ar7 φ ar8 φ α ma1 θ ma2 θ ma3 θ ma4 θ B Python Code for vietnam.py import urllib import re import time FILE = open("vietnam.dat","w") FILE.write("month\tdeaths\n") i = 0 for year in range(1960,1976,1): for month in range(1,13,1): page = " + str(month) + "&Casul_Date_Day=0" + \ "&Casul_Date_Year=" + str(year) + "&Submit=+++Submit+++" class AppURLopener(urllib.FancyURLopener): version = "Mozilla/5.0 (Windows: U; Windows NT 5.0; en-us; rv: ) " +\ "Gecko/ Firefox/ " urllib._urlopener = AppURLopener() handle = urllib.urlopen(page).read() pattern = re.compile("of\s(\d+)") m = re.search(pattern,handle) if m == None: deaths = 0 else: deaths = int(m.group(1)) i = i + 1 FILE.write(str(deaths) + \n ) time.sleep(3) 7

8 C Plots Vietnam War Deaths: U.S Number of U.S. Military Deaths Month # Starting with January 1, 1960 Figure 1: Plot of raw data showing war deaths a function of time, starting at January Correlogram of Raw Data autocorrelation lag (months) Figure 2: Correlogram of raw time series. 8

9 Deaths during Vietnam War period Residuals Plot Deaths Residuals Months Since January Fitted Values Figure 3: Deaths from January 1960 to December 1965 Vietnam War Deaths as Three Phases: U.S Number of U.S. Military Deaths Month # Starting with January 1, 1960 Figure 4: Original time series separated into three phases. 9

10 Vietnam War Deaths during Phase I: Deaths Months since January 1960 Figure 5: time series subset corresponding to Phase I. Correlogram of Phase 1, No Fit autocorrelation lags(months) Figure 6: Correlogram of subset of time series corresponding to Phase I. 10

11 Phase 2 Fit with Quadratic Residuals of Fit Deaths Residual Months since January Fitted Value Figure 7: Quadratic fit to Phase II time series and its residuals. Correlogram of Fit for Phase 2 autocorrelation lags(months) Figure 8: Correlogram of quadratic fit residuals for phase II. Vietnam War Deaths during Phase 3: 1968 through August 1973 Vietnam War Deaths during Phase 3: 1968 through August 1973 Deaths Deaths Months Since January Months Since January 1960 Figure 9: Time series plot and scatterplot of Phase III. 11

12 Residuals of Fit Residual Fitted Value Figure 10: Residuals from phase III after order 6 polynomial fit. 12

13 Correlogram of Fit for Phase 3 autocorrelation lags(months) Figure 11: Correlogram from residuals of order 6 polynomial fit to phase III. Periodogram of Residuals for Phase III Fit Spectral Density Estimate Frequency (cycles/yr) Figure 12: Danielle smoothed periodogram from residuals of order 6 polynomial fit to phase III subset of time series. 13

14 Seasonal Decomposition of Phase III Time Series remainder trend seasonal data time Figure 13: Seasonal decomposition of phase III time series. Seasonality Component of Phase III Deaths Expected Residual from Detrending Month of Year Figure 14: Seasonal component of phase III time series shows winter and spring as deadlier to U.S. troops. Correlogram of Detrended/Deseasoned Series, Phase 3 autocorrelation lags(months) Figure 15: Final correlogram of deseasoned/detrended phase III data. 14

15 Periodogram of Residuals for Phase III Fit Spectral Density Estimate Frequency (cycles/yr) Figure 16: Final periodogram Correlogram of Fit for Entire Time Series autocorrelation lags(months) Figure 17: Final correlogram of entire time series in terms of autocorrelation function, after detrending and removing seasonality from each phase. Correlogram of Fit for Entire Time Series autocorrelation lags(months) Figure 18: Final correlogram of entire time series in terms of partial autocorrelation function, after detrending and removing seasonality from each phase. 15

16 Standardized Residuals Time ACF of Residuals ACF Lag p values for Ljung Box statistic p value Figure 19: Diagnostic plot for my ARIMA(8,0,4)/ARMA(8,4) model. Only the p-values for the Ljung-Box statistic are important. lag 16

17 References [1] Battlefield Vietnam: Timeline [2] Vietnam War Memorial Fund: [3] Ljung-Box Statistic 17