ams-514-lec-10-m.nb 1

Transcription

1 ams--lec-0-m.nb Set Up << JLink` << Miscellaneous`Calendar` << Statistics`MultiDescriptiveStatistics` Downloading Data from a URL This function is based on the technical note on the Wolfram Research website:elli, Brendan, "Using Java to get Stock Data from the Internet into Mathematica", This function reads data from the specified URL, stores it in a temporary file and returns the name of the file for further processing. xgeturl@surl_stringd := JavaBlock@Module@ 8uBuffer, icount, sotmp, ourl, siurl<, InstallJava@ sotmp = OpenTemporary@DOSTextFormat -> False H* Create a temp file. *L ourl = JavaNew@"java.net.URL", surl H* $Failed if not valid. *L siurl = ourl ü openstream@ H* Create a Java input stream. *L If@siUrl === $Failed, Return@$FailedD ubuffer = JavaNew@"@B", 000 H* 000 is an arbitrary size. *L While@ H* Read data from the stream saving it to the temp file. *L HiCount = siurl ü read@ubufferdl > 0, WriteString@soTmp, FromCharacterCode@Take@Val@uBufferD, icountddd siurl ü close@ H* Close the stream. *L Close@soTmpD H* Close the temp file, which also returns the filename. *L D

2 ams--lec-0-m.nb Utility Functions Regularizing field names We want to be able to regularize field names by squeezing out spaces and extraneous (non-letter and non-digit) characters. xregularizename = StringJoin ü Pick@#, HLetterQ@#D»» DigitQ@#DL & êü #D & ü Characters@#D &; The Yahoo!Finance returns CSV data in which one one of the fields is "Adj. Close*". xregularizename@"adj. Close*"D AdjClose Date formatting The dates are returned as strings, e.g., {, 3, 7} is "7-Mar-06". The following set of functions convert these strings into Mathematica date vectors. The first converts 3-letter month names to integers. xmonthnametointeger@ss_stringd := Which@ ss ã "Jan",, ss ã "Feb",, ss ã "Mar", 3, ss ã "Apr",, ss ã "May",, ss ã "Jun", 6, ss ã "Jul", 7, ss ã "Aug", 8, ss ã "Sep", 9, ss ã "Oct", 0, ss ã "Nov",, ss ã "Dec",, True, $Failed These two functions convert a -digit year into a four digit year. The first makes a guest at what point to switch from 900 to 000. The second lets you specify the dividing line.

3 ams--lec-0-m.nb 3 xcompleteyear@iyear_integerd := Module@ 8iDivide<, idivide = Round@00 FractionalPart@First@Date@DD ê 00.D Which@ iyear 900, iyear, iyear 00, If@iYear idivide, iyear + 000, iyear + 900D, True, $Failed D xcompleteyear@iyear_integer, idivide_integerd := Which@ iyear 900, iyear, iyear 00, If@iYear idivide, iyear + 000, iyear + 900D, True, $Failed xcompleteyear@d 00 xcompleteyear@98d 998 xcompleteyear@30d 930 xcompleteyear@30, 0D 030 This function ties it together. xyahoodatetomathematicadate@ss_stringd := Module@ 8vsS<, vss = StringSplit@sS, "-" 8xCompleteYear@ToExpression@vsSP3TDD, xmonthnametointeger@vssptd, ToExpression@vsSPTD<

4 ams--lec-0-m.nb Here's a demonstration. xyahoodatetomathematicadate@"7-mar-06"d 8, 3, 7< Date@D 8,,,, 3, 7.936< This function converts a Mathematica date into a real number. xmathematicadatetoyear@8iy_integer, im_integer, id_integer<d := NA DaysBetween@8iY -,, 3<, 8iY, im, id<d iy + ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ DaysBetween@8iY -,, 3<, 8iY,, 3<D E; Time expressed as a real number is easier to process and plot, e.g., xmathematicadatetoyear@8, 6, 30<D. Extracting subsets of data matrices These are examples of functions that we've used in earlier classes. xextract@vvudata_list, vsfields_list, sfield_string, xselectq_: HTrue &LD := Module@ 8iPosition<, iposition = Position@vsFields, sfield If@ Length@iPositionD ã 0, Null, Select@vvuDataPAll, ipositionp, TT, xselectqd D

5 ams--lec-0-m.nb vsfields_list, vsfieldsubset_list, xselectq_: HTrue &LD := 8vvuBuffer<, vvubuffer = Transpose@xExtract@vvuData, vsfields, #D & êü vsfieldsubset Select@vvuBuffer, xselectqd History Data from Yahoo!Fianance The example in the Technical Note referenced above uses an MSN site. For various reasons,we have decided to use Yahoo!Finance as our data source primarily because its split and dividend adjusted close is in keeping with CRSP standards. Building a URL This function builds a URL which will download daily data in comma-separated-value (CSV) format. xyahoofinancehistoryurl@ ssymbol_string, 8iStartYear_Integer, istartmonth_integer, istartday_integer<, 8iEndYear_Integer, iendmonth_integer, iendday_integer< D := Module@ 8xF, ss<, xf@ss_d := StringTake@"0" <> ToString@sSD, - StringJoin@ " H* Main call *L "s=" <> ssymbol, H* Symbol string *L "&a=" <> xf@istartmonth - D, H* Start month - *L "&b=" <> ToString@iStartDayD, H* Start day *L "&c=" <> ToString@iStartYearD, H* Start year *L "&d=" <> xf@iendmonth - D, H* End month - *L "&e=" <> ToString@iEndDayD, H* End day *L "&f=" <> ToString@iEndYearD, H* End year *L "&g=d", H* Frequency: daily *L "&ignore=.csv" D For example, to download the history for the 3M Corporation (ticker "MMM") for {970,, } to {, 3, 7} we have the following:

6 ams--lec-0-m.nb 6 xyahoofinancehistoryurl@"mmm", 8970,, <, 8, 3, 7<D MMM&a=00&b=&c=970&d=0&e=7&f=&g=d&ignore=.csv Downloading, reading in and processing the data. Given the data specification, as above, this function passes it to xyahoofinancehistoryurl, uses the URL to download the data, reads the data into a local variable and performs some postprocessing to put it into a convenient form for subsequent processing in Mathematica. The function returns a 3-vector consisting of the original arguments, a list of field names, and a data matrix. xgetyahoofinancehistorydata@ssymbol_string, vistart_, viend_d := Module@ 8vvsBuffer, vvudata, vsfields<, vvsbuffer = Import@ xgeturl@ xyahoofinancehistoryurl@ssymbol, vistart, viendd D, "CSV" vsfields = xregularizename êü First@vvsBuffer vvudata = Rest@vvsBuffer vvudatapall, T = xyahoodatetomathematicadate êü vvudatapall, T; vvudata = Sort@vvuData, HFirst@#D.80000, 00, <L < HFirst@#D.80000, 00, <L & 88sSymbol, vistart, viend<, vsfields, vvudata< We'll provide a detailed example in the next section. Example We download the data for the 3M Corporation, (ticker "MMM") for 970 to. Note that dates not containing data (non-trading days, such as New Years Day, and future days) are not returned.

7 ams--lec-0-m.nb 7 8vuCall, vsfields, vvudata< = xgetyahoofinancehistorydata@"mmm", 8970,, <, 8,, 3< Print@"Arguments = ", vucall Print@"Field Names = ", vsfields Print@ "First & Last 3 Data Rows = \n", Join@Take@vvuData, 3D, 8"..."<, Take@vvuData, -3DD Arguments = 8MMM, 8970,, <, 8,, 3<< Field Names = 8Date, Open, High, Low, Close, Volume, AdjClose< First & Last 3 Data Rows = ,, <, 09.6, 0., 09., 09.6, 7000,.<, 88970,, <, 09.7, 0.37, 09.7, 0., 600,.<, 88970,, 6<, 0.,.37, 0.,.37, 76000,.<,..., 88,, 3<, 7.97, 77.9, 7.76, 7.8, , 7.8<, 88,, <, 76., 77.99, 76., 77., , 77.<, 88,, <, 77.89, 77.9, 76.8, 77.6, 7800, 77.6<< 7.8 ÅÅÅÅÅÅÅÅÅÅÅÅÅÅ This statement extracts the "Date" and "AdjClose" columns, converts the dates to real numbers, takes the log of the prices and plots the result.

8 ams--lec-0-m.nb 8 ListPlot@ 8xMathematicaDateToYear@First@#DD, Log@Last@#DD< & êü xextract@vvudata, vsfields, 8"Date", "AdjClose"<D, PlotJoined Ø True, Frame Ø True, FrameLabel Ø 8"date", "log price", StyleForm@"MMM", FontSize Ø 6D, ""<, ImageSize Ø 00 MMM 3 l og pric e date

9 ams--lec-0-m.nb 9 ListPlot@ 8xMathematicaDateToYear@First@#DD, Last@#D< & êü xextract@vvudata, vsfields, 8"Date", "AdjClose"<D, PlotJoined Ø True, Frame Ø True, FrameLabel Ø 8"date", "price", StyleForm@"MMM", FontSize Ø 6D, ""<, ImageSize Ø 00 MMM price date Assignment The list of tickers of the Dow Jones Industrial Average. vsdowjonessymbol = 8"MMM", "AA", "MO", "AXP", "AIG", "T", "BA", "CAT", "C", "KO", "DD", "XOM", "GE", "GM", "HPQ", "HD", "HON", "INTC", "IBM", "JNJ", "JPM", "MCD", "MRK", "MFST", "PFE", "PG", "UTX", "VZ", "WMT", "DIS"< 8MMM, AA, MO, AXP, AIG, T, BA, CAT, C, KO, DD, XOM, GE, GM, HPQ, HD, HON, INTC, IBM, JNJ, JPM, MCD, MRK, MFST, PFE, PG, UTX, VZ, WMT, DIS< Produce a matrix of monthly rates of return (ROR) based on the adjusted closing price. Monthly RORs are defined from month end to month end. Each row of the matrix represents a given date. Each column represents a given stock. In addition, produce two additional vectors: one containing the dates corresponding to the rows of the data matrix and the other containing the tickers corresponding to the columns.

10 ams--lec-0-m.nb 0 As many have noted by now, the ticker for Microsoft is incorrect. 8vuCall, vsfields, vvudata< = xgetyahoofinancehistorydata@"mfst", 8970,, <, 8,, 3< Java::excptn : A Java exception occurred: java.io.filenotfoundexception: yahoo.comêtable.csv?s=mfst&a=00&b=&c=970&d=&e=3&f=&g=d&ignore=.csv at sun.net. getinputstreamhhttpurlconnection.java:798l at java.net.url.openstreamhurl.java:93l at sun.reflect.nativemethodaccessorimpl.invoke0hnative MethodL at sun.reflect.nativemethodaccessorimpl.invokehnativemethodaccessorimpl.java:39l at sun.reflect.delegatingmethodaccessorimpl. invokehdelegatingmethodaccessorimpl.java:l. Import::chtype : First argument $Failed is not a valid channel type. More Import::guess : Could not guess format from channel CSV. More Part::partd : Part specification Import@CSVDPAll, T is longer than depth of object. More Part::pspec : Part specification xyahoodatetomathematicadate@alld is neither an integer nor a list of integers. More Set::partd : Part specification vvudata$833pall, T is longer than depth of object. More The correct ticker is "MSFT". 8vuCall, vsfields, vvudata< = xgetyahoofinancehistorydata@"msft", 8970,, <, 8,, 3< Print@"Arguments = ", vucall Print@"Field Names = ", vsfields Print@ "First & Last 3 Data Rows = \n", Join@Take@vvuData, 3D, 8"..."<, Take@vvuData, -3DD Arguments = 8MSFT, 8970,, <, 8,, 3<< Field Names = 8Date, Open, High, Low, Close, Volume, AdjClose< First & Last 3 Data Rows = , 3, 3<,., 9.,., 8., , 0.08<, 88986, 3, <, 8., 9., 8., 9., , 0.09<, 88986, 3, 7<, 9., 9.7, 9., 9., 33700, 0.09<,..., 88,, 3<, 7.67, 7.73, 7., 7.6, , 7.6<, 88,, <, 7.6, 7.8, 7.7, 7.6, 70000, 7.6<, 88,, <, 7.88, 7.9, 7.6, 7.7, 39300, 7.7<<

11 ams--lec-0-m.nb & êü vsfields, 8"Date", "AdjClose"<D, PlotJoined Ø True, Frame Ø True, FrameLabel Ø 8"date", "log price", FontSize Ø 6D, ""<, ImageSize Ø 00 MSFT 3 l og pric e This is the th ticker and must be corrected in the vector date Position@vsDowJonesSymbol, "MFST"DP, T vsdowjonessymbolp Position@vsDowJonesSymbol, "MFST"DP, T T = "MSFT"; vsdowjonessymbol 8MMM, AA, MO, AXP, AIG, T, BA, CAT, C, KO, DD, XOM, GE, GM, HPQ, HD, HON, INTC, IBM, JNJ, JPM, MCD, MRK, MSFT, PFE, PG, UTX, VZ, WMT, DIS<

12 ams--lec-0-m.nb We can now download the data for all 30. We can simply Map xgetyahoofinancialhistorydata over the list of tickers. Note that dataabstract is a list each element of which is the return of a single xgetyahoofinancialhistorydata call. dataabstract = xgetyahoofinancehistorydata@#, 8970,, <, 8,, 3<D & êü vsdowjonessymbol; We are not sure at this point that all tickers are defined for all periods. This piece of code prints the firsst and last date for each ticker. TableForm@ Join@8#P, T<, 8First@#D, Last@#D< &@First êü #P3TDD & êü dataabstract, TableHeadings Ø 8None, 8"Ticker", "Start", "End"<< D Ticker Start End MMM 970 AA MO AXP AIG T BA CAT C KO DD XOM

13 GE 970 GM 970 HPQ 970 HD HON 970 INTC IBM 970 JNJ 970 JPM MCD 970 MRK 970 MSFT PFE 98 PG 970 UTX 970 VZ 983 WMT 97 8 DIS 970 ams--lec-0-m.nb 3

14 ams--lec-0-m.nb Ultimately, we want only month ending dates and it appears that we can only as far back as {986, 7, 9}. There also isn't much point including anything past {, 3, 3} since we're still in April. dataabstract = xgetyahoofinancehistorydata@#, 8986, 7, 9<, 8, 3, 3<D & êü vsdowjonessymbol; We only need the Date and AdjClose fields, so we'll restructure dataabstract to that effect. dataabstract = 8#PT, #P, 8, 7<T, #P3, All, 8, 7<T< & êü dataabstract; We only need the month ending dates. The Split function can be used to split a list into sublists of "same" elements. Here we employ Split with a function that defines two entries as the same if the month of their dates are the same. Applied to a single element of the top-level dataabstract list this returns sublists of month data. We then simply pull out the last element of each sublist as the last trading day of the month. For@i =, i Length@dataAbstractD, i ++, dataabstractpi, 3T = Last êü Split@dataAbstractPi, 3T, H#P, T ã #P, TL & We want to be very careful and check that all of the dates across all of the stocks are equal. They are. SameQ üü H#P3, All, T & êü dataabstractl True dataabstractpall,, T 8MMM, AA, MO, AXP, AIG, T, BA, CAT, C, KO, DD, XOM, GE, GM, HPQ, HD, HON, INTC, IBM, JNJ, JPM, MCD, MRK, MSFT, PFE, PG, UTX, VZ, WMT, DIS< Finally, we'll build a returns data structure consisting of three elements, The first is a list of dates as row names. The second is a list of tickers as column nams. The third is a matrix of returns over the dates and tickers. vureturns = 9 Rest@dataAbstractP, 3, All, TD, dataabstractpall,, T, Rest@#D ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ - &@Transpose@#P3, All, T & êü dataabstractdd Most@#D =;

15 ams--lec-0-m.nb At the set up section at the top of the notebook we've added a call to <<Statistics`MultiDescriptiveStatistics` which we allow us to compute mean vectors and covariance matrices. vnmean = Mean@vuReturnsP3TD 80.06, , , , , , 0.006, 0.063, , 0.08, , , 0.078, , , , , 0.07, , , 0.009, , , , , 0.03, 0.038, , 0.06, 0.066< mncovariance = CovarianceMatrix@vuReturnsP3TD , , , , , , , , , , 0.009, , , , , , , , , , , , , 0.009, , 0.003, , , , <, , , , , , , 0.007, 0.009, , , , , , , , , , , , 0.00, , , , , , , , , , <, , , , , , , , 0.003, , , , , , , , , 0.00, , , , , , , , 0.008, 0.006, , , , <, , , , 0.006, , , , , , , , , , , , 0.007, , , , , , , , , , , , , , <, , , , , , , , , , , , , , , , , 0.006, , , , , , , , , , , , , <, , , , , , 0.008, , , 0.007, , , , , , , , , , , 0.007, 0.008, , , , , <

16 ams--lec-0-m.nb , , , , <, , 0.007, , , , , , , , , , , , , , , , , , , , , , , 0.003, , , , , <, , 0.009, 0.003, , , , , , , , , , , , , 0.006, , , , , , 0.003, , , , , , , 0.008, <, , , , , , 0.007, , , , , , , , , , , , , , 0.007, , 0.007, , , , , , , , <, , , , , , , , , , , , , , , 0.009, , , , , 0.003, , , , 0.00, , , , , , <, , , , , , , , , , , , , , , , , , , , , , , , 0.003, , , , , , <, , , , , , , , , , , , , , , , , , , 0.008, , , , , , , , , , , <, , , , , , , , , , , , , , 0.003, , , , , 0.007, , , , , , , , , , , <, , , , , , , , , , , , , 0.003, , , , , , , , , , , , , , , , , <, , , , , , , , , , 0.009, , , , , , , , , , , , , , , <

17 ams--lec-0-m.nb , , , 0.008, , 0.00<, , , , 0.007, , , , 0.006, , , , , , , , , 0.000, , , 0.00, , 0.006, , , , , , 0.000, , <, , , 0.00, , 0.006, , , , , , , , , , , 0.000, , , , , , , , , , , , 0.000, , <, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , <, , , , , , , , , , , , 0.008, 0.007, , , , , , , , , , , , , , , , , <, , 0.00, , , , 0.007, , , 0.007, 0.003, , , , , , 0.00, , , , , , , , , , , , , , <, , , , , , 0.008, , , , , , , , , , , , , , , , , , , , 0.006, , , , <, , , , , , , , 0.003, 0.007, , , , , , , 0.006, , , , , , , , 0.000, , , , , , <, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , <, , , , , , , , , , 0.00, 0.003, , , , , , , , , , , 0.000, , , <

18 ams--lec-0-m.nb , 0.00, 0.009, , , <, , , 0.008, , , , 0.003, , , , , , , , , , , , , , , , , , , , , , , <, , , 0.006, , , , , , , , , , , , , , , , , , 0.006, , , 0.00, , 0.003, 0.009, , , <, , , , , , , , , , , , , , , , , , , , , , , , 0.009, , 0.009, , , , <, , , , , , , , , , , , , , , 0.008, 0.000, 0.000, , , , , , , , , , , , , <, , , , , , , , 0.008, , , , , , , , , , , , , , , , , , , , , 0.008, <, , , , , , , , , , , , , , , 0.00, , , , , , , , , , , , , , , << For example, the spectrum of the covariance matrix is svd = SingularValueDecomposition@mnCovariance

19 ams--lec-0-m.nb 9 ListPlot@ Table@svdP, i, it, 8i,, 30<D, PlotRange Ø All, PlotJoined Ø True, ImageSize Ø