Lustick Consulting s ABM Verification & Validation
|
|
- Ellen Hall
- 5 years ago
- Views:
Transcription
1 Lustick Consulting s ABM Verification & Validation September 2015 The key to Lustick Consulting s Verification & Validation (V&V) process is iterative analysis of model inputs, theory, outputs, and empirical indicators. V&V should not be a static, one- time event, but rather an ongoing process of hypothesizing, testing, and confirmation or disconfirmation. 1 In order to implement this goal, Option I of our project called for six quarterly Verification & Validation reports from August 2012 to December 2013 that included our system status, data inputs, and model forecasts. In addition, we performed a MESA Epistemological Decomposition of our Venezuela model in May 2012 to unpack the data and theory behind the model. 2 Lastly, we held multiple sessions with two Subject Matter Experts Dr. Allen Hicken and Dr. David Faris who helped us better understand how our models relate to the real- world. Both experts found our approach compelling with relation to their countries (Egypt, Indonesia, the Philippines, and Thailand). There are many other ways to measure verification and validation, including a model s consistency, prominence, and accuracy with respect to other research in the discipline. We can also measure the model's utility to users, since an effective model should be useful for some specified purpose. Of course, the classic way to validate a model in modern science is to create a ground truth dataset and use that real- world data to test the model in- and out- of- sample. In this section, we test the validity of our country models by fitting a simple logistic regression using key indicators from the ABM output to predict the likelihood of five Events of Interest (EOIs) from the ICEWS project: Domestic Political Crisis, Insurgency, Rebellion, Ethnic/Religious Violence, and International Crisis. However, forecasting of discrete events, per se, is not a fully appropriate measure of performance for our modeling technique. This shortcoming stems from the greater flexibility of our model, which seeks to provide information not only about what could happen or did happen, but also how and why those things could or did happen. Our model also can examine the likelihood of what could have happened and how, and why that event might have occurred. From a causal, theoretical, and policy planning point of view, it is crucial to understand that much of what actually happens is random, i.e. theoretically uninteresting and causally inaccessible. In other words, a model can be excellent, and be wrong about a particular forecast. The better and more difficult validation procedure must include comparison of patterns of forecasts to patterns of outcomes and not merely forecast discrete events. 1 For more information on Lustick Consulting s deep Verification & Validation work, see (Lustick and Tubin, 2012). 2 For more information on the Model Evaluation, Selection, and Assessment (MESA) project, see (Alicia, et.al., 2012).
2 Our computational steering process used for updating the models allows us to start running our models back to the beginning of 2001, but we treat the first three years of the run as a burn- in period, a common practice in agent- based modeling applications. We also start four of our models later than 2001 due to lack of data and severe regime shifts in some countries. In all models we gather thirteen key output variables that we believe are the most likely drivers of the ICEWS EOIs. These include: 1. Mobilization indicators: attacks, protests, lobbies, and victims 2. Dynamic Political Hierarchy (DPH) indicators: dominant, incumbent, regime, system, and non- system subscription and activation (for more information on the Dynamic Political Hierarchy, see Lustick et.al. 2012) We then fit a simple logistic regression using these variables to our five binary EOIs. All variables were included for all models, and no attempt was made to tweak the model for performance. We intend these results to be a proof of concept, not a set of stable LC forecasts. In order to gauge the accuracy of our results, we compare our in- sample and out- of- sample forecasts to the Ensemble Bayesian Model Averaging (EBMA) model from the ICEWS project (Montgomery, et al., 2012). In general, we have found that our models forecast events well and rival the EBMA in some cases. Figure 1: Brier scores by EOI and Country, broken down by the ABM and EBMA results Figure 1 above compares the Brier Scores 3 for the ABM and EBMA forecasts for each EOI and country. We can visualize these metrics by creating separation plots by EOI to show how many cases we have classified correctly in Figure 2. In the separation plots, predictions are ordered from least likely to most likely, and a red bar indicates a true EOI. An accurate separation plot tends to have high predictions during true events on the right side of the graph and low predictions during false events on the left side. We can also check our model s accuracy by looking at the Sensitivity (percentage of 1 s correct), Specificity (percentage of 0 s correct), and 3
3 statistics 4 (see Table 1.) To calculate the metrics that require a cut- point, the value.5 was used. (This cut- point was chosen ahead of time, not calibrated to improve performance.) Figure 2: Separation Plot of in- sample and out- of- sample results Table 1: Agent- based Model Forecast Metrics, In- and Out- of- sample Domestic Political Crisis Rebellion International Crisis Insurgency Ethnic/Religious Violence is a method for measuring the correlation of binary variables.
4 Figure 3: EOI forecasts for Yemen To give an example, Figure 3 shows four EOI forecasts for our Yemen model, comparing the predictions for both the EBMA and ABM models. Both models seem to do poorly in predicting the Yemen Rebellion (REB) or Domestic Political Crisis (DPC), but do modestly well in forecasting the Insurgency (INS) and Ethnic/Religious Violence (ERV) before it occurred. Note that forecasts made after July 2013 are completely out- of- sample. To show the volume of data being used for this analysis, we can look at the ABM output data in Figure 4 for each country- month combination within our timeframe. Changes in the ABM output data are caused by a combination of exogenous punctuations, continuous computational steering, and internal model dynamics. The x- axis for each country shows the month and the y- axis shows the value for each particular ABM output variable. The y- axis scales may vary for each country. Lobby, protest, attack, and victim are measures of the average number of agents during each country- month that are mobilizing in different ways, representing levels of discontent and isolation that rise to the level of mobilization. The DPH subscription variables are the average number of agents for each DPH level for which that DPH level is the highest to which it is subscribed. For example, high levels of system and non- system subscription tell us that there are many agents in the landscape that do not have a dominant, incumbent, or regime identity in their repertoire and are therefore alienated from the center of politics. The last set of variables, DPH activation, is the average number of agents that are activated on an identity in
5 each level of the DPH. High regime activation simply means that there are many agents that are activated on some regime identity, but could be subscribed to dominant, incumbent, or system identities. All of these variables have consistent meanings across countries and are therefore ideal for using in a large- N cross- country analysis. Figure 4: Main ABM output variables over time We also ran a similar experiment as part of the ME- CEWS project to forecast the number of violent events per province per week in seven Middle Eastern countries. We used the same method as described above, but used a simple ordinary least squares linear model with a floor of zero. Also, we used only a subset of the variables from the EOI country- level forecasts, including attack, protest, lobby, and our DPH subscription measures (eight variables). We again compare our model to the EBMA results, but different metrics need to be used in order to measure the accuracy of a count variable as opposed to a binary variable. In Table 2 we show the comparison between the ABM and EBMA validation metrics for weekly forecasts of violent events for 125 provinces between January 2004 and September We can see that the correlation is weak for both models, meaning that neither model did particularly well in forecasting violent events. Although the ABM Root Mean Squared Error is higher, our Mean Absolute Error is lower, indicating more outlier forecasts for the ABM. Perhaps
6 most interesting is that the correlation between the two models is only.38, which means that our model would likely improve the EBMA forecast if included in the averaging algorithm. Table 2: Empirical validation for violent event forecasts Agent- based Model Ensemble Bayesian (ABM) Model Averaging (EBMA) Root Mean Squared Error Mean Absolute Error Pearson s Correlation To show the richness of the output, Figure 5 shows the weekly ABM output for each Syrian province between 2012 and 2017 (our forecast end date). We can see that there is significant variation from province to province due to differences in support for factions (Assad, Kurds, ISIS, Rebels), identity complexion (ethnic, religious, political), and steering data coming from media reports picked up by the ICEWS event data. Figure 5: ABM Output for Syrian Provinces The main advantage to a large N approach for problems like these is that the logistic regression can find patterns that a human could not. The downside is that all of the results are correlational, meaning that they don t necessarily describe real causal processes. Building a
7 large set of agent- based models, running and updating them for a fifteen- year period, and then validating the results using real- world data is an unprecedented step in the field of computational social science. Even more, showing that those results can compete with and complement the state- of- the- art in statistical forecasting is strong evidence that ABM holds great potential in the future of the discipline. Bibliography Lustick, Ian S. and Matthew Tubin Verification as a Form of Validation: Deepening Theory to Broaden Application of DoD Protocols to the Social Sciences. Advances in Design for Cross- Cultural Activities. Lustick, Ian S., et al From theory to simulation: the dynamic political hierarchy in country virtualisation models. Journal of Experimental & Theoretical Artificial Intelligence 24(3) Montgomery, Jacob M., Florian M. Hollenbach, and Michael D. Ward Improving predictions using ensemble Bayesian model averaging. Political Analysis Ruvinsky, Alicia I., Janet E. Wedgwood, and John J. Welsh Establishing bounds of responsible operational use of social science models via innovations in verification and validation. Advances in design for cross- cultural activities Part II.
Package EBMAforecast
Type Package Title Ensemble BMA Forecasting Version 0.52 Date 2016-02-09 Package EBMAforecast February 10, 2016 Author Jacob M. Montgomery, Florian M. Hollenbach, and Michael D. Ward Maintainer Florian
More informationMixed Methods Stability Forecasting and Mitigation for the DARPA ICEWS Program
2009 Lockheed Martin Mixed Methods Stability Forecasting and Mitigation for the DARPA ICEWS Program Third international Conference on Computational Cultural Dynamics (ICCCD 2009) Brian Kettler, PhD, Team
More informationForecasting Conflict Lecture 4 Models and Metrics
Forecasting Conflict Lecture 4 Models and Metrics Philip A. Schrodt Parus Analytical Systems schrodt735@gmail.com Graduate School of Decision Sciences University of Konstanz 14-17 October 2013 Overview
More informationEXAMINATION: QUANTITATIVE EMPIRICAL METHODS. Yale University. Department of Political Science
EXAMINATION: QUANTITATIVE EMPIRICAL METHODS Yale University Department of Political Science January 2014 You have seven hours (and fifteen minutes) to complete the exam. You can use the points assigned
More informationStatistical Models for Causal Analysis
Statistical Models for Causal Analysis Teppei Yamamoto Keio University Introduction to Causal Inference Spring 2016 Three Modes of Statistical Inference 1. Descriptive Inference: summarizing and exploring
More informationRemoval and Destruction of Syrian Chemical Weapons
Removal and Destruction of Syrian Chemical Weapons Chemical Demilitarisation Branch UN Investigation of CW Use in Syria UN team (including OPCW and WHO experts) arrived in Damascus on 18 August 2013 Alleged
More informationIssues around verification, validation, calibration, and confirmation of agent-based models of complex spatial systems
Issues around verification, validation, calibration, and confirmation of agent-based models of complex spatial systems David O Sullivan 1 Mark Gahegan 2 1 School of Geography, Geology, and Environmental
More informationVerification and Validation. CS1538: Introduction to Simulations
Verification and Validation CS1538: Introduction to Simulations Steps in a Simulation Study Problem & Objective Formulation Model Conceptualization Data Collection Model translation, Verification, Validation
More informationDo we need Experts for Time Series Forecasting?
Do we need Experts for Time Series Forecasting? Christiane Lemke and Bogdan Gabrys Bournemouth University - School of Design, Engineering and Computing Poole House, Talbot Campus, Poole, BH12 5BB - United
More informationReview of Panel Data Model Types Next Steps. Panel GLMs. Department of Political Science and Government Aarhus University.
Panel GLMs Department of Political Science and Government Aarhus University May 12, 2015 1 Review of Panel Data 2 Model Types 3 Review and Looking Forward 1 Review of Panel Data 2 Model Types 3 Review
More informationECE521 week 3: 23/26 January 2017
ECE521 week 3: 23/26 January 2017 Outline Probabilistic interpretation of linear regression - Maximum likelihood estimation (MLE) - Maximum a posteriori (MAP) estimation Bias-variance trade-off Linear
More informationMath 6330: Statistical Consulting Class 5
Math 6330: Statistical Consulting Class 5 Tony Cox tcoxdenver@aol.com University of Colorado at Denver Course web site: http://cox-associates.com/6330/ What is a predictive model? The probability that
More informationBasic Verification Concepts
Basic Verification Concepts Barbara Brown National Center for Atmospheric Research Boulder Colorado USA bgb@ucar.edu Basic concepts - outline What is verification? Why verify? Identifying verification
More informationA Wisdom of the Crowd Approach to Forecasting
A Wisdom of the Crowd Approach to Forecasting Funded by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior National Business Center contract number D11PC20059 Brandon
More informationCHAPTER 4: DATASETS AND CRITERIA FOR ALGORITHM EVALUATION
CHAPTER 4: DATASETS AND CRITERIA FOR ALGORITHM EVALUATION 4.1 Overview This chapter contains the description about the data that is used in this research. In this research time series data is used. A time
More informationData Mining. CS57300 Purdue University. March 22, 2018
Data Mining CS57300 Purdue University March 22, 2018 1 Hypothesis Testing Select 50% users to see headline A Unlimited Clean Energy: Cold Fusion has Arrived Select 50% users to see headline B Wedding War
More information3. If a forecast is too high when compared to an actual outcome, will that forecast error be positive or negative?
1. Does a moving average forecast become more or less responsive to changes in a data series when more data points are included in the average? 2. Does an exponential smoothing forecast become more or
More information2/26/2017. PSY 512: Advanced Statistics for Psychological and Behavioral Research 2
PSY 512: Advanced Statistics for Psychological and Behavioral Research 2 When and why do we use logistic regression? Binary Multinomial Theory behind logistic regression Assessing the model Assessing predictors
More informationLearning: Binary Perceptron. Examples: Perceptron. Separable Case. In the space of feature vectors
Linear Classifiers CS 88 Artificial Intelligence Perceptrons and Logistic Regression Pieter Abbeel & Dan Klein University of California, Berkeley Feature Vectors Some (Simplified) Biology Very loose inspiration
More informationNOWCASTING THE OBAMA VOTE: PROXY MODELS FOR 2012
JANUARY 4, 2012 NOWCASTING THE OBAMA VOTE: PROXY MODELS FOR 2012 Michael S. Lewis-Beck University of Iowa Charles Tien Hunter College, CUNY IF THE US PRESIDENTIAL ELECTION WERE HELD NOW, OBAMA WOULD WIN.
More informationLogic and Artificial Intelligence Lecture 13
Logic and Artificial Intelligence Lecture 13 Eric Pacuit Currently Visiting the Center for Formal Epistemology, CMU Center for Logic and Philosophy of Science Tilburg University ai.stanford.edu/ epacuit
More informationForecasting: The First Step in Demand Planning
Forecasting: The First Step in Demand Planning Jayant Rajgopal, Ph.D., P.E. University of Pittsburgh Pittsburgh, PA 15261 In a supply chain context, forecasting is the estimation of future demand General
More informationGDP growth and inflation forecasting performance of Asian Development Outlook
and inflation forecasting performance of Asian Development Outlook Asian Development Outlook (ADO) has been the flagship publication of the Asian Development Bank (ADB) since 1989. Issued twice a year
More informationMachine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function.
Bayesian learning: Machine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function. Let y be the true label and y be the predicted
More informationComponents for Accurate Forecasting & Continuous Forecast Improvement
Components for Accurate Forecasting & Continuous Forecast Improvement An ISIS Solutions White Paper November 2009 Page 1 Achieving forecast accuracy for business applications one year in advance requires
More informationA Random-effects construction of EMV models - a solution to the Identification problem? Peter E Clarke, Deva Statistical Consulting
A Random-effects construction of EMV models - a solution to the Identification problem? Peter E Clarke, Deva Statistical Consulting Credit Scoring and Credit Control Conference XV Edinburgh, August 2017
More informationTargeted Maximum Likelihood Estimation in Safety Analysis
Targeted Maximum Likelihood Estimation in Safety Analysis Sam Lendle 1 Bruce Fireman 2 Mark van der Laan 1 1 UC Berkeley 2 Kaiser Permanente ISPE Advanced Topics Session, Barcelona, August 2012 1 / 35
More informationNiche Modeling. STAMPS - MBL Course Woods Hole, MA - August 9, 2016
Niche Modeling Katie Pollard & Josh Ladau Gladstone Institutes UCSF Division of Biostatistics, Institute for Human Genetics and Institute for Computational Health Science STAMPS - MBL Course Woods Hole,
More informationIntroduction to Statistical Inference
Introduction to Statistical Inference Kosuke Imai Princeton University January 31, 2010 Kosuke Imai (Princeton) Introduction to Statistical Inference January 31, 2010 1 / 21 What is Statistics? Statistics
More informationFreeman (2005) - Graphic Techniques for Exploring Social Network Data
Freeman (2005) - Graphic Techniques for Exploring Social Network Data The analysis of social network data has two main goals: 1. Identify cohesive groups 2. Identify social positions Moreno (1932) was
More informationRon Heck, Fall Week 8: Introducing Generalized Linear Models: Logistic Regression 1 (Replaces prior revision dated October 20, 2011)
Ron Heck, Fall 2011 1 EDEP 768E: Seminar in Multilevel Modeling rev. January 3, 2012 (see footnote) Week 8: Introducing Generalized Linear Models: Logistic Regression 1 (Replaces prior revision dated October
More informationGOVERNMENT MAPPING WORKSHOP RECOVER Edmonton s Urban Wellness Plan Mapping Workshop December 4, 2017
GOVERNMENT MAPPING WORKSHOP 12.4.17 RECOVER Edmonton s Urban Wellness Plan Mapping Workshop December 4, 2017 In July of 2017, City Council directed administration to develop RECOVER, Edmonton s Urban Wellness
More informationForecasting Time Frames Using Gann Angles By Jason Sidney
Forecasting Time Frames Using Gann Angles By Jason Sidney In the previous lesson, we looked at how Gann Angles can be used in conjunction with eighths and third retracements. In that lesson you were shown
More informationEC 331: Research in Applied Economics
EC 331: Research in Applied Economics Terms 1 & 2: Thursday, 1-2pm, S2.133 Vera E. Troeger Office: S0.75 Email: v.e.troeger@warwick.ac.uk Office hours: Friday 9.30 11.30 am Research Design The Purpose
More informationUNIVERSITY of PENNSYLVANIA CIS 520: Machine Learning Final, Fall 2013
UNIVERSITY of PENNSYLVANIA CIS 520: Machine Learning Final, Fall 2013 Exam policy: This exam allows two one-page, two-sided cheat sheets; No other materials. Time: 2 hours. Be sure to write your name and
More informationJagdish Maheshri ABSTRACT. Introduction
Modern Planets and Wars, Uprisings, Revolutions and Violence Jagdish Maheshri ABSTRACT The article explores possible correlations between the major world events such as wars, uprising, revolutions and
More informationModel Selection. Frank Wood. December 10, 2009
Model Selection Frank Wood December 10, 2009 Standard Linear Regression Recipe Identify the explanatory variables Decide the functional forms in which the explanatory variables can enter the model Decide
More informationBrief Glimpse of Agent-Based Modeling
Brief Glimpse of Agent-Based Modeling Nathaniel Osgood Using Modeling to Prepare for Changing Healthcare Need January 9, 2014 Agent-Based Models Agent-based model characteristics One or more populations
More informationIntroduction to Artificial Intelligence. Prof. Inkyu Moon Dept. of Robotics Engineering, DGIST
Introduction to Artificial Intelligence Prof. Inkyu Moon Dept. of Robotics Engineering, DGIST Chapter 3 Uncertainty management in rule-based expert systems To help interpret Bayesian reasoning in expert
More informationIndustrial Engineering Prof. Inderdeep Singh Department of Mechanical & Industrial Engineering Indian Institute of Technology, Roorkee
Industrial Engineering Prof. Inderdeep Singh Department of Mechanical & Industrial Engineering Indian Institute of Technology, Roorkee Module - 04 Lecture - 05 Sales Forecasting - II A very warm welcome
More informationCrop / Weather Update
Crop / Weather Update This is our last for the 2018 planting, growing and harvesting seasons. USDA will publish their Crop Progress Report again in April 2019, and we will continue our coverage sometime
More informationNorthwestern Consolidated Schools of Shelby County. Curriculum. World Studies (Eastern Hemisphere) Prepared by. Rich Ballard
Northwestern Consolidated Schools of Shelby County Curriculum World Studies (Eastern Hemisphere) 7th Grade Prepared by Rich Ballard 2012-2013 1 Triton Central Middle School Vision Statement We are committed
More informationCSL302/612 Artificial Intelligence End-Semester Exam 120 Minutes
CSL302/612 Artificial Intelligence End-Semester Exam 120 Minutes Name: Roll Number: Please read the following instructions carefully Ø Calculators are allowed. However, laptops or mobile phones are not
More informationEconomic and Social Council
United Nations Economic and Social Council Distr.: General 2 July 2012 E/C.20/2012/10/Add.1 Original: English Committee of Experts on Global Geospatial Information Management Second session New York, 13-15
More informationAnalyzing Residuals in a PROC SURVEYLOGISTIC Model
Paper 1477-2017 Analyzing Residuals in a PROC SURVEYLOGISTIC Model Bogdan Gadidov, Herman E. Ray, Kennesaw State University ABSTRACT Data from an extensive survey conducted by the National Center for Education
More informationLogistic Regression: Regression with a Binary Dependent Variable
Logistic Regression: Regression with a Binary Dependent Variable LEARNING OBJECTIVES Upon completing this chapter, you should be able to do the following: State the circumstances under which logistic regression
More informationDecision Trees: Overfitting
Decision Trees: Overfitting Emily Fox University of Washington January 30, 2017 Decision tree recap Loan status: Root 22 18 poor 4 14 Credit? Income? excellent 9 0 3 years 0 4 Fair 9 4 Term? 5 years 9
More informationCSISS Resources for Research and Teaching
CSISS Resources for Research and Teaching Donald G. Janelle Center for Spatially Integrated Social Science University of California, Santa Barbara Montreal 26 July 2003 Workshop on Spatial Analysis for
More informationSTOCKHOLM UNIVERSITY Department of Economics Course name: Empirical Methods Course code: EC40 Examiner: Lena Nekby Number of credits: 7,5 credits Date of exam: Friday, June 5, 009 Examination time: 3 hours
More informationLearning Objectives. c D. Poole and A. Mackworth 2010 Artificial Intelligence, Lecture 7.2, Page 1
Learning Objectives At the end of the class you should be able to: identify a supervised learning problem characterize how the prediction is a function of the error measure avoid mixing the training and
More informationCONCEPT 4 Scientific Law. CONCEPT 3 Scientific Theory
CONCEPT 1 Nature of Science is a way of learning about the natural world that is based on evidence and logic. The goal of science is to understand and things happen. Science advances as new evidence accumulates
More information(quantitative or categorical variables) Numerical descriptions of center, variability, position (quantitative variables)
3. Descriptive Statistics Describing data with tables and graphs (quantitative or categorical variables) Numerical descriptions of center, variability, position (quantitative variables) Bivariate descriptions
More informationLECTURE 15: SIMPLE LINEAR REGRESSION I
David Youngberg BSAD 20 Montgomery College LECTURE 5: SIMPLE LINEAR REGRESSION I I. From Correlation to Regression a. Recall last class when we discussed two basic types of correlation (positive and negative).
More informationMapping Communities of Opportunity in New Orleans
Mapping Communities of Opportunity in New Orleans December 11, 2009 Samir Gambhir Senior Research Associate Kirwan Institute for the study of Race and Ethnicity The Ohio State University Gambhir.2@osu.edu
More informationEconomics 390 Economic Forecasting
Economics 390 Economic Forecasting Prerequisite: Econ 410 or equivalent Course information is on website Office Hours Tuesdays & Thursdays 2:30 3:30 or by appointment Textbooks Forecasting for Economics
More informationLEARNING OUTCOMES SST (G1-G12)
LEARNING OUTCOMES SST (G1-G12) Grade 1 Learning Outcomes Students will be expected to: - demonstrate an understanding of the importance of interactions between people - demonstrate an understanding of
More informationEver since elections for office have been held, people. Accuracy of Combined Forecasts for the 2012 Presidential Election: The PollyVote FEATURES
FEATURES Accuracy of Combined Forecasts for the 2012 Presidential Election: The PollyVote Andreas Graefe, LMU Munich J. Scott Armstrong, University of Pennsylvania and University of South Australia Randall
More informationAUTO SALES FORECASTING FOR PRODUCTION PLANNING AT FORD
FCAS AUTO SALES FORECASTING FOR PRODUCTION PLANNING AT FORD Group - A10 Group Members: PGID Name of the Member 1. 61710956 Abhishek Gore 2. 61710521 Ajay Ballapale 3. 61710106 Bhushan Goyal 4. 61710397
More informationGlobal Perspectives Goals & Objectives: (include numbers) Learning Essentials Materials/ Resources
August/ September Early Civilizations Goal 1.6: Explain the rise of human civilization. 6-9.WHC.1.6.1 Describe types of evidence used by anthropologists, archaeologists, and other scholars to reconstruct
More informationCS 188: Artificial Intelligence Spring Announcements
CS 188: Artificial Intelligence Spring 2010 Lecture 22: Nearest Neighbors, Kernels 4/18/2011 Pieter Abbeel UC Berkeley Slides adapted from Dan Klein Announcements On-going: contest (optional and FUN!)
More informationCalibrated Uncertainty in Deep Learning
Calibrated Uncertainty in Deep Learning U NCERTAINTY IN DEEP LEARNING W ORKSHOP @ UAI18 Volodymyr Kuleshov August 10, 2018 Estimating Uncertainty is Crucial in Many Applications Assessing uncertainty can
More informationVBM683 Machine Learning
VBM683 Machine Learning Pinar Duygulu Slides are adapted from Dhruv Batra Bias is the algorithm's tendency to consistently learn the wrong thing by not taking into account all the information in the data
More informationChapter 10 Regression Analysis
Chapter 10 Regression Analysis Goal: To become familiar with how to use Excel 2007/2010 for Correlation and Regression. Instructions: You will be using CORREL, FORECAST and Regression. CORREL and FORECAST
More informationAccuracy of Combined Forecasts for the 2012 Presidential Election: The PollyVote
University of Pennsylvania ScholarlyCommons Marketing Papers Wharton Faculty Research 4-2014 Accuracy of Combined Forecasts for the 2012 Presidential Election: The PollyVote Andreas Graefe J. S. Armstrong
More informationTRADE AND DEVELOPMENT: THE ASIAN EXPERIENCE. All F. Darrat
TRADE AND DEVELOPMENT: THE ASIAN Introduction EXPERIENCE All F. Darrat One development issue that has attracted much attention in recent years is the potential impact of a country s exports on its economic
More informationForecasting Using Time Series Models
Forecasting Using Time Series Models Dr. J Katyayani 1, M Jahnavi 2 Pothugunta Krishna Prasad 3 1 Professor, Department of MBA, SPMVV, Tirupati, India 2 Assistant Professor, Koshys Institute of Management
More informationMapping granite outcrops in the Western Australian Wheatbelt using Landsat TM data
Journal of the Royal Society of Western Australia, 109-113, 2000 Mapping granite outcrops in the Western Australian Wheatbelt using Landsat TM data N A Campbell 1, S D Hopper 2 & P A Caccetta 1 1 CSIRO
More informationRecommendations on trajectory selection in flight planning based on weather uncertainty
Recommendations on trajectory selection in flight planning based on weather uncertainty Philippe Arbogast, Alan Hally, Jacob Cheung, Jaap Heijstek, Adri Marsman, Jean-Louis Brenguier Toulouse 6-10 Nov
More informationDo Markov-Switching Models Capture Nonlinearities in the Data? Tests using Nonparametric Methods
Do Markov-Switching Models Capture Nonlinearities in the Data? Tests using Nonparametric Methods Robert V. Breunig Centre for Economic Policy Research, Research School of Social Sciences and School of
More informationIntroduction to Machine Learning Midterm Exam
10-701 Introduction to Machine Learning Midterm Exam Instructors: Eric Xing, Ziv Bar-Joseph 17 November, 2015 There are 11 questions, for a total of 100 points. This exam is open book, open notes, but
More informationSocial Studies: The World (End of the Year TEST)
Name: Period: TEST A Social Studies: The World (End of the Year TEST) Map and Globe Skills: Label the continents and oceans on the lines below the map. 1 11 3 4 5 8 2 9 10 8 6 7 1. 8. 2. 9. 3. 10. 4. 11.
More informationLinear classifiers: Overfitting and regularization
Linear classifiers: Overfitting and regularization Emily Fox University of Washington January 25, 2017 Logistic regression recap 1 . Thus far, we focused on decision boundaries Score(x i ) = w 0 h 0 (x
More informationCreating Non-Gaussian Processes from Gaussian Processes by the Log-Sum-Exp Approach. Radford M. Neal, 28 February 2005
Creating Non-Gaussian Processes from Gaussian Processes by the Log-Sum-Exp Approach Radford M. Neal, 28 February 2005 A Very Brief Review of Gaussian Processes A Gaussian process is a distribution over
More informationRegression Analysis. A statistical procedure used to find relations among a set of variables.
Regression Analysis A statistical procedure used to find relations among a set of variables. Understanding relations Mapping data enables us to examine (describe) where things occur (e.g., areas where
More informationPOLICY GAMES. u t = θ 0 θ 1 π t + ε t. (1) β t (u 2 t + ωπ 2 t ). (3)
Eco5, Part II Spring C. Sims POLICY GAMES The policy authority believes 1. THE SAN JOSE MODEL u t = θ θ 1 π t + ε t. (1) In fact, though, u t = ū α (π t E t 1 π t ) + ξ t. () They minimize. POLICYMAKERS
More informationPOLICY GAMES. u t = θ 0 θ 1 π t + ε t. (1) β t (u 2 t + ωπ 2 t ). (3) π t = g t 1 + ν t. (4) g t = θ 1θ 0 ω + θ 2 1. (5)
Eco504, Part II Spring 2004 C. Sims POLICY GAMES The policy authority believes In fact, though, 1. THE SAN JOSE MODEL u t = θ 0 θ 1 π t + ε t. (1) u t = ū α (π t E t 1 π t ) + ξ t. (2) They minimize 2.
More informationSCIENTIFIC INQUIRY AND CONNECTIONS. Recognize questions and hypotheses that can be investigated according to the criteria and methods of science
SUBAREA I. COMPETENCY 1.0 SCIENTIFIC INQUIRY AND CONNECTIONS UNDERSTAND THE PRINCIPLES AND PROCESSES OF SCIENTIFIC INQUIRY AND CONDUCTING SCIENTIFIC INVESTIGATIONS SKILL 1.1 Recognize questions and hypotheses
More informationNOAA 2015 Updated Atlantic Hurricane Season Outlook
NOAA 2015 Updated Atlantic Hurricane Season Outlook Dr. Gerry Bell Lead Seasonal Forecaster Climate Prediction Center/ NOAA/ NWS Collaboration With National Hurricane Center/ NOAA/ NWS Hurricane Research
More informationA Plot of the Tracking Signals Calculated in Exhibit 3.9
CHAPTER 3 FORECASTING 1 Measurement of Error We can get a better feel for what the MAD and tracking signal mean by plotting the points on a graph. Though this is not completely legitimate from a sample-size
More informationApplied Microeconometrics I
Applied Microeconometrics I Lecture 6: Instrumental variables in action Manuel Bagues Aalto University September 21 2017 Lecture Slides 1/ 20 Applied Microeconometrics I A few logistic reminders... Tutorial
More informationNOTES AND CORRESPONDENCE. Improving Week-2 Forecasts with Multimodel Reforecast Ensembles
AUGUST 2006 N O T E S A N D C O R R E S P O N D E N C E 2279 NOTES AND CORRESPONDENCE Improving Week-2 Forecasts with Multimodel Reforecast Ensembles JEFFREY S. WHITAKER AND XUE WEI NOAA CIRES Climate
More informationCourse Introduction and Overview Descriptive Statistics Conceptualizations of Variance Review of the General Linear Model
Course Introduction and Overview Descriptive Statistics Conceptualizations of Variance Review of the General Linear Model PSYC 943 (930): Fundamentals of Multivariate Modeling Lecture 1: August 22, 2012
More informationImplementation Status & Results Burkina Faso TRANSPORT SECTOR PROJECT (P074030)
losure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Implementation Status & Results Burkina Faso TRANSPORT SECTOR PROJECT (P074030) Operation Name:
More informationFall 2013 Monday 6:30pm-9:00pm
History & Methods Geography Fall 2013 Monday 6:30pm-9:00pm Instructor: Dr. Nate Gabriel Email: gabrieln@rowan.edu Course Description This course will cover the history and development of geographic study,
More informationDemand Forecasting Reporting Period: 19 st Jun th Sep 2017
N A T I O N A L G R I D P A G E 1 O F 21 C O M M E R C I A L, E L E C T R I C I T Y C O M M E R C I A L O P E R A T I O N S Demand Forecasting Reporting Period: 19 st Jun 2017 10 th Sep 2017 EXECUTIVE
More informationBasic Verification Concepts
Basic Verification Concepts Barbara Brown National Center for Atmospheric Research Boulder Colorado USA bgb@ucar.edu May 2017 Berlin, Germany Basic concepts - outline What is verification? Why verify?
More informationCourse Introduction and Overview Descriptive Statistics Conceptualizations of Variance Review of the General Linear Model
Course Introduction and Overview Descriptive Statistics Conceptualizations of Variance Review of the General Linear Model EPSY 905: Multivariate Analysis Lecture 1 20 January 2016 EPSY 905: Lecture 1 -
More informationFire frequency in the Western Cape
Fire frequency in the Western Cape First year progress report Diane Southey 3 May 27 This report is a summary of the work I have done in the first year of my masters. Each section is briefly discussed
More informationAssessing probabilistic forecasts about particular situations
MPRA Munich Personal RePEc Archive Assessing probabilistic forecasts about particular situations Kesten C. Green Business and Economic Forecasting Unit, Monash University 25. April 2008 Online at http://mpra.ub.uni-muenchen.de/8836/
More informationDay 6: Classification and Machine Learning
Day 6: Classification and Machine Learning Kenneth Benoit Essex Summer School 2014 July 30, 2013 Today s Road Map The Naive Bayes Classifier The k-nearest Neighbour Classifier Support Vector Machines (SVMs)
More informationMeasuring Social Influence Without Bias
Measuring Social Influence Without Bias Annie Franco Bobbie NJ Macdonald December 9, 2015 The Problem CS224W: Final Paper How well can statistical models disentangle the effects of social influence from
More informationLecture I. What is Quantitative Macroeconomics?
Lecture I What is Quantitative Macroeconomics? Gianluca Violante New York University Quantitative Macroeconomics G. Violante, What is Quantitative Macro? p. 1 /11 Qualitative economics Qualitative analysis:
More information13.7 ANOTHER TEST FOR TREND: KENDALL S TAU
13.7 ANOTHER TEST FOR TREND: KENDALL S TAU In 1969 the U.S. government instituted a draft lottery for choosing young men to be drafted into the military. Numbers from 1 to 366 were randomly assigned to
More informationVariable selection and machine learning methods in causal inference
Variable selection and machine learning methods in causal inference Debashis Ghosh Department of Biostatistics and Informatics Colorado School of Public Health Joint work with Yeying Zhu, University of
More informationSRI Briefing Note Series No.8 Communicating uncertainty in seasonal and interannual climate forecasts in Europe: organisational needs and preferences
ISSN 2056-8843 Sustainability Research Institute SCHOOL OF EARTH AND ENVIRONMENT SRI Briefing Note Series No.8 Communicating uncertainty in seasonal and interannual climate forecasts in Europe: organisational
More informationDavid G. DeWitt Director, Climate Prediction Center (CPC) NOAA/NWS
CPC Current Capabilities and Metrics (Part I) and Key Science Challenges to Improving S2S Forecast Skill (Part II) David G. DeWitt Director, Climate Prediction Center (CPC) NOAA/NWS 1 Outline Disclaimer
More informationC1: From Weather to Climate Looking at Air Temperature Data
C1: From Weather to Climate Looking at Air Temperature Data Purpose Students will work with short- and longterm air temperature data in order to better understand the differences between weather and climate.
More informationNATCOR. Forecast Evaluation. Forecasting with ARIMA models. Nikolaos Kourentzes
NATCOR Forecast Evaluation Forecasting with ARIMA models Nikolaos Kourentzes n.kourentzes@lancaster.ac.uk O u t l i n e 1. Bias measures 2. Accuracy measures 3. Evaluation schemes 4. Prediction intervals
More informationModels for Count and Binary Data. Poisson and Logistic GWR Models. 24/07/2008 GWR Workshop 1
Models for Count and Binary Data Poisson and Logistic GWR Models 24/07/2008 GWR Workshop 1 Outline I: Modelling counts Poisson regression II: Modelling binary events Logistic Regression III: Poisson Regression
More informationDeep Learning Architecture for Univariate Time Series Forecasting
CS229,Technical Report, 2014 Deep Learning Architecture for Univariate Time Series Forecasting Dmitry Vengertsev 1 Abstract This paper studies the problem of applying machine learning with deep architecture
More information