Economics 2010c: Lecture 2 Iterative Methods in Dynamic Programming
|
|
- Owen Charles
- 5 years ago
- Views:
Transcription
1 Economics 2010c: Lecture 2 Iterative Methods in Dynamic Programming David Laibson 9/04/2014
2 Outline: 1. Functional operators 2. Iterative solutions for the Bellman Equation 3. Contraction Mapping Theorem 4. Blackwell s Theorem (Blackwell: , see obituary) 5. Application: Search and stopping problem
3 1 Functional operators: Sequence Problem: Find ( ) such that ( 0 )= sup { +1 } =0 subject to +1 Γ( ) with 0 given. X =0 ( +1 ) Bellman Equation: Find ( ) such that ( ) = sup { ( +1 )+ ( +1 )} +1 Γ( )
4 Bellman operator operating on function is defined ( )( ) sup { ( +1 )+ ( +1 )} +1 Γ( ) Definition is expressed pointwise for one value of butappliestoall values of Operator maps function to a new function. Operator maps functions; itisreferredtoasafunctional operator. The argument of the function may or may not be a solution to the Bellman Equation.
5 Let be a solution to the Bellman Equation. If = then = ( )( ) = sup +1 Γ( ) { ( +1)+ ( +1 )} = ( ) Function is a fixed point of ( maps to ).
6 2 Iterative Solutions for the Bellman Equation 1. (Guess a solution from last lecture. This is not iterative.) 2. Pick some 0 and analytically iterate 0 until convergence. (This is really just for illustration.) 3. Pick some 0 and numerically iterate 0 until convergence. (This is the way that iterative methods are used in most cases.)
7 What does it mean to iterate? ( )( ) = sup +1 Γ( ) { ( +1)+ ( +1 )} ( ( ))( ) = sup +1 Γ( ) { ( +1)+ ( )( +1 )} ( ( 2 n ))( ) = sup +1 Γ( ) ( +1 )+ ( 2 )( +1 ) o. =. ( ( ))( ) = sup +1 Γ( ) { ( +1)+ ( )( +1 )}
8 What does it mean for functions to converge? Notation: lim 0 = Informally, as gets large, the set of remaining elements of the sequence of functions n o are getting closer and closer together. What does it mean for one function to be close to another? maximum distance between the functions is bounded by The We will not worry about these technical issues in this mini-course.
9 3 Contraction Mapping Theorem Why does converge as? Answer: is a contraction mapping. Definition 3.1 Let ( ) be a metric space and : be a function mapping into itself. is a contraction mapping if for some (0 1) ( ) ( ) for any two functions and. Intuition: is a contraction mapping if operating on any two functions moves them strictly closer together: and are strictly closer together than and
10 Remark 3.1 A metric (distance function) is just a way of representing the distance between functions (e.g., the supremum pointwise gap between the two functions).
11 Theorem 3.1 If ( ) is a complete metric space and : is a contraction mapping, then: 1) has exactly one fixed point 2) for any 0 lim 0 = 3) 0 has an exponential convergence rate at least as great as ln( )
12 Consider the contraction mapping ( )( ) ( )+ ( ) with (0 1) ( )( ) = ( )+ ( ) ( ( ))( ) = 2 ( )+ ( )+ ( ) ( ( 2 ))( ) = 3 ( )+ 2 ( )+ ( )+ ( ) ( )( ) = ( )+ 1 ( )+ + ( ) lim( )( ) = ( ) (1 ) So fixed point, ( ) is ( ) = ( ) (1 ) Note that ( )( ) = ( )
13 ProofofContractionMappingTheorem Broad strategy: 1. Show that { 0 } =0 is a Cauchy sequence. 2. Show that the limit point is a fixed point of. 3. Show that only one fixed point exists. 4. Bound the rate of convergence.
14 Choose some 0 Let = 0 Since is a contraction Continuing by induction, ( 2 1 )= ( 1 0 ) ( 1 0 ) ( +1 ) ( 1 0 ) We can now bound the distance between and when ( ) ( 1 )+ + ( )+ ( +1 ) h i ( 1 0 ) = h i ( 1 0 ) So { } =0 is Cauchy. X 1 ( 1 0 )
15 Since is complete We now have a candidate fixed point To show that = note ( ) ( 0 )+ ( 0 ) ( 1 0 )+ ( 0 ) These inequalities must hold for all And both terms on the RHS converge to zero as So ( ) =0 implying that = X
16 Now we show that our fixed point is unique. Suppose there were two fixed points: 6= Then = and = (since fixed points). Also have ( ) ( ) (since is contraction) So, ( )= ( ) ( ) Contradiction. So the fixed point is unique. X
17 Finally, note that ( 0 ) = ( ( 1 0 ) ) ( 1 0 ) So ( 0 ) ( 0 ) follows by induction. X This completes the proof of the Contraction Mapping Theorem.
18 4 Blackwell s Theorem These are sufficient conditions for an operator to be contraction mapping. Theorem 4.1 (Blackwell s sufficient conditions) Let < and let ( ) be a space of bounded functions : <, with the sup-metric. Let : ( ) ( ) be an operator satisfying two conditions: 1) (monotonicity) if ( ) and ( ) ( ), then ( )( ) ( )( ) 2) (discounting) there exists some (0 1) such that [ ( + )]( ) ( )( )+ ( ) 0 Then is a contraction with modulus
19 Remark 4.1 Note that is a constant and ( + ) is the function generated by adding to the function Remark 4.2 Blackwell sconditionsaresufficient but not necessary for to be a contraction.
20 ProofofBlackwell stheorem: For any ( ) ( ) ( )+ ( ) We ll write this as + ( ) So property 1) and 2) imply ( + ( )) + ( ) ( + ( )) + ( )
21 Combining the last two lines, we have ( ) ( ) ( )( ) ( )( ) ( ) sup ( )( ) ( )( ) ( ) ( ) ( ) X
22 Example 4.1 Check Blackwell conditions for Bellman operator in a consumption problem (with stochastic asset returns, stochastic labor income, and a liquidity constraint). ( )( ) = sup [0 ] n ( )+ ( +1 ( )+ +1 ) o Monotonicity. Assume ( ) ( ). Suppose is the optimal policy when the continuation value function is ( )( ) = sup [0 ] n ( )+ ( +1 ( )+ +1 ) o = ( )+ ( +1 ( )+ +1) ( )+ ( +1 ( )+ +1) n ( )+ ( +1 ( )+ +1 ) o sup [0 ] = ( )( )
23 Discounting. Adding a constant ( ) to an optimization problem does not affect optimal choice, so n h [ ( + )]( ) = sup ( )+ ( +1 ( )+ +1 )+ io [0 ] n = sup ( )+ ( +1 ( )+ +1 ) o + [0 ] = ( )( )+
24 5 Application: Search/ stopping. An agent draws an offer, from a uniform distribution with support in the unit interval. The agent can either accept the offer and realize net present value (ending the game) or the agent can reject the offer and draw again a period later. All draws are independent. Rejections are costly because the agent discounts the future with discount factor. This game continues until the agent receives an offer that she is willing to accept. The Bellman equation for this problem is: ( ) =max{ h ( 0 ) i }
25 In the first lecture, we showed that the solution would be a stationary threshold: REJECT if ACCEPT if This implies that ( ) = ( if if ) Weshowedthatif = 1 µ 1 q1 2 then, ( ) solves the Bellman Equation.
26 Iteration of Bellman Operator starting with 0 ( ) =0 This is the Bellman Operator for our problem: ( )( ) max{ h ( 0 ) i } Iterating the Bellman Operator once on 0 ( ) yields 1 ( ) = ( 0 )( ) = max{ h 0 ( 0 ) i } = max{ 0} = So associated threshold cutoff is 0 In other words, accept all offers greater than or equal to 0 since the continuation payoff function is 0 =0
27 General notation. Iteration of the Bellman Operator,. ( ) = ( 0 )( ) = = ( 1 0 )( ) max{ h ( 1 0 )( 0 ) i } Let h ( 1 0 )( 0 ) i, which is the continuation payoff for ( ) is also the cutoff threshold associated with ( ) =( 0 )( ) ( ) ( ) =( if 0 )( ) = if is a sufficient statistic for ( 0 )( )
28 Another illustrative example. Iterate the functional operator on 0 2 ( ) = ( 2 0 )( ) = ( 0 )( ) = max{ ( 0 )( 0 )} = max{ 1 ( 0 )} = max{ 0 } 2 = 0 = 2 2 ( ) =( 2 0 )( ) =( )
29 The proof itself: ( 1 0 )( ) =( 1 if 1 if 1 ) = ( 1 0 )( 0 ) = = 2 " Z 1 1 ( ) + =0 ³ Z =1 = 1 ( ) # Set = 1 to confirm that this sequence converges to µ lim = 1 1 q1 2
30 Iterating functional for search/stopping problem with discount rate =.1 = -ln(delta) xn value x x iteration number
31 Review of Today s Lecture: 1. Functional operators 2. Iterative solutions for the Bellman Equation 3. Contraction Mapping Theorem 4. Blackwell s Theorem 5. Application: Search and stopping problem (analytic iteration)
Economics 2010c: Lectures 9-10 Bellman Equation in Continuous Time
Economics 2010c: Lectures 9-10 Bellman Equation in Continuous Time David Laibson 9/30/2014 Outline Lectures 9-10: 9.1 Continuous-time Bellman Equation 9.2 Application: Merton s Problem 9.3 Application:
More informationDYNAMIC LECTURE 5: DISCRETE TIME INTERTEMPORAL OPTIMIZATION
DYNAMIC LECTURE 5: DISCRETE TIME INTERTEMPORAL OPTIMIZATION UNIVERSITY OF MARYLAND: ECON 600. Alternative Methods of Discrete Time Intertemporal Optimization We will start by solving a discrete time intertemporal
More informationBasic Deterministic Dynamic Programming
Basic Deterministic Dynamic Programming Timothy Kam School of Economics & CAMA Australian National University ECON8022, This version March 17, 2008 Motivation What do we do? Outline Deterministic IHDP
More informationStochastic Dynamic Programming: The One Sector Growth Model
Stochastic Dynamic Programming: The One Sector Growth Model Esteban Rossi-Hansberg Princeton University March 26, 2012 Esteban Rossi-Hansberg () Stochastic Dynamic Programming March 26, 2012 1 / 31 References
More information1 Stochastic Dynamic Programming
1 Stochastic Dynamic Programming Formally, a stochastic dynamic program has the same components as a deterministic one; the only modification is to the state transition equation. When events in the future
More informationLecture 5: The Bellman Equation
Lecture 5: The Bellman Equation Florian Scheuer 1 Plan Prove properties of the Bellman equation (In particular, existence and uniqueness of solution) Use this to prove properties of the solution Think
More informationd(x n, x) d(x n, x nk ) + d(x nk, x) where we chose any fixed k > N
Problem 1. Let f : A R R have the property that for every x A, there exists ɛ > 0 such that f(t) > ɛ if t (x ɛ, x + ɛ) A. If the set A is compact, prove there exists c > 0 such that f(x) > c for all x
More informationEconomics 2010c: Lecture 3 The Classical Consumption Model
Economics 2010c: Lecture 3 The Classical Consumption Model David Laibson 9/9/2014 Outline: 1. Consumption: Basic model and early theories 2. Linearization of the Euler Equation 3. Empirical tests without
More informationMacro 1: Dynamic Programming 1
Macro 1: Dynamic Programming 1 Mark Huggett 2 2 Georgetown September, 2016 DP Warm up: Cake eating problem ( ) max f 1 (y 1 ) + f 2 (y 2 ) s.t. y 1 + y 2 100, y 1 0, y 2 0 1. v 1 (x) max f 1(y 1 ) + f
More informationSlides II - Dynamic Programming
Slides II - Dynamic Programming Julio Garín University of Georgia Macroeconomic Theory II (Ph.D.) Spring 2017 Macroeconomic Theory II Slides II - Dynamic Programming Spring 2017 1 / 32 Outline 1. Lagrangian
More information2. What is the fraction of aggregate savings due to the precautionary motive? (These two questions are analyzed in the paper by Ayiagari)
University of Minnesota 8107 Macroeconomic Theory, Spring 2012, Mini 1 Fabrizio Perri Stationary equilibria in economies with Idiosyncratic Risk and Incomplete Markets We are now at the point in which
More informationEconomics 204 Fall 2011 Problem Set 2 Suggested Solutions
Economics 24 Fall 211 Problem Set 2 Suggested Solutions 1. Determine whether the following sets are open, closed, both or neither under the topology induced by the usual metric. (Hint: think about limit
More informationEconomics 204 Summer/Fall 2011 Lecture 5 Friday July 29, 2011
Economics 204 Summer/Fall 2011 Lecture 5 Friday July 29, 2011 Section 2.6 (cont.) Properties of Real Functions Here we first study properties of functions from R to R, making use of the additional structure
More informationADVANCED MACROECONOMIC TECHNIQUES NOTE 3a
316-406 ADVANCED MACROECONOMIC TECHNIQUES NOTE 3a Chris Edmond hcpedmond@unimelb.edu.aui Dynamic programming and the growth model Dynamic programming and closely related recursive methods provide an important
More informationDynamic Programming Theorems
Dynamic Programming Theorems Prof. Lutz Hendricks Econ720 September 11, 2017 1 / 39 Dynamic Programming Theorems Useful theorems to characterize the solution to a DP problem. There is no reason to remember
More information1 Markov decision processes
2.997 Decision-Making in Large-Scale Systems February 4 MI, Spring 2004 Handout #1 Lecture Note 1 1 Markov decision processes In this class we will study discrete-time stochastic systems. We can describe
More informationUniversity of Warwick, EC9A0 Maths for Economists Lecture Notes 10: Dynamic Programming
University of Warwick, EC9A0 Maths for Economists 1 of 63 University of Warwick, EC9A0 Maths for Economists Lecture Notes 10: Dynamic Programming Peter J. Hammond Autumn 2013, revised 2014 University of
More informationMarkov Decision Processes Infinite Horizon Problems
Markov Decision Processes Infinite Horizon Problems Alan Fern * * Based in part on slides by Craig Boutilier and Daniel Weld 1 What is a solution to an MDP? MDP Planning Problem: Input: an MDP (S,A,R,T)
More informationGeneral Examination in Macroeconomic Theory SPRING 2013
HARVARD UNIVERSITY DEPARTMENT OF ECONOMICS General Examination in Macroeconomic Theory SPRING 203 You have FOUR hours. Answer all questions Part A (Prof. Laibson): 48 minutes Part B (Prof. Aghion): 48
More informationContents. An example 5. Mathematical Preliminaries 13. Dynamic programming under certainty 29. Numerical methods 41. Some applications 57
T H O M A S D E M U Y N C K DY N A M I C O P T I M I Z AT I O N Contents An example 5 Mathematical Preliminaries 13 Dynamic programming under certainty 29 Numerical methods 41 Some applications 57 Stochastic
More informationLecture 4: Completion of a Metric Space
15 Lecture 4: Completion of a Metric Space Closure vs. Completeness. Recall the statement of Lemma??(b): A subspace M of a metric space X is closed if and only if every convergent sequence {x n } X satisfying
More informationECON 2010c Solution to Problem Set 1
ECON 200c Solution to Problem Set By the Teaching Fellows for ECON 200c Fall 204 Growth Model (a) Defining the constant κ as: κ = ln( αβ) + αβ αβ ln(αβ), the problem asks us to show that the following
More informationECON 582: Dynamic Programming (Chapter 6, Acemoglu) Instructor: Dmytro Hryshko
ECON 582: Dynamic Programming (Chapter 6, Acemoglu) Instructor: Dmytro Hryshko Indirect Utility Recall: static consumer theory; J goods, p j is the price of good j (j = 1; : : : ; J), c j is consumption
More informationAdvanced Economic Growth: Lecture 21: Stochastic Dynamic Programming and Applications
Advanced Economic Growth: Lecture 21: Stochastic Dynamic Programming and Applications Daron Acemoglu MIT November 19, 2007 Daron Acemoglu (MIT) Advanced Growth Lecture 21 November 19, 2007 1 / 79 Stochastic
More informationMath 118B Solutions. Charles Martin. March 6, d i (x i, y i ) + d i (y i, z i ) = d(x, y) + d(y, z). i=1
Math 8B Solutions Charles Martin March 6, Homework Problems. Let (X i, d i ), i n, be finitely many metric spaces. Construct a metric on the product space X = X X n. Proof. Denote points in X as x = (x,
More informationSimple Consumption / Savings Problems (based on Ljungqvist & Sargent, Ch 16, 17) Jonathan Heathcote. updated, March The household s problem X
Simple Consumption / Savings Problems (based on Ljungqvist & Sargent, Ch 16, 17) subject to for all t Jonathan Heathcote updated, March 2006 1. The household s problem max E β t u (c t ) t=0 c t + a t+1
More informationEconomics 210B Due: September 16, Problem Set 10. s.t. k t+1 = R(k t c t ) for all t 0, and k 0 given, lim. and
Economics 210B Due: September 16, 2010 Problem 1: Constant returns to saving Consider the following problem. c0,k1,c1,k2,... β t Problem Set 10 1 α c1 α t s.t. k t+1 = R(k t c t ) for all t 0, and k 0
More informationUnderstanding (Exact) Dynamic Programming through Bellman Operators
Understanding (Exact) Dynamic Programming through Bellman Operators Ashwin Rao ICME, Stanford University January 15, 2019 Ashwin Rao (Stanford) Bellman Operators January 15, 2019 1 / 11 Overview 1 Value
More information9 - Markov processes and Burt & Allison 1963 AGEC
This document was generated at 8:37 PM on Saturday, March 17, 2018 Copyright 2018 Richard T. Woodward 9 - Markov processes and Burt & Allison 1963 AGEC 642-2018 I. What is a Markov Chain? A Markov chain
More informationMarkov Decision Processes
Markov Decision Processes Lecture notes for the course Games on Graphs B. Srivathsan Chennai Mathematical Institute, India 1 Markov Chains We will define Markov chains in a manner that will be useful to
More informationAn Application to Growth Theory
An Application to Growth Theory First let s review the concepts of solution function and value function for a maximization problem. Suppose we have the problem max F (x, α) subject to G(x, β) 0, (P) x
More informationRecursive Methods. Introduction to Dynamic Optimization
Recursive Methods Nr. 1 Outline Today s Lecture finish off: theorem of the maximum Bellman equation with bounded and continuous F differentiability of value function application: neoclassical growth model
More informationFixed Point Theorems
Fixed Point Theorems Definition: Let X be a set and let f : X X be a function that maps X into itself. (Such a function is often called an operator, a transformation, or a transform on X, and the notation
More informationStochastic Shortest Path Problems
Chapter 8 Stochastic Shortest Path Problems 1 In this chapter, we study a stochastic version of the shortest path problem of chapter 2, where only probabilities of transitions along different arcs can
More informationLecture 1. Evolution of Market Concentration
Lecture 1 Evolution of Market Concentration Take a look at : Doraszelski and Pakes, A Framework for Applied Dynamic Analysis in IO, Handbook of I.O. Chapter. (see link at syllabus). Matt Shum s notes are
More informationStochastic Dynamic Programming. Jesus Fernandez-Villaverde University of Pennsylvania
Stochastic Dynamic Programming Jesus Fernande-Villaverde University of Pennsylvania 1 Introducing Uncertainty in Dynamic Programming Stochastic dynamic programming presents a very exible framework to handle
More informationOutline Today s Lecture
Outline Today s Lecture finish Euler Equations and Transversality Condition Principle of Optimality: Bellman s Equation Study of Bellman equation with bounded F contraction mapping and theorem of the maximum
More informationLecture 4: Dynamic Programming
Lecture 4: Dynamic Programming Fatih Guvenen January 10, 2016 Fatih Guvenen Lecture 4: Dynamic Programming January 10, 2016 1 / 30 Goal Solve V (k, z) =max c,k 0 u(c)+ E(V (k 0, z 0 ) z) c + k 0 =(1 +
More informationMacroeconomic Theory II Homework 2 - Solution
Macroeconomic Theory II Homework 2 - Solution Professor Gianluca Violante, TA: Diego Daruich New York University Spring 204 Problem The household has preferences over the stochastic processes of a single
More informationCSE250A Fall 12: Discussion Week 9
CSE250A Fall 12: Discussion Week 9 Aditya Menon (akmenon@ucsd.edu) December 4, 2012 1 Schedule for today Recap of Markov Decision Processes. Examples: slot machines and maze traversal. Planning and learning.
More informationLecture Pakes, Ostrovsky, and Berry. Dynamic and Stochastic Model of Industry
Lecture Pakes, Ostrovsky, and Berry Dynamic and Stochastic Model of Industry Let π n be flow profit of incumbant firms when n firms are in the industry. π 1 > 0, 0 π 2
More informationMATH3283W LECTURE NOTES: WEEK 6 = 5 13, = 2 5, 1 13
MATH383W LECTURE NOTES: WEEK 6 //00 Recursive sequences (cont.) Examples: () a =, a n+ = 3 a n. The first few terms are,,, 5 = 5, 3 5 = 5 3, Since 5
More informationLecture 1: Dynamic Programming
Lecture 1: Dynamic Programming Fatih Guvenen November 2, 2016 Fatih Guvenen Lecture 1: Dynamic Programming November 2, 2016 1 / 32 Goal Solve V (k, z) =max c,k 0 u(c)+ E(V (k 0, z 0 ) z) c + k 0 =(1 +
More informationSome AI Planning Problems
Course Logistics CS533: Intelligent Agents and Decision Making M, W, F: 1:00 1:50 Instructor: Alan Fern (KEC2071) Office hours: by appointment (see me after class or send email) Emailing me: include CS533
More informationLecture 4: The Bellman Operator Dynamic Programming
Lecture 4: The Bellman Operator Dynamic Programming Jeppe Druedahl Department of Economics 15th of February 2016 Slide 1/19 Infinite horizon, t We know V 0 (M t ) = whatever { } V 1 (M t ) = max u(m t,
More informationAbstract Dynamic Programming
Abstract Dynamic Programming Dimitri P. Bertsekas Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology Overview of the Research Monograph Abstract Dynamic Programming"
More informationTime is discrete and indexed by t =0; 1;:::;T,whereT<1. An individual is interested in maximizing an objective function given by. tu(x t ;a t ); (0.
Chapter 0 Discrete Time Dynamic Programming 0.1 The Finite Horizon Case Time is discrete and indexed by t =0; 1;:::;T,whereT
More informationSequences. Limits of Sequences. Definition. A real-valued sequence s is any function s : N R.
Sequences Limits of Sequences. Definition. A real-valued sequence s is any function s : N R. Usually, instead of using the notation s(n), we write s n for the value of this function calculated at n. We
More informationa) Let x,y be Cauchy sequences in some metric space. Define d(x, y) = lim n d (x n, y n ). Show that this function is well-defined.
Problem 3) Remark: for this problem, if I write the notation lim x n, it should always be assumed that I mean lim n x n, and similarly if I write the notation lim x nk it should always be assumed that
More information6.2 Deeper Properties of Continuous Functions
6.2. DEEPER PROPERTIES OF CONTINUOUS FUNCTIONS 69 6.2 Deeper Properties of Continuous Functions 6.2. Intermediate Value Theorem and Consequences When one studies a function, one is usually interested in
More informationAn approximate consumption function
An approximate consumption function Mario Padula Very Preliminary and Very Incomplete 8 December 2005 Abstract This notes proposes an approximation to the consumption function in the buffer-stock model.
More informationMath 328 Course Notes
Math 328 Course Notes Ian Robertson March 3, 2006 3 Properties of C[0, 1]: Sup-norm and Completeness In this chapter we are going to examine the vector space of all continuous functions defined on the
More informationCharacterisation of Accumulation Points. Convergence in Metric Spaces. Characterisation of Closed Sets. Characterisation of Closed Sets
Convergence in Metric Spaces Functional Analysis Lecture 3: Convergence and Continuity in Metric Spaces Bengt Ove Turesson September 4, 2016 Suppose that (X, d) is a metric space. A sequence (x n ) X is
More information1 Stochastic Dynamic Programming
1 Stochastic Dynamic Programming Formally, a stochastic dynamic program has the same components as a deterministic one; the only modification is to the state transition equation. When events in the future
More informationL p Functions. Given a measure space (X, µ) and a real number p [1, ), recall that the L p -norm of a measurable function f : X R is defined by
L p Functions Given a measure space (, µ) and a real number p [, ), recall that the L p -norm of a measurable function f : R is defined by f p = ( ) /p f p dµ Note that the L p -norm of a function f may
More informationSequences. Chapter 3. n + 1 3n + 2 sin n n. 3. lim (ln(n + 1) ln n) 1. lim. 2. lim. 4. lim (1 + n)1/n. Answers: 1. 1/3; 2. 0; 3. 0; 4. 1.
Chapter 3 Sequences Both the main elements of calculus (differentiation and integration) require the notion of a limit. Sequences will play a central role when we work with limits. Definition 3.. A Sequence
More informationOnline Appendix Durable Goods Monopoly with Stochastic Costs
Online Appendix Durable Goods Monopoly with Stochastic Costs Juan Ortner Boston University March 2, 2016 OA1 Online Appendix OA1.1 Proof of Theorem 2 The proof of Theorem 2 is organized as follows. First,
More informationDynamic Problem Set 1 Solutions
Dynamic Problem Set 1 Solutions Jonathan Kreamer July 15, 2011 Question 1 Consider the following multi-period optimal storage problem: An economic agent imizes: c t} T β t u(c t ) (1) subject to the period-by-period
More informationMath 104: Homework 7 solutions
Math 04: Homework 7 solutions. (a) The derivative of f () = is f () = 2 which is unbounded as 0. Since f () is continuous on [0, ], it is uniformly continous on this interval by Theorem 9.2. Hence for
More informationECON607 Fall 2010 University of Hawaii Professor Hui He TA: Xiaodong Sun Assignment 2
ECON607 Fall 200 University of Hawaii Professor Hui He TA: Xiaodong Sun Assignment 2 The due date for this assignment is Tuesday, October 2. ( Total points = 50). (Two-sector growth model) Consider the
More informationModule 16: Signaling
Module 16: Signaling Information Economics (Ec 515) George Georgiadis Players with private information can take some action to signal their type. Taking this action would distinguish them from other types.
More informationPolitical Economy of Institutions and Development: Problem Set 1. Due Date: Thursday, February 23, in class.
Political Economy of Institutions and Development: 14.773 Problem Set 1 Due Date: Thursday, February 23, in class. Answer Questions 1-3. handed in. The other two questions are for practice and are not
More information5.5 Deeper Properties of Continuous Functions
5.5. DEEPER PROPERTIES OF CONTINUOUS FUNCTIONS 195 5.5 Deeper Properties of Continuous Functions 5.5.1 Intermediate Value Theorem and Consequences When one studies a function, one is usually interested
More informationPlanning in Markov Decision Processes
Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Planning in Markov Decision Processes Lecture 3, CMU 10703 Katerina Fragkiadaki Markov Decision Process (MDP) A Markov
More informationThe Principle of Optimality
The Principle of Optimality Sequence Problem and Recursive Problem Sequence problem: Notation: V (x 0 ) sup {x t} β t F (x t, x t+ ) s.t. x t+ Γ (x t ) x 0 given t () Plans: x = {x t } Continuation plans
More informationLecture notes for Analysis of Algorithms : Markov decision processes
Lecture notes for Analysis of Algorithms : Markov decision processes Lecturer: Thomas Dueholm Hansen June 6, 013 Abstract We give an introduction to infinite-horizon Markov decision processes (MDPs) with
More informationMath 320-2: Midterm 2 Practice Solutions Northwestern University, Winter 2015
Math 30-: Midterm Practice Solutions Northwestern University, Winter 015 1. Give an example of each of the following. No justification is needed. (a) A metric on R with respect to which R is bounded. (b)
More informationArea I: Contract Theory Question (Econ 206)
Theory Field Exam Summer 2011 Instructions You must complete two of the four areas (the areas being (I) contract theory, (II) game theory A, (III) game theory B, and (IV) psychology & economics). Be sure
More informationSeptember Math Course: First Order Derivative
September Math Course: First Order Derivative Arina Nikandrova Functions Function y = f (x), where x is either be a scalar or a vector of several variables (x,..., x n ), can be thought of as a rule which
More informationMDP Preliminaries. Nan Jiang. February 10, 2019
MDP Preliminaries Nan Jiang February 10, 2019 1 Markov Decision Processes In reinforcement learning, the interactions between the agent and the environment are often described by a Markov Decision Process
More informationConsumption-Savings Decisions with Quasi-Geometric Discounting: The Case with a Discrete Domain
Consumption-Savings Decisions with Quasi-Geometric Discounting: The Case with a Discrete Domain Per Krusell Anthony A. Smith, Jr. 1 December 2008 Abstract How do individuals with time-inconsistent preferences
More informationMacro 1: Dynamic Programming 2
Macro 1: Dynamic Programming 2 Mark Huggett 2 2 Georgetown September, 2016 DP Problems with Risk Strategy: Consider three classic problems: income fluctuation, optimal (stochastic) growth and search. Learn
More informationMetric Spaces and Topology
Chapter 2 Metric Spaces and Topology From an engineering perspective, the most important way to construct a topology on a set is to define the topology in terms of a metric on the set. This approach underlies
More informationMath 117: Infinite Sequences
Math 7: Infinite Sequences John Douglas Moore November, 008 The three main theorems in the theory of infinite sequences are the Monotone Convergence Theorem, the Cauchy Sequence Theorem and the Subsequence
More informationT 1. The value function v(x) is the expected net gain when using the optimal stopping time starting at state x:
108 OPTIMAL STOPPING TIME 4.4. Cost functions. The cost function g(x) gives the price you must pay to continue from state x. If T is your stopping time then X T is your stopping state and f(x T ) is your
More informationECOM 009 Macroeconomics B. Lecture 2
ECOM 009 Macroeconomics B Lecture 2 Giulio Fella c Giulio Fella, 2014 ECOM 009 Macroeconomics B - Lecture 2 40/197 Aim of consumption theory Consumption theory aims at explaining consumption/saving decisions
More informationHeterogeneous Agent Models: I
Heterogeneous Agent Models: I Mark Huggett 2 2 Georgetown September, 2017 Introduction Early heterogeneous-agent models integrated the income-fluctuation problem into general equilibrium models. A key
More informationCMU Lecture 11: Markov Decision Processes II. Teacher: Gianni A. Di Caro
CMU 15-781 Lecture 11: Markov Decision Processes II Teacher: Gianni A. Di Caro RECAP: DEFINING MDPS Markov decision processes: o Set of states S o Start state s 0 o Set of actions A o Transitions P(s s,a)
More informationEstimating Single-Agent Dynamic Models
Estimating Single-Agent Dynamic Models Paul T. Scott New York University Empirical IO Course Fall 2016 1 / 34 Introduction Why dynamic estimation? External validity Famous example: Hendel and Nevo s (2006)
More informationMath 410 Homework 6 Due Monday, October 26
Math 40 Homework 6 Due Monday, October 26. Let c be any constant and assume that lim s n = s and lim t n = t. Prove that: a) lim c s n = c s We talked about these in class: We want to show that for all
More informationSequences. We know that the functions can be defined on any subsets of R. As the set of positive integers
Sequences We know that the functions can be defined on any subsets of R. As the set of positive integers Z + is a subset of R, we can define a function on it in the following manner. f: Z + R f(n) = a
More informationconverges as well if x < 1. 1 x n x n 1 1 = 2 a nx n
Solve the following 6 problems. 1. Prove that if series n=1 a nx n converges for all x such that x < 1, then the series n=1 a n xn 1 x converges as well if x < 1. n For x < 1, x n 0 as n, so there exists
More informationECOM 009 Macroeconomics B. Lecture 3
ECOM 009 Macroeconomics B Lecture 3 Giulio Fella c Giulio Fella, 2014 ECOM 009 Macroeconomics B - Lecture 3 84/197 Predictions of the PICH 1. Marginal propensity to consume out of wealth windfalls 0.03.
More informationValue and Policy Iteration
Chapter 7 Value and Policy Iteration 1 For infinite horizon problems, we need to replace our basic computational tool, the DP algorithm, which we used to compute the optimal cost and policy for finite
More informationAssignment-10. (Due 11/21) Solution: Any continuous function on a compact set is uniformly continuous.
Assignment-1 (Due 11/21) 1. Consider the sequence of functions f n (x) = x n on [, 1]. (a) Show that each function f n is uniformly continuous on [, 1]. Solution: Any continuous function on a compact set
More informationScalar multiplication and addition of sequences 9
8 Sequences 1.2.7. Proposition. Every subsequence of a convergent sequence (a n ) n N converges to lim n a n. Proof. If (a nk ) k N is a subsequence of (a n ) n N, then n k k for every k. Hence if ε >
More informationExample I: Capital Accumulation
1 Example I: Capital Accumulation Time t = 0, 1,..., T < Output y, initial output y 0 Fraction of output invested a, capital k = ay Transition (production function) y = g(k) = g(ay) Reward (utility of
More informationMarkov Decision Processes and Dynamic Programming
Markov Decision Processes and Dynamic Programming A. LAZARIC (SequeL Team @INRIA-Lille) Ecole Centrale - Option DAD SequeL INRIA Lille EC-RL Course In This Lecture A. LAZARIC Markov Decision Processes
More information, and rewards and transition matrices as shown below:
CSE 50a. Assignment 7 Out: Tue Nov Due: Thu Dec Reading: Sutton & Barto, Chapters -. 7. Policy improvement Consider the Markov decision process (MDP) with two states s {0, }, two actions a {0, }, discount
More information1 THE GAME. Two players, i=1, 2 U i : concave, strictly increasing f: concave, continuous, f(0) 0 β (0, 1): discount factor, common
1 THE GAME Two players, i=1, 2 U i : concave, strictly increasing f: concave, continuous, f(0) 0 β (0, 1): discount factor, common With Law of motion of the state: Payoff: Histories: Strategies: k t+1
More informationPolitical Economy of Institutions and Development. Lecture 8. Institutional Change and Democratization
14.773 Political Economy of Institutions and Development. Lecture 8. Institutional Change and Democratization Daron Acemoglu MIT March 5, 2013. Daron Acemoglu (MIT) Political Economy Lecture 8 March 5,
More informationAdvanced Calculus I Chapter 2 & 3 Homework Solutions October 30, Prove that f has a limit at 2 and x + 2 find it. f(x) = 2x2 + 3x 2 x + 2
Advanced Calculus I Chapter 2 & 3 Homework Solutions October 30, 2009 2. Define f : ( 2, 0) R by f(x) = 2x2 + 3x 2. Prove that f has a limit at 2 and x + 2 find it. Note that when x 2 we have f(x) = 2x2
More informationAppendix (For Online Publication) Community Development by Public Wealth Accumulation
March 219 Appendix (For Online Publication) to Community Development by Public Wealth Accumulation Levon Barseghyan Department of Economics Cornell University Ithaca NY 14853 lb247@cornell.edu Stephen
More informationReinforcement Learning and Optimal Control. ASU, CSE 691, Winter 2019
Reinforcement Learning and Optimal Control ASU, CSE 691, Winter 2019 Dimitri P. Bertsekas dimitrib@mit.edu Lecture 8 Bertsekas Reinforcement Learning 1 / 21 Outline 1 Review of Infinite Horizon Problems
More information1 MDP Value Iteration Algorithm
CS 0. - Active Learning Problem Set Handed out: 4 Jan 009 Due: 9 Jan 009 MDP Value Iteration Algorithm. Implement the value iteration algorithm given in the lecture. That is, solve Bellman s equation using
More informationIn last semester, we have seen some examples about it (See Tutorial Note #13). Try to have a look on that. Here we try to show more technique.
MATH202 Introduction to Analysis (2007 Fall and 2008 Spring) Tutorial Note #4 Part I: Cauchy Sequence Definition (Cauchy Sequence): A sequence of real number { n } is Cauchy if and only if for any ε >
More informationMAS221 Analysis Semester Chapter 2 problems
MAS221 Analysis Semester 1 2018-19 Chapter 2 problems 20. Consider the sequence (a n ), with general term a n = 1 + 3. Can you n guess the limit l of this sequence? (a) Verify that your guess is plausible
More informationDynamic Discrete Choice Structural Models in Empirical IO
Dynamic Discrete Choice Structural Models in Empirical IO Lecture 4: Euler Equations and Finite Dependence in Dynamic Discrete Choice Models Victor Aguirregabiria (University of Toronto) Carlos III, Madrid
More informationWe have been going places in the car of calculus for years, but this analysis course is about how the car actually works.
Analysis I We have been going places in the car of calculus for years, but this analysis course is about how the car actually works. Copier s Message These notes may contain errors. In fact, they almost
More informationMS&E338 Reinforcement Learning Lecture 1 - April 2, Introduction
MS&E338 Reinforcement Learning Lecture 1 - April 2, 2018 Introduction Lecturer: Ben Van Roy Scribe: Gabriel Maher 1 Reinforcement Learning Introduction In reinforcement learning (RL) we consider an agent
More information