Markov Decision Processes: Biosens II

Transcription

1 Markov Decision Processes: Biosens II E. Jørgensen & Lars R. Nielsen Department of Genetics and Biotechnology Faculty of Agricultural Sciences, University of Århus / 008

2 : Markov Decision Processes Examples : Biosens project Dairy Replacement models Sow Replacement models Delivery strategy... Electronic ID and recording at each milking Robot milking Yield, temperature In-line laboratory Lactate DeHydrogenase (LDH) Electric Conductivity (EC) Somatic cell counts Progesterone... : Biosens project Methods Biosens II Kalman-filter and DLM/State Space Models Multiprocess: outlier and trend model Decisions via heuristic decision rules Implemented in Herd navigator Biosens II: Improved monitoring and management of dairy production based on on-farm biosensors Goal: Better detection of oestrus and illnesses Focus on biomarkers in milk (progesterone, LDH, yield, etc.) Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) Five year project ( ). Budget 5 mill EUR Commercial product Herd Navigator TM based on Biosens project ( OR50 Sep. 11 th 008 / 1 Biosens II Biosens II Biosens II: Improved monitoring and management of dairy production based on on-farm biosensors Goal: Better detection of oestrus and illnesses Focus on biomarkers in milk (progesterone, LDH, yield, etc.) Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) Five year project ( ). Budget 5 mill EUR Commercial product Herd Navigator TM based on Biosens project ( OR50 Sep. 11 th 008 / 1 Biosens II: Improved monitoring and management of dairy production based on on-farm biosensors Goal: Better detection of oestrus and illnesses Focus on biomarkers in milk (progesterone, LDH, yield, etc.) Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) Five year project ( ). Budget 5 mill EUR Commercial product Herd Navigator TM based on Biosens project ( OR50 Sep. 11 th 008 / 1

3 Biosens II Biosens II Biosens II: Improved monitoring and management of dairy production based on on-farm biosensors Goal: Better detection of oestrus and illnesses Focus on biomarkers in milk (progesterone, LDH, yield, etc.) Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) Five year project ( ). Budget 5 mill EUR Commercial product Herd Navigator TM based on Biosens project ( OR50 Sep. 11 th 008 / 1 Biosens II: Improved monitoring and management of dairy production based on on-farm biosensors Goal: Better detection of oestrus and illnesses Focus on biomarkers in milk (progesterone, LDH, yield, etc.) Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) Five year project ( ). Budget 5 mill EUR Commercial product Herd Navigator TM based on Biosens project ( OR50 Sep. 11 th 008 / 1 Biosens II Use decision theoretical for optimizing of decisions BioSens II Biosens II: Improved monitoring and management of dairy production based on on-farm biosensors Goal: Better detection of oestrus and illnesses Focus on biomarkers in milk (progesterone, LDH, yield, etc.) Commercial partner Lattec I/S (FOSS A/S and DeLaval AB) Five year project ( ). Budget 5 mill EUR Commercial product Herd Navigator TM based on Biosens project ( Base decisions on herd specific parameters Optimize treatment strategies on cow level (economic point of view). Need model for predicting milk yield, degree of infection, oestrus state space models (SSMs). Need model for calculating the optimal strategy Markov decision processes (MDPs). OR50 Sep. 11 th 008 / 1 Use decision theoretical for optimizing of decisions BioSens II Project Phases Phases Base decisions on herd specific parameters Optimize treatment strategies on cow level (economic point of view). Need model for predicting milk yield, degree of infection, oestrus state space models (SSMs). Need model for calculating the optimal strategy Markov decision processes (MDPs). Extend existing Dairy replacement models. Include decisions and registrations related to reproduction Include decisions and registrations related to disease (Mastitis)

4 Daily yield Reproduction Involuntary culling Model components Phase 1 Growth and economic parameters MDPs applied to dairy Many papers about the dairy cow replacement problem but limited use in pratice. Reasons could be: - Often large and complex models with no friendly user interface. - Parameters in the model must be estimated, i.e. data collection frameworks at herd level must exist. - Stage length: one month up to a year no assistance when to inseminate, treat or cull the cow in the current month. Bio-sensors and cow specific traits/interventions exists in modern dairy herds parameters can be estimated on a daily basis. OR50 Sep. 11 th / 1 MDPs applied to dairy MDPs applied to dairy Many papers about the dairy cow replacement problem but limited use in pratice. Reasons could be: Many papers about the dairy cow replacement problem but limited use in pratice. Reasons could be: - Often large and complex models with no friendly user interface. - Often large and complex models with no friendly user interface. - Parameters in the model must be estimated, i.e. data collection frameworks at herd level must exist. - Parameters in the model must be estimated, i.e. data collection frameworks at herd level must exist. - Stage length: one month up to a year no assistance when to inseminate, treat or cull the cow in the current month. - Stage length: one month up to a year no assistance when to inseminate, treat or cull the cow in the current month. Bio-sensors and cow specific traits/interventions exists in modern dairy herds parameters can be estimated on a daily basis. Bio-sensors and cow specific traits/interventions exists in modern dairy herds parameters can be estimated on a daily basis. OR50 Sep. 11 th / 1 OR50 Sep. 11 th / 1 MDPs applied to dairy MDPs applied to dairy Many papers about the dairy cow replacement problem but limited use in pratice. Reasons could be: Many papers about the dairy cow replacement problem but limited use in pratice. Reasons could be: - Often large and complex models with no friendly user interface. - Often large and complex models with no friendly user interface. - Parameters in the model must be estimated, i.e. data collection frameworks at herd level must exist. - Parameters in the model must be estimated, i.e. data collection frameworks at herd level must exist. - Stage length: one month up to a year no assistance when to inseminate, treat or cull the cow in the current month. - Stage length: one month up to a year no assistance when to inseminate, treat or cull the cow in the current month. Bio-sensors and cow specific traits/interventions exists in modern dairy herds parameters can be estimated on a daily basis. Bio-sensors and cow specific traits/interventions exists in modern dairy herds parameters can be estimated on a daily basis. OR50 Sep. 11 th / 1 OR50 Sep. 11 th / 1

5 MDPs applied to dairy What is an MDP? Many papers about the dairy cow replacement problem but limited use in pratice. Reasons could be: - Often large and complex models with no friendly user interface Parameters in the model must be estimated, i.e. data collection frameworks at herd level must exist. - Stage length: one month up to a year no assistance when to inseminate, treat or cull the cow in the current month. Bio-sensors and cow specific traits/interventions exists in modern dairy herds parameters can be estimated on a daily basis. Develop MDP with daily stages based on daily yield measurements. OR50 Sep. 11 th / 1 OR50 Sep. 11 th / 1 What is an MDP? What is an MDP? S 1 S S S 4 S 1 S S S 4 s 1,1 s 1, s 1, s 1,4 s 1,1 s 1, s 1, s 1,4 p s,1 s, s, s,4 s,1 r s, s, s,4 s,1 s, s, s,4 s,1 s, s, s,4 OR50 Sep. 11 th / 1 OR50 Sep. 11 th / 1 What is an MDP? What is an MDP? S 1 S S S 4 S 1 S S S 4 s 1,1 s 1, s 1, s 1,4 s 1,1 s 1, s 1, s 1,4 s,1 s, s, s,4 s,1 s, s, s,4 s,1 s, s, s,4 s,1 s, s, s,4 OR50 Sep. 11 th / 1 OR50 Sep. 11 th / 1

6 What is an MDP? Hierarchical MDP (HMDP) 1 4 S 1 S S S 4 Level 0 child process child process s 1,1 s 1, s 1, s 1,4 Level 1 child process child processes child process s,1 s, s, s,4 Level s,1 s, s, s,4 OR50 Sep. 11 th / 1 OR50 Sep. 11 th / 1 Dairy HMDP properties Dairy HMDP properties Hierarchical MDP (HMDP) based on lactation cycles of the cow. Hierarchical MDP (HMDP) based on lactation cycles of the cow. Daily stages during the lactation except for the dry period (49 days). Daily stages during the lactation except for the dry period (49 days). levels and running over an infinite time-horizon. levels and running over an infinite time-horizon. State variables are dry week and state variables related to the milk yield SSM embedded into the HMDP + IC state. State variables are dry week and state variables related to the milk yield SSM embedded into the HMDP + IC state. Decisions Replace, Keep and Dry. Decisions Replace, Keep and Dry. Maximize the net present reward of the cow. Maximize the net present reward of the cow. OR50 Sep. 11 th / 1 OR50 Sep. 11 th / 1 Dairy HMDP properties Dairy HMDP properties Hierarchical MDP (HMDP) based on lactation cycles of the cow. Hierarchical MDP (HMDP) based on lactation cycles of the cow. Daily stages during the lactation except for the dry period (49 days). Daily stages during the lactation except for the dry period (49 days). levels and running over an infinite time-horizon. levels and running over an infinite time-horizon. State variables are dry week and state variables related to the milk yield SSM embedded into the HMDP + IC state. State variables are dry week and state variables related to the milk yield SSM embedded into the HMDP + IC state. Decisions Replace, Keep and Dry. Decisions Replace, Keep and Dry. Maximize the net present reward of the cow. Maximize the net present reward of the cow. OR50 Sep. 11 th / 1 OR50 Sep. 11 th / 1

7 Dairy HMDP properties Dairy HMDP properties Hierarchical MDP (HMDP) based on lactation cycles of the cow. Hierarchical MDP (HMDP) based on lactation cycles of the cow. Daily stages during the lactation except for the dry period (49 days). Daily stages during the lactation except for the dry period (49 days). levels and running over an infinite time-horizon. levels and running over an infinite time-horizon. State variables are dry week and state variables related to the milk yield SSM embedded into the HMDP + IC state. State variables are dry week and state variables related to the milk yield SSM embedded into the HMDP + IC state. Decisions Replace, Keep and Dry. Decisions Replace, Keep and Dry. Maximize the net present reward of the cow. Maximize the net present reward of the cow. OR50 Sep. 11 th / 1 OR50 Sep. 11 th / 1 Hierarchical overview Tilstande for hver tidstrin Level 0 calving cow 1 cow cow cow 4 cow 5 new cow new cow calving replace replace 1 parity 1 parity parity parity 1 Markov beslutningsprocesser MDP cow model MDP for malkekoen Hierarkisk ordning Milk yield model Tilstande for hver tidstrin Netto indtægt time t time t + 1 replaced replaced (return to founder process) replaced invol invol m (1), π (1) m (1), π (1) insemination starts insemination ends dry Sandsynligheder Fremtiden m (i), π (i) keep replace m (i), π (i) m (q), π (q) m (q), π (q) π beskriver tidspunkt hvor skal goldes (-1 hvis tidspunkt ikke kendt endnu). I den nuværende model ca. 11 mill tilstande totalt. OR50 Sep. 11 th / 1 BFG April 4 th / 1 Rewards Lactation yield For each state and decision we can calculate the net reward Replace: Slaughter value of cow Keep: Value of milk - feeding costs etc. Dry - off : Price of calv - feeding costs etc. Obs. yield time from calving, d

8 t Y t E(Y t D t 1) E(A D t 1) t Lactation yield censorering Lactation yield Characteristics Obs. yield time from calving, d Start and end of registrations Culling Drying-off Obs. yield time from calving, d Mean lactation curve Effect of cow Short term fluctuations... Milk yield SSM Observed milk yield intensity Lactation yield Estimation M tc = µ t + A c + X tc + ν tc Subtract herd effect (remove index c) Y t = M t µ t = F θ t + ν t ( ) A = ( 1 1 ) + ν X t t θ t = Gθ t 1 + ω t ( ) ( 1 0 A = 0 ρ X t 1 ) ( ) 0 + ǫ t milk yield M t µ t The State Space model can be formulated as a linear normal mixed model and estimated e.g. via R or PROC MIXED in SAS. A spline function is used to estimate the mean lactation curve. where residual milk yield (θ t Y 0,..., Y t ) N(m t, C t ) OR50 Sep. 11 th / 1 Lactation yield Estimation Lactation yield 1 + The State Space model can be formulated as a linear normal mixed model and estimated e.g. via R or PROC MIXED in SAS. A spline function is used to estimate the mean lactation curve. mean yield (kg) Complications! DFC

9 t t Y t E(Y t D t 1) E(A D t 1) Milk yield SSM Embedding the SSM into a MDP Observed milk yield intensity M tc = µ t + A c + X tc + ν tc Subtract herd effect (remove index c) Y t = M t µ t = F θ t + ν t ( ) A = ( 1 1 ) + ν X t t milk yield M t µ t Can find P (m t+1 m t ) if store the mean m t and variance C t in each state. Discrete states discretize m t with { m (1),..., m (q) } and calculate P ( m (i) t+1 m(j) t ) Discretization can be done non-uniform (m t = (E(A t ), E(X t ))). θ t = Gθ t 1 + ω t ( ) ( 1 0 A = 0 ρ X t 1 ) ( ) 0 + ǫ t where residual milk yield (θ t Y 0,..., Y t ) N(m t, C t ) OR50 Sep. 11 th / 1 OR50 Sep. 11 th 008 / 1 Embedding the SSM into a MDP Embedding the SSM into a MDP Can find P (m t+1 m t ) if store the mean m t and variance C t in each state. Can find P (m t+1 m t ) if store the mean m t and variance C t in each state. Discrete states discretize m t with { m (1),..., m (q) } and calculate P ( m (i) t+1 m(j) t ) Discrete states discretize m t with { m (1),..., m (q) } and calculate P ( m (i) t+1 m(j) t ) Discretization can be done non-uniform (m t = (E(A t ), E(X t ))). Discretization can be done non-uniform (m t = (E(A t ), E(X t ))). OR50 Sep. 11 th 008 / 1 OR50 Sep. 11 th 008 / 1 Embedding the SSM into a MDP Discretization, m t state Can find P (m t+1 m t ) if store the mean m t and variance C t in each state. Discrete states discretize m t with { m (1),..., m (q) } and calculate P ( m (i) t+1 m(j) t ) Discretization can be done non-uniform (m t = (E(A t ), E(X t ))) m (6.54) (14.06) 17 (0.0) (9) 8 9 (.9) (11.45) (.6) (5.08) (7.54) ( 6.5) (.9) ( 1.9) (8.84) 1 (0.01) (.47) (4.9) ( 8.97) ( 5.06) (.46) (0) (.46) (5.06) ( 4.9) (.47) ( 0.01) ( 14.04) ( 11.44) ( 8.84) (1.9) ( 7.54) ( 5.08) (.6) 0 ( 11.45) (.9) 6 7 ( 17.97) ( 9) 6 8 ( 14.06) ( 6.54) 5 (17.97) 14 9 (11.44) (14.04) 0 (8.97) 15 4 (.9) (6.5) 7 ( 0.0) uniform non-uniform m 1 OR50 Sep. 11 th 008 / 1

10 Reproduction model component (π t state) (5) Oestrus Start Observable Oestrus (6) Oestrus End. Pregnancy defines end of lactation period Decision: Inseminate when oestrus is observed Observe heat/oestrus Quality of insemination Observe pregnancy (1) Induction Weaning CL regression. Puberty () Oestrogen peak () LH peak (4) Ovulation Start (7) Follicle rupture Pregnancy recognition (1) CL regression () Oestrogen Peak Figure: Outline of events in the oestrus cycle. NB! drawing not scaled according to time.. Example: Pregnancy rate {0.5, 0.6, 0.7} Days from calving to drying off S P S S S4 S1. Figure: The oestrus-cycle as a semi-markov Process C Density Days from Calving Example: Pregnancy rate {0.5, 0.6, 0.7} Days from calving to drying off Example: Pregnancy rate {0.5, 0.6, 0.7} Days from calving to drying off Probability Marg. Probability Days from Calving Days from Calving

11 Model component: Involuntary culling Model component: Involuntary culling Calving to start of insemination (Voluntary Waiting Period) From start of insemination until either pregnancy is confirmed or end of insemination Pregnancy confirmation to dry-off End of insemination to culling Daily prob. of involuntary culling Accumulated prob. of involuntary culling DFC DFC Other model components Based on approach in SIMHERD Other model components Based on approach in SIMHERD Total body weight Total body weight dfc dfc Current state of model Examples Cow lac = 1 dfc = 11 dry = 1 Model running (Linux) Manuscript in prep Parameters may need fine-tuning Verification of results residual yield RPO days from start m ave. yield days from start days from start m 1

12 Prelim. results: Examples Implementation Start of lactation vs. milk yield Value of calf Replacement vs recovery R-package(s) MLHMP Java library C ++ program parts Linux (large model) Challenges in final part of project Number of state variables Complexity Observed vs. latent variables Insemination known / imperfect pregnancy test Latent disease state Information from other herds.