Modelling HDD Failure Rates with Bayesian Inference

Transcription

1 Modelling HDD Failure Rates with Bayesian Inference Sašo Stanovnik Mentor: prof. Erik Štrumbelj, PhD Faculty of Computer and Information Science University of Ljubljana, Slovenia May 2016 Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 1 / 30

2 Problem Some hard drives fail earlier than others. An enormous amount of (unreliable) anecdotal evidence. People don t know what they re talking about. People make conclusions on sample sizes where n = 2. How long will a typical hard drive last? Are there differences between manufacturers? Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 2 / 30

3 About the data We re using the Backblaze dataset 1. Consumer, not enterprise drives. Drives used in a datacenter environment. 60 drives in a single enclosure 2. Temperature and vibration are a factor. Daily SMART data for each disk in the datacenter days of data (from 2013Q2 to 2016Q1). More than total drives. We only use SMART 9: Power On Hours Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 3 / 30

4 Useful information (Hitachi Global Storage Technologies) was founded when Hitachi acquired IBM s HDD branch. In 2012, was acquired by and has been completely merged into in late The analysis merges and Hitachi brandings 3. 3 As reported by the drive s model attribute. Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 4 / 30

5 Exploration Disk lifetime Power On Hours (alive) Power On Hours (failed) count 1000 count hours hours Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 5 / 30

6 Exploration Disk lifetime manufacturers Live Power On Hours (alive) Live Live 2e 04 1e 04 density 0e+00 Dead Dead Dead manufacturer 2e 04 1e 04 0e hours Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 6 / 30

7 Exploration Disk counts manufacturer count Live. Dead. Live. Dead. Live. Dead. Failure.Manufacturer Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 7 / 30

8 About the Weibull distribution x Weibull(α, σ) 0.5 { α pdf (x) = σ ( x σ )α 1 e ( x σ )α x 0 0 x < 0 α, σ (0, ) x [0, ) α has an important meaning alpha < 1: early failure alpha = 1: random failure alpha > 1: age failure pdf(x) x alpha Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 8 / 30

9 Model The Weibull distribution is used for modelling failure rates. A single or a mixture of Weibull distributions? Separate modelling of early random age failure characteristics. Early failure α is 1 early_mortality_offset. Age failure α is 1 + age_mortality_offset. Non informative priors, we have enough data. Right censoring 4 to include data about disks that are still alive. Separate modelling of different manufacturers. The top 3 manufacturers:,,. 4 Picture it as including a probability that a certain disk has survived for as long as it has. Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 9 / 30

10 Model Plate notation Single Weibull Three Weibulls Separate runs of the same model for different manufacturers. Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 10 / 30

11 Model Convergence We ran 5000 iterations, 500 of which were warmup. All parameters across all models have 1200 effective samples. The generated Power On Hours have 4500 effective samples ˆR is always 1. Therefore, no reason to doubt convergence. Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 11 / 30

12 Model Convergence: single Weibull Manufacturer subsets (poh_generated) 3e+06 4e+05 2e+06 1e+06 3e+05 2e+05 1e+05 0e e e+05 4e+05 5e+04 2e+05 0e e Note: other parameters also show no reason to doubt convergence, but there are too many to show. Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 12 / 30

13 Model Convergence: triple Weibull Manufacturer subsets (poh_generated) e+05 2e e e e e e e Note: other parameters also show no reason to doubt convergence, but there are too many to show. Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 13 / 30

14 Results Single Weibull, manufacturers, α alpha 500 appears to predominantly have age related failures. and are similar in this characteristic. count group all alpha Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 14 / 30

15 Results Single Weibull, manufacturers, σ drives have proportionately little failures (2% of all drives) and high lifetimes. The model likely overestimates based on limited data. Here, is estimated to 1000 generally have a higher lifetime than 0 count sigma 0e+00 2e+05 4e+05 6e+05 sigma group all Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 15 / 30

16 Results Single Weibull, manufacturers, Predicted Power On Hours 600 s prediction does not seem realistic. has a longer tail than. count group all 0 0e+00 1e+05 2e+05 3e+05 4e+05 5e+05 Predicted Power On Hours Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 16 / 30

17 Results Triple Weibull, manufacturers, early failure offsets early mortality offset 300 This mostly tends to 0 (to achieve α = 1). is different here in that it shows more confidence in (some) early mortality. count group all early_mortality_offset Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 17 / 30

18 Results Triple Weibull, manufacturers, age failure offsets trends most heavily towards age mortalities, followed by. All drives combined appear to fail randomly, likely because of the mixture of different manufacturers. count age mortality offset age_mortality_offset group all Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 18 / 30

19 Results Triple Weibull, manufacturers, sigmas The parameter for early failures is the same across all drives, but is very confident in very early failures. Random failures occur latest for. has a very wide spread and all drives combined fail early if they fail randomly. Age related failures occur latest for, then, then. group all all all sigma value group all Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 19 / 30

20 Results Triple Weibull, manufacturers, mixture ratios Early failures are fairly infrequent for all manufacturers. has most random failures (at the expense of age related failures), followed by. count mixture value group all Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 20 / 30

21 Results Triple Weibull, manufacturers, Predicted Power On Hours stands out as the most reliable. is better than, but has a spike in early failures. Modelling all drives together 100 does not give useful information as different manufacturers are blended together. 0 count Predicted Power On Hours group all Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 21 / 30

22 Model quality estimation PSIS leave one out approximation using loo elpd.estimate elpd.se Single Weibull Triple Weibull The triple Weibull model outperforms the single model, but is within the error tolerance. Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 22 / 30

23 Interpretation Proportion of failed drivers by year Ratio of failed drives by year (predicted, single Weibull) 1.00 Ratio of failed drives by year (predicted, three Weibull) group 0.75 group failure 0.50 failure 0.50 all all year The three Weibull model is more rational. is most reliable, followed by, then. has more early failures year Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 23 / 30

24 Interpretation Hazard function Hazard Rate manufacturer Power On Time (years) 1 and have similar hazards. s hazard rises later, but then experiences more random failures. 2 fails primarily with age. Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 24 / 30

25 Conclusions The mixture of Weibulls is more effective than a single one. We can model detailed disk characteristics. There are differences between manufacturers. drives appear most reliable, followed by, then. Early failures are characteristic for. We need to remember that we have a (relatively) short time window of observation. Also the environment in which these drives operate. Think about all these interpretations for yourself, anectodal evidence is bad. Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 25 / 30

26 Addendum Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 26 / 30

27 AD 1: stan code for the single Weibull model data { int <lower=0> n ; i n t poh [ n ] ; i n t <lower =0, upper=1> f a i l u r e b o o l s [ n ] ; parameters { real <lower=0.0> alpha ; real <lower=0.0> sigma ; model { f o r ( i i n 1 : n ) { i f ( f a i l u r e b o o l s [ i ] == 1) { poh [ i ] w e i b u l l ( alpha, sigma ) ; e l s e { // r i g h t c e n s o r i n g i n c r e m e n t l o g p r o b ( w e i b u l l c c d f l o g ( poh [ i ], alpha, sigma ) ) ; generated q u a n t i t i e s { r e a l p o h g e n e r a t e d ; r e a l l o g l i k e l i h o o d [ n ] ; poh generated < w e i b u l l r n g ( alpha, sigma ) ; f o r ( i i n 1 : n ) { i f ( f a i l u r e b o o l s [ i ] == 1) { l o g l i k e l i h o o d [ i ] < w e i b u l l l o g ( poh [ i ], alpha, sigma ) ; e l s e { l o g l i k e l i h o o d [ i ] < w e i b u l l c c d f l o g ( poh [ i ], alpha, sigma ) ; Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 27 / 30

28 AD 2: stan code for the triple Weibull model (1/3) data { int <lower=0> n ; i n t poh [ n ] ; i n t <lower =0, upper=1> f a i l u r e b o o l s [ n ] ; real <lower=0.0> prior gamma alpha0 ; real <lower=0.0> prior gamma beta0 ; parameters { s i m p l e x [ 3 ] t h e t a ; r e a l <lower =0.0, upper =1.0> e a r l y m o r t a l i t y o f f s e t ; r e a l <lower =0.0, upper=100> a g e m o r t a l i t y o f f s e t ; real <lower =0.0, upper=100000> sigma [ 3 ] ; Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 28 / 30

29 AD 2: stan code for the triple Weibull model (2/3) model { r e a l temp [ 3 ] ; sigma gamma( prior gamma alpha0, prior gamma beta0 ) ; f o r ( i i n 1 : n ) { i f ( f a i l u r e b o o l s [ i ] == 1) { temp [ 1 ] < l o g ( t h e t a [ 1 ] ) + w e i b u l l l o g ( poh [ i ], 1 e a r l y m o r t a l i t y o f f s e t, sigma [ 1 ] ) ; temp [ 2 ] < l o g ( t h e t a [ 2 ] ) + w e i b u l l l o g ( poh [ i ], 1 / random m o r t a l i t y /, sigma [ 2 ] ) ; temp [ 3 ] < l o g ( t h e t a [ 3 ] ) + w e i b u l l l o g ( poh [ i ], 1 + a g e m o r t a l i t y o f f s e t, sigma [ 3 ] ) ; e l s e { // r i g h t c e n s o r i n g temp [ 1 ] < l o g ( t h e t a [ 1 ] ) + w e i b u l l c c d f l o g ( poh [ i ], 1 e a r l y m o r t a l i t y o f f s e t, sigma [ 1 ] ) ; temp [ 2 ] < l o g ( t h e t a [ 2 ] ) + w e i b u l l c c d f l o g ( poh [ i ], 1 / random m o r t a l i t y /, sigma [ 2 ] ) ; temp [ 3 ] < l o g ( t h e t a [ 3 ] ) + w e i b u l l c c d f l o g ( poh [ i ], 1 + a g e m o r t a l i t y o f f s e t, sigma [ 3 ] ) ; increment log prob ( log sum exp ( temp ) ) ; Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 29 / 30

30 AD 2: stan code for the triple Weibull model (3/3) generated q u a n t i t i e s { r e a l p o h g e n e r a t e d ; i n t c a t ; r e a l l o g l i k e l i h o o d [ n ] ; r e a l temp [ 3 ] ; c a t < c a t e g o r i c a l r n g ( t h e t a ) ; i f ( c a t == 1) { p o h g e n e r a t e d < w e i b u l l r n g (1 e a r l y m o r t a l i t y o f f s e t, sigma [ 1 ] ) ; e l s e i f ( c a t == 2) { poh generated < w e i b u l l r n g (1, sigma [ 2 ] ) ; e l s e { p o h g e n e r a t e d < w e i b u l l r n g (1 + a g e m o r t a l i t y o f f s e t, sigma [ 3 ] ) ; f o r ( i i n 1 : n ) { i f ( f a i l u r e b o o l s [ i ] == 1) { temp [ 1 ] < l o g ( t h e t a [ 1 ] ) + w e i b u l l l o g ( poh [ i ], 1 e a r l y m o r t a l i t y o f f s e t, sigma [ 1 ] ) ; temp [ 2 ] < l o g ( t h e t a [ 2 ] ) + w e i b u l l l o g ( poh [ i ], 1 / random m o r t a l i t y /, sigma [ 2 ] ) ; temp [ 3 ] < l o g ( t h e t a [ 3 ] ) + w e i b u l l l o g ( poh [ i ], 1 + a g e m o r t a l i t y o f f s e t, sigma [ 3 ] ) ; e l s e { temp [ 1 ] < l o g ( t h e t a [ 1 ] ) + w e i b u l l c c d f l o g ( poh [ i ], 1 e a r l y m o r t a l i t y o f f s e t, sigma [ 1 ] ) ; temp [ 2 ] < l o g ( t h e t a [ 2 ] ) + w e i b u l l c c d f l o g ( poh [ i ], 1 / random m o r t a l i t y /, sigma [ 2 ] ) ; temp [ 3 ] < l o g ( t h e t a [ 3 ] ) + w e i b u l l c c d f l o g ( poh [ i ], 1 + a g e m o r t a l i t y o f f s e t, sigma [ 3 ] ) ; l o g l i k e l i h o o d [ i ] < log sum exp ( temp ) ; Sašo Stanovnik, UL FRI Modelling HDD Failure Rates with Bayesian Inference 30 / 30