A Dynamic Programming Approach for Sequential Preventive Maintenance Policies with Two Failure Modes

Chapter 1 A Dynamic Programming Approach for Sequential Preventive Maintenance Policies with Two Failure Modes Hiroyuki Okamura 1, Tadashi Dohi 1 and Shunji Osaki 2 1 Department of Information Engineering, Graduate School of Engineering, Hiroshima University, 1 4 1 Kagamiyama, Higashi-Hiroshima 739-8527, Japan 2 Faculty of Information Sciences and Engineering, Nanzan University, Seto 489-0863, Japan 1 Introduction Stochastic preventive maintenance problem consists of formalism (modeling) and algorithm (optimization) for real applications. Since the seminal contribution by Barlow and Hunter [Barlow and Hunter (1965)], a huge number of stochastic preventive maintenance problems have been discussed from both viewpoints of modeling and optimization. Sequential preventive maintenance policies to determine aperiodic maintenance schedules may be regarded as the most critical but complex ones because they are reduced to nonlinear optimization problems with multiple decision variables. First Nguyen and Murthy [Nguyen and Murthy (1981)] consider two aperiodic preventive maintenance models with/without minimal repairs, and extend the Barlow and Hunter model [Barlow and Hunter (1965)]. Nakagawa [Nakagawa (1986)] independently considers the similar model and extends it in the subsequent paper [Nakagawa (1988)]. Since the above papers by Nakagawa [Nakagawa (1986, 1988)], the stochastic preventive maintenance models with aperiodic maintenance schedules are called the 3

4 Reliability Modeling with Applications sequential preventive maintenance models, and have attracted much attentions from many authors. Lin et al. [Lin et al. (2000)] extend Nakagawa models [Nakagawa (1986, 1988)] in terms of age reduction effect and hazard rate adjustment. The same authors [Lin et al. (2001)] also take account of two failure modes and extend their models [Lin et al. (2000)]. El-Ferik and Ben-Daya [El-Ferik and Ben-Daya (2005)] also extend Nguyen and Murthy model [Nguyen and Murthy (1981)] with a different policy from Nakagawa [Nakagawa (1988)] and Lin et al. [Lin et al. (2001)]. Kim et al. [Kim et al. (2007)] consider a problem to determine both the optimal number of preventive maintenance and its associated time sequence. Sheu and Liou [Sheu and Liou (1995)] and Sheu and Chang [Sheu and Chang (2002)] formulate the different sequential preventive maintenance problems with the same line as Nguyen and Murthy [Nguyen and Murthy (1981)] and Nakagawa [Nakagawa (1986, 1988)]. Recently, Nakagawa and Mizutani [Nakagawa and Mizutani (2009)] give a modification of Nakagawa model [Nakagawa (1986, 1988)] with a finite planning time horizon. It is worth mentioning that these preventive maintenance policies are not impractical models. A good illustrative example of the sequential preventive maintenance policy is given by Jayabalan and Chaudhuri [Jayabalan and Chaudhuri (1992)], where they apply it to the maintenance problem of bus engines in a large transport network with more than 2500 buses. In this way, considerable attentions have been paid to the sequential preventive maintenance models. However, the main concern devoted in the related work was just the modeling, but not the computation algorithm. More specifically, the underlying optimization problems for the sequential preventive maintenance policies can be reduced to nonlinear optimization problems with multiple decision variables. In almost all papers, it is shown that the optimal time sequence of preventive maintenance must satisfy the first-order condition of optimality, but no effective computation algorithms are not developed. It should be surprised to see that the above related papers to the sequential preventive maintenance policies give very small toy exercise and never solve a realistic level of problem with more than 100 time points. In this paper, we take an example model with two different failure modes corresponding to the minimal repair and replacement, and formulate this generalized problem by means of the dynamic programming (DP). The resulting algorithm gives an effective algorithm to compute the aperiodic optimal maintenance schedule which minimizes the relevant expected cost rate, and is independent of the kind of model. So, the proposed computation

A Dynamic Programming Approach for Sequential Preventive Maintenance Policies 5 scheme provides a unified framework to determine the sequential preventive maintenance policies. 2 Sequential Imperfect PM Nguyen and Murthy [Nguyen and Murthy (1981)] and Nakagawa [Nakagawa (1986, 1988)] propose the sequential preventive maintenance (PM) policies. Unlike the periodic PM policies with equidistant PM time period, the sequential PM policies allow the aperiodic PM time sequence at which the system should undergo preventive maintenance optimally. Consider a system under minimal repairs and replacement. The system has two failure modes: Type I and Type II failures, where Type I failure is an error of any system component and can be fixed by a minimal repair, and Type II failure is a fatal error and is repaired by only corrective replacement. In this paper, we assume that two types of failure occur independently. More precisely, the system undergoes preventive maintenance at each time points, t 1 < t 2 < < t N, where the k-th preventive maintenance is performed at t k. From the assumption of imperfect preventive maintenance [Lin et al. (2001); Nakagawa (1988)], it is supposed that the failure rates for Type I and Type II failures are changed at each preventive maintenance point. Define the failure rate functions for Type I and Type II failures in the k-th period of preventive maintenance by Type I failure: Type II failure: h I k(t t k 1 ), 0 t < t k t k 1, (1) h II k (t t k 1 ), 0 t < t k t k 1. (2) Suppose that the failure rate functions in the k-th period depend on the time point of the last preventive maintenance, t k 1. For the sake of convenience, we set t 0 = 0. Define the following cost parameters: c 1 : the cost of a minimal repair, c 2 : the cost of preventive maintenance, c 3 : the cost of replacement, c 4 : the cost of corrective maintenance.

6 Reliability Modeling with Applications Let S k (t k t k 1 ) and T k (t k t k 1 ) be the expected total cost incurred in the k-th period and the expected time length of the k-th period, provided that the k 1-st and k-th preventive maintenances are preformed at t k 1 and t k, respectively. Then we have S k (t k t k 1 ) =c 1 tk t k 1 0 h I k(t t k 1 )R II k (t t k 1 )dt + c 2 R II k (t k t k 1 t k 1 ) + c 4 ( 1 R II k (t k t k 1 t k 1 ) ), (3) k = 1,... N 1, S N (t N t N 1 ) =c 1 tn t N 1 0 h I N (t t N 1 )R II N (t t N 1 )dt + c 3 R II N (t N t N 1 t N 1 ) ( + c 4 1 R II N (t N t N 1 t N 1 ) ), (4) tk t k 1 T k (t k t k 1 ) = Rk II (t t k 1 )dt, (5) 0 k = 1,... N, where Rk II(t t k 1) is the reliability function for Type II failure in the k-th period: R II k (t t k 1 ) = exp ( t 0 ) h II k (s t k 1 )ds, 0 t < t k t k 1. (6) Using S k ( ) and T k ( ), the expected cost rate under the sequential preventive maintenance policy π N = {t 1,..., t N }, provided that the number of total maintenances is N, is given by k 1 N k=1 l=1 C(π N, N) = RII l (t l t l 1 t l 1 )S k (t k t k 1 ) N k 1 k=1 l=1 RII l (t l t l 1 t l 1 )T k (t k t k 1 ). (7) The problem is to find the optimal N and π N which minimize the expected cost rate. The above formula comprehends the existing sequential PM models. For example, when we set Rk II(t t k 1) = 1, the model can be reduced to the original sequential PM model in [Nakagawa (1986)]. Also, if the failure rate is defined by the base failure rate h I 0(t); h I k(t t k 1 ) = β k h I 0(t + α k t k 1 ), (8)

A Dynamic Programming Approach for Sequential Preventive Maintenance Policies 7 the model represents the hazard rate PM and age reduction PM models [Lin et al. (2000)]. In addition, when the failure rate h I k (t t k 1) is replaced with h I+III k (t t k 1 ) = h I k(t t k 1 ) + h III (t + t k 1 ), (9) the model corresponds to the sequential PM with unmaintainable failure mode (Type III failure) [Lin et al. (2001)]. 3 DP Algorithm Since the expected cost rate for each policy is given as a function of N and π N, the optimization problem is reduced to a non-linear programming to obtain min N,πN C(π N, N), given the number of total maintenances N. It is worth noting that there is no effective algorithm to find the optimal pair (N, π N ) simultaneously. Hence the total number of preventive maintenances must be carefully adjusted according to any heuristic manner. Instead, we focus on finding the optimal maintenance schedule π N under a fixed N. In the case of a fixed N, the most popular method to find the optimal maintenance schedule might be Newton s method or its iterative variants. However, since Newton s method is a general-purpose non-linear optimization algorithm, it may not often work well to solve the minimization problem with many parameters. In our minimization problem, the decision variables π N = {t 1,..., t N } have the constraint t 1 < < t N. For such sequential optimization problems, it is well known that the dynamic programming (DP) can be used effectively. In this section, we develop a DP algorithm for finding the optimal maintenance schedule π N. The idea behind our DP algorithm is the iterative algorithm based on the optimality equations which are typical functional equations. It is straightforward to give the optimality equations which the optimal maintenance schedule π N must satisfy. Suppose that there exists the unique minimum expected cost rate ρ. From the principle of optimality, we obtain the following optimality equations for the minimization problem of the expected cost rate: J k = min t k W k (t k t k 1, J 1, J k+1, ρ), k = 1,..., N 1, (10) J N = min t N W N (t N t N 1, J 1, ρ), (11) where functions W k (t k t k 1, J 1, J k+1, ρ) and W N (t N t N 1, J 1, ρ) are given

8 Reliability Modeling with Applications by W k (t k t k 1, J 1, J k+1, ρ) = S k (t k t k 1 ) ρt k (t k t k 1 ) + J k+1 Rk II ( (t k t k 1 t k 1 ) + J 1 1 R II k (t k t k 1 t k 1 ) ), (12) W N (t N t N 1, J 1, ρ) = S N (t N t N 1 ) ρt N (t N t N 1 ) + J 1. (13) In the above equations, J k, k = 1,..., N, are called the relative value functions. Equations (10) and (11) are necessary and sufficient conditions of the optimal maintenance schedule. That is, the problem can be reduced into finding the maintenance schedule which satisfies the optimality equations. In the long history of the DP research, there are a couple of algorithms to solve the optimality equations. In this paper, we apply the policy iteration scheme to derive the optimal maintenance schedule. Our algorithm is twofold: the policy improvement under given relative value functions and the computation of relative value functions under a maintenance schedule. These two steps are repeatedly executed until the maintenance schedule converges. In the policy improvement, we find a new maintenance schedule based on the following functions under given relative value functions J 1,..., J N : and W k (t k t k 1, J 1, J k+1, ρ), k = 1,..., N 1 (14) W N (t N t N 1, J 1, ρ). (15) However, when J 1,..., J N are constants, the above functions are not always convex with respect to decision variables t k. Thus our policy improvement algorithm is based on the following composite functions for two successive periods, instead of W k ( ): W k (t k t k 1, t k+1, J 1, J k+2, ρ) = W k (t k t k 1, J 1, W k+1 (t k+1 t k, J 1, J k+2, ρ), ρ), (16) t k 1 t k t k+1, k = 1,..., N 2, W N 1 (t N 1 t N 2, t N, J 1, ρ) = W N 1 (t N 1 t N 2, J 1, W N (t N t N 1, J 1, ρ), ρ), (17) t N 2 t N 1 t N. The above composite functions are convex in the respective ranges t k 1 t k t k+1, k = 1,..., N 1. In addition, the function W N (t N t N 1, J 1, ρ) is also convex in the range t N 1 t N <. Thus it is possible to find

A Dynamic Programming Approach for Sequential Preventive Maintenance Policies 9 the improved maintenance schedule by performing the one-dimensional optimization for each period of preventive maintenance. Under a given maintenance schedule t 1,..., t N, the computation step gives corresponding relative value functions and ρ by solving the following linear system: Mx = b, (18) where Ri II (t i t i 1 t i 1 ) if i = j and j N, 1 if i = j + 1, [M] i,j = T i (t i t i 1 ) if j = N, 0 otherwise, (19) x = (J 2,..., J N, ρ), (20) b = (S 1 (t 1 t 0 ),..., S N (t N t N 1 )). (21) In Eq. (19), [ ] i,j denotes the (i, j)-element of matrix, and the prime ( ) represents transpose of vector. The above linear system comes from the optimality equations (10) and (11) directly. Note that J 1 = 0, since we are here interested in the relative value function J i and ρ. Finally, we derive the DP algorithm to derive the optimal maintenance schedule as follows. Step 1: Give initial values π (0) N k := 0, t 0 := 0, := {t(0) 1,..., t(0) N }. Step 2: Compute J (k) 1,..., J (k) N, ρ(k) for the linear system (18) under the maintenance schedule π (k) N. Step 3: Solve the following optimization problems: t (k+1) i := argmax t (k+1) N 1 t (k+1) N t (k) i 1 t t(k) i+1 i = 1,..., N 2, := argmax t (k) N 2 t t(k) N W i (t t (k) i 1, t(k) i+1, J (k) 1, J (k) i+2, ρ(k) ), W N 1 (t t (k) N 2, t(k) N, J (k) 1, ρ (k) ) := argmax W N (t t (k) N 1, J (k) 1, ρ (k) ). t (k) N 1 t<

10 Reliability Modeling with Applications Step 4: For all i = 1,..., N, if t (k+1) i t (k) i < δ, stop the algorithm, where δ is an error tolerance. Otherwise, let k := k + 1 and go to Step 2. In Step 3, an arbitrary optimization technique can be applied. Since the composite functions are convex functions having a unique solution in the ranges [t i 1, t i+1 ), i = 1,..., N 1, it is not so difficult to calculate the optimal preventive maintenance time. In fact, the golden section method is effective to find the solution. 4 Numerical Examples We first consider the case where the failure rate for Type I failure is given by the following Weibull-type failure rate: h I k(t t k 1 ) = α k β k t β k 1, (22) where 1/α k = 100 0.81 k 1 and β k = 2.0. The other parameters are c 1 = 1.0, c 2 = 3.0, c 3 = 100.0 and c 4 = 1000.0. These parameters are cited from [Nakagawa (1986)]. We investigate the sensitivity of the failure rate for Type II failure on the optimal PM sequence and the minimum cost. Table 1 presents the optimal PM timing when the total number of PMs is fixed under h II k (t t k 1) = 0, i.e., Rk II(t t k 1) = 1. As mentioned before, our model is reduced to Nakagawa s model when Rk II(t t k 1) = 1. From the table, the optimal number of PMs is N = 11, which minimizes the total cost rate. In [Nakagawa (1986)], the optimal PM sequence in the case of N = 11 was presented, and it is exactly same as our result. That is, our DP algorithm can solve the sequential PM policy stably. Figure 1 illustrates the optimal PM sequences for N = 2 through 16. In the figure, x-axis represents PM timing and the optimal PM sequence was plotted as points on the horizontal line. From the figure, it can be founded that the time interval of PMs becomes monotonically decreasing sequence for all the cases. Moreover, when we focus on the first PM timing, it decreases as the number of total PMs increases from N = 2 to N = 11, but it decreases from N = 11 to N = 16. In this case, the optimal number of total PMs is N = 11. The tendency of PM sequences are changed at the optimal number of PMs. Next we investigate the case where the type II failure rate is given by a constant; h II k (t t k 1 ) = λ. (23)

A Dynamic Programming Approach for Sequential Preventive Maintenance Policies 11 Table 1 failure. Optimal sequential PM timing for N = 8, 9, 10, 11, 12, 13, 14 without type II N 8 9 10 11 12 13 14 t 1 53.1 52.6 52.4 52.3 52.4 52.5 52.8 t 2 96.1 95.3 94.8 94.7 94.8 95.0 95.5 t 3 131.0 129.8 129.2 129.1 129.2 129.5 130.1 t 4 159.2 157.8 157.0 156.9 157.0 157.3 158.2 t 5 182.1 180.4 179.6 179.4 179.6 179.9 180.9 t 6 200.6 198.8 197.9 197.6 197.8 198.3 199.3 t 7 215.6 213.7 212.7 212.4 212.6 213.1 214.2 t 8 227.8 225.7 224.6 224.4 224.6 225.1 226.2 t 9 235.5 234.4 234.1 234.3 234.8 236.0 t 10 242.2 241.9 242.2 242.7 243.9 t 11 248.3 248.6 249.1 250.3 t 12 253.7 254.3 255.5 t 13 258.5 259.8 t 14 263.2 C(π N, N) 1.062 1.053 1.048 1.047 1.048 1.051 1.056 N=2 N=3 N=4 N=5 N=6 N=7 N=8 N=9 N=10 N=11 N=12 N=13 N=14 N=15 N=16 0 50 100 150 200 250 300 PM time Fig. 1 Optimal PM sequences for fixed N without Type II failure. Figures 2 through 7 depict the optimal PM sequences for respective type II failure rates λ = 1/100, 1/200, 1/300, 1/400, 1/500, 1/1000 with a fixed N. Also Table 2 presents the minimum cost rates for each optimal PM sequence with a fixed N. The last column indicates the minimum cost rates in the case where the Type II failure does not occur, i.e., h II k (t t k 1) = 0. The

12 Reliability Modeling with Applications N=2 N=3 N=4 N=5 N=6 N=7 N=8 N=9 N=10 N=11 N=12 N=13 N=14 N=15 N=16 0 50 100 150 200 250 300 PM time Fig. 2 Optimal PM sequences for fixed N (λ = 1/100). N=2 N=3 N=4 N=5 N=6 N=7 N=8 N=9 N=10 N=11 N=12 N=13 N=14 N=15 N=16 0 50 100 150 200 250 300 PM time Fig. 3 Optimal PM sequences for fixed N (λ = 1/200). asterisk means the optimal number of PMs minimizing the cost rates. For every case, we apply the DP algorithm to obtain the optimal PM sequences. Even if the number of PMs is large, e.g., N = 16, we can get the optimal PM sequences stably.

A Dynamic Programming Approach for Sequential Preventive Maintenance Policies 13 N=2 N=3 N=4 N=5 N=6 N=7 N=8 N=9 N=10 N=11 N=12 N=13 N=14 N=15 N=16 0 50 100 150 200 250 300 PM time Fig. 4 Optimal PM sequences for fixed N (λ = 1/300). N=2 N=3 N=4 N=5 N=6 N=7 N=8 N=9 N=10 N=11 N=12 N=13 N=14 N=15 N=16 0 50 100 150 200 250 300 PM time Fig. 5 Optimal PM sequences for fixed N (λ = 1/400). From the figures, even when the Type II failure occurs, the optimal PM sequence has the similar tendency as the case where the Type II failure does not occur. Also, the PM timing tends to be earlier in the case where Type II failure rate is high; λ = 1/100. Moreover, it can be seen that the

14 Reliability Modeling with Applications N=2 N=3 N=4 N=5 N=6 N=7 N=8 N=9 N=10 N=11 N=12 N=13 N=14 N=15 N=16 0 50 100 150 200 250 300 PM time Fig. 6 Optimal PM sequences for fixed N (λ = 1/500). N=2 N=3 N=4 N=5 N=6 N=7 N=8 N=9 N=10 N=11 N=12 N=13 N=14 N=15 N=16 0 50 100 150 200 250 300 PM time Fig. 7 Optimal PM sequences for fixed N (λ = 1/1000). minimum cost rate strongly depend on Type II failure rate from the table. In addition, in the case of the high Type II failure rate, the optimal number of PMs is bigger than the other cases.

A Dynamic Programming Approach for Sequential Preventive Maintenance Policies 15 Table 2 Minimum cost rates. N 1/100 1/200 1/300 1/400 1/500 1/1000 w/o Type II 2 11.009 6.236 4.656 3.867 3.394 2.451 1.509 3 10.851 6.055 4.468 3.676 3.202 2.256 1.311 4 10.766 5.957 4.367 3.574 3.100 2.152 1.206 5 10.716 5.899 4.306 3.513 3.038 2.089 1.143 6 10.683 5.861 4.268 3.474 2.998 2.050 1.104 7 10.662 5.836 4.242 3.448 2.973 2.024 1.078 8 10.647 5.819 4.225 3.431 2.956 2.008 1.062 9 10.637 5.808 4.214 3.421 2.946 1.998 1.053 10 10.630 5.801 4.208 3.414 2.939 1.992 1.048 11 10.625 5.797 4.204 3.411 2.937 1.990 1.047* 12 10.622 5.795 4.203* 3.410* 2.936* 1.990* 1.048 13 10.620 5.794* 4.203 3.411 2.937 1.992 1.051 14 10.620 5.795 4.205 3.413 2.940 1.996 1.056 15 10.619* 5.796 4.207 3.417 2.944 2.001 1.062 16 10.619 5.798 4.210 3.421 2.948 2.006 1.068 5 Concluding Remarks In this paper, we have considered a sequential preventive maintenance model with minimal repair and replacement. The model under consideration is an extension of Nakagawa [Nakagawa (1986, 1988)] by introducing two different failure modes. To derive the optimal preventive maintenance schedule which minimizes the expected cost rate, we have developed a DPbased algorithm which is twofold; policy improvement and computation of relative value functions. In particular, we have applied composite functions for two successive periods of preventive maintenance to derive the improved maintenance schedule. Acknowledgment The authors are grateful to Prof. Toshio Nakagawa who stimulated their research interests in preventive maintenance modeling through his many papers. This research was partially supported by the Ministry of Education, Science, Sports and Culture, Grant-in-Aid for Scientific Research (C), Grant No. 23500047 (2011 2013) and Grant No. 23510171 (2011 2013). References Barlow, R. E. and Hunter, L. C. (1965). Optimum preventive maintenance policies, Operations Research 8, pp. 90 100.

16 Reliability Modeling with Applications El-Ferik, S. and Ben-Daya, M. (2005). Age-based hybrid model for imperfect preventive maintenance, IIE Transactions 38, pp. 365 375. Jayabalan, V. and Chaudhuri, D. (1992). Sequential imperfect preventive maintenance policies: a case study, Microelectronics and Reliability 32, pp. 1223 1229. Kim, H. S., Kwon, Y. S. and Park, D. H. (2007). Adaptive sequential preventive maintenance policy and Bayesian consideration, Communications in Statistics Theory and Methods 36, pp. 1251 1269. Lin, D., Zuo, M. J. and Yam, R. C. M. (2000). General sequential imperfect preventive maintenance models, International Journal of Reliability, Quality and Safety Engineering 7, pp. 253 266. Lin, D., Zuo, M. J. and Yam, R. C. M. (2001). Sequential imperfect preventive maintenance models with two categories of failure modes, Naval Research Logistics 48, pp. 172 183. Nakagawa, T. (1986). Periodic and sequential preventive maintenance policies, Journal of Applied Probability 23, 536 542. Nakagawa, T. (1988). Sequential imperfect preventive maintenance policies, IEEE Transactions on Reliability 37, 295 298. Nakagawa, T. and Mizutani, S. (2009). A summary of maintenance policies for a finite interval, Reliability Engineering and System Safety 94, pp. 89 96. Nguyen, D. G. and Murthy, D. N. P. (1981). Optimal preventive maintenance policies for repairable systems, Operations Research 29, pp. 1181 1194. Sheu, S.-H. and Chang, T.-H. (2002). Generalized sequential preventive maintenance policy of a system subject to shocks, International Journal of Systems Science 33, pp. 267 276. Sheu, S.-H. and Liou, C.-T. (1995). A generalized sequential preventive maintenance policy for repairable systems with general random minimal repair costs, International Journal of Systems Science 26, pp. 681 690.