IEEE Transactions on Aerospace and Eectronic Systems. Voume 38, #, pp.388-403, 00. Data Mining Technoogy for Faiure Prognostic of Avionics V.A. Skormin, Binghamton University, Binghamton, NY, 1390, USA vskormin@binghamton.edu V.I.Gorodetski, St. Petersburg Institute for Informatics and Automation St. Petersburg, 199178, Russia fax 7(81)384450 ph.7(81)384411 gor@mai.iias.spb.su Abstract. L.J.Popyack Air Force Research Laboratory Rome, NY, 13441-5700, USA popyack@r.af.mi Safe, reiabe, and efficient operation of avionics is crucia for a modern aircraft or spacecraft. Whie in operation, avionics components are exposed to eectrica perturbations, mechanica vibrations, excessive temperatures, humidity, etc. These adverse conditions, acting individuay and in combination, are known to have cumuative effects eading to avionics performance degradation and faiures. Cassica reiabiity had addressed statisticay-generic devices. Unti recenty, it was virtuay impossibe to obtain data characterizing performance of individua units. At the present, avaiabiity of dedicated monitoring systems and ike devices aows for the coection of arge amounts of actua data of any particuar unit of aircraft hardware. Based on this data, modern Data Mining techniques, common in Knowedge Discovery from Data (KDD) technoogy, faciitate formuation and soution of important on-ine and off-ine prognostic-reated probems. The research presented in this paper is aimed at the accurate assessment of the probabiity of faiure of hardware, such as avionics, on the basis of its known history of abuse by environmenta and operationa factors. Such a probem statement is prompted by the modern concept of maintenance known as the "service when needed". It is expected that the prognostic mode presented in this paper is deveoped on the basis of information downoaded from dedicated monitoring systems of fight-critica hardware and stored as a database. The deveoped KDD technoogy is demonstrated numericay. The KDD technoogy proposed in the paper coud be efficienty used in a wide area of appied tasks of Data Mining and KDD. 1 1 Parts of this work have been funded by the United States Air Force office of Scientific Research.
1. INTRODUCTION Safe, reiabe, and efficient operation of avionics is crucia for a modern aircraft or spacecraft. Whie in operation, avionics components are exposed to eectrica perturbations, mechanica vibrations, excessive temperatures, humidity, radiation, etc. These adverse conditions, acting individuay and in combination, are known to have cumuative effects eading to avionics performance degradation and faiures. Traditionay, reiabiity of any technica device (eectronic, eectro-mechanica, and mechanica) is defined in terms of such characteristics as the average time of norma (no-faiure) operation. These reiabiity concepts referring to a statisticay-generic device may be considered acceptabe as ong as the faiures are caused by the factors reated to manufacturing. At the present, this approach is not aways acceptabe. Manufacturers of eectronics, due to competey automated processes, have achieved a very high degree of reiabiity of their products and very itte variation in properties from device to device. Manufacturing-reated effects on faiures of eectronics are graduay becoming ess significant. The main causes of faiures are traced now to the individua operationa and environmenta conditions of particuar units. Therefore, the average time of norma operation and other "traditiona" reiabiity characteristics, defined without taking into account actua history of abuse of a device, are becoming ess important. Cassica reiabiity had a good reason for addressing a statisticay-generic device, at that time it was virtuay impossibe to obtain data characterizing performance of individua units in various operating environments. At the present, avaiabiity of Time- Stress Measurement Devices (TSMD) [1], smart sensors and data acquisition systems (ike Anaog Devices AD uc81 []), and MEMS makes possibe to coect arge amount of actua data of any particuar unit of aircraft hardware. Based on this data, modern Data Mining techniques, common in Knowedge Discovery from Data (KDD) technoogy, faciitate formuation and soution of important on-ine and off-ine reiabiity-reated probems. The most important probem is forecasting the probabiity of faiure of fight-critica units of aircraft hardware during a forthcoming sortie. Soving such a probem impies the investigation of the roe of various environmenta factors in the deveopment of particuar faiures, investigation of combined effects of severa factors, reevauation of probabiity of faiure on the basis of known exposure to particuar adverse conditions, as we as deveopment of specia types of mathematica modes and mode-based techniques. Data Mining and KDD address specific practica needs for soving abovementioned probems. Data Mining provides a wide spectrum of avaiabe techniques and toos to deveop a KDD technoogy focusing on design of a mathematica mode for particuar appication ([3], [4], [5], [6], [7]). It is we known that every particuar appication possesses specific properties that require either the abiity to adapt aready existing Data Mining techniques or deveop new ones to buid an adequate and efficient technoogy of origina data processing aimed at a particuar mode deveopment. As per common understanding ([8]), a KDD process considered herein consists of a number of Data Mining procedures that, regardess of domain and particuar task, conceptuay, incude such steps as (1) definition of the goa of the task, () coection or mode-based generation of adequate statistica data and its preprocessing, (3) data reduction, transformation to find usefu specifications and representations, incuding visuaization if possibe, (4) deveopment of a KDD strategy that, actuay, corresponds to the outine of the future technoogy as a number of steps of Data
Mining, (5) seection, adaptation or deveopment of Data Mining methods and agorithms intended for the reaization of the accepted KDD technoogy (search of informative subsets of attributes and pattern of interest, separation and decision making rues creation, features, regression mode deveopment, etc.), (6) interpretation of the Data Mining resuts and incorporation of these resuts into a target mode, (7) testing and vaidation of the resutant mode. Steps of this KDD process are usuay iterative and interactive and are common for any KDD process. Nevertheess, from the agorithmic and impementation points of view, particuar KDD processes may be impemented in very different ways. It is we known that the best universa approach does not exist. Moreover, the wider the area of possibe utiization of an agorithm or approach, the esser its efficiency. Therefore, taking into account the domain and task specifics, combined with the experience in KDD technoogy and Data Mining, assures the successfu soution of any particuar appication probem. Then, foowing such a principe, we deveop an approach that consists of traditiona steps of KDD process, but its appication refects the foowing framework: pecuiarities of the goa of the task (prognosis of probabiity of faiure of avionics); origina statistica data avaiabe for diagnostic and prognostic mode design (TSMD-based records of cumuative exposure to environmenta factors and operationa conditions); the need for a highy dependabe mode-based prognostic procedure; requirement of a reiabe assessment of probabiities invoved in the cacuation of the probabiity of faiure of a hardware even if the size of statistica data is sma; The remainder of the paper is organized as foows. In the next section we give a brief outine of the KDD technoogy appied to the deveopment of the mode-based prognosis system to assess probabiity of faiure of avionics. Section 3 contains a brief description of the heuristic informativey criteria that are used in the genera case for the preiminary seection of informative subspaces of ow dimension. In the paper we consider ony two-dimensiona subspaces. In Section 4, a notion of a cassification predicate is defined and a number of approaches to obtain such predicates are described. We propose a visuaization technique that makes it possibe for a deveoper to draw a separation bound and to generate the associated cassification predicate automaticay. Then we describe the main principe behind the design of decision trees and associated probabiistic spaces that form a set of decision procedures. Since the major purpose of the mode under deveopment is the assessment of the probabiity of faiure of a hardware unit, in Section 5 we consider in detai the improvement of the precision and reiabiity of this assessment using the sma size of experimenta data and experts knowedge. Section 6 features an exampe of an impementation of the outined technoogy for the deveopment of a mode-based prognostic procedure for a particuar avionics modue. In Section 7, we compare the deveoped KDD technoogy with the known techniques faciitating soution of the same or simiar KDD subtasks, emphasizing the advancements proposed in the paper. In concusion we outine the main resuts and our intentions for the future work.. GENERAL SCHEME OF THE DEVELOPED KDD TECHNOLOGY The foowing is the outine of the deveoped approach that forms a KDD technoogy for the prognostic mode deveopment. Origina statistica data specifies current vaues of cumuative exposure of a number of avionics modues of the same type to environmenta factors and operationa conditions. This data forms a factor space that is divided into two custers formed by data records interpreted as the
«norma performance» («no faiure») and «faiure» respectivey. The KDD process aims at the deveopment of a mode and a mode-based prognostic procedure. The atter is utiized for estimating the probabiity of faiure of a particuar modue at a given time in the future, say, during a forthcoming sortie, on the basis of the current "history of abuse of this modue. From the Data Mining point of view, this probem constitutes the cassification task. As a whoe, the target cassification mode is specified as a number of decision trees. Decision making procedure consists of making a decision by each decision tree separatey and joint processing of decisions obtained by each of them utiizing the reativey simpe procedure which, to make a fina decision, utiizes an additiona background probabiistic knowedge. Deveopment of each above-mentioned decision tree incudes the deveopment of a partiay ordered set of nodes. Consider creating i-th node (i=1,,..., r) having as input argument a subset of training data S i which contains cases of both custers ('no faiure" and "faiure"). Then the procedure of a node design consists of the foowing steps. 1. Ranking of two-dimensiona subspaces (-d subspaces) of the entire factor space in accordance with the chosen criterion of informativity cacuated over training data S i. It is avaiabe for a deveoper to use a number of heuristic criterions of informativity. Seection of the most informative -d subspaces is the most important task.. Visuaization of the projections of both custers of training data S i onto the seected -d subspace and providing a deveoper with the opportunity to adjust the separation rue manuay using a computer interface. Automated generation of the associated cassification predicate. 3. Division of the experimenta data S i into two non-overapping subsets S + i, S i, S i = S + i S i. The subset S + i contains the cases of S i such that the cassification predicate obtained at the previous step is "true", and the subset S i contains the cases of S i over which it is "fase". Based on an additiona criterion, each of two subsets S + i and S i is cassified as a eaf R j of the decision tree under deveopment or as its new intermediate node. 4. Decision tree deveopment is ended if it does not contain intermediate nodes that were not processed in accordance with the procedures described above in the steps 1-3. 5. Each eaf R j of the decision tree is mapped to the predicate P j, which is constituted as conjunction of cassification predicates met aong the way from root node up to the eaf R j. In addition, each eaf R j is mapped to a subset of cases of experimenta data for which predicate P j is "true". Note that each eaf R j may contain cases of experimenta data beonging to both custers "no faiure" and "faiure" or ony to one of them. 3 Let r decision trees deveoped. Then the foowing task is performed to constitute the decision-making procedure. Note, that the deveoped software makes it possibe to draw manuay a non-inear and even non-convex separation rue and automated generation of the associated cassification predicate (see beow Section 4). 3 As a particuar case, we consider a decision tree that consists ony of a root node.
6. For each decision tree number k, (k=1,,...,r), definition of the probabiistic space in k k which every its eaf R j constitutes an eementary event. Each eementary event R j is characterized by a point and an confidence interva probabiities p k (X/"no faiure") and p k (X/"faiure ) estimate defined empiricay using training and testing data for any vector of factors X R k j. The next two steps correspond to decision making procedure itsef and are utiized for mode-based estimating (forecasting) the probabiity of faiure of a particuar modue at a given time on the basis of its current "history of abuse". k 7. For given vector of factors X, definition of the eaf R j which it beongs to within each decision tree number k, and hence, definition of the vaues of probabiities p k (X/"no faiure") and p k (X/"faiure") (see step 5) obtained by each decision tree. Joint processing of the above probabiities on the basis of so caed "Agebraic Bayes' Network" forma framework deveoped in [9], [10], and cacuation of the fina vaues of probabiities p(x/"no faiure") and p(x/"faiure"). 8. Definition of the decision making scheme based on Bayes approach [11]. This procedure aims at cacuation of the probabiity of faiure p("faiure"/x) of the device having given vaue of the vector of factors X. 9. Testing the deveoped prognostic mode and mode-based prognostic procedure using an array of both training and examination data to assess properties of the mode and the decision-making procedure. Of course, the opportunity to utiize our procedure for assessing the probabiity of faiure during the future cyce of the modue operation depends upon the abiity to forecast exposure to adverse factors at the time in question. The task of deveopment of the appropriate means of forecasting is out of the scope of this paper. Note once more that the first five steps resut in the definition of a prognostic mode. The next three steps constitute a mode-based decision-making procedure. The ast step is aimed at vaidation of the resuting mode and mode-based decision-making procedure. The mathematica mode, discussed herein, is appicabe to soving such practica prognostic reated probems as: ranking particuar environmenta conditions as factors responsibe for genera and particuar types of faiures, determination of particuar groups of environmenta conditions and assessment of their combined effects on faiures in genera and on particuar types of faiures, tracking the dynamics of deveopment of custer modes and their statistica characteristics in the process of obtaining new experimenta data, justification of the deveopment of devices protecting avionics from adverse environmenta conditions, deveopment of the recommendations on the avoidance of the combined effects of adverse conditions. This coud be performed in rea-time, on an aircraft or spacecraft. Beow we consider the above-described steps of the proposed technoogy in more detais. 3. HEURISTIC SELECTION OF INFORMATIVE SUBSPACES It is we known that earning procedures aimed at extracting knowedge from data are computationay intensive. To decrease the amount of computations, researchers often use heuristic and intuitive notions such as "informativity", "simiarity", etc. Formay specified, heuristic and intuitive notions are aways probem-
or domain-oriented. In our approach we use the intuitive notion of informativity to rank the subspaces of factors (features) and to seect a more compressed specification of experimenta data for further processing on this basis. We have investigated a number of forma specifications of informativity criteria. A of them can be interpreted as mean square normaized and, possiby, weighted distance between two custers of statistica data. For arge amounts of data the same criteria can be specified in terms of corresponding statistics assessed over data empiricay. Herein and beow we use the foowing notations: X = { x1, x,..., x n } - is a vector of factors representing cumuative exposure to adverse conditions in hours; Q {0,1} - is an integer variabe symboizing the output discrete event ("norma operation of the device" corresponds to Q=1 and "the device faied" corresponds to Q=1; observed data are indexed by the symbo t=1,,...,n, N= K 0 +K 1, where K 0, K 1 are the tota number of reaizations of custer "0" and custer "1" respectivey. Therefore, experimenta database consists of the subsets of K 0 reaizations of custer 0 0 0 "0" marked by superscript "0", for exampe, { x1 ( r), x ( r),..., xn ( r)}, and K 1 reaizations of custer "1" marked by superscript "1", for exampe, 1 1 1 { x1 ( s), x ( s),..., xn ( s)}. Additiona notations wi be introduced ater. The informativity criteria presented beow were seected as the most appropriate due to their adequate representation of experts intuitive interpretation of the subspace informativity, and the compexity of the subspace ranking task. Athough criteria (1) - (3) beow correspond to a -d case, they coud be defined in a subspace of arbitrary dimension. 1 K1 K0 1 0 1 0 D q, = ( K1K 0) {[x (r) x (s)] / σ + [x q(r) x q(s)] / σ q }, (1) r= 1 s = 1 MD [ ] = w ( 0)/ σ + w ( 0)/ σ + w ( 1)/ σ + w ( 1)/ σ + q x xq q x xq q,, + ( x ) / σ + ( x ) / σ 01 01 q q where σ, σ q - are standard deviations of variabes x and x q estimated over the 01 entire range of experimenta data, (, 01 x ),( x, q ) are squared distances between mathematica expectations of vectors of factors within custers "0" and "1" respectivey in the subspace comprising factors x, x. 1 K1 K0 0 1 1 0 1 0 Dw q, = ( K1K0) aк a s{[x (r) x (s)] / + [x q(r) x (s)] } r 1 s 1 q / = = σ σ q, (3) where a 0 r, a 1 s are weights assigned to cases (reaizations) number r and number s of custers "1" and "0" respectivey. Weights a 0 r and a 1 s are cacuated according to the agorithm given beow. First, the vaues of coordinates of the unit vector eq =< e, eq > of the ine connecting mean vectors of custers in the subspace < x, xq> are computed as foows: 0 1 0 1 x = ( x x )/ σ, xq = ( xq xq)/ σ q, b = ( x ) + ( xq) r eq =< x / b, xq / b >=< e,, eq > Then the vaues of weights for every reaization of custer "1" (they are marked by argument r) and of the custer "0" (they are marked by argument s) are computed as foows: q ()
x q For a cases of custer «1» do (r=1,,... K 1 ) 0 0 0 0 0 d = e ( x ( r) x )/ σ + e ( x ( r) x )/ σ r q q q q 1 0 1 0 1 d = e ( x ( r) x )/ σ + e ( x ( r) x )/ σ 0 1 if d d 0 1 r r, ar = 0 1 0, if dr < dr. For a cases of custer «0» do (s=1,,... K 0 ) r q q q q 0 1 0 1 0 d = e ( x ( s) x )/ σ + e ( x ( s) x )/ σ s q q q q 1 1 1 1 1 e d = e ( x ( s) x )/ σ + e ( x ( s) x )/ σ x Fig.1.Geometric interpretation of the criterion 3. 1 0 1 if d s d 1 s, - mean of the custer "0," - mean of the custer "1" as = 0 1 0, if d s < d s. - cases of the custer "0" - cases of the custer "1" The roe of the weighs in criterion 3 for -d case is expained in Fig.1. Note, that criterion () is a statistica equivaent of criterion (1) and is intended to be used for arge amounts of experimenta data. Criteria (1) - () are additive and this property makes possibe to design an efficient optimization procedure of subspace ranking according to their informativity for any arbitrary dimension. Let us demonstrate this procedure for criterion (1). At the first step of the optimization procedure, we consider the seection task for one-dimensiona subspaces. For an arbitrary subspace associated with a factor x, =1,,...,n, the vaue of function (1) may be computed as foows: s q q q q 1 1 1 0 D = ( K K ) {[x ( r) x ( s )] /σ }. (4) 1 0 = = K K0 r 1 s 1 The computationa compexity of the function (4) for a =1,,...,n is equa to CnK 1 0K1. The ranking procedure takes nog(n) operations ([1]), therefore, the tota theoretica compexity for the case under consideration is equa to Cnog( nkk ) 0 1. At the second step, a -d case is considered and it is necessary to cacuate a trianguar matrix of paired distances (1). Fortunatey, the vaue of {, q}-th eement of the above mentioned matrix may be cacuated in a simpe way if we woud utiize the resuts obtained for the one-dimensiona case as foows: 1 K1 K0 1 0 1 0 D q, = ( K1K 0) {[x (r) x (s)] / + [x q(r) x (s)] } D D r 1 s 1 q / q = + = = σ σ q (5) The computationa compexity of this step, incuding the ranking procedure, is expressed as Cn 6 + Cn 7 og( n), therefore, the tota compexity is Cn 7 og( n). At the third step, the vaue of the modified distance (4.3) is cacuated as D,, = D + D + D (6) qp q p Hence, even for an exhaustive search, the computationa compexity of the seection of the most informative three-dimensiona subspaces, is expressed as Cn + Cn og( n), therefore, the tota compexity is Cn og( n) 4. In the same way 3 e q 4 "0" 0 0 0 X (, q) =< x ( r), x ( r) > r q e 0 d r "1" d r 1 1 0 0 d < d, then a =0 r r r 5 4 Note that at the second and a consequent steps, computationa compexity is evauated with respect to summations but not to mutipications.
the compexity anaysis coud be performed for the case of subspaces of four and higher dimensions. In section 6 we demonstrate the utiization of the above seection procedure numericay. 4. CLASSIFICATION PREDICATES AND DECISION TREE DESIGN According to the accepted methodoogy of prognostic mode deveopment based on numerica experimenta data, at the next step so-caed cassification predicates ([13]) are deveoped. Actuay, the meaning of cassification predicates introduced beow is twofod. First, they form a basis for the definition of prognostic rues. Second, one can consider cassification predicates as a feature that represents experimenta data on a binary scae instead of the origina numeric scae. The atter view is very usefu: since the origina experimenta data contains both continuous and discrete coumns (factors), the utiization of cassification predicates faciitates the transformation of a coumns of the origina experimenta data to a discrete format. This transformation is typica in performing Data Mining and KDD tasks. Conceptuay, a cassification predicate is viewed as the predicate associated with a separation rue designed within a subspace of ow dimension. Reca, that in order to faciitate visuaization, in this study we consider ony -d subspaces. Deveopment of cassification predicates and decision trees is achieved in a number of steps outined in brief in Section. Let us describe these steps in more detai. In Section 6 we demonstrate the basic ideas numericay. 4.1. Synthesis of separation rues Consider two techniques for the definition of separation rues and associated cassification predicates. The first technique is based on visuaization and with the aid of an interactive software too, resuts in optima poygon-ike separation rues. The second technique works automaticay and resuts in eipse-based noninear separation. 4.1.1. Visua synthesis of arbitrary separation rue approximated by inear spine Let us consider projection of two origina custers of experimenta data on a -d subspace of factors, i.e. onto a pane as shown in Fig.. Assume that a software too aows a user to draw inear separation bounds that are perceived as good or optima. Assume that as a user draws a inear separation bound, the software too automaticay generates the inear equation f ( x, x ) of the corresponding bound and defines the q appropriate predicate as foows: if fk( x, xq) 0 then P k is true (7) if fk( x, xq)<0 thenp k is fase. Geometricay, (7) impies that in a haf-pane < x, xq> predicate P k is true and it is fase in the aternative haf-pane. It is expected that a user has the abiity to draw a number of inear separation bounds and the software too automaticay generates equations f1( x, xq),..., fm( x, xq) and associated predicates P 1, P,..., P m. Each predicate divides pane < x, xq> in two haf-panes. Generay, pane < x, xq> wi be divided into no more than m m convex regions Li, i= 1,,..., that do not overap and m in combination cover the entire subspace < x, x >. Within each region L, i=1,,..., q i
Fig.. Linear spine approximation of separation rue. Associated cassification predicate is as foows: CP 1 = P 1 & P & P 3. Predicate CP 1 is true within the grayed region. are cases of the custer "1" and are cases of the custer "0". Broken ine was cacuated by software too exacty one conjunction of the ength m of predicates P 1, P,..., P m taken with and without negation is true. Hence, m each region L, i = 1,,..., is i defined formay by a conjunction of predicates (7), an arbitrary combination of such regions is defined by disjunction of abovementioned conjunctions. One shoud understand that if a software too provides a user with the capabiity to define a number of visuayjustified inear separation bounds and associated predicates, then the user is capabe of designing a very wide cass of separation rues. This cass contains inear and poygonike bounds and arbitrary convex and non-convex regions that may be obtained as a combination of convex regions of truth of some above mentioned conjunctions. To iustrate this concept, consider the situation depicted in Fig.. Assume that a user defines three inear separation bounds by visuaizing the computer-generated custering pattern in the pane <x 9,x 1 >. The predicates P 1, P, and P 3 are associated with these bounds and are assigned vaue of "true" within the haf-pane ocated above and on the right from the corresponding inear bounds. Hence, the highighted region corresponds to the truth domain of the predicate CP 1 = P 1 & P & P 3. The separation bound and the corresponding predicate, CP 1 = P 1 & P & P 3 are designed usuay to meet standard requirements that formay can be specified as foows: N 11 > M 10, М 00 > N 01 (8) where N 11 is the number of reaizations of custer "1" for which predicate CP 1 is assigned the vaue of true"; N 01 is the number of reaizations of custer "1" for which predicate CP 1 is assigned the vaue of fase ; M 10 is the number of reaizations of custer "0" for which predicate CP 1 is assigned the vaue of true, and M 00 is the number of reaizations of custer "0" for which predicate CP 1 is assigned the vaue of fase. It is cear that (N 11 + М 00 ) reaizations of experimenta data are correcty cassified by predicate CP 1, and (M 10 + N 01 ) reaizations of data are cassified by predicate CP 1 erroneousy. For exampe, these numbers for predicate CP 1 are (see Fig.): N 11 =7, M 10 =6, N 01 =3, М 00 =99. Definition 1. Predicate that meets conditions (7) is a cassification predicate.
Definition 1 is non-forma and introduces the term that is used esewhere. Based on experimenta data, every cassification predicate CP k, k=1,,...,m, can be assigned a number of attributes that represent the quaity of cassification that it is expected to achieve. Consider empirica estimates of probabiities of the correct and erroneous cassifications of reaizations of experimenta data represented as a matrix: pk( / ) pk( / ) p( CPk ) = 1 1 1 0 pk( 0/ 1) pk( 0/ 0 ), (9) Note that the first argument within the brackets corresponds to the decision made by cassification predicate, and the second argument corresponds to the actua status of the reaization in question. These estimates can be cacuated as foows: p k ( 1/ 1 ) =N 11 (k)/[n 11 (k)+m 10 (k)], p k ( 1/ 0 ) =M 10 (k)/[m 10 (k)+м 00 (k)], p k ( 0 / 1 ) =N 01 (k)/[m 00 (k)+n 01 (k)], p k ( 0 / 0 ) = M 00 (k)/[n 01 (k)+ М 00 (k)], For exampe, for cassification predicate CP 1 (see Fig.) the above estimates are as foows: p k ( 1/ 1 ) =7/78=0.93; p k ( 1/ 0 ) =6/78=0.077; p k ( 0 / 1 ) =3/1=0.189; p k ( 0 / 0 ) =99/1=0.811.
4.1.. Automaticay obtained eipse-based separation rues An automatic procedure resuting in eipse-based separation rues has been proven to be quite convenient and efficient. It is impemented under the assumption that a separation rue in the space of two factors x and x p is defined on the basis of an eipse equation fi( A, x, xp) = ( x a1) / a3 + ( xp a) + a5( x a1)( xp a)/( a3a4) 1, (10) where A= [ a1, a,..., a5 ] - is the vector of unknown parameters of the eipse. This assumption is justified by most common mutua topoogies of two custers, such as faiures and no-faiures are ineary separated, faiures are surrounded by no-faiures, and nofaiures are surrounded by faiures. Initia vaues of parameters A= [ a1, a,..., a5 ] coud be defined in terms of estimated means and variances of variabes x 1 (), r x 1 p(), r r=1,,..., K 1. The fine-tuning of these parameters, resuting in the optima position, orientation, size, and shape of the eipse can be achieved by numerica minimization of the foowing heuristic criterion: K 1 1 K 0 0 Q( A) = 1 0 V[ A, x ( r), xp( r)] + V[ A, x ( s), xp( s) r= 1 s =, 1 (11) where 1 1 1 1 V[ Ax, ( r), x ( r)] = [ x ( r) a ) + [ x ( r) a ] - is the penaty assigned to each point of custer p 1 p 1 1 1 1 faiure that does not satisfy condition f(, Ax, x ) > 0, otherwise V[ Ax, ( r), x( r)] = 0; and i p 0 0 0 0 V[ Ax, ( s), x ( s)] = [ x ( s) a ) + [ x ( s) a ] - is the penaty assigned to each point of custer p 1 p 0 0 0 0 no faiure that satisfies condition fi( A, x, xp) > 0, otherwise V[ Ax, ( s), xp( s)] = 0. A very efficient numerica optimization procedure (the direct search by Neder and Mead [14]) has been successfuy utiized to carry out the minimization task eading to the estimation of parameters A=[ a1, a,..., a5 ] of the separating function. Note in order to accommodate for a different custer topoogy, it is suggested to suppement the above anaysis by deveoping separation rues on the basis of eipses g( X ) = 0, 1 1 0 0 i=1,,..,k, such that gx (, xp) <0 and gx (, xp) 0 and optimize their parameters by numerica minimization. Then for each i=1,,..,k the goodness of the separating surface g( X ) = 0 must be compared with the one of fi ( X)=0 and the best surface sha be seected. When the task of estabishing separation surfaces has been competed, ogic functions of cassification ike (7) can be defined. The set of conditiona probabiities (9) of correct and incorrect cassification can be estabished empiricay for every i=1,,...,k. 4..Decision tree deveopment The genera idea of decision tree was outined in Section. The decision tree deveopment procedure aims at finding the non-correated informative subspaces over the training subset of experimenta data. Let us expain the term "correation" within the context of the paper. Let P1 and P be the cassification predicates associated with a seected pair of the most informative subspaces. Let each of them divide the entire set of cases S into two subsets, i.e. into subspaces {S 1 1,S 1 } and into subspaces {S 1,S } respectivey. It may turn out that S 11 S 1 and S 1 S. If such non-forma equaities are hed, then the second informative subspace is not abe to improve remarkaby the cassification procedure. Hence, the second subspace is informative in itsef, but is not informative if it is added to the first one. This exampe expains in what sense we use the term "subspaces correation". The atter may be specified formay, but it is not p
necessary because it is cear how to take into account correation of subspaces in the seection procedure to avoid utiization of highy "correated" subspaces in the cassification procedure. Let us expain how it can be performed. In the Fig.3, a sampe of a decision tree is depicted. In the first step the deveoper has seected the -d subspace < x9, x1> as the most S 1 informative. The upper screen (Fig.3) depicts projections of both custers "0" and "1" onto the pane fase 4 Leaf R 1 Cases of custer "faiure" true S Leaf R 1 3 Cases of custer "no faiure" Leaf R 1 fase <X9,X1> <X9,X10 fase true Fig.3. Decision tree No. 1. <X3,X11> <X6,X17> true <X6,X1> Leaf R 1 1 < x9, x1> and a inear separation rue (continuous ine) estabished by an expert manuay as the optima one (The broken ine corresponds to the separation rue cacuated automaticay but rejected by the deveoper.). This separation rue divides the entire set of training experimenta data into two non-overapping subsets S 1,S. The next steps of subspaces seection are appied separatey to subsets S 1 and S which form two new nodes. Therefore, on the second and third steps we have to sove two tasks of informative subspaces seection for the two above mentioned subsets S 1 and S of experimenta data. In the Fig.3 the resuts of the second step seections are visuaized and represented as printouts. The next and a further steps can be reaized in the same way. As a resut, a decision tree depicted in the Fig.3 is obtained. This decision tree contains a reated information, i.e. partiay ordered set of -d subspaces, equations of separation rues, cassification predicates, reated probabiities, etc. One can see that some separation bounds are chosen in a noninear form. 4.3. Probabiistic decision making procedure The resut of the above procedure is the decision tree such that each its eaf is mapped to a subset of cases of origina training data. These subsets are non-overapping and their union covers the entire set of training data. On the other hand, each eaf is mapped to its own cassification predicate constituted as conjunction of cassification predicates of decision tree nodes met aong the way from tree root up to respective eaf. Each such predicate is true in the concrete region of
the factor space, and the regions corresponding to the different eaves are not overapping and cover the entire factor space. Hence, they can be used as the eementary events to design a probabiistic space. Let us consider in more forma way how the probabiistic space is constituted and how it is utiized to assess the probabiity of faiure of a device having given "history of abuse". Let {R 1, R,..., R S } is the set of eementary events which are mapped to the set of eaves of a designed decision tree and P 1, P,..., P S is the set of the respective (mapped to corresponding eaves of decision tree) cassification predicates. Each such eementary event R i {R 1, R,..., R S } can be mapped to an empiricay estimated probabiity p ( X ) R = p ( i X ) on the basis of testing of designed i decision tree over both training and testing data in a traditiona way. We suppose that these estimations are cacuated as confidence intervas for given eve of confidence probabiity (see exampe in Section 6). In the same way, confidence intervas of probabiities p ( i X / 0 ) and p ( i X / 1 ) can be estimated. Note, that p ( i X )= p( X/ 0 ) + p( X/ ) i i 1. As a resut, for each predicate P 1, P,..., P S the foowing probabiities wi be estimated by confidence intervas: ph( 1/ 1) = p[ Ph( X) = " true"/ X " 1"], ph( 1/ 0) = p[ Ph( X) = " true"/ X " 0"], (1) ph( 0/ 1) = p[ Ph( X ) = " fase"/ X " 1"], ph( 0/ 0) = p[ Ph( X) = " fase"/ X " 0"]. The avaiabiity of the probabiities (1) makes it possibe to cacuate the target probabiity of faiure for any point of the factor space X * subject to the condition that P ( h X ) = " true " using Bayes' formua as foows: p() 1 p )* h ( 1/ 1) P[ 1 / Ph ( X ] = (13) p() 1 ph( 1/ 1) + p() 0 ph( 1/ 0) where p(1), p(0) - are prior probabiities of faiure and norma operation. It is obvious that probabiity of norma operation under the adverse exposures X * is ph( 0/ Ph( X * )] = 1 p[ 1/ Ph( X * )] In the genera case when the avaiabe size of training and testing data is too sma, we are not abe to obtain the satisfactory accuracy of estimations of task reated probabiities to forecast a probabiity of faiure. However, the quaity of designed prognostic mode depends criticay on the above accuracy. Therefore, we have to undertake specia efforts to provide the needed accuracy. In the next section we investigate an approach to cope with the above probem. 5. GENERAL SCHEME OF CALCULATION OF THE TASK RELATED PROBABILITIES Reca that the main goa of the prognostic mode under deveopment is the reevauation of the probabiity of faiure of an avionics modue on the basis of its actua history of abuse represented by the vector of adverse exposures. It is esewhere adopted that, the accuracy of the assessment of this probabiity depends on two factors: (1) the tota amount of experimenta data that is used for training (prognostic mode design) and testing (evauation of the mode quaity), () the quaity of prognostic mode and mode-based decision-making procedure. We assume that in our case the amount of experimenta data is sma and we have no information about the distribution of reaizations within data custers. Therefore, in a genera case even the best prognostic mode may be not abe to provide precise probabiity assessment. However, we are abe to assign a confidence interva for the probabiity of each eementary event associated with each eaf of the designed decision tree which the probabiistic space that constitute a guaranteed estimation.
To narrow these confidence intervas and, hence, to improve the accuracy of probabiity assessment, we proposed the foowing two approaches: It is proposed to use a set of prognostic modes (a set of decision trees) that differ in factor subspaces utiized by each decision tree. This approach eads to a coective of decisionmaking procedures. Then the redundancy of information invoved in a decision making procedure resuts in the accuracy improvement of the probabiity estimation. Of course, the appication of this approach requires that a specia agorithm for joint processing of probabiities resuting from each decision tree be deveoped. We propose to use the agorithm based on the Agebraic Bayes' Network (ABN) approach deveoped in [15]. It is proposed to repace the traditiona Bayes' formua (13) resuting in a posterior probabiity, by its equivaent deveoped on the basis of methods of interva mathematics which faciitates the cacuation of a guaranteed estimate of the probabiity of faiure. Consider the brief discussion of the agorithms impementing the above approaches. In Section 6 these agorithms are iustrated by a numerica exampe of the assessment of faiure probabiity of an avionics modue. It is understood that the quantities generated by a decisionmaking procedure, represented by a decision tree, depend on the choice of the root and intermediate nodes. Typicay, we are abe to design a number of decision trees utiizing different subspaces of the factor space that resuts in information redundancy. Consider the impact of this redundancy on the accuracy of the prognostic procedure. Assume that a set of three decision trees has been estabished within a particuar prognostic mode. Consider the appication of this mode for the assessment of the probabiity of faiure of an avionics modue subjected to adverse exposures X. Assume that vector X resuts in eementary events R 1 i, R j and R 3 k or, using some specific jargon, beongs to the appropriate eaves of the first, second and third decision trees. The respective probabiities are defined as R 1 i : a 1 i p(x) b 1 i, a 1 i () 0 p(x/0) b 1 i ( 0), a 1 i () 1 p(x/1) b 1 i (); 1 (14) R j : a i p(x) b j, a j ( 0) p(x/0) b j ( 0), a j () 1 p(x/1) b j (); 1 (15) R 3 k : a 3 k p(x) b 3 k, a 3 k ( 0) p(x/0) b 3 k ( 0), a 3 k () 1 p(x/1) b 3 k (); 1 (16) It coud be seen that vector X beongs to a tree subspaces, therefore, the probabiity of the event R i1 R j R 3 k, i.e. the probabiity of the event p( R i1 R j R 3 k ) coud be defined on the basis of Agebraic Bayes' Network (ABN) approach deveoped in [16]. This approach refects the basics of the probabiity theory and requires that the interva constraints (14) - (16) be suppemented by some fundamenta axioms. In the case under consideration, we specify the interreationships between probabiities that are defined by each decision tree (they are given in (14) - (16)) and the probabiity p( R i1 R j R 3 k ). Foowing the ABN approach we represent the interreationships between probabiities by a Hasse diagram [17] as shown in Fig.4. Denote the probabiities of events constituted by intersections of events R 1 i, R j and R 3 k as foows: p(x R 1 i )=p(x 1 ), p(x R j )=p(x ), p(x R 3 k )=p(x 3 ), p[(x R 1 i )&(X R j )]=p(x 1 X ), p[(x R 1 i )&(X R 3 k )]=p(x 1 X 3 ), p[(x R j )&(X R 3 k )]=p(x X 3 ), p[(x R 1 i )&(X R j )&(X R 3 k )]=p(x 1 X X 3 ). The foowing are the interreationships between probabiities refecting the axioms of norm and additivity: px ( 1 ) + px ( ) pxx ( 1 ) 1 px ( 1 ) + px ( ) + px ( 3 ) pxx ( 1 ) pxx ( 1 3) px ( 1 ) + px ( 3 ) pxx ( 1 3) 1 - pxx ( 3 ) + pxxx ( 1 3) 1. (17) px ( ) + px ( 3 ) pxx ( 3) 1, pxx ( 3) pxxx ( 1 3) 0, pxx ( 1 3) pxxx ( 1 3) 0, p( X 1X ) pxxx ( ) 1 3 0,
px ( ) pxx ( 1 ) 0 px ( 1) pxx ( 1 ) 0 px ( 3) pxx ( 1 3) 0 px ( 3 ) pxx ( 1 3) pxx ( 3 ) + pxxx ( 1 3) 0, (17) px ( 1) pxx ( 1 3) 0 px ( ) pxx ( 1 ) pxx ( 3 ) + pxxx ( 1 3) 0, px ( 3) pxx ( 3) 0 px ( 1) pxx ( 1 ) pxx ( 1 3 ) + pxxx ( 1 3) 0, px ( ) pxx ( 3) 0 and, as usua, px ( ) 0, i= 13,,, pxx ( ) 0, ij, = 13,,, pxxx ( ). i i j 1 3 0 These interreationships are viewed as the background knowedge, that coud be incorporated in the estimation of probabiity p(x)= pxxx ( 1 3 ) in the form of two inear programming probems as foows: 1.min{ p( X 1X X 3 )) under constraints (14), (15) (16) and (17) (ower bound); (18).max{ p( X 1X X 3 )) under constraints (14), (15) (16) and (17); (upper bound). (19) Note that based on experimenta data it is possibe to assess intervas for probabiities pxx ( 1 ), pxx ( ) 1 3 and pxx ( 3 ) as we. This may be usefu for further narrowing intervas of probabiities in question. This approach to p( X1XX3) improving the accuracy of interva probabiities is very fruitfu. It wi be demonstrated numericay by p( X1X) p( X1) pxx ( 1 3) p( XX3) p( X ) p( X ) Fig.4.Agebraic Bayes' Network for three- propositiona case 3 the exampe in Section 6. Now consider the use of Bayes' formua (13) for cacuation of the posterior probabiity p(1/x) in the case when probabiities p(x/1) and p(x/0) are given by their confidence intervas. Our goa is to cacuate the upper bound of probabiity p(1/x). Therefore, the task is to find its maximum vaue subject to constraints a P( X / 1) b, a P( X / 0) b. (0) 1 1 0 0 Simpe anaysis of this optimization task shows that it is equivaent to the task of maximization of the quotient p(x/1)/p(x/0). This provides the justification for the foowing formua p() 1 b1 max{ p( 1 / X) = (1) p() 1 b + p() 0 a 1 0 6. APPLICATION TO FAILURE ANALYSIS AND PREDICTION OF A PARTICULAR AVIONICS MODULE For the purpose of vaidation of the proposed technoogy and verification of the particuar procedures a software too impementing the basic computationa tasks presented herein was deveoped. This software provided the opportunity to iustrate the proposed KDD technoogy for a prognostic mode design by numerica exampes. Consider an exampe utiizing a training set of simuated TSMD data representing history of abuse and a binary status of 00 avionics modues of the same type. Additiona 300 reaizations are utiized to test the deveoped mode. The most interesting intermediate and fina resuts obtained by the impementation of the deveoped KDD technoogy for the cassification probem are presented beow. The database utiized for design and testing of the prognostic mode consists of 0 coumns that corresponds to 19 adverse exposures and the binary status of a modue. Adverse exposures (factors) and their denotations are given in the Tabe1.
Tabe 1. Database Composition X 1 Vibration X 6 Environmenta Temperature 15 - RMS, 1 - g, 0 C X Vibration RMS, 3-4 g X 7 Environmenta Temperature 0-15 C X 11 Power Suppy 1.1 - - 1.3 nomina Vdc X 1 Power Suppy over 1.3 nomina Vdc X 16 X 17 Functiona Overoad 31-40% X 1 Functiona Overoad 41-50% X 3 X 4 X 5 Vibration RMS, over 4g Humidity, 0 50% Humidity, 70 95% X 8 X 9 X 10 Environmenta Temperature 50-75 C Environmenta Temperature 76-100 C Power Suppy.7 -.9 nomina Vdc X 13 X 14 X 15 Functiona Overoad 5-10% X 1 Functiona Overoad 11-0% X 1 Functiona Overoad 1-30% X 18 X 19 X 0 Air Pressure.3 -.7 nomina Air Pressure 1.1-1.3 nomina Status of the modue (norma / faiure) a) a) b) b) Fig.5. Lists of ordered subspaces and corresponding histograms obtained by a) criterion (1),() and b) criterion (3) respectivey. <X1,X9> fase true <X10,X1> <X1,X17> fase true fase true Leaf R 1 Leaf R 1 Cases of custer «faiure» <X13,X17> Leaf R Leaf R 3 <X9,X18> Fig.6. Printouts forming decision tree No.. Visuaization of manuay obtained separation rues in subspaces of ow dimension. Cases of custer «no faiure»
<X9,X17> <X1,X17> <X10,X1> 1 Leaf R 3 Cases of custer "faiure" <X7,X17 <X9,X13> Leaf R 3 4 Cases of custer "no faiure" Leaf R 3 Leaf R 3 3 Fig.7. Decision tree No. 3. Appication of the deveoped method of seection of the most informative -d subspaces (see Section 3) has resuted in the ist of subspaces and their informativity vaues in the descending order. The first 0 subspaces are given in the printouts depicted in Fig. 5.a,b. Fig. above demonstrates pecuiarities of the visuaization procedure that makes it possibe to obtain noninear separation rues, associated cassification predicates and corresponding probabiistic attributes. To deveop a prognostic mode and decision making procedure, three decision trees were designed. They are depicted in the Fig.3, Fig.6, and Fig.7. Tabe contains information about probabiistic attributes of each decision tree eaf, i.e. interva estimations of probabiities of eementary events of each of three probabiistic spaces corresponding to the designed decision trees. These probabiistic attributes are used in the decision making procedure for every input For exampe, to assess probabiity of faiure of an avionics unit which "history of abuse" is specified by the vector of adverse factors is as foows: X * =[16.40, 19.09, 0.71, 10.87, 18.18, 0.05, 0.01, 8.91, 13.41, 13.58, 11.85, 13.68, 13.78, 1.49, 8.14, 1.15, 19.5, 17.7, 1.36]. The decision making procedure consists of three steps. The first of them is estabishing for each decision tree what its eementary event the input vector of adverse factors beongs to and the confidence intervas of probabiities are assigned vector X * by each decision tree. For a given input vector () the above eementary events and corresponding confidence intervas obtained empiricay are given in the Tabe 3. The second step of decision making procedure corresponds to soving of the tasks (18), (19) and aims at narrowing confidence intervas of the probabiity of faiure assigned by each ()
decision tree taking into account background knowedge (17). We can see that input vector () is cassified by the first, the second and the third decision tree as beonging to the subspaces 3 R 1, R and R 3 respectivey. We can sove the tasks (18), (19) under constraints (17) and interva constraints (14)-(16) taking or not taking into account probabiistic dependencies that exist over subspaces R 1, R and R 33. If we don t take into account the above dependencies then we obtain a soution as foows: 0.035 p(x * /1) 0.105, 0.0 p(x * /0) 0.08. Tabe. Confidence intervas assigned probabiity of faiure by each decision tree and obtained as soutions of the tasks (18) and (19). t-number of decision tree R i t - denotation of eementary event p( R t i )=p(x * /R t i ) p(x * /1, R t i ) p(x * /0, R t i ) a i t b i t a t i ( 0 ) b t i ( 0 ) a t i () 1 b t i () 1 R 1 1 0.36 0.464 0 0.040 0.308 0.34 t=1 R 1 0.163 0.77 0.035 0.105 0.100 0.00 R 3 1 0.083 0.177 0.083 0.177 0 0 R 4 1 0.194 0.316 0.194 0.316 0 0 R 1 0.70 0.400 0 0 0.70 0.400 t= R 0.100 0.00 0.059 0.141 0.00 0.080 R 3 0.199 0.31 0.075 0.165 0.09 0.188 R 4 0.185 0.315 0.185 0.315 0 0 R 1 3 0.37 0.363 0 0 0.37 0.363 t=3 R 3 0.105 0.05 0.038 0.11 0.04 0.118 R 3 3 0.185 0.305 0.058 0.14 0.096 0.194 R 4 3 0.37 0.363 0 0 0.37 0.363 Tabe 3. Confidence intervas assigned probabiity of faiure by each decision tree for input vector of adverse factors () t- decision tree number R i t -eementary event denotation p( R i t )=p(x * /R i t ) p(x * /1, R i t ) p(x * /0, R i t ) a i t b i t a t i ( 0 ) b t i ( 0 ) a t i () 1 b t i () 1 t=1 R 1 0.163 0.77 0.035 0.105 0.100 0.00 t= R 0.100 0.00 0.059 0.141 0.00 0.080 t=3 R 3 3 0.185 0.305 0.058 0.14 0.096 0.194 However, based on an experimenta database (say, training set of data), we are abe to obtain joint probabiities p( R 1 i & R j ), p( R 1 j &R 3 k ), p( R j &R 3 k ) and p( R 1 i & R j &R 3 k ) and in this case we are abe to obtain a more precise soution. However, it makes sense ony for the case when the size of database is not too high. Finay, the third step of the decision making procedure is the assessment of the probabiity of faiure based on Bayes' procedure transformed for the case when probabiities are assessed as
confidence intervas. Assume, that a-priori probabiities of the custer "0" and custer "1" are as foows: p('0")=0.999, p('1")=0.001. Then, the guaranteed upper estimations of probabiity of faiure corresponding to the vector of adverse factors () is as foows: p("1"/ X * )=0.105 0.001/(0.105 0.001+0.0 0.999)=0.005. 7. RELATED RESEARCH AND NEW RESULTS OF THE PAPER The research presented in this paper is aimed at the accurate assessment of the probabiity of faiure of hardware, such as avionics, on the basis of its known «history of abuse» by environmenta and operationa factors. The successfu soution of this probem aows us to forecast the probabiity of faiure during a forthcoming sortie thus providing a quantitative basis for mission panning and timey maintenance as we as preventing emergencies. This appication cannot be regarded as a conventiona reiabiity probem because cassica reiabiity does not view exposure to specific environmenta conditions and operationa factors as a main cause of faiures. The probem stated herein does not constitute a conventiona prognostic task, aso, because the faiure may not occur at a. The probem statement considered in the paper was first formuated in the paper [18]. Such a probem statement is prompted by the modern concept of maintenance known as the «service when needed». It is expected that the prognostic mode presented in this paper is deveoped on the basis of information downoaded from dedicated monitoring systems of fight-critica hardware and stored in a database. Therefore, the stated probem is reated to the area of tasks of Data Mining and KDD ([19], [0], [1], []). According to the existing topics of Data Mining prognostic mode design is a cassification probem [3]. Cassification probems as such are we known and are being investigated during at east four decades ([4], [5], [6], [7], [8]). Nevertheess, a number of principe tasks of cassification are sti of great interest and deserve further investigation. For exampe, a hot area of cassification is the so-caed probem of feature informativity and agorithms of their seection as we as methods and agorithms of earning for synthesis of cassification rue [9]. In addition, there exist a number of probems very important from the appications point of view that sti do not have efficient soutions. For exampe, deveopment of cassification modes based on Data Mining and KDD for the case when databases contain coumns measured on both continuous and discrete scaes. Let us summarize the new resuts presented in this paper that constitute its main contribution to the appied cassification probem soving and to the area of Data Mining and KDD. 1. The basis of the cassification mode proposed in the paper is formed by so-caed cassification predicates. They are associated with subspaces of factors of ow dimension, in particuar, with -d subspaces. Such predicates are defined over the entire factor space and according to its truth vaues («true» and «fase») divide it into two regions such that each region contains mosty reaizations of one of two custers of data. A cassification predicate is true within a region of the factor space bounded by a set of separation functions of two arguments, which are particuar components of the entire factor space. For a -d separation bound, two very efficient procedures resuting in optima separation functions of arbitrary shape (incuding non- convex case) and associated cassification predicates are investigated. The first procedure is based on the visuaization of custer projections onto arbitrary -d subspaces, and is impemented in an interactive software too deveoped by authors. This procedure provides a user an opportunity to draw any separation rue manuay utiizing its approximation by a poygon, i.e. an arbitrary inear spine. Moreover, the regions estabished by this procedure coud be many-connected and non-convex. A user is required to draw a separation bound whie the software too generates the associated cassification predicate automaticay. The second procedure resuting in an automated generation of cassification