Mine Your Own Business

Transcription

1 33 Mine Your Own Business Using process mining to turn big data into better processes and systems prof.dr.ir. Wil van der Aalst PAGE 0

2 Season 1, Episode 4 (1969) PAGE 1

3 "Mine Your Own Business" (2006) the world's first anti-environmentalist documentary PAGE 2

4 Mine your own business: Turning big data into real value

5 process mining as the missing link aligning model and reality divide and conquer process discovery Big (Event) Data challenges getting started PAGE 4

6 process mining as the missing link aligning model and reality divide and conquer process discovery Big (Event) Data challenges getting started PAGE 5

7 process model analysis (simulation, verification, optimization, gaming, etc.) performanceoriented questions, problems and solutions process mining complianceoriented questions, problems and solutions data-oriented analysis (data mining, machine learning, business intelligence) PAGE 6

8 PAGE 7

9 PAGE 8

10 PAGE 9

11 let's play PAGE 10

12 Play-Out process model event log PAGE 11

13 Play-Out (Classical use of models) B A p1 E p3 D start end p2 C p4 A B C D A E D A C B D A B C D A E D A C B D A E D A C B D PAGE 12

14 Play-In event log process model PAGE 13

15 Play-In A B C D A C B D A E D A E D A B C D A C B D A E D A C B D B A p1 E p3 D start end p2 C p4 PAGE 14

16 Example Process Discovery (Vestia, Dutch housing agency, 208 cases, 5987 events) PAGE 15

17 Example Process Discovery (ASML, test process lithography systems, events) PAGE 16

18 Example Process Discovery (AMC, 627 gynecological oncology patients, events) PAGE 17

19 Replay event log process model extended model showing times, frequencies, etc. diagnostics predictions recommendations PAGE 18

20 Replay A B C D B A p1 E p3 D start end p2 C p4 PAGE 19

21 Replay A E D B A p1 E p3 D start end p2 C p4 PAGE 20

22 Replay can detect problems A C D Problem! token left behind B Problem! missing token A p1 E p3 D start end p2 C p4 PAGE 21

23 Conformance Checking (WOZ objections Dutch municipality, 745 objections, 9583 event, f= 0.988) PAGE 22

24 Replay can extract timing information A 5 B 8 C 9 D B A p1 E p3 D start 5 4 p C 9 p end PAGE 23

25 Performance Analysis Using Replay (WOZ objections Dutch municipality, 745 objections, 9583 event, f= 0.988) PAGE 24

26 Models are like the glasses required to see and understand event data! PAGE 25

27 process mining as the missing link aligning model and reality divide and conquer process discovery Big (Event) Data challenges getting started PAGE 26

28 Language identification in the limit (Mark Gold 1967) abc? ab(c d)? (ad) (ab(c d))? ab*(c d)? abc abd ad abbc ac A language is learnable in the limit if there exists a perfect child that generates only finitely many hypotheses. Language identification in the limit by E Mark Gold, Information and Control, 10(5): , PAGE 27

29 Learning is not easy Even simple languages like regular languages are not learnable in the limit. Many settings: evil or wellbehaving mothers, with or without negative examples, frequencies, etc. sentence trace in event log language process model PAGE 28

30 Process discovery algorithms (small selection) automata-based learning heuristic mining distributed genetic mining language-based regions genetic mining stochastic task graphs state-based regions LTL mining neural networks fuzzy mining mining block structures hidden Markov models α algorithm α# algorithm conformal process graph multi-phase mining partial-order based mining α++ algorithm ILP mining PAGE 29

31 Quiz Question: How to remove behavior? B A p1 E p3 D start end p2 C p4 PAGE 30

32 Quiz Question: How to remove behavior? B A p1 E p3 D start end p2 C p4 Add places or remove transitions! PAGE 31

33 Quiz Question: How to add behavior? B A p1 E p3 D start end p2 C p4 PAGE 32

34 Quiz Question: How to add behavior? B A p1 E p3 D start end p2 C p4 F Add transition or remove places! PAGE 33

35 Places limit behavior B A E D C abcd ad abed abccd acbd aebcd aed aad caed aded PAGE 34

36 Places limit behavior B A E D start C abcd ad abed abccd acbd aeccd aed aad caed aded PAGE 35

37 Places limit behavior B A E D start end C abcd ad abed abccd acbd aebcd aed aad caed aded PAGE 36

38 Places limit behavior B A p1 E D start end C abcd ad abed abccd acbd aebcd aed aad caed aded PAGE 37

39 Places limit behavior B A p1 E p3 D start end p2 C p4 abcd ad abed abccd acbd aebcd aed aad caed aded PAGE 38

40 Example: Process Discovery Using Language-Based Regions f c1 A place is feasible if it can be added without disabling any of the traces in the event log. a1 b1 e c d a2 p R b2 X Y PAGE 39

41 Genetic process mining: Overview create initial population event log mutation termination compute fitness tournament next generation elitism children crossover select best individual parents dead individuals PAGE 40

42 Example: crossover PAGE 41 a start register request b examine thoroughly c examine casually d check ticket decide pay compensation reject request reinitiate request e g h f end a start register request b examine thoroughly c examine casually d check ticket decide pay compensation reject request reinitiate request e g h f end a start register request b examine thoroughly c examine casually d check ticket decide reinitiate request e f a start register request b examine thoroughly c examine casually d check ticket decide pay compensation reject request reinitiate request e g h f end pay compensation reject request g h end

43 Example: mutation remove place b b examine thoroughly g examine thoroughly g start a register request c examine casually d check ticket decide f e reinitiate request pay compensation h reject request end start a register request added arc c examine casually d check ticket e decide f reinitiate request pay compensation h reject request end PAGE 42

44 models are like maps, their usefulness is determined by the intended use, i.e., there is not a single "perfect map" PAGE 43

45 process mining as the missing link aligning model and reality divide and conquer process discovery Big (Event) Data challenges getting started PAGE 44

46 Conformance checking an activity that should not happen happened an activity was executed by the wrong person an activity was executed too late an activity that should happen did not happen two activities were swapped PAGE 45

47 Alignments are essential! conformance checking to diagnose deviations squeezing reality into the model to do model-based analysis PAGE 46

48 process model event log synchronous move move on model only move on log only PAGE 47

49 Example: BPI Challenge 2012 (Dutch financial institute, doi: /uuid:3926db30-f aebc e91f) Loops of W_Completeren aanvraag and W_Nabellen offertes are often performed O_DECLINED and W_Wijzigen contractgegevens are often skipped Many moves on log of O_CANCELLED, O_CREATED, O_SELECTED, O_SENT occurred with the same frequency value (i.e. 60) before parallel branch Work of Arya Adriansyah (Replay project) Many moves on log of W_Afhandelen leads ( > 2200 times) occurred in the end of traces PAGE 48

50 Synchronous moves of Completeren aanvraag Loops of W_Completeren aanvraag and W_Nabellen offertes are often performed O_DECLINED and W_Wijzigen contractgegevens are often skipped Move on log of Completeren aanvraag Many moves on log of O_CANCELLED, O_CREATED, O_SELECTED, O_SENT occurred with the same frequency value (i.e. 60) before parallel branch The average waiting time for the input place of W_Nabellen offertes+start is very long (2.83 days) compares to the average waiting time of other places Move on log of O_CANCELLED and A_CANCELLED Moves on model towards end of traces Many moves on log of W_Afhandelen leads ( > 2200 times) occurred in the end of traces O_ACCEPTED has average sojourn time of minutes, while A_REGISTERED, A_ACTIVATED, and A_APPROVED have average sojourn time of minutes Activity W_Wijzigen contractgegevens is the bottleneck, but it occured rarely (only 4 times) PAGE 49

51 PAGE 50

52 process mining as the missing link aligning model and reality divide and conquer process discovery Big (Event) Data challenges getting started PAGE 51

53 In 10 years we will have 50 times as much data! PAGE 52

54 PAGE 53

55 PAGE 54

56 PAGE 55

57 PAGE 56

58 Big Data? PAGE 57

59 Big or fast and efficient? PAGE 58

60 process mining as the missing link aligning model and reality divide and conquer process discovery Big (Event) Data challenges getting started PAGE 59

61 Big Data: Opportunities and Challenges PAGE 60

62 Divide and Conquer A D 1: AC 2: BC 3: BC 4: AC 5: AC 6: AC 7: BC 8: BC B C 1: CDEG 2: CFG 3: CFG 4: CEDG 5: CFG 6: CEDG 7: CEDG 8: CDEG C F E G split horizontally split vertically 1: ACDEG 2: BCFG 3: BCFG 4: ACEDG 5: ACFG 6: ACEDG 7: BCEDG 8: BCDEG start A B C D F E G complete merge horizontally merge vertically D 1: ACDEG 4: ACEDG 6: ACEDG A C E G D 7: BCEDG 8: BCDEG B C E G 5: ACFG A C F G 2: BCFG 3: BCFG B C F G PAGE 61

63 Vertical Decomposition A D split vertically 1: ACDEG 2: BCFG 3: BCFG 4: ACEDG 5: ACFG 6: ACEDG 7: BCEDG 8: BCDEG start B C F E G complete merge vertically D 1: ACDEG 4: ACEDG 6: ACEDG A C E G D 7: BCEDG 8: BCDEG B C E G 5: ACFG A C F G 2: BCFG 3: BCFG B C F G PAGE 62

64 Horizontal Decomposition A D 1: AC 2: BC 3: BC 4: AC 5: AC 6: AC 7: BC 8: BC B C 1: CDEG 2: CFG 3: CFG 4: CEDG 5: CFG 6: CEDG 7: CEDG 8: CDEG C F E G split horizontally 1: ACDEG 2: BCFG 3: BCFG 4: ACEDG 5: ACFG 6: ACEDG 7: BCEDG 8: BCDEG start A B C D F E G complete merge horizontally PAGE 63

65 Decomposing Conformance Checking e.g., maximal decomposition, passage-based decomposition, or SESE/RPST-based decomposition e.g., A* based alignments, token-based replay, or simple replay until first deviation decomposition technique yields a (valid) activity partitioning conformance checking technique SN process model decompose model M 1 submodel M 2 submodel M n submodel conformance check conformance check conformance check L event log decompose event log L 1 sublog L 2 sublog L n sublog conformance diagnostics PAGE 64

66 Example of a valid decomposition f t7 c7 t8 c8 t11 start a t1 c1 c2 t2 b t3 c t4 c3 c4 d t5 e c5 c6 g t9 h t10 c9 end Log can be split in the same way! t6 SN 1 start a t1 SN 2 a t1 c1 t2 b t3 c3 d t5 e t6 SN 5 t7 d t5 e c5 c7 c6 f t8 g t9 h t10 f t8 g t9 h t10 c8 c9 t11 end SN 6 a t6 t1 c2 c SN 4 d t5 SN 3 t4 e t6 c t4 c4 PAGE 65

67 Example of alignment for observed trace a,b,c,d,e,c,d,g,f a,b,c,d,e,c,d,g,f t2 c1 b a t3 start t1 c2 c t4 c3 c4 t7 d t5 e t6 c5 c7 c6 f t8 g t9 h t10 c8 c9 t11 end SN 1 start a t1 SN 3 a t1 c2 SN 2 a t1 c t4 c1 e t6 t2 b t3 c3 SN 4 c t4 d t5 e t6 c4 SN 5 t7 d t5 e t6 d t5 c5 c7 c6 f t8 g t9 h t10 f t8 g t9 h t10 c8 c9 t11 end SN 6 Etc. PAGE 66

68 Conformance checking can be decomposed!!! General result for any valid decomposition: Any event log or trace is perfectly fitting the overall model if and only if it is also fitting all the individual fragments start a t1 c1 c2 t2 b t3 c t4 c3 c4 t7 d t5 e t6 c5 c7 c6 f t8 g t9 h t10 c8 c9 t11 end SN 1 start a t1 SN 3 a t1 c2 SN 2 a t1 c t4 c1 e t6 t2 b t3 c3 SN 4 c t4 d t5 e t6 c4 SN 5 t7 d t5 e t6 d t5 c5 c7 c6 f t8 g t9 h t10 f t8 g t9 h t10 c8 c9 t11 end SN 6 Wil van der Aalst, Decomposing Petri nets for process mining: A generic approach. Distributed and Parallel Databases, Volume 31, Issue 4, pp , 2013 PAGE 67

69 Decomposing Process Discovery e.g., causal graph based on frequencies is decomposed using passages or SESE/RPST e.g., language/state-based region discovery, variants of alpha algorithm, genetic process mining decomposition technique yields a (valid) activity partitioning process discovery technique L event log decompose event log L 1 sublog L 2 sublog L n sublog discovery discovery discovery M process model compose model M 1 submodel M 2 submodel M n submodel PAGE 68

70 Learn more about decomposing process mining problems? W.M.P. van der Aalst. Decomposing Petri Nets for Process Mining: A Generic Approach. Distributed and Parallel Databases, 31(4): , W.M.P. van der Aalst. A General Divide and Conquer Approach for Process Mining. In M. Ganzha, L. Maciaszek, and M. Paprzycki, editors, Federated Conference on Computer Science and Information Systems (FedCSIS 2013), pages IEEE Computer Society, W.M.P. van der Aalst. Decomposing Process Mining Problems Using Passages. In S. Haddad and L. Pomello, editors, Applications and Theory of Petri Nets 2012, volume 7347 of Lecture Notes in Computer Science, pages Springer-Verlag, Berlin, J. Munoz-Gama, J. Carmona, and W.M.P. van der Aalst. Hierarchical Conformance Checking of Process Models Based on Event Logs. In J.M. Colom and J. Desel, editors, Applications and Theory of Petri Nets 2013, volume 7927 of Lecture Notes in Computer Science, pages Springer-Verlag, Berlin, J. Munoz-Gama, J. Carmona, and W.M.P. van der Aalst. Conformance Checking in the Large: Partitioning and Topology. In F. Daniel, J. Wang, and B. Weber, editors, International Conference on Business Process Management (BPM 2013), volume 8094 of Lecture Notes in Computer Science, pages Springer-Verlag, Berlin, E. Verbeek and W.M.P. van der Aalst. Decomposing Replay Problems: A Case Study. In D. Moldt and H. Roelke, editors, Proceedings of the International Workshop on Petri Nets in Software Engineering (PNSE 2013), volume 989 of CEUR Workshop Proceedings, pages CEUR- WS.org, PAGE 69

71 process mining as the missing link aligning model and reality divide and conquer process discovery Big (Event) Data challenges getting started PAGE 70

72 Distributing process mining problems to cope with big data PAGE 71

73 streaming event data (sensors, RFID, messages, etc.) PAGE 72

74 process discovery: finding sheep with five or more legs 1 formal (not just a picture) fast (should not take years) 2 ability to balance all conformance dimensions (fitness, precision, generalization, and simplicity) incl. noise provide guarantees (not just a best effort) sound (result should at least be free of deadlocks, etc.) PAGE 73

75 On-the-fly process mining Operational support PAGE 74

76 Concept drift Concept drift PAGE 75

77 Cross-organizational mining cross-organizational / comparative process mining PAGE 76

78 Supporting the process of process mining PAGE 78

79 process mining as the missing link aligning model and reality divide and conquer process discovery Big (Event) Data challenges getting started PAGE 79

80 How to get started? PAGE 80

81 600+ plug-ins available covering the whole process mining spectrum open-source (L-GPL) Download from: 81

82 Commercial Alternatives Disco (Fluxicon) Perceptive Process Mining (before Futura Reflect and BPM one) ARIS Process Performance Manager QPR ProcessAnalyzer Interstage Process Discovery (Fujitsu) Discovery Analyst (StereoLOGIC) XMAnalyzer (XMPro) 82

83 How to Get Started? Collect event data Collect questions Minimal requirement: events referring to an activity name and a process instance. Good to have: timestamps, resource information, additional data elements. Challenges: scoping and sometimes correlation. What kind problems would you like to address (cost, time, risk, compliance, service, etc.)? Related to discovery, conformance, enhancement? Iterative process: can be curiosity driven initially. 83

84 Join our expedition: Mine your processes! performance-oriented questions, problems and solutions process model analysis (simulation, verification, etc.) process mining data-oriented analysis (data mining, machine learning, business intelligence) compliance-oriented questions, problems and solutions PAGE 84

85 Learn more? Informal PM Meeting (15.50 today). Thanks to Krzysztof Kluza! Building C2 (entrance through buildings C1 or C3), room no. 316 (3rd floor). PAGE 85