Database Data Mining: Practical R Enterprise and Oracle Advanced Analytics

Transcription

1 Database Data Mining: Practical R Enterprise and Oracle Advanced Analytics Husnu Sensoy husnu.sensoy@globalmaksimum.com October 2, 2012

2 Content 1 Introduction 2 Oracle Enterprise R in Practice Data Visualization A Bit of Probability and Information Theory Optimization Text Analysis & Decision Trees 3 Wrap up

3 Who am I? Founder at Global Maksimum Data & Information Technologies in BI Domain Oracle Magazine DBA of the Year in 2009

4 A bunch of people who know what they are doing mainly focused on data and the transformation of data into information.

5 A bunch of people who know what they are doing mainly focused on data and the transformation of data into information. Complex Event Processing

6 A bunch of people who know what they are doing mainly focused on data and the transformation of data into information. Complex Event Processing 1.2 Million Event in a second on 2x2 Socket Nehalem Blades Data Mining

7 A bunch of people who know what they are doing mainly focused on data and the transformation of data into information. Complex Event Processing 1.2 Million Event in a second on 2x2 Socket Nehalem Blades Data Mining Churn Prediction Models for Telcos Marketing Target Selection Models Large Scale Database Management System Projects

8 A bunch of people who know what they are doing mainly focused on data and the transformation of data into information. Complex Event Processing 1.2 Million Event in a second on 2x2 Socket Nehalem Blades Data Mining Churn Prediction Models for Telcos Marketing Target Selection Models Large Scale Database Management System Projects 120+ TB Exadata migration from UNIX systems. Exadata Master Class all over the EMEA region for Exadata customers, Oracle partners, and Oracle staff at the region.

9 Advanced Analytics For the first version of BI we just filter rows, project columns, aggragate them using some functions, and give only what customer asks for.

10 Advanced Analytics For the first version of BI we just filter rows, project columns, aggragate them using some functions, and give only what customer asks for. After we have focused on machine generated data, or Big Data dealing the data as we did before becomes more and more fruitless.

11 Advanced Analytics For the first version of BI we just filter rows, project columns, aggragate them using some functions, and give only what customer asks for. After we have focused on machine generated data, or Big Data dealing the data as we did before becomes more and more fruitless. That s mainly because of the fact that there is only tiny amount of information available in this pile of data.

12 Advanced Analytics For the first version of BI we just filter rows, project columns, aggragate them using some functions, and give only what customer asks for. After we have focused on machine generated data, or Big Data dealing the data as we did before becomes more and more fruitless. That s mainly because of the fact that there is only tiny amount of information available in this pile of data. So it requires better tricks, automation, and post-analysis capabilities.

13 In-database Advanced Analytics 80% of data mining activity for enterprise means feature engineering.

14 In-database Advanced Analytics 80% of data mining activity for enterprise means feature engineering. Feature Engineering requires an iterative process of

15 In-database Advanced Analytics 80% of data mining activity for enterprise means feature engineering. Feature Engineering requires an iterative process of Filtering data (WHERE) Aggragating data (GROUP BY) Transforming data (CASE, DECODE, COALESCE, etc.)

16 In-database Advanced Analytics 80% of data mining activity for enterprise means feature engineering. Feature Engineering requires an iterative process of Filtering data (WHERE) Aggragating data (GROUP BY) Transforming data (CASE, DECODE, COALESCE, etc.) It is almost impossible to maintain an integrated mining environment (Scripts, files, metafiles,etc.) out of database

17 Oracle Advanced Analytics Toolkit SQL-2003 & Extensions Oracle Data Mining Oracle Spatial Extensions Flow based mining with SQL Developer Oracle Enterprise R

18 R is a free software environment for statistical computing and graphics. Majority of newbies (young data scientists) recently graduate or to be graduated from top universities use R. Batteries are included. Runs on all modern platforms.

19 Oracle R Enterprise Data you can process with standard R is limited with the amount of memory available on the server running R.

20 Oracle R Enterprise Data you can process with standard R is limited with the amount of memory available on the server running R. In order to bypass this problem people implement their own solutions in order to off-store data or utilize data sampling techniques.

21 Oracle R Enterprise Data you can process with standard R is limited with the amount of memory available on the server running R. In order to bypass this problem people implement their own solutions in order to off-store data or utilize data sampling techniques. ORE is an extension to standard R adding Oracle steroids into it.

22 Oracle R Enterprise Data you can process with standard R is limited with the amount of memory available on the server running R. In order to bypass this problem people implement their own solutions in order to off-store data or utilize data sampling techniques. ORE is an extension to standard R adding Oracle steroids into it. The basic idea is to off-load R commands seemless to Oracle Database or Oracle Big Data Appliance.

23 This session This session is not a R tutorial session but rather a fly over some possible solutions to real life scenarios using R. If you need some R tutorial please refer to Rob Kabacoff. R in Action. Manning, 2010 Oracle R Enterprise Training 2 - Introduction to R R Studio

24 Data Visualization Data Visualization Advance data analysis usually starts and ends with data visualization. Before modeling anything data scientists use graphs & charts to figure out behaviour of data After modeling in order to report the results they again refer to charts. R supports tens of different charting & graphing packages. Just to mention two of them lattice is used to generate conditioned graphs (a.k.a. trellis graphs) ggplot2 is used to make graph generation more consistent in R.

25 Data Visualization Histogram Do you see any significant pattern in distribution? Do you like the way histogram is represented? s o u r c e ( / r s n i p p l e t s /oow2012/mydata. r, l o c a l=true) d a t a s e t = generatecustomer ( ) h=h i s t ( d a t a s e t $ B i l l p e r P e r i o d, f r e q=true, y l a b= Number o f Customers, x l a b= B i l l Amount, main= B i l l Amount D i s t r i b u t i o n )

26 Data Visualization Remove the Outliers Do you see any significant pattern in distribution? s o u r c e ( / r s n i p p l e t s /oow2012/mydata. r, l o c a l=true) d a t a s e t = generatecustomer ( ) n o o u t l i e r = f u n c t i o n ( data, column, q =0.99, i n c=true){ q = q u a n t i l e ( data [, column ], na. rm=true, p r o b s = q u a n t i l e, names=false) i f ( i n c l u s i v e ){ pruned = s u b s e t ( data, data [, column ] <= q ) } e l s e { pruned = s u b s e t ( data, data [, column ] < q ) } } pruned pruned = n o o u t l i e r ( d a t a s e t, B i l l p e r P e r i o d, ) h=h i s t ( pruned $ B i l l p e r P e r i o d, f r e q=true, y l a b= Number o f Customers, x l a b= B i l l Amount, main= B i l l Amount D i s t r i b u t i o n )

27 Data Visualization Conditional Histograms s o u r c e ( / r s n i p p l e t s /oow2012/mydata. r, l o c a l=true) s o u r c e ( / r s n i p p l e t s /oow2012/commons. r, l o c a l=true) d a t a s e t = generatecustomer ( ) pruned = n o o u t l i e r ( d a t a s e t, B i l l p e r P e r i o d, ) l i b r a r y ( l a t t i c e ) h i s t o g r a m ( B i l l p e r P e r i o d U s i n g S e r v i c e X, data=pruned )

28 Data Visualization Too Many Columns to Visualize s o u r c e ( / r s n i p p l e t s /oow2012/mydata. r, l o c a l=true) s o u r c e ( / r s n i p p l e t s /oow2012/commons. r, l o c a l=true) d a t a s e t = generatecustomer ( ) head ( d a t a s e t ) pruned = n o o u t l i e r ( d a t a s e t, B i l l p e r P e r i o d, ) l i b r a r y ( l a t t i c e ) h i s t o g r a m ( B i l l p e r P e r i o d CarBrand, data=pruned )

29 A Bit of Probability and Information Theory Comparing Histograms We need a way to calculate similarity between those histograms. A strong tool from information theory Kullback Leibler Divergence allows us to define a distance metric between two distributions. equiwidth = function ( data, col, n=10, s f=1e 6){ q l i s t = q u a n t i l e ( data [, c o l ], na. rm=true, p r o b s = seq ( 0. 1, 1. 0, by =1./n ), names=false) } r e s u l t=c ( ) f o r ( q u a n t i l e i n q l i s t ){ r e s u l t = c ( r e s u l t, ( nrow ( s u b s e t ( data, data [, c o l ] <= q u a n t i l e ) ) / nrow ( data ) ) ) } r e s u l t [ 1 : n] c ( 0, r e s u l t [ 1 : ( n 1) ] ) + r e p ( s f, n )

30 A Bit of Probability and Information Theory KL Divergence & Symmetry D KL (P Q) = i Notice that D KL (P Q) D KL (Q P) P(i)ln P(i) Q(i) So we simply take the average of two to obtain a symmetric metric. k l d i s t a n c e = f u n c t i o n ( d i s t 1, d i s t 2 ){ k l 1 = 0. 0 f o r ( i i n 1 : l e n g t h ( d i s t 1 ) ){ k l 1 = k l 1 + d i s t 1 [ i ] l o g ( d i s t 1 [ i ] / d i s t 2 [ i ], 2 ) } k l 2 = 0. 0 f o r ( i i n 1 : l e n g t h ( d i s t 1 ) ){ k l 2 = k l 2 + d i s t 2 [ i ] l o g ( d i s t 2 [ i ] / d i s t 1 [ i ], 2 ) } ( k l 1+k l 2 ) /2 }

31 A Bit of Probability and Information Theory Top 5 Car Brands whose Owners Diverge from Baseline Brand KL Lancia Lincoln Proton Daewoo Pontiac ddf = NULL b a s e l i n e = e q u i w i d t h ( pruned, B i l l p e r P e r i o d ) f o r ( brand i n d a t a s e t [! d u p l i c a t e d ( d a t a s e t [, c ( CarBrand ) ] ), 1 ] ) { } b r a n d D i s t = e q u i w i d t h ( s u b s e t ( pruned, pruned [, CarBrand ] == brand ), B i l l p e r P e r i o d ) ddf = r b i n d ( ddf, data. frame ( carbrand=brand, k l=k l d i s t a n c e ( b a s e l i n e, b r a n d D i s t ) ) ) head ( ddf [ o r d e r ( ddf $ k l, d e c r e a s i n g=true), ] )

32 Optimization Problem Definition We a have a terrain covered by several stations and each point on the terrain has one of the following status GREEN Region is in the LoS of at least one station. YELLOW Region is in the LoS of at least on station but far away. RED Region is out of LoS. For a fixed number of stations we need to cover as much as we can.

33 Optimization Model Sketch Up 1 1 Define a function to calculate the ratio of green zones on terrain. 2 Give this function to one of optimization modules of R (Nelder Mead Technique) which can handle non-smooth target functions. 3 Get the optimal station distribution. t a r g e t f u n c=f u n c t i o n ( o b s e r v e r ){ m = matrix ( data=observer, ncol =2, byrow=true) # Compute merged s t a t u s o f a l l o b s e r v e r s m e r g e d s t a t u s < r e p ( r e d, l e n g t h ( t e r r $ h e i g h t ) ) f o r ( i i n seq ( 1 : dim (m) [ 1 ] ) ){ t e r r $ d i s t 2 o b s e r v e r = d i s t a n c e ( t e r r, c (m[ i, ], 7 ) ) s t a t u s = LoS ( t e r r, c (m[ i, ], 7 ), maxdist ) mergedstatus = updatestatus ( mergedstatus, s t a t u s ) } sum ( mergedstatus== green ) } optim < optim ( observers, targetfunc, c o n t r o l= l i s t ( f n s c a l e = 1, t r a c e =5, REPORT=1) ) 1 Refer to LoS Analysis (Part 4)

34 Optimization 1 Station (54%)

35 Optimization 3 Stations (83%)

36 Optimization 6 Stations (99%)

37 Text Analysis & Decision Trees Problem Definition For a given string which is written intentionally or erroneously wrong by subscribers, how can we build a model which can deduce the most probable string among 3 possibilities (or chose to not making any decision).

38 Text Analysis & Decision Trees Problem Definition For a given string which is written intentionally or erroneously wrong by subscribers, how can we build a model which can deduce the most probable string among 3 possibilities (or chose to not making any decision). Our legitamate strings are mom, dad, and brother. And we have brothe brother bro brother brother1 brother p? 1234? mom.i.came.home mom mmomyy mom dad[atwork] dad dod dad

39 Text Analysis & Decision Trees Model Sketch Up 1 Do some feature engineering

40 Text Analysis & Decision Trees Model Sketch Up 1 Do some feature engineering Length of the string

41 Text Analysis & Decision Trees Model Sketch Up 1 Do some feature engineering Length of the string Prefix flag (3 attributes for each)

42 Text Analysis & Decision Trees Model Sketch Up 1 Do some feature engineering Length of the string Prefix flag (3 attributes for each) Contains flag (3 attributes for each)

43 Text Analysis & Decision Trees Model Sketch Up 1 Do some feature engineering Length of the string Prefix flag (3 attributes for each) Contains flag (3 attributes for each) Anything else? 2 Build a classifier to classify those texts based on those features. 3 Evaluate your classifier

44 Text Analysis & Decision Trees First Model s o u r c e ( / r s n i p p l e t s /oow2012/mydata. r, l o c a l=true) df = generatetext ( ) l i b r a r y ( r p a r t ) # grow t r e e f i t < r p a r t ( c o r r e c t e d l e n g t h+p r e f i x B r o t h e r+p r e f i x D a d+prefixmom+i n s t r B r o t h e r+ instrdad+instrmom, method= c l a s s, data=d f ) t a b l e ( pred = p r e d i c t ( f i t, df, t y p e= c l a s s ), true = df $ corrected ) true pred? brother dad mom? brother dad mom

45 Text Analysis & Decision Trees More Feature Engineering using Jaro-Winkler Algorithm Jaro-Winkler distance is a distance metric between strings which can be used as a fuzzy string matching algorithm resilient to typo errors. l i b r a r y ( R e c o r d L i n k a g e ) enhanced = data. frame ( df, momscore = j a r o w i n k l e r ( mom, d f $ o r g i n a l ), dadscore = j a r o w i n k l e r ( dad, d f $ o r g i n a l ), b r o t h e r S c o r e = j a r o w i n k l e r ( b r o t h e r, d f $ o r g i n a l ) ) true pred? brother dad mom? brother dad mom

46 Conclusion R contains lots of libraries to help you model a physical phenomenon in anyway you like and visualize it.

47 Conclusion R contains lots of libraries to help you model a physical phenomenon in anyway you like and visualize it. Oracle Enterprise R makes it possible to handle large volumes of data without changing your R environment basics.

48 Conclusion R contains lots of libraries to help you model a physical phenomenon in anyway you like and visualize it. Oracle Enterprise R makes it possible to handle large volumes of data without changing your R environment basics. Don t take ODM and Oracle Enterprise R as alternatives of each other but rather complimentary solutions of the same problem.

49 Question & Answer

50 Stay in Touch