PLANNING. ANALYSIS. REPORTING. Real-Time Quantification Filters for Multidimensional Databases Peter Strohm, Jedox AG
Jedox: In-Memory OLAP Database 00 Founded in Freiburg, Germany Today - 00+ Employees - Offices in Freiburg, Frankfurt, Düsseldorf, Paris - 00+ Business partners globally Jedox Suite Version 5. Business Intelligence, Analytics & Performance Management Excel-, Web-, Mobile-Client GPU Acceleration Jedox for Excel GPU Accelerator ERP, CRM, SCM Jedox Web Jedox OLAP Server Jedox ETL RDB, DWH Jedox Mobile Jedox SAP Connector SAP/R3 3 rd Party Tools ODBO XMLA SAP BI/BW
Jan Feb Mar Q Apr May Jun Q Jul Aug Sep Q3 Oct Nov Dec Q4 Year What is an OLAP-Database? 3 Multidimensional Cube Hierarchical Structure Consolidated Elements All regions Europe France Italy UK North America USA Canada Mexico 7 4 4 55 9 0 0 59 0 8 5 53 7 0 8 75 7 5 6 8 0 0 3 4 5 3 3 0 4 75 6 0 5 0 9 0 9 3 3 8 0 0 0 0 8 0 3 4 6 0 8 0 0 7 7 0 0 0 0 9 0 3 4 0 0 4 0 6 7 3 0 4 8 0 9 8 37 9 9 0 38 7 4 0 4 0 0 4 00 0 0 8 6 3 7 0 0 0 0 0 0 30 9 3 4 6 0 0 7 7 5 0 6 4 0 0 4 33 5 4 0 3 0 4 0 3 0 0 0 0 37 Deviation Actual Budget 4 Elements as dimension path, value pairs Year Dimension Path Val Jan Europe Actual 4 Q Q Q3 Q4 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Jan Feb Mar Q Apr May Jun Q Jul Aug Sep Q3 Oct Nov Dec Q4 Year In-Memory OLAP-Database All data in main memory All regions Europe France Italy UK North America USA Canada Mexico 7 4 4 55 9 0 0 59 0 8 5 53 7 0 8 75 7 5 6 8 0 0 3 4 5 3 3 0 4 75 6 0 5 0 9 0 9 3 3 8 0 0 0 0 8 0 3 4 6 0 8 0 0 7 7 0 0 0 0 9 0 3 4 0 0 4 0 6 7 3 0 4 8 0 9 8 37 9 9 0 38 7 4 0 4 0 0 4 00 0 0 8 6 3 7 0 0 0 0 0 0 30 9 3 4 6 0 0 7 7 5 0 6 4 0 0 4 33 5 4 0 3 0 4 0 3 0 0 0 0 37 Deviation Actual Budget
Jan Feb Mar Q Apr May Jun Q Jul Aug Sep Q3 Oct Nov Dec Q4 Year In-Memory OLAP-Database All data in main memory Store only base elements All regions Europe France Italy UK North America USA Canada Mexico 6 0 5 0 9 0 3 3 0 0 0 0 3 6 0 0 0 7 0 0 0 0 4 0 0 0 6 3 0 0 0 8 6 3 0 0 0 0 0 9 3 4 0 0 7 5 0 4 0 0 5 4 3 0 0 0 0 0 Deviation Actual Budget
Jan Feb Mar Q Apr May Jun Q Jul Aug Sep Q3 Oct Nov Dec Q4 Year In-GPU-Memory In-Memory OLAP-Database 3 GPU All data in main memory Store only base elements Store only non-zero values All regions Europe France Italy UK North America USA Canada Mexico 9 6 5 9 3 3 5 7 3 6 0 7 8 6 8 4 6 3 4 0 8 6 3 5 9 3 4 7 5 4 3 5 4 3 Deviation Actual Budget 4! Calculate consolidated elements on the fly Save Memory, be up to date Year Q Q Q3 Q4 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Jan Feb Mar Q Apr May Jun Q Jul Aug Sep Q3 Oct Nov Dec Q4 Year What is a Quantification Filter? ANY and ALL Quantifier on one dimension Conditional Filter All regions Europe France Italy UK North America USA Canada Mexico 7 4 4 55 9 0 0 59 0 8 5 53 7 0 8 75 7 5 6 8 0 0 3 4 5 3 3 0 4 75 6 0 5 0 9 0 9 3 3 8 0 0 0 0 8 0 3 4 6 0 8 0 0 7 7 0 0 0 0 9 0 3 4 0 0 4 0 6 7 3 0 4 8 0 9 8 37 9 9 0 38 7 4 0 4 0 0 4 00 0 0 8 6 3 7 0 0 0 0 0 0 30 9 3 4 6 0 0 7 7 5 0 6 4 0 0 4 33 5 4 0 3 0 4 0 3 0 0 0 0 37 Deviation Actual Budget Ex Time period with any element > 0
Jan Feb Mar Q Apr May Jun Q Jul Aug Sep Q3 Oct Nov Dec Q4 Year What is a Quantification Filter? ANY and ALL Quantifier Conditional Filter All regions Europe France Italy UK North America USA Canada Mexico 7 4 4 55 9 0 0 59 0 8 5 53 7 0 8 75 7 5 6 8 0 0 3 4 5 3 3 0 4 75 6 0 5 0 9 0 9 3 3 8 0 0 0 0 8 0 3 4 6 0 8 0 0 7 7 0 0 0 0 9 0 3 4 0 0 4 0 6 7 3 0 4 8 0 9 8 37 9 9 0 38 7 4 0 4 0 0 4 00 0 0 8 6 3 7 0 0 0 0 0 0 30 9 3 4 6 0 0 7 7 5 0 6 4 0 0 4 33 5 4 0 3 0 4 0 3 0 0 0 0 37 Deviation Actual Budget Ex Time period with any element > 0 Ex Region with all elements < 0
Jan Feb Mar Q Apr May Jun Q Jul Aug Sep Q3 Oct Nov Dec Q4 Year What is a Quantification Filter? ANY and ALL Quantifier Conditional Filter All regions Europe France Italy UK North America USA Canada Mexico 7 4 4 55 9 0 0 59 0 8 5 53 7 0 8 75 7 5 6 8 0 0 3 4 5 3 3 0 4 75 6 0 5 0 9 0 9 3 3 8 0 0 0 0 8 0 3 4 6 0 8 0 0 7 7 0 0 0 0 9 0 3 4 0 0 4 0 6 7 3 0 4 8 0 9 8 37 9 9 0 38 7 4 0 4 0 0 4 00 0 0 8 6 3 7 0 0 0 0 0 0 30 9 3 4 6 0 0 7 7 5 0 6 4 0 0 4 33 5 4 0 3 0 4 0 3 0 0 0 0 37 Deviation Actual Budget Ex Time period with any element > 0 Ex Region with all elements < 0
Quantification Filter: Challenges Source Cells Preprocessing Aggregated Cells Any/All Hashtable Result Cells Pre-processing, e.g. Aggregation, Rules, etc. Condition for one dimension, e.g. value > 5
Is AnyProcessor FALSE TRUE Quantification Filter: Challenges TRUE Is Zero In Result? FALSE Any && 0 included Any && 0 excluded satisfied satisfied Not satisfied - Counter!= slicecellcount flag > 0 All && 0 included All && 0 excluded Not satisfied Not Satisfied - satisfied flag == 0 Counter == slicecellcount
Multi-GPU Quantification Filter: Algorithm Any/All Processor Preprocessed Cells condition? Check cell No Yes Put cell into hash table discard Count Result Cells Insert zeros Check Postprocessing
Quantification: Algortihm 3 Using hash table Skipping already checked elements Avoiding atomics and locks
Wikipedia Page Stats Example http://blog.jedox.com/03//7/big-data-analytics-jedox-example-wikipedia-part-/ 3 Starting point: Big Data (743GB reduced to GB) Getting the data into the cube Getting amazing speed-up with GPU See also: www.saphana.com, www.wikipedia.org, blog.gbrueckl.at
Wikipedia Page Stats cube Pages (, Million) 7 4 4 55 9 0 0 59 0 8 5 53 7 0 8 75 7 5 6 8 0 0 3 4 5 3 3 0 4 75 6 0 5 0 9 0 9 3 3 8 0 0 0 0 8 Languages (~360) 0 3 4 6 0 8 0 0 7 7 0 0 0 0 9 0 3 4 0 0 4 0 6 7 3 0 4 8 0 9 8 37 9 9 0 38 7 4 0 4 0 0 4 00 0 0 8 6 3 7 0 0 0 0 0 0 30 3 Hours (4), Date (~360) 9 3 4 6 0 0 7 7 5 0 6 4 0 0 4 33 5 4 0 3 0 4 0 3 0 0 0 0 37 4 Projects (6) 5 Measures (~4) 5 Cube has ~,48 Trillion possible cells (about 76 Million filled)
Wikipedia Example: Superbowl Super_Bowl 946.783,00 Frank_Ocean 99.53,00 Tunguska_event 909.63,00 Gangnam_Style 897.48,00 Martin_Luther_King,_Jr. 893.704,00 Baltimore_Ravens 809.333,00 Joe_Flacco 777.894,00 Mardi_Gras 768.077,00 Mumford_%6_Sons 757.956,00 List_of_Super_Bowl_champions 747.68,00 03_in_UFC 74.66,00 George_Washington 70.503,00 Michael_Oher 69.753,00 Chinese_zodiac 633.067,00 Mohandas_Karamchand_Gandhi 6.80,00 Roman_numerals 55.79,00 List_of_Downton_Abbey_episodes 54.99,00 Beasts_of_the_Southern_Wild 54.338,00 Alabama_Shakes 540.304,00 San_Francisco_49ers 538.84,00,000,0000 0,8000 0,6000 0,4000 0,000 0,0000 Top 50 Superbowl - Peak in - February Peak in February 03 - Superbowl 03 3 4 5 6 7 8 9 0
Wikipedia: Peak + QFilter-ALL 3 Top 50 Peaks in 0/3 ALL QFilter < 0.7 Correlations,000,0000 0,8000 0,6000 0,4000 0,000 0,0000 Top Elements Superbowl Peak in February && ALL other months < 0.7 3 4 5 6 7 8 9 0 Baltimore_Ravens Joe_Flacco Michael_Oher Roman_numerals San_Francisco_49ers Flag_of_the_United_States Super_Bowl
Wikipedia: P+QfALL Performance Top 50 Peaks in 0/3 QFilter ALL < 0.7 on Pages 80.000ms 73.745ms ALL QFilter < 0.7 70.000ms 60.000ms GPU(xK40) CPU (Xeon E5-643) 50.000ms 46.700ms 3 Correlations 40.000ms 30.000ms 83x 63x 0.000ms 0.000ms 0ms 558ms en (63.303.959cells).6ms Natural languages (0.965.76 cells)
Wikipedia: What s new in June? ALL elements > 4 June compared to Jan-May.000.000,00 900.000,00 3 ALL QFilter > 0.5 June compared to Jul-Dec No Peak but steady interest 800.000,00 700.000,00 600.000,00 500.000,00 400.000,00 300.000,00 Lycos List_of_PlayStation_4_games Aishwarya_Rai_Bachchan Nothing_Was_the_Same Pitbull_(Rapper) Billy_Ray_Cyrus 00.000,00 00.000,00 0,00 3 4 5 6 7 8 9 0
Wikipedia: WNiJ Performance ALL elements > 4 June compared to Jan-May ALL QFilter > 0.5 June compared to Jul-Dec 0.000ms 00.000ms 80.000ms GPU(xK40) CPU (Xeon E5-643) 73.745ms 03.9ms 3 No Peak but steady interest 60.000ms 40.000ms 46.700ms 83x 63x 70x 0.000ms 0ms 558ms.6ms.466ms en (63.303.959cells) Natural languages (0.965.76 cells) What's new (7.484.560 cells)
Wikipedia: What s new (event)? Aggregation + DFilter.000.000 On a daily base 00.000 Francisco_(papa) Jorge_Bergoglio 3 Even more data 0.000.000 Jorge_Mario_Bergoglio Pope_Francis Franziskus_(Papst) Papa_Francesco 00
Future works Multi-Node-GPU performance Fast massive & continuous insertion 3 New OLAP Features
Visit us in the exhibit hall! Visit at booth 030! Thanks to: Download at www.jedox.com Tweet to @JedoxAG Mail to peter.strohm@jedox.com Alex Haberstroh, Jedox AG Tobias Lauer, Jedox AG Steffen Wittmer, Jedox AG http://blog.jedox.com/03//7/big-data-analytics-jedox-example-wikipedia-part-/