2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Size: px

Start display at page:

Maude Hines
5 years ago
Views:

1 B A

2 ER C M R S a C A R A R W C

3 eo l # ( k eo i a i a eog c b eog r ( T GW ) t k eo ns

4 2/ 39 : ws 2 e ej i zvl ) ) 2 r Ln d su R J. 3 T 9C ) () 2mL p dhs tlu J C a e S Rb I a WRI j i zvl p o L cmiu :CC ) DC 9E9 C 9 A

5 ( ) j nl Mcauv S b /. hiojgm 0 A w W ss Mdb uv /. hiojgm uv t /. hiojgm uvf W b uv b My q prf e f z W b /. A C A / A C1:82: 0 08:01: 0 8A AC /.,CA :01: 0 0 A :C 8 8A 2C 8A 8 : 0A 0 A 8 0 CA A /. A 82 A 10A C 0 8 : A 8:: 1 10A C 02 C0: CA /. A 82 A A 8 0 A 8

6 DWH DWH DB Administrator AWS Amazon S3 / AWS Glue / Amazon Redshift

10 Structured data Unstructured and Semi-structured data Web and mobile data Logs Social Media data Spreadsheets Streaming data IOT data

11 Data Volume Ge ne rat ed Data Availa ble for Ana lysis

12 Data Scientists Business Users Agile Real time Flexible Scale Analysts Applications

13 Business Intelligence Data Warehouse OLTP ERP CRM LOB

14 Business Intelligence Machine Learning DW Queries Big data processing Interactive Data Catalog Real-time R ( ) Data Warehouse OLTP ERP CRM LOB Data Lake Devices Web Sensors Social S

15 on AWS Redshift Athena Kinesis EMR Elasticsearch Service Amazon S3 AWS Glue D e U SG a n t Bhd s o s U c Snowball Snowmobile Kinesis Data Streams Kinesis Data Firehose m5 /

17 ( ) )

18 > > >

19 à AWS Glue

20 AWS Glue

21 AWS Glue Data Catalog Job Authoring Job Execution Apache Hive AWS PySpark Scala ETL

22 AWS Glue Data Catalog Amazon RDS Amazon Redshift Amazon S3 ( ) View 1 Amazon Athena Amazon Redshift Spectrum Apache Hive metastore Amazon EMR

23 on Amazon S3 with AWS Glue Your data Web app data AWS GLUE ETL Amazon RDS Other databases AMAZON QUICKSIGHT On premises data Streaming data

24 Glue Data Catalog Hive DDL Glue API

25 Glue Data Catalog Data Catalog 1.

26 Crawler Crawler Data Catalog Data Catalog classifier Grok classifier Amazon S3 Hive Crawler

27 Glue Data Catalog Nested fields

28 Crawler classifier Data Catalog classifier classifier ( ) Crawler classifier Crawler Crawler classifier Glue classifier classifier

29 Crawler classifier IAM Role Databases Built-In Classifiers Glue Crawler Amazon RDS MySQL MariaDB PostreSQL Oracle Microsoft SQL Server Amazon Aurora Data Warehouse Amazon Redshift JDBC Connection Object Connection Amazon Redshift Data Lakes Amazon S3 Avro Parquet ORC XML JSON & BSON Logs (A pache (Grok), Linu x(grok), M S(Grok), Ru by, Redis, and m any others) Delimited (com m a, pipe, tab, sem icolon) Compressions (ZIP, BZIP, G ZIP, LZ4, Snappy)

30 classifier Grok classifier Grok 1 Example: %{TIMESTAMP_ISO8601:timestamp} [%{MESSAGEPREFIX:message_prefix} ] %{CRAWLERLOGLEVEL:loglevel} : %{GREEDYDATA:message}

31 classifier 1. classifier 2. Crawler classifier

32 Crawlers: enumerate S3 objects identify file type and parse files semi-structured per-file schema semi-structured unified schema file 1 custom classifiers Grok based parser char int char int file 2 file N built-in classifiers JSON parser CSV parser Parquet parser bool int array int bool struct int char int array char struct array char int

33 Schema A root Schema B root name: str id: num addr name: str id: num addr: str street: str city: str zip: num Schema similarity heuristic +1 point +1 point sim > 0.7 sim à intersection min(a,b)

35 S3

37 Import/Export Import from an external metastore Export to an external metastore Apache Hive Metastore Apache Hive Metastore AWS GLUE ETL AWS GLUE DATA CATALOG AWS GLUE ETL. : -. - / / /./.....

38 AWS Glue

39 Amazon Athena (Amazon Athena )

41 Amazon Athena SQL Amazon S3 Query Instantly Pay per query Open Easy $ SQL Amazon S % ANSI SQL JDBC/ODBC join Amazon QuickSight (BI)

42 Amazon EMR 20 Apache Spark Apache Hive Presto AWS Glue Data Catalog Latest versions Low cost Use S3 storage Easy $ Data Lake EC % Amazon S3 EMRFS Apache Hadoop & Apache Spark

43 Amazon Redshift Spectrum S3 Amazon Redshift Spectrum query engine S3 Amazon Redshift SQL Redshift S3 Amazon Redshift data S3 data lake Parquet, ORC, Grok, Avro, CSV

44 à Amazon Redshift Spectrum

45 Amazon Redshift Data Warehousing 1/10 I ; Fast at any scale Open file formats Secure Inexpensive $ / SSD Amazon S3 1 1,000 USD O 1/10

46 Amazon Redshift Spectrum ( ) S3 Amazon Redshift Spectrum query engine S3 Amazon Redshift SQL Redshift S3 Amazon Redshift data S3 data lake Parquet, ORC, Grok, Avro, CSV

47 Amazon Redshift Spectrum SQL Clients/BI Tools Leader Node JDBC/ODBC SQL Leader Node Compute Node load / unload / backup / restore Amazon Redshift Spectrum Node Amazon S3 Redshift Spectrum Load Unload Backup Restore Compute Node Amazon S3 Compute Node Compute Node N Amazon Redshift Spectrum

48 Amazon Redshift Spectrum JDBC/ODBC 1 SELECT COUNT(*) FROM S3.EXT_TABLE GROUP BY 8 9 Amazon Redshift Amazon Redshift 2 3 Spectrum 4 Data Catalog (Dynamically prune partitions) 7 Amazon Redshift Spectrum N... 5 Amazon Redshift Spectrum 6 Amazon Redshift Spectrum S3 Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Metastore

49 (Schema on Read) Data Catalog Amazon Redshift CREATE external schema archived_trips from data catalog database 'sampledb' iam_role 'arn:aws:iam:: :role/myspectrumrole' region 'us-east-2 ; select * from svv_external_schemas select * from svv_external_tables

50 Amazon Redshift AWS Glue Data Catalog Amazon S3 AWS Identity and Access Management (IAM) Amazon Redshift ARN(Amazon Resource Name)

51 on Amazon S3 with AWS Glue ( ) Your data Web app data AWS GLUE ETL Amazon RDS Other databases AMAZON QUICKSIGHT On premises data Streaming data

52 Redshift Spectrum

53 - 1 / 5 Amazon Redshift Spectrum Redshift Spectrum Redshift Redshift Spectrum Redshift

54 - 2 / 5 Apache Parquet Apache Parquet SVL_S3QUERY_SUMMARY Parquet S3 s3_scanned_rows s3query_returned_rows 2 CSV Redshift Spectrum Redshift

55 - 3 / 5 Parquet SQL : SELECT query, segment, max(assigned_partitions) as total_partitions, max(qualified_partitions) as qualified_partitions FROM svl_s3partition WHERE query=<query-id> GROUP BY 1,2;

56 - 4 / 5 Amazon Redshift Spectrum S3 S3

57 - 5 / 5 Predicate pushdown Amazon Redshift Spectrum SQL ) GROUP BY LIKE COUNT/SUM/AVG/MIN/MAX/ Regex_replace

58 - 5 / 5 (Cont.) Predicate pushdown DISTINCT ORDER BY SQL Amazon Redshift Spectrum Amazon Redshift ) DISTINCT GROUP BY

59 Amazon Redshift Spectrum 10

61 2 1. à AWS Glue 2. à Amazon Redshift Spectrum

62 EMR Athena Kinesis Elasticsearch Redshift Service Amazon S3 AWS Glue papers/storage/data-lake-onaws.pdf Snowball Snowmobile Kinesis Data Streams Kinesis Data Firehose

63 AWS Glue AWS Glue AWS Glue Amazon Redshift Amazon Redshift

65 & S a. / / & :/ Q W A. -

66 /: / 88 F S AR

67 c S h W e a : /.-. /.- : A :

68 .. d r t c h e A jlm i g. /... - /. Sou na s:psw

69 /:86: 0 96:2 l p S n ) 10 upks i d 10 O re l a W S c n ) 10 8 t d t C 10 o O z n ) d ) m x ( n ) 92B : 8 A A / :

70 -:8 D C DK FDBF 8 EBF y S ) p -E O F 5 K F ) z -E O F 1D KB 2BD 8N K E -E O F 128 ) p -:8R u g eiac o ) z 9 K - MB ) ) p vn ml kots S-:8R u 0B K 7 M N ) z -:8 8 MB / K D A ) p -E O F 1D KB / FK BF 8 MB 3 F K -E O F 138 -:8 2 A K ) ( z -:8 8N K E 4 F A ) p 8 a a t fawb d er A / :

71 Q A )1 - Q A - ) Q Q Q Q 1) & - 1

Visualizing Big Data on Maps: Emerging Tools and Techniques. Ilir Bejleri, Sanjay Ranka

Visualizing Big Data on Maps: Emerging Tools and Techniques Ilir Bejleri, Sanjay Ranka Topics Web GIS Visualization Big Data GIS Performance Maps in Data Visualization Platforms Next: Web GIS Visualization