Deep dive into analytics using Aggregation. Boaz

Transcription

1 Deep dive into analytics using Aggregation Boaz

2 Elasticsearch an end-to-end search and analytics platform.

3 full text search highlighted search snippets search-as-you-type did-you-mean suggestions Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited.

4 combines full text search with geolocation uses more-like-this to find related questions and answers Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited.

5 search repositories, users, issues, pull requests search 130 billion lines of code track all alerts, events, logs Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited.

6 index and analyse 5TB of log data every day Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited.

7 combine visitor logs with social network data real-time feedback to editors Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited.

8 Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited.

9 Feature summary Fully-featured search Relevance-ranked text search Scalable search High-performance geo, temporal, numeric range and key lookup Highlighting Support for complex document types (nested structures) * Spelling suggestions Powerful query DSL * Standing queries * Real-time results * Extensible via plugins *! Powerful faceting/analysis Summarise large sets by any combinations of time, geo, category and more. * Kibana visualisation tool *! * Features we see as differentiators Management Simple and robust deployments * REST APIs for handling all aspects of administration/monitoring * Marvel console for monitoring and administering clusters * Special features to manage the life cycle of content * Integration Hadoop (MapRed,Hive, Pig, Cascading..)* Client libraries (Python, Java, Ruby, javascript ) Data connectors (Twitter, JMS ) Logstash ETL framework * Support Development and Production support with tiered levels Support staff are the core developers of the product *

10 Let s talk data data === json

11 { "created_at": "Mon Apr 28 12:31: ", "id": , "text": "Prepping up for my talk tomorrow at #noslq14 Cologne, where I ll spend the coming two days. Drop by for everything #elasticsearch related.", "user": { "id": , "name": "Boaz Leskes", "screen_name": "bleskes", "location": "Amsterdam", "description": "Coder at Elasticsearch", "time_zone": "Amsterdam",, "geo": null, "retweet_count": 1, "entities": { "hashtags": [ { "text": noslq14, { "text": elasticsearch" ], "symbols": [], "urls": [], "user_mentions": [], "favorited": false, "retweeted": false, "lang": "en"

12 { "dt": " T02:01:48.026Z", "url": " "querystring": "", "host": " "path": "/film/2014/mar/03/oscars-2014-winners-list", "section": "film", "platform": "r2", "useragent": { "type": "Browser", "family": "Safari 5.1.9", "os": "OS X ", "device": "Personal computer", "documentreferrer": " "browser": { "id": "ga6ruflhwnqvwdt0rw4r78fg", "isnew": false, "referringhost": "theguardian.com", "referringpath": "/world", "iscontent": true, "contentpublicationdate": " ", "countrycode": "US", "countryname": "United States", "location": { "lonlat": [ , ]

13 Data can be anything Questions Code Logs Credit card transactions Click logs

14 aggregations == 50km view == patterns

15 aggregations == 50km view == patterns insights

16 a simple UI element

17 or more complex

18 .. even more complex?

19 Underpinnings back to search

20 Powerful search engine GET tweets/tweet/_search { "query": { "filtered": { "query": { "match": { "text": "jumping", "filter": { "range": { "created_at": { "from": " T05:16:29+00:00", "to": "now"

21 Inverted index nosql 128 New York lat=6.9 lon=50 F

22 Our goal results

23 Counting results Cameras 6 8 Optics Accessories 153 Lenses

24 Counting results Cameras Optics Accessories Lenses 1 2 0

25 Field data nosql New York 4 lat=6.9 lon= F

26 Introducing aggregations analysis lego

27 Before there were facets facets are awesome the serve well and long but they are not scalable from a functionality perspective

28 Analysis lego

29 Buckets and metrics results Cameras Optics Accessories Lenses buckets metrics

30 Buckets and metrics results Cameras 8 Optics Accessories Lenses buckets metrics

31 json style GET localhost:9200/_search { "aggs" : { "countries" : { "terms" : { "field" : "country", "aggs" : { "subjects" : { "terms" : { field" : "subject", "aggs" : { "avg_score" : { "avg" : { "field" : "score" { "hits" : {..., "aggregations" : { "countries" : { "buckets" : [ { "key" : "USA", "doc_count" : 5 "aggregations" : { "subjects" : "buckets" : [ { "key" : "Mathematics", "doc_count" : 3, "aggregations" : { "avg_score" : { "value" : 87.5,... ],... ]

32 measure all the things doc count (free!) avg stats extended stats min max sum count

33 Buckets global filter missing terms range date range ip range date histogram geo distance nested geohash grid cardinality percentiles significant terms histogram

34 Example - range bucket GET localhost:9200/grades/grade/_search { "aggs" : { "age_groups" : { "range" : { "field" : "age", "ranges" : [ { "from" : 5, "to" : 10, { "from" : 10 ], "aggs" : { "avg_grade" : { "avg" : { "field" : "grade" ' "age_groups": { "buckets": [ { "from": 5, "to": 10, "doc_count": 911, "avg_grade": { "value": , { "from": 10, "doc_count": 2276, "avg_grade": { "value": ]!

35 Example - histogram GET grades/grade/_search { "aggs" : { "grades_distribution" : { "histogram" : { "field" : "grade", "interval" : 10

36 Example - histogram GET grades/grade/_search { "aggregations": { "aggs" : { "grades_distribution": { "grades_distribution" "buckets": [ : { "histogram" { : { "field" "key": 60, : "grade", "doc_count": 467 "interval" : 10, { ] "key": 70, "doc_count": 873, { "key": 80, "doc_count": 930, { "key": 90, "doc_count":

37 Significant terms analytics as search

38 Common crimes GET ukcrimes/_search { "query": { "aggregations" : { "map" : { "geohash_grid" : { "field":"location", "precision":5,, "aggregations":{ "most_popular_crime_type":{ "terms":{ "field" : "crime_type", "size" : 1

39 Geo-what?

40 Geo-what?

41 Geo-what?

42 The common terms problem

43 Uncommonly common GET ukcrimes/_search { "query": { "aggregations" : { "map" : { "geohash_grid" : { "field":"location", "precision":5,, "aggregations":{ "most_popular_crime_type":{ "significant_terms":{ "field" : "crime_type", "size" : 1

44 Uncommonly common

45 Demo!

46 Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited