Exact data mining from in- exact data Nick Freris

Transcription

1 Exact data mining from in- exact data Nick Freris Qualcomm, San Diego October 10, 2013

2 Introduc=on (1) Informa=on retrieval is a large industry.. Biology, finance, engineering, marke=ng, vision/graphics, etc...but data are hardly ever maintained in original form Compression Rights protection Original Quantized Watermarking

3 Preserva=on of the mining outcome (2) this talk: provable guarantees for distanced- based mining Optimal distance estimation K-means preservation D=11 ˆD 2 [11, 12]

4 Op=mal distance es=ma=on (3) This work presents op=mal es=ma=on of Euclidean distance in the compressed domain. This result cannot be further improved D = 11.5 ˆD 2 [11, 12] Our approach is applicable on any orthonormal data compression basis (Fourier, Wavelets, Chebyshev, PCA, etc.) Our method allows up to 57% bewer distance es=ma=on 80% less computa=on effort 128 : 1 compression efficiency

5 Mo=va=on/Benefits Time- series data are customarily compressed in order to: Save storage space Reduce transmission bandwidth Remove noise Achieve faster processing / data analysis (4) Distance es=ma=on has various data analy=cs applica=ons Clustering / Classifica=on Anomaly detec=on Similarity search Now we can do all this very efficiently directly on the compressed data!

6 Previous work vs current approach First coeffs + error Best coeffs + error Best coeffs + error + constraints à Op=mal (our method) (5) Approach 1 Approach 2: (only one compressed) Our approach

7 (6) applica=ons Similarity Search/Grouping/Clustering find seman=cally similar searches, deduce associa=ons dynamic inventory op=miza=on Five key supply chain areas Chief supply chain officer IBM s dynamic inventory integrated supply chain Smarter supply chain Retail Supply Management Business consultant supply chain management QUERY: Supply Chain Results with similar query pawerns

8 (6) Similarity Search/Grouping/Clustering applica=ons Adver=sing/recommenda=ons Buying keywords that are cheaper but exhibit same demand pawern Requests Query: business services Query: business analytics Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

9 (6) Burst Detec=on Discovery of important events applica=ons Query: Acquisition Feb 10: Intelliden Inc. Acquisition Requests May 10: Sterling Commerce Acquisition (1.4B) Jan Feb Mar Apr May Jun Jul Aug Sep Okt Nov Dec

10 (7) underlying opera=on: k- NN search K- Nearest Neighbor (k- NN) similarity search Issues that arise: How to compress data? How to speedup search We have to es=mate =ght bounds on the distance metric using just the compressed representa=on

11 (8) similarity search problem query D = 7.3 Distance Linear Scan: Objec've: Compare the query with all sequences in DB and return the k most similar sequences to the query. D = 10.2 D = 11.8 D = 17 D = 22

12 (8) speeding up similarity search simplified DB Candidate Superset original DB Final Answer set Upper / lower bounds on distance Verify against original DB simplified query keyword 1 keyword 2 keyword 3 keyword

13 (9) compressing weblog data Use Euclidean distance to match =me- series. But how should we compress the data? Query: analytics and optimization 1 year span The data is smooth and highly periodic, so we can use Fourier decomposition. Instead of using the first Fourier coefficients we can use the best ones instead. Query: consulting services

14 (10) Discrete Fourier Transform (DFT) decomposi=on of =me- series into sinusoids Every signal can be represented as a superposition of sines and cosines X(f) i i i i i i i i i i i i i i i i i i i i i i i i x(n) Jean Baptiste Fourier ( )

15 (10) spectrum P(f) X(f) x(n) i Keep just some frequencies of periodogram i i i i i i Sequence: i i i i i i Periodogram: i i i i i f i i i i i i

16 (10) spectrum P(f) X(f) x(n) i Keep just some frequencies of periodogram i i i i i i Sequence: i i i i i i Periodogram: i i i i i f i i day period i i i i

17 (10) spectrum P(f) X(f) x(n) i Keep just some frequencies of periodogram i i i i i i Sequence: i i i i i i Periodogram: i i i i i f i i day period i i i i

18 (10) spectrum P(f) X(f) x(n) i Keep just some frequencies of periodogram i i i i i i Sequence: i i i i i i Periodogram: i i i i i Reconstruction: f i i i i i i

19 (11) similarity Search Approximate the Euclidean distance using DFT First 5 Coefficients +symmetric ones x(n) X(f) i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i

20 (11) similarity Search Approximate the Euclidean distance using DFT Best 5 Coefficients + symmetric ones x(n) X(f) i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i

21 (12) objec=ve Calculate the =ghtest possible upper/lower bounds using the coefficients with the highest energy This will result in bewer pruning of the search space - > faster search

22 (13) x X Q q i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i

23 (13) Find best k coefficients of X (k=4) Magnitude vector X X Q i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i Q

24 (13) Find best 4 coefficients of T X X Q i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i Q

25 (13) minpower Iden=fy smallest magnitude (à power) X X Q i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i Q

26 (13) minpower The remaining powers are less than minpower X X Q i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i Q

27 (13) e 2 We keep also the sum of squares of the remaining powers X X Q i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i Q

28 (13) Calculate distance from k known coeffs X X Q i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i Q

29 (13) minpower Op=mize distance from remaining coefficients X X Q i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i Q Op=miza=on Generaliza=on

30 (14) water- filling d 2 Q X = -α Q d 1

31 (14) water- filling d 2 Q e Boundary condi=on on total energy: d d 22 < e 2 X = -α Q d 1

32 (14) water- filling d 2 Q minp e Boundary condi=on on each dimension: d 1 < minpower, d 2 < minpower X = -α Q d 1 minp

33 (14) water- filling d 2 Q minp e We hit one of our constraints X = -α Q d 1 minp

34 (14) water- filling d 2 Q minp e We start moving on the other dimension un=l we use all our energy X = -α Q d 1 minp

35 (14) in N- dimensions Q X minpower Available Energy

36 (14) in N- dimensions Q X minpower Available Energy

37 (14) in N- dimensions Q X minpower Available Energy

38 (14) in N- dimensions Q X minpower Available Energy

39 (14) in N- dimensions Q X minpower Available Energy

40 (14) in N- dimensions Q X minpower Available Energy

41 (14) in N- dimensions Q X minpower Available Energy

42 (14) in N- dimensions Q X minpower Available Energy

43 (14) in N- dimensional space Q X minpower Available Energy

44 (15) When both sequences are compressed the op=mal distance es=ma=on can be solved using a double waterfilling process. single waterfilling double waterfilling

45 the bounds are op=mally =ght (16) Theorem (Freris et al. 2012): The computa=on of lower and upper bounds given the aforemen=oned compressed representa=ons can be solved exactly using double water- filling problem. The lower and upper bounds are op=mally =ght; no =ghter solu=ons can be provided Sketch of proof: - convex re- parametriza=on - KKT condi=ons - proper=es of op=mal solu=ons

46 Double Water- filling algorithm (17) Intui=on: Water- fill for the discarded coefficients of the two vectors separately....using the appropriate energy alloca=on Zero- overhead algorithm Complexity: Θ(nlong)

47 Experiments (18) Unica database. IBM web traffic for year of 2010 Marke=ng/Adwords recommenda=on Analysis/Storage of weblog queries (1TB of data per month) GBS: Scheduling adver=sing campaigns / pricing BUSINESS DYNAMICS IBM YIN YANG OF FINANCIAL DISRUPTION EINSURANCE CUSTOMER EXPERIENCE. IBM GLOBAL BUSINESS ANDREW STEVENS BUSINESS CONSULTING GLENN FINCH IBM AMERICA MEDIA PLAYER INDUSTRY STRATEGIE ENTREPRISE RENTABILIT

48 our analy=c solu=on is 300x faster than convex op=mizers (19)

49 Our algorithm converges very fast: in 2-3 itera=ons irrespec=ve of the compression rate (20)

50 the derived LB/UB are 20% bewer than state- of- art (21)

51 when searching in a DB this results in up to 80% data pruning the previous (10-20%) improvement in distance es=ma=on can significantly reduce the search space when searching for k- NN results (22) We retrieve 20%-80% fewer sequences than other approaches

52 Conclusions (23) Increasing data sizes is a perennial problem for data analysis ª It is important to support mining directly on the compressed data We have shown an exact algorithm for obtaining op=mally =ght Euclidean bounds on compressed representa=ons ª Mining directly on compressed data with provable guarantees Many generaliza=ons: Cosine Similarity (text documents): cos(x,y) = 1 - L 2 (x,y) 2 /2 Correla=on (financial analysis): corr(x,y) = 1 - L 2 (x,y) 2 /2 (for normalized signals x,y) Dynamic Time Warping (flexible similarity metric):

53 Compression with provable K- means preserva=on

54 Scope of this work (24) Mul=- bit compression and inver=ble transforma=on of high- dimensional data Ø with provable K- means preserva=on cluster 1 cluster 2 cluster 3 cluster 1 cluster 2 cluster 3 identical clustering results K-means" K-means" original data Original data quantized data Compressed data

55 Applica=ons (25) Reduced storage requirements Reduced processing =me Data hiding / anonymiza=on Original values are hidden via an inver=ble cluster contrac=on Encoder- decoder scheme Shape preserva=on High- quality data reconstruc=on Extends applicability to other mining opera=ons such as Nearest- Neighbor search, data visualiza=on, etc. Tunable compression level based on number of allocated bits in order to achieve good storage/quality trade- off

56 Overview (26) 1. Pre- clustering 2. Quan=za=on per cluster and dimension (one or mul=ple bits) 3. Data contrac=on transforma=on 4. Post- quan=za=on clustering 5. Decoding using key (contrac=on parameter)

57 Our method effec=vely compresses the data (27) 1. MMSE quan=za=on per cluster and dimension (one or mul=ple bits) Preserves mean and reduces variance and dynamic range Example: 1- bit MMSE quan=za=on Quan=zer:

58 (28) How does our scheme apply to high- dimensional data? These are the points for one dimension and for one cluster of objects. Dimension d (or time instance )d) Process is repeated for all dimensions and for all clusters. We have one quan=zer per class

59 (29) We can also support Data- Hiding with our scheme 2. Data contrac=on transforma=on Inver=ble Preserves centroids and reduces variance and dynamic range BeWer cluster separa=on

60 Results (30) Theore=cal guarantee on cluster preserva=on for sufficiently contracted clusters (sufficiently small a) Analy=cal calcula=on of op=mal contrac=on coefficient (a) K(K- 1) calcula=ons, K: number of clusters Lower distor=on (MSE) than MPQ which decreases with the number of allocated bits Mul=- bit alloca=on (per cluster) Generaliza=on of MMSE quan=zers for mul=ple bits Efficient algorithm for op=mal alloca=on given storage constraints

61 (31) We preserve 100% the clustering structure Stock values 2100 stocks during a period of three years Clustered in 3,5,8 clusters Cluster preserva=on was always 100% Shape preserva=on

62 Mul=- Bit quan=za=on improves quality up to 96%! MSE due to quan=za=on decreases with number of bits and number of clusters (32) 37%, 70% and 96% over MPQ using MMSE with 1, 2, 4 bits 76%, 84% using 5, 8 clusters over 3 clusters

63 Compress > 8x without loss in clustering accuracy (33) Compression efficiency: 1.8x 8x reduc=on

64 Conclusions (34) Efficient K- means preserving compression of high- dimensional data Data hiding (encoder- decoder data transforma=on) Tunable compression level (storage / distor=on trade- off) Theore=cal guarantees Our scheme is the first one which always guarantees 100% cluster preserva=on Efficient bit alloca=on algorithm for op=mal distor=on given storage constraints Experiments 100% cluster preserva=on Excellent shape preserva=on even for 1 bit quan=zers High compression efficiency Extensions Our approach works with any quan=za=on scheme Example: Minimum Absolute Error Quan=za=on (MAE)

65 1. Distributed es=ma=on in sensor networks Other research (35) 2. Randomized algorithms Ax = b 3. Compressed sensing 4. Clock synchroniza=on Convergence lab (CSL, Univ. Illinois) 5. Wireless resource alloca=on 6. Video streaming SmartSense, EPFL

66 Thank you! Ques=ons