Impression Store: Compressive Sensing-based Storage for Big Data Analytics Jiaxing Zhang, Ying Yan, Liang Jeff Chen, Minjie Wang, Thomas Moscibroda & Zheng Zhang Microsoft Research
The Curse of O(N) in Big Data Era In the old days, an O(N) algorithm was efficient But what if N is increasing fast? Parallelism is only a partial solution O(N) O N k k: # of machines It s an illusion that we can compute against all the data What we collect is always a sample By the time we finish computing, new data has generated Approximate Results Suffice! N k
Impression Store Provides an abstraction of big data vectors Support retrieval of big data components Store impression information rather than raw data Improvements the performance Save storage capacity Save IO bandwidth High scalability
System Design Query Top-K/Outlier-K Δx 1 Δx 2 Δx L 1 Δx L Update Synchronization f(δx 1 ) f(δx 2 ) Eventually consistent f(δx L 1 ) f(δx L ) Key technique: Compressive Sensing Node 1 Node 2 Node L 1 Node L 1. High Scalability: Any node can process any update / query Synchronization 2. Storage, memory, IO and communication cost efficient 3. High throughput to Impression Query- Top, Outlier & mode Compression Incremental updates on compression domain Uncompressing big components only
Introduction of Compressive Sensing Data Vector (Sparse) compression Length N Length M Measurement decompression x 1 x 2 x N φ 1 φ M y = Φ x y 1 y 2 y M M M N Recovery Algorithms: Orthogonal Matching Pursuit (OMP) Random Projection, Φ is a random matrix = N Recovered Data Vector x 1 x 2 x N
Compressive Sensing vs. Compression Decomposable Compression: x = x 1 + x 2 y = Φx = Φ x 1 + x 2 = Φx 1 + Φx 2 = y 1 + y 2 Distributed Aggregation y = y 1 + y 2 Continuous Updated Data y = y 1 + y 2 y 1 = Φx 1 y 2 = Φx 2 y 1 = Φx 0 y 2 = ΦΔx x 1 x 2 Data: x 1 + x 2 x 0 Δx Data: x0 + Δx Node 1 Node 2 Recovery Algorithm (OMP)/Our BOMP Big components have more precision Base data Update
Architecture Update Client Impression Store API Query Issue to any node Accumulated Update Δx 1 Impression Store Query Top-K/Outlier-K Measurement: Δy 1 = ΦΔx 1 Δy 2 1 Recovery: from y L Update Synchronization y 1 Δy 1 Δy 1 +Δy 2 Σ L 2 i=1 Δy i Σ L 1 i=1 Δy i Σ i=2 Δy i y 2 y L 1 y y = Σ i=1 Δy i Σ i=3 Δy i 1 + L Oracle Measurement Node 1 Node 2 Node L 1 Node L
Client Update Client Impression Store API Query 1. Map a table into data vectors 2. Translation: SQL->operations on Vector Issue to any node Accumulated Update Δx 1 Impression Store Query Top-K/Outlier-K Measurement: Δy 1 = ΦΔx 1 Δy 2 1 Recovery: from y L Update Synchronization y 1 Δy 1 Δy 1 +Δy 2 Σ L 2 i=1 Δy i Σ L 1 i=1 Δy i Σ i=2 Δy i y 2 y L 1 y y = Σ i=1 Δy i Σ i=3 Δy i 1 + L Oracle Measurement Node 1 Node 2 Node L 1 Node L
Architecture Update (i, x) Client Impression Store API Top(K): returns top-k Outlier(K): returns outlier-k and mode Issue to any node Accumulated Update Δx 1 Impression Store Query Top-K/Outlier-K Measurement: Δy 1 = ΦΔx 1 Δy 2 1 Recovery: from y L Update Synchronization y 1 Δy 1 Δy 1 +Δy 2 Σ L 2 i=1 Δy i Σ L 1 i=1 Δy i Σ i=2 Δy i y 2 y L 1 y y = Σ i=1 Δy i Σ i=3 Δy i 1 + L Oracle Measurement Node 1 Node 2 Node L 1 Node L
Query Processing Update Issue to any node Client Impression Store API Query Each node continuously works on three tasks: 1. Aggregate and compress data updates 2. Update Synchronization O(M) 3. Top/outlier-K recovery Matrix computation->gpu Impression Store Accumulated Update Δx 1 Query Top-K/Outlier-K Top/outlier-K Recovery Top/outlier-K Recovery Measurement: Δy 1 2 L 1 Recovery: Recovery: from from y 1 = ΦΔx 1 Δy 2 1 y L L Update Synchronization y 1 Δy 1 Δy 1 +Δy 2 Σ L 2 i=1 Δy i Σ L 1 i=1 Δy i Σ i=2 Δy i y 2 y L 1 y y = Σ i=1 Δy i Σ i=3 Δy i 1 + L Oracle Measurement Node 1 Node 2 Node L 1 Node L
Update Synchronization Client Update Impression Store API Query Goal: y i converges to y quickly Randomly issue Issue to to any node Accumulated Update Δx 1 Impression Store Query Top-K/Outlier-K Measurement: Δy 1 = ΦΔx 1 Δy 2 1 L Recovery: from y L Update Synchronization y 1 Δy 1 Δy Δy 1 1 +Δy +Δy 2 2 Σ i=1 Σ L 2 i=1 Δy ii ΣΣ L 1 L 1 i=1 i=1 Δy ii Oracle Measurement y 2 y L 1 y y = Σ L Σ L i=1 Δy i Σ L Σ L i=2 Δy i i=3 Δy i 1 +Δy Σ L i=2 Δy i i=3 Δy i L 1 + L Node 1 Node 2 Node L 1 Node L
Update Synchronization Synchronization policy ψ p l Δy y ψ w l Loop-free topology Master-Slave tree structure Small latency Load is not balanced Node p Node l Topology in between - trade off y = Δy + ψ q l q N(l) Send to p: ψ l p = Δy + q N(l) ψ q l ψ p l = y ψ p l Line structure Long latency Load is balanced Each pair of Send-Receive copies my not be the same all the time! The policy is proved to achieve eventual consistency.
Optimizations Speed up the recovery O(M 2 N) GPU: 30~40X speed-up For continuous updates Optimizing the recover algorithm by keeping the positions of the last recovery Reduce the complexity to O(M 3 )
Experiment Setup Error metrics Example Ground truth Key Value 1 100 2 80 3 50 4 30 5 10 Approximation Key Value 1 100 2 80 3 48 4 25 6 11 E p = 1 4 5 = 20% E v = 3.88% Workload: Revenue on Ads entries in Bing Search engine Group by 6 attributes (Market, Vertical, QueryClass ) Totally N=12,891 user-interested entries in the vector
Preliminary Results (1) Effect of M and N on Recover Quality E p E v
Preliminary Results (2) Bigger value can be recovered much more accurate with smaller M
Preliminary Results (3) Compare with traditional Top-K only approach (K=M) Approximated x 1 + x 2 (top-k) Approximated x 1 + x 2 (top-k) Recovery Algorithm Merge y = y 1 + y 2 Top-K in x 1 Top-K in x 2 y 1 = Φx 1 y 2 = Φx 2 x 1 x 2 x 1 x 2 Data: x 1 + x 2 Traditional Top-K Approach Compressive Sensing Approach
Ongoing and Future work Support more sophisticated queries Exploring CS and other techniques Work together with sampling Multiple parallel queries to different nodes can improve confidence
Thanks!