OSIsoft Data Compression Analysis

Transcription

1 OSIsoft Data Compression Analysis Raymond de Callafon Charles H. Wells University of California, San Diego (UCSD) OSIsoft NASPI Work Group Meeting, Knoxville, TN, March 11-12, 2014

2 Swinging Door Compression 2

3 Swinging Door Compression Compression in action: Done in real-time Compression level chosen here at Hz 3

4 Contents of This Talk Given swinging door data compression technique, consider: Analysis of data compression Investigate data loss for different compression settings Recommended data compression setting for synchrophasor data 4

5 Test Data as a Usecase Data: Frequency Sample at 30Hz Data has: Trend Noise Small jumps 5

6 Test Data as a Usecase Notation: e(k) = f(k)-f c (k) where f(k) = raw f c (k) = compressed Observations: Despite trend, e(k) <0.001 Due to small compression level, e(k) looks white noise 6

7 Test Data as a Usecase Notation: e(k) = f(k)-f c (k) where f(k) = raw f c (k) = compressed Observations: Despite trend, e(k) <0.001 Due to small compression level, e(k) looks white noise Effect of compression clear in time domain: 901 points -> 196 points (21.75%) 7

8 Basic Analysis (1/2) Let {tt kk, ff kk } = raw data, {tt cc kk, ff cc kk } = compressed data, then ff ll (kk) = LL{tt cc kk, ff cc kk, tt(kk)} ee cc kk = ff ll (kk) ff kk where LL{tt cc kk, ff cc kk, tt(kk)} is linear interpolation of {tt cc kk, ff cc kk } at tt kk. Let tt be sampling time and let DFT be given by FF ll ωω nn NN = ff ll kk ee jjωω nnkk tt kk=1 Since DFT is linear operation: FF ll ωω nn = FF ωω nn + EE cc ωω nn where EE cc ωω nn is DFT of ee cc kk. Note that FF ll ωω nn 2 = FF ωω nn + EE cc ωω nn 2 8

9 Basic Analysis (2/2) With FF ll ωω nn = FF ωω nn + EE cc ωω nn following observations can be made: Fourier transform (and spectra) is influenced linearly by compression error ee cc kk. Compression error ee cc kk is always bounded ee cc kk < cc with EE{ee cc kk } = 0 and thus EE ee 2 cc kk < cc 2 /3 if ee cc kk has uniform distribution (however, not always uniform) Aliasing is bounded due to re-interpolation LL{tt cc kk, ff cc kk, tt(kk)} of {tt cc kk, ff cc kk } at tt kk creating again a sampling frequency of 1/ tt If ee cc kk is pure white noise, EE cc ωω nn is a complex number with EE cc ωω nn = cc 2 /(6ππ) aaaaaaaaaa{ee cc ωω nn [ ππ, ππ] 9

10 Back to our Test Data as a Usecase Check if white : Compute autocorrelation function: RR ee ττ = EE{ee cc (kk)ee cc (kk ττ) Result: If indeed error ee cc kk is white, spectrum (periodogram) EE cc ωω nn 2 = constant (but noisy) 10

11 Back to our Test Data as a Usecase Result: Indeed, spectrum (periodogram) EE cc ωω nn 2 = constant (but noisy) Conclusion: Spectrum for given compression level where ee cc (kk) is white noise provides base level 11

12 Back to our Test Data as a Usecase Conclusion: Compression only influences DFT significantly when DFT comes close to base level When ee cc (kk) is NOT a white noise! Hence: Compression level c must be chosen such that ee cc kk is a white noise! 12

13 Back to our Test Data as a Usecase Observe: Increasing compression level c At low levels of c=0.001, ee cc (kk) is a white noise For higher levels of c>0.001, ee cc (kk) is NOT a white noise anymore 13

14 Back to our Test Data as a Usecase Observe: Increasing compression level c ee cc (kk) is NOT a white noise anymore Effect can also be seen in correlation function (for c=0.002) 14

15 Back to our Test Data as a Usecase Observe: Increasing compression level c ee cc (kk) is NOT a white noise anymore Effect can also be seen in spectra 15

16 Another usecase: noise free fixed sine Effect of compression is immediately clear when signal does not have noise: Data: Single sinusoid with compression level c= > 91 points ee cc (kk) is NOT a white noise but periodic! Periodic error signal may have several harmonics 16

17 Another usecase: noise free fixed sine 17 Data: Single sinusoid with compression level c= > 91 points ee cc (kk) is NOT a white noise but periodic! Periodic error signal may have several harmonics Spectrum hardly influenced at main harmonic!

18 Another usecase: noise free fixed sine 18 Data: Single sinusoid with compression level c= > 91 points ee cc (kk) is NOT a white noise but periodic! Periodic error signal may have several harmonics Effect of harmonics in error signal can be seen in spectra!

19 Another usecase: fixed sine with noise Data: Single sinusoid with compression level c=0.001 Typically data has some noise 900 -> 235 points Due to noise, ee cc kk can be a white noise, provided c is picked properly. 19

20 Another usecase: fixed sine with noise 20 Data: Single sinusoid with compression level c=0.001 Typically data has some noise 900 -> 235 points Due to noise, ee cc kk can be a white noise, provided c is picked properly. Spectrum is flat again (but noisy)

21 Wrap up/conclusions Main conclusions: Compression creates an error signal ee cc kk < c Compression level bounds aliasing as linear (re)interpolation preserves sampling frequency. Suggestions: Compression level c must be chosen such that ee cc kk is a white noise. Compression level c must be such that it resembles noise level of sensor data. For typical frequency application/oscillations c=0.001 gives about 20-25% compression. 21