Statistical procedures searching for suspicious measurements and their sequences in measurement series. Ing. Marek Brabec, PhD CHMU, SZU, UI Praha

Transcription

1 Saisical procedures searching for suspicious measuremens and heir sequences in measuremen series Ing. Marek Brabec, PhD CHMU, SZU, UI Praha 8h AQ EIONET meeing Oslo, November 2003

2 Goals: Propose procedures for checking suspicious spos in ime series of air qualiy measuremens. Look: - no only for ouliers (large values) - bu also for more complicaed scenarios (e.g. aypical sequences, unusual dynamics) Formalize ad hoc procedures uilized by air polluion expers for manual checking Suppress subjeciviy in human exper appraisal - make rules for finding suspicious values explici, objecive and open o conrol Techniques applicable in a sofware for rouine use - help expers o find problemaic spos - wihou going hrough masses of normal daa - proposed mehods find only suspicious obs. - ulimae decision is reserved for human expers Use mehods ha can be employed in on-line mode - possible applicaions in real ime (conrol sofware of he measuremen devices)

3 Our approach is: based on saisical modelling measuremen series we use several flexible ime series models of DLM (Dynamic Linear Model) ype unforunaely, we do no have pure raining daa available wih reliable a priori classificaion problem / no- problem for all daa poins (e.g. gold sandard classificaion by expers) if such caegorizaion were available, model for problem free series would be more precise and deecion of problemaic daa would be more sharp from he heoreical poin of view, we have o derive a classifier wihou having raining classificaion consequenly, he fied model describes ypical properies of an essenially problem free series

4 Subsequenly: we look for measuremens and sequences ha are: - in some sense non-conforming wih he model developed for ypical daa, - i.e. hose ha are suspicious we check many pre-specified scenarios (i.e. pre-specified suspicious siuaions o look for) lis of suspicious scenarios - was prepared according o exper opinion abou wha are he mos frequen and mos imporan problems occurring in measuremen daabases - can be expanded based on furher exper advice o localize dubious observaions, or heir paches, we apply saisical qualiy conrol mehods (o residuals afer he all righ model)

5 Series of air qualiy measuremens have raher complex saisical properies - call for more complicaed ime series mehods when subsanial feaures are no prop. modelled, subsequen ess/checks are invalid: - inflaion of false alarms (compared o nominal) - increasing # of undeeced real problems Subsanial saisical feaures of observed series: auocorrelaion - measuremens closer in ime are more alike - more auocorrelaion for frequen measuremens (half hourly daa) - auoregressive models show close o uni roos (Dickey-Fuller problem) long erm rends - conneced o seasonal changes, long erm evoluion of he polluion level and oher facors - measuremens are ypically nonsaionary - shape of he rend is ofen complicaed (canno be described by simple parameric f.) periodiciy (day/nigh, weekdays/weekend changes, ohers) polluan concenraions are no observed direcly blurred by a measuremen error from: - device imprecision - local influences in space and ime

6 General saisical model of measuremen series: Measuremen equaion: T Y = F θ + ε Sae equaion(s): θ Gθ + η = 1 Where: Y value observed a ime. θ sae vecor (m x 1) G ime-invarian ransiion marix (m x m). T F (possibly) ime-varying vecor (1x m) 2 ε ~ N 0; σ ε observaion error, ( ) σ 2 is unknown, will be esimaed η sae error (srucural disurbance), η ~ N ( 0; ) W is unknown W To complee he model: ε and η are assumed independen (muually and across differen s) o ge sared, a priori iniial sae disribuion is θ ~ N 0 C assumed, 0 ( ; 0 ) W is specified by discouning, 1 δ = δ W GC 1 for a specific discoun facor 0 < δ < 1 G T

7 Imporan model feaures: T rue air qualiy sae ( F θ ) is no dir. observable, - wha is observed is only a noisy version - blurred by measuremen error ( ε ) decomposiion of he oal variabiliy ino: 2 - measuremen error variance ( σ ) - srucural variabiliy ( W ) 2 σ relaive amoun of srucural variabiliy w.r.. - amouns o signal-o-noise-raio - when S/N is larger, measuremens reflec rue sae of he naure closer and vice versa rue concenraion dynamics is linear (DLM) - vecor auoregressive model (VAR1) - simple Markov sysem modelled observaion Y can consis from - eiher he original measuremens - or heir ransformaion, like Y = log( Z a) effecively, shifed LN disribuion is assumed for original measuremens: Z T ( F θ ). exp( ) a = exp ε + - useful e.g. for modeling skewed daa (like O 3 )

8 Some concree model specificaions: he general DLM: - is flexible and modular in naure - easily adaped o paricular polluan s properies - unified saisical and programming framework modulariy G11 G12 T T T - block marix specificaion of G =, F = ( F 1, F 2 ) - each block can address a specific modeled feaure G flexibiliy T - locally consan model: G = 1, F = T - local linear rend: G =, F = ( 1 0) 0 i.e. linear rend, parameers randomly changing Y = µ + ε µ = µ β = β β + η η 1 1 (adapable alernaive o global linear regression) - wihin day periodiciy for half-hourly daa harmonics can be chosen o represen paricular polluan (srong d. periodiciy for O 3, vs. PM10) randomly changing rigonomeric coefficiens e.g. for k-h harmonic: a k 2πk sin + b 48 k 2πk cos 48 a b k k = a = b k, 1 k, 1 + η + η - relaionship beween (spaially close) saions T G = 1, F = Y 21 k1 k 2 G 22

9 Applying model o daa: esimae srucural parameers 2 - like σ ; δ, or elemens of W - hese are needed for subsequen seps - choices: maximum likelihood fiing Bayesian esimaion T esimae rue concenraions µ = F θ, or θ several asks: - when observaions Y1, hy are available filraion - when only Y1, l Y 1 are available one sep ahead (e.g. 30 min) predicion - when Y1,f Y, fyt for some T > are available smoohing smoohing produces revises esimaes when hisory of he process beyond ime of ineres is available we use only filraion, (one sep ahead) predicion - hese can be compued on-line (hisory up o he curren ime is available) - hese are mos useful for inspecion of suspicious measuremens in rouine regimen

10 Compuaionally, filraion, predicion can be done via Kalman algorihm swiching beween predicion/updaing seps relaively simple, linear calculaions advanageous marix formulaion dimension of marices (and inversions involved) is small (compared o sandard LS approach) due o he recursive naure of he Kalman filraion, - sorage requiremens are small - implemenaion easy and fas (yearlong, 30-min) can easily handle missing daa (jus skipping updaing sep) no only esimaes ( a, f sae and observaion predicions) - bu also heir variances can be easily obained ( R, Q sae predicion error covariance and observaion predicion error variance) - when normal disribuion of Y, θ is assumed, confidence inervals can be compued easily e.g. f ± q0.95( n 1) Q

11 Kalman filering Iniialisaion: m 0 = 0 (m x 1) vecor C 0 = I (m x m) marix S 0 = n ( Yi Y ) i= 1 n 1 2 scalar Predicion ( parameers of an a priori disribuion before a new observaion comes): Variance of one sep ahead predicion error ( Y predicion error, fromy1,, Y 1 ): T R = GC 1 G + W (m x m) marix see below for C calculaion 1 δ T W = GC 1G (due o he discoun simplificaion), hence δ 1 T R = GC 1G δ Predicion of he sae vecor one sep ahead: a = Gm 1 (m x 1) vecor predicion of he observaion one sep ahead: T f = F a scalar esimae of is variance is: T Q F R F + S scalar = 1 Updaing he parameers afer a new observaion comes (when we know obs. Y, 1,Y ): The residual afer predicing one sep ahead: e = Y f scalar auxiliary vecor: A = R F / Q (m x 1) vecor degrees of freedom for he esimae of he observaion variance: n = n 1 +1 scalar esimae of he observaion variance: 2 S 1 e S = + S 1 1 scalar n Q esimae of he sae vecor: m = a + A e esimae of is covariance marix: S T C = ( R A A Q ) S 1 This concludes one enire cycle of compuaions: calculaion reurns o R + 1 again (for he subsequen ime +1).

12 To check: how he model fis daa overall wheher paricular daum conform o he model look a: residuals afer he model or sandardized residuals e U = y y f = Q for various saisical reasons, we work wih sepahead residuals (orhogonaliy, independence) Idea: Any subsanial sysemaic feaures of he daa: - should be capured by he model - and hence should be removed from he residuals herefore, residuals should: - behave (approximaely) as a simple whie noise sequence (-disribued, under normaliy) - be ideal for checking suspicious deparures form he all righ model subsanial saisical heory available for he checks saisical qualiy conrol mehods are useful some ess amenable o graphical implemenaion, - - suiable for scanning large amouns of daa by human expers f

13 Ouline of he sraegy for suspicious runs checks: fi a model ha describes daa behavior under all righ siuaion reasonably well inspec residuals and sandardized residuals - o check he proposed model qualiy as he normal siuaion descripor (overall residual behavior should no show subsanial sysemaic feaures) - o localize suspicious observaions (showing unusually large or small residuals) - sd. res. are comparable o heor. criical values - residual checks can deec broad range of problems and deparures, bu wih smaller power han a focused es inspec model coefficiens (esimaed saes) o address more specific problems (changing/loosing periodiciy, loosing relaionship o geographically close saion, presence of a srong unidirecional rend, lack of any rend, ec.) o deec deparures from a ypical paern, use mehods analogous o saisical qualiy conrol - conrol chars wih rules, EWMA, CUSUM - applied o eiher residuals or coefficiens

14 Example of real daa (30-min. ozone, saion 777) Original daa: log(o3+0.1) cas (pulhodina) Sandardized residuals (from one-sep-ahead predicion): sandardizovane reziduum predikce o krok dopredu cas

15 ACF from residuals: ACF original daa ACF (wihou model correcing sysemaic feaures): lag ACF lag

16 Hisogram of sandardized residuals: sandardizovane reziduum z predikce o krok dopredu In addiion o residual checks, look a sae vecor elemens (behavior of model coefficiens): Variabiliy conneced wih differen harmonics (periodic model, ozone, 1011): cislo harmonicke slozky

17 Checking residuals for unusual behavior: 1. Tesing for gross, isolaed deparures from ypical behavior (large U or e ): Shewhar conrol scheme: - If U > ku, give an alarm signal (prediced value is oo small, compared o ypical rajecory) - If U < k L, give an alarm signal (prediced value is oo large, compared o ypical rajecory) - If wu < U ku or wl > U k L, give a warning signal - If wl U wu, do no do anyhing suiable choice of consans k L < wl < wu < ku - ofen k L = ku = 3, w L = wu = 2 (normaliy) - oher choices for heavy-ailed disribuions - larger absolue values of k L, ku, wl, wu number of false alarms, undeeced problems for consans seleced o saisfy prescribed false alarm rae, his is a sandard saisical es can be easily implemened in graphical form

18 Shewhar conrol char: Limis: warning, w L = wu = 2 decision, k L = ku = 3 sandardizovane reziduum cas

19 2. Tesing for slow, bu consisen deparures from ypical behavior (slow rends in U or e ): EWMA(Exponenially Weighed Moving Average) Defined recursively: E = 0 = 1,2, 0 E = ( 1 λ ) E 1 + λu for 0 < λ < 1 can be viewed as a degree of innovaion λ means subsanial E upgrade by U each ime λ leads o E smooher han U Decision scheme (wo-sided): - If E > k, give alarm signal (large residual; predicions are oo low) - If E < k, give alarm signal (small residual) - If k < E < k, do no do anyhing Choice of k : - ARL (Average Run Lengh) beween wo alarms - should be for all OK siuaion - should be for a specified problemaic scenario - analogous o ype I and ype II error raes, ( α, β ) - heoreical compuaions, ables, Crowder (1987) - Markov chain compuaions for simple siuaions - Mone Carlo simulaions for complicaed schemes CUSUM - ypically yields similar resuls, Wes, Harrison (1997)

20 EWMA: Procedure parameers: λ = 0. 5, k = 5 Decisions: +1 for large posiive residual, curren measuremen is much smaller han ypical -1 for large negaive residual Theoreical ARL s: for all OK siuaion 7.31 for sysemaic under/over-esimaion by 2 se s (0.0062% and respecively) Observed alarm rae: 0.80% wih posiive residues, 0.83% wih negaive

21 Checks can be applied no only o residuals, bu also o sae componens (esimaed model coefficiens) o address specific problems: sandardized i-h sae vecor componen, U i mi mi = C (local) lin. regr. coefficien, saion 771 on 777: ii sandardizovany regresni koeficien cas warning and decision limis as before unusually large pos./neg. coefficiens show unusual posiive/negaive relaionship beween wo geographically close saion zeros and oher values can be esed (deparure from long erm average relaionship)

22 3. Tesing for specific scenarios: Shewhar char wih addiional rules Example: implemened in CHMI for rouine checks wih doubly exponenial disribuion, wih quaniles: ( ) Q p log 2 p = for p 0. 5 = 1 ( 2( p) ) log for p 0. 5 use a simplified (random walk) model: Y = Y 1 + η work wih differences: e = Y Y 1 and heir sandardized forms: U = e σ

23 define following evens: P1 U ( Q ; ) P3 U ( Q 0.999;Q ] P2 U ( Q 0.99;Q0.999 ] Z U ( Q 0.499;Q0.501] N2 U ( Q 0.001;Q0.01] N3 U ( Q ;Q0.001] N1 ( ) U ;Q furher, define 8 conrolled siuaions A1 P1 A2 N1 A3 P3-1 and N3 A4 N3-1 and P3 A5 Z -3 and P2-2 and N2-1 and Z A6 Z -3 and N2-2 and P2-1 and Z A7 P2-1 and N2 and U U 1 k 1 A8 N2-1 and P2 and k inerpreaion in erms of original series ( Y ): A1 jump A2 jump A3 jump, followed by jump A4 jump, followed by jump A5 almos consan, jump, jump, almos cons. A6 almos consan, jump, jump, almos cons. A7 jump, jump wih seep decline A8 jump, jump wih seep rise U U 1

24 Example of an applicaion on real daa: Saion 771 (Prague, Náměsí Republiky) yearlong (2002), ozone, 30-minue measuremens Theoreical all OK ARL=11880 (abou % of checked daa should be alarms of any kind) The procedure found 6 alarms in daa (0.0356%) Observed frequencies of various alarm ypes: A1 A2 A3 A4 A5 A6 A7 A Exremely simple implemenaion in a sofware for rouine applicaions Furher rules can be added, according o air polluion specialiss experise Underlying saisical model is raher simple, can be improved: - srucured DLM, alernaive o simple differencing - more deailed model leads o improvemen in false alarm and undeeced error raes ( α, β )

25 Examples of siuaions labeled as suspicious by he procedure among real daa: Horizonal axis: ime since he alarm signalled, verical axis: O 3 O cas od poplachu O cas od poplachu

26 O cas od poplachu Beware of he imporan feaure: by design, procedure produces some false alarms in order no o miss problem spos hence, i is imporan ha an exper will check suggesed suspicious spos manually before declaring hem o be in error checking procedures produce lis of suspicious values/sequences, hey are no inended as auomaic ools for deleion of measured values he lis of suspicious values is raher shor, so ha he exper can concenrae on heir inspecion, wihou going hrough masses of normal daa

27 Special procedures for deecing specific problems: example of device jamming conrol Problem descripion provided by A-P expers: occasionally, measuring device freezes on a single value jamming occurs for various reasons, e.g. daa pre-processing, ec., he final daa are no compleely consan, bu oscillae somehow around he jammed value hence, he jamming deecion is no as rivial as checking for zero differences Formally, we consider wo models: Model 1: normal measuremens - This is a more complicaed DLM (e.g. local linear rend) Model 2: jamming - This is a simpler alernaive (locally consan model) Alhough we know abou possibiliy of: normal o jammed change (M1->M2) and he reverse change (M2->M1) For a paricular ime : one isn sure which model holds

28 Afer inroducing formal mixure m. (dynamic linear mixure model) having apriori probabiliy π ha M1 holds (and 1 π for jammed M2) one can compue a poseriori probabiliy p ha normal model (M1) holds, given he observed daa up o ime (using Bayesian approach, see e.g. Wes-Harrison (1997)) he mos imporan inspecion ool in his conex is p and is changes hroughou he ime whenever p drops below a specified hreshold P, jamming is suspeced Example: Saion 1011(Úsí nad Labem, Kočkov) wih parameers: π = 0. 95, P = % of he 2002 daa swiched he alarm on p() cas

29 Real daa example of spos where alarm was se off Horizonal axis: ime since he alarm signalled, verical axis: log(o ) log(o3+0.1) cas sandardizovane reziduum cas

30 Behavior of he procedure for simulaed daa 2 M1 is locally linear, Var η = diag( 1, 0.01 ) M2 is locally consan σ = 0. 1 for boh of hem simulaed 200 M1 poins and 200 M2 (jammed) y cas Mixure model was applied wih: = 5 π, P = , δ = 0. 5 p cas

31 Conclusions: Inroduced saisical checks for problemaic spos The ess are broader han jus checking for ouliers include ess for: - unusually large/small values - unusual rajecories - specific feaures ( spaial cor.) - paricular problems (online jamming monior) Model srucure and subsequen esing is: - explici - open o criicism - open o furher improvemen (AP exp. suggesion) - modular - flexible Procedures are based on unified framework: - saisically - from programming poin of view The goal is o provide a ool for air polluion expers when checking he daa - provide lis of suspicious siuaions - o be checked manually - replaces he need o go hrough los of normal daa (usable for years-long 30-min daa) - removes parially subjeciviy in exper judgemen Presened mehods, procedures and ools - can be programmed easily - ino a rouinely usable sofware