SEQUENTIAL KERNEL ESTIMATION OF A MULTIVARIATE REGRESSION FUNCTION

Transcription

1 1 SEQUENTIAL KERNEL ESTIMATION OF A MULTIVARIATE REGRESSION FUNCTION D.N. Politis Uiversity of Califoria, Sa Diego USA, Sa Diego, La Jolla, CA dpolitis@ucsd.edu V.A. Vasiliev 1 Tomsk State Uiversity Russia, , Tomsk, Lei Ave., 36 vas@mail.tsu.ru Key words: Kerel regressio estimatio; depedet observatios; sequetial approac; guarateed mea square accuracy; fiite sample size Tis paper presets a sequetial estimatio procedure for a ukow multivariate regressio fuctio. Observed regressors ad oises of te model are supposed to be depedet ad form sequeces of depedet vectors ad umbers respectively. Two types of estimators are cosidered. Bot estimators are costructed o te basis of Nadaraya Watso kerel estimators. First, sequetial estimators wit give bias ad mea square error are defied. Accordig to te sequetial approac te duratio of observatios is a special stoppig time. Te o te basis of tese estimators, trucated sequetial estimators of a regressio fuctio are costructed o a time iterval of a fixed legt. At te same time te variace of tese estimators is also bouded by a o-asymptotic boud. Togeter wit fiite-sample, asymptotic properties of te preseted estimators are ivestigated. It is sow, i particular, tat by te appropriately cose badwidts bot estimators ave optimal as compared to te case of idepedet data) rates of covergece. 1. Itroductio I may problems of idetificatio te solutio is reduced to fidig some statistics i te form of a ratio of some radom fuctios as estimators e.g., ratio estimators). Suc situatio arises i te regressio estimatio problem by use of te Nadaraya Watso estimators. Tere are a lot of results ad publicatios dedicated tis problem for idepedet ad depedet observatios, i asymptotic ad oasymptotic problem statemets see, e.g, [1], [3]-[6], [8], [9], [16], [20]-[27], [30] etc.) It sould be oted, tat ofte regressors X see 1) below) are supposed to be o-radom or bouded ad oises of te model are idepedet [1, 4, 8, 17], [22]-[26] etc. If, i geeral, te deomiator of suc ratio may be small eoug wit positive probability, te te fuctioal is o loger fiite ad oe as, e.g., to use some 1 Researc was supported by RFBR Grat N Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

2 2 variats of Cramer s teorem [3] to fid eve te first momet of suc estimator. Furtermore it is ot trivial to fid domiat sequeces for momets of ratio estimators see, e.g., [30]). Te fuctioals of tis type were uified i [30] see also refereces terei) ito te class of fuctioals wit sigularities ad special estimatio procedures for tem, called piecewise smoot approximatio, were cosidered. Te metod proposed i [30] gives a possibility to get estimators based o depedet observatios wit kow pricipal term of te mea square error MSE) ad kow asymptotic properties we te sample size uboudedly icreases). However i practice te observatio time of a system is always fiite wic does ot allow us to judge te quality of suc estimators. Oe of te possibilities for fidig estimators wit a guarateed quality of iferece is provided by te stad-poit of sequetial aalysis; tis approac presumes a special coice of te time for stoppig observatios wit te elp of some fuctioal of te process uder observatio. Te priciple of sequetial aalysis as bee primarily proposed by A.Wald for a sceme of idepedet observatios [33]; later, te sequetial approac was applied to parameter estimatio problem of oe-parameter stocastic differetial equatios i [15, 18] ad for multiparameter cotiuous ad discretetime dyamic systems i may papers ad books see [2], [7], [10]-[14], [28]-[32] amog oters). Sequetial approac as also bee applied to o-parametric regressio ad desity fuctio estimatio problems as well see, e.g., [4]-[6], [9], [23, 28, 30, 32] ad refereces i [6]). Peculiarities ad difficulties of sequetial approac to oparametric problems are discussed i detail by Efroimovic i [6]. I tis paper see Sectio 3) we solve te problem of estimatio of a regressio fuctio wit a prescribed statistical quality usig a sequetial approac, wic supposes uboudedess of a sample size. I Sectio 4 we cosider te most realistic problem statemet, were te observatio time of a system is ot oly fiite but bouded. Oe of te possibilities for fidig estimators wit te guarateed quality of iferece from a sample of fixed size is provided by te approac of trucated sequetial estimatio. Trucated sequetial estimatio metod was developed by Koev ad Pergamectcikov [11]-[13], as well as i [7] for parameter estimatio problems i discrete ad cotiuous time dyamic models. Usig sequetial approac, tey ave costructed estimators of dyamic systems parameters wit kow variace from samples of fixed size. No-parametric trucated sequetial estimators of a regressio fuctio preseted i tis paper costitute Nadaraya Watso estimators calculated at a special stoppig time. Tese estimators ave kow mea square errors as well. Te duratio of observatios is also radom but bouded from above by a o-radom fixed umber. No-asymptotic ad asymptotic properties of estimators of bot types are ivestigated. It is sow, tat trucated sequetial estimators coicide wit te afore-metioed sequetial estimators i.e. wit te Nadaraya Watso estimators calculated at a special stoppig time) for sufficietly large sample size. Te assumptio o te idepedece of observatios is ofte ot satisfied i practice ad is a essetial restrictio. I tis paper te sequetial aalysis approac is employed i te procedures of o-parametric estimatio of a regressio fuctio from depedet observatios. I particular, model iputs ad oises ca Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

3 3 be depedet ad ubouded wit a positive probability. Te example of estimatio of a oliear autoregressive fuctio is cosidered. 2. Problem statemet Tis paper cosiders te problem of o-parametric kerel estimatio of a regressio fuctio fx) wit multivariate argumet x R m, m 1 at a poit x = x 0 satisfyig te equatio 1) Y i = fx i ) + δ i, i 1, i te case of depedet regressors X = x i ) i 1 ad oises = δ i ) i 1. It is supposed, tat pairs Y i, x i ), i = 1, 2,... are uder observatios but te errors δ i are uobservable. We allow for geeral depedece coditios o radom variables x i ) ad δ i ) supposig mutual depedece of te processes X ad. I particular, model iputs ca be ubouded wit a positive probability. Te mai goal of te paper is twofold. We will costruct sequetial ad trucated sequetial estimators. Estimators of bot types ave kow upper bouds of mea square errors. No-parametric trucated sequetial estimators of a regressio fuctio preseted i tis paper are costitute te Nadaraya Watso estimators calculated at a special stoppig time. Tese estimators ave kow upper boud of mea square errors as well. Te duratio of observatios is also radom but bouded from above to a o-radom fixed umber. Now we give some eeded otatio ad defiitios. Te otatio 1, m meas te set of itegers 1, 2,..., m, ad χa) is te idicator fuctio of set A. For a fixed vector of oegative itegers a = α 1,..., α m ), we cosider te partial derivative f α) a x) = α fx) x α x αm m, f 0) 0 x) = fx) of a fuctio fx) at a give poit x R m, were α 1 + α α m = α. Deote by βk) te set of all vectors b = β 1,..., β m ) wit oegative itegervalued compoets β 1,..., β m suc tat β β m = k. Omittig te subscript b = β 1,..., β m ) of partial derivatives f k) b β 1,..., β m is ot specified. I te sequel, we will use te followig otatio: βk) x) will mea tat te set of idices meas te summatio over all b βk); we write x k = x β x βm m for vector x = x 1,..., x m ) ad k! = β 1!... β m!, were k = β β m. We suppose tat te fuctio to be estimated satisfies te followig assumptio Assumptio 1. Te fuctio fx) is p-t differetiable at a poit x 0, p 0 ad for some positive umbers C f,i, i = 0, p p 0), L f, γ f 0, 1], suc tat f i) x 0 ) C f,i, i = 0, p Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

4 4 ad for every x, y R m, f p) x) f p) y) L f x y γ f, x 2 = m x 2 j. j=1 For estimatio of te fuctio f ) at a poit x 0, f = fx 0 ) we use so-called sequetial estimators, costructed o te basis of kerel Nadaraya Watso estimators of te form N ) ) x 0 + x i 2) ˆfN = K N ) x 0 x i Y i K, N 1, were a + = a 1 for a 0; a + = 0 we a = 0; K ) is a kerel fuctio ad is a badwidt, satisfyig te Assumptios 4 ad 5 below respectively. Defie for every i 1 desity fuctios f i 1 t) of coditioal distributio fuctios F i 1 t) = P x i t F x i 1), F x i = σ{x 1,..., x i }. As regards to te regressors X ad we suppose Assumptio 2. Assume te coditioal desity fuctios f i 1 t), i 1 are q-t differetiable at a poit x 0, q 0 ad for some positive umbers c 0 x, C x,k, k = 0, q q 0), L x ad γ x 0, 1] te followig relatios if i 1 f i 1x 0 ) c 0 x, ad for every x, y R m, sup f k) i 1 x0 ) C x,k, k = 0, q i 1 sup i 1 f q) q) i 1 x) f i 1 y) L x x y γx are fulfilled. Assumptio 3. Let te oises be coditioally zero mea Eδ i F x i ) = 0, i 1 ad for some mootoically o-icreasig as k o-egative fuctio ϕk), k 1 te followig "geeral mixig coditio" for te sequece olds true Eδ i δ j F x i j) ϕ i j ), i, j 1. We ow defie admissible sets of kerels K ) ad badwidts as follows. Let T j = u α u αm m Ku) du, R m were α 1,..., α m are oegative itegers. m α i = j, Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

5 5 Assumptio 4. Let te fuctio K ) is uiformly bouded from above: sup Kz) K < z ad suc tat Kz)dz = 1, z s Kz) dz <, 0 s q + 2p + 3. Moreover, we suppose tat te itegrals T j = 0, j = 1, p + q. Assumptio 5. Let ) 1 be a sequece of mootoically decreasig to zero positive umbers suc tat m as ad 1 1. It sould be oted tat, accordig to Assumptio 4, te kerel K ) does ot ave to be oegative. For example, te ifiite-order flat-top kerels of Politis [19] are allowed, icludig te ifiitely differetiable flat-top kerel of McMurry ad Politis [16]. 3. Sequetial estimatio of te regressio fuctio f I tis sectio we cosider sequetial estimators of te regressio fuctio f at a poit x 0 wit a give variace, calculated at a special stoppig time. Tese estimators are costructed o te basis of Nadaraya-Watso kerel estimators. Badwidt givig a optimal covergece rate of estimators wic coicides wit te rate of Nadaraya-Watso estimators costructed by idepedet observatios) is foud. Accordig to sequetial approac we itroduce te followig defiitio. Defiitio 1. Let H ) 1 be a uboudedly mootoically icreasig give sequece of positive umbers ad ) 1 satisfies Assumptio 5. Te sequetial plas τ, f ), 1 of estimatio of te fuctio f = fx 0 ) will be defied by te formulae τ = if{j 1 : j ) x 0 x i K H }, 3) f = 1 τ ) x 0 x i Y i K, Q = Q τ ) x 0 x i K, were τ is te duratio of estimatio, ad f accuracy i te mea square sese. is te estimator of f wit give We ave itroduced i Defiitio 1 te sequece of sequetial estimatio plas. At te same time Teorem 1 below gives o-asymptotic properties of tese plas for every 1, i particular, upper boud for te MSE of te estimator f. It Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

6 6 gives a possibility to ivestigate asymptotic properties of defied plas as well see Sectio 3.1.). To formulate tis teorem we eed te followig otatio. Assume l ) 1 to be a arbitrary o-decreasig sequece of positive umbers. Defie te fuctio γ x,δ = 0 if processes X ad are idepedet ad 1 oterwise; γ δ = 0 if ϕk) = 0, k > 0 ad 1 oterwise; te umbers ϕ = ϕl ), K α γ) = z α z γ Kz) dz, α 0, γ 0, K 0 = K 0 0), κ = L x q! K q γ x ), C 1 = βq) C 2 = 1 p! C x,0 p j=1 βq+j) K p γ f ), βp) q+1 p+1 C 4 = 4Kp + 2) j=0 k=0 βj+2k) ν j,k = j+2k χj = q+1) χk = p+1), as well as + [C 3 C x,0 + κ q+γx w = 1 q + j! C f,jk q+j γ x )χp > 0), C 3 = C x,0 K 2 y) dy, 1 j! k!) 2 C x,jc 2 f,kk νj,k µ j,k ), C x,q+1 = 1 q! L x, C f,p+1 = 1 p! L f, 0 = if{ 1 : c 0 x κ q+γx > 0}, t = C x,0 + κ q+γx ) C 3 H + K) + c 0 x κ q+γx ) 5/2 m ) 2 c 0 x κ q+γx µ j,k = γ x χj = q+1)+2γ f χk = p+1), ) 3 + 2K 2 c 0 x κ q+γx c 0 x κ q+γx ) m ) 1 ]H + K) + H 2, ϕ0)k 2 l γ δ 1 + 3l H 2 ) + C 3 ϕ0)c 0 x κ q+γx ) K )+ H H +2γ δ ϕ0)k 0 Kc 0 x κ q+γx ) l K ) + 2γ δ K 0 Kϕ m t 2, H H b = c 0 x κ q+γx ) 1 [C 1 q+1+γx + C 2 p+γ f ] H 2 ) 1 + KH + w ad { V = 2 2C 3 Cf, γ x,δ )C 4 )c 0 x κ q+γx ) K )+ H H +1 + γ x,δ )w 2 + 2C 1 q+1+γx + C 2 p+γ f H 2 ) 2 2m t 2 }. Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

7 7 Teorem 1. Uder model 1), let te Assumptios 1-5 be fulfilled ad 0. Te te sequetial estimatio plas τ, f ) are closed, i.e. τ < a.s. ad ave te followig o-asymptotic properties: a) 1. for te expected duratio of observatios 4) C x,0 + κ q+γx b) 5) Eτ 2 t 2 ; ) 1 m H Eτ c 0 x κ q+γx ) 1 m H + K), 2. for te bias 3. for te MSE Ef f b ; Ef f) 2 V. Remark 1. From Defiitio 1 it follows tat te preseted sequetial estimators coicide wit te usual Nadaraya Watso estimators 2) calculated at a special stoppig time. It follows tat, at least i te case of idepedet iputs of te model, tese estimators ave te same asymptotic properties as Nadaraya Watso estimators see Sectio 3.1.). However, as sow i Teorem 1, sequetial estimators ave te above exact, o-asymptotic properties, tat may be importat for practitioers Badwidt coice for sequetial estimators From Teorem 1 follows, tat it is atural to defie H = m. I tis case ad uder te coditio o te badwidt m as from Assumptio 5 te mea of te duratio of observatios is proportioal, accordig to 4) to, i.e., Cx,0 1 C 1 x,0κ q+γx Eτ C x,0 + κ q+γx as well as, from 5) it follows, ad Ef f = O c 0 x) 1 + c0 x) 1 κ q+γx c 0 x κ q+γx + K m, 0, Eτ 2 c 0 x) o 2 ) as ϱ + ϕ m γ ) δ l, ϱ = q γ x ) p + γ f ), m Ef f) 2 = O 2ϱ + ϕ m γ ) δl m as. Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

8 8 From tese formulae it follows, tat for te asymptotic ubiasessy ad L 2 - covergecy of f we ave to use te sequeces l ) ad ) satisfyig te coditio 6) ϕ m + l = o1) as. Remark 2. If te regressors x i form a sequece of i.i.d.r.v s, te c 0 x = C x,0. Moreover if te umber c 0 x = f i x 0 ) is kow, we ca put H = c 0 x m. I tis case ad κq+γx c 0 x + κ q+γx were a b = mi{a, b}, as well as Eτ + κq+γx + K m c 0 x κ q+γx lim [q+γx ) 1 m ] Eτ <, Eτ o 2 ) as. It sould be oted tat i te case of i.i.d. oises δ i ) ad by q p te MSE of te costructed sequetial estimator as similar to te i.i.d. or o-radom case for te regressors x i ) optimal decreasig rate Ef f) 2 = O 2p+γ f ) + 1 ) as. m I particular, for te case p = 0, γ f = 1 see, for compariso, [8] amog oters), Ef f) 2 = O ) as. m 4. Trucated sequetial estimatio of te regressio fuctio f We sall cosider i tis sectio te problem of estimatio of te regressio fuctio f wit a kow mea square accuracy based o observatios of te process Y i, x i ) for i = 1,..., N o te time iterval [1, N] for a fixed time N. Suc possibility gives te trucated sequetial estimatio metod, developed by Koev ad Pergamectcikov [11]-[13], as well as i [7] for parameter estimatio problems i discrete ad cotiuous time dyamic systems. Defiitio 2. Let H ad be positive umbers. Te trucated sequetial plas τ N, H), fn, H)), N 1 of estimatio of te fuctio f = fx 0 ) will be defied by te formulae τ N, H) = if{j [1, N] : j ) x 0 x i K H} Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

9 9 wit te provisio tat if{ø} = N, fn, H) = ˆf N ) ) x 0 x i N, H) χ K H + ˆf N, H) = + C N f,0 2 χ ) ) x 0 x i K < H, τ N,H) 1 Y i K Q N, H) x 0 x i ) τ N,H), Q N, H) = K x 0 x i were τ N, H) is te duratio of estimatio, wic is bouded by costructio τ N, H) N) ad fn, H) is te estimator of f. Te parameters H ad will be defied i a special way as fuctios of N for optimizatio of a covergece rate of te estimator fn, H) as N i te mea square sese. Assume lh) to be a arbitrary o-decreasig fuctio of positive umbers. Defie { V N, H) = 2 2C 3 Cf, γ x,δ )C 4 ) m N γ H 2 x,δ )[2γ δ K 0 KϕlH)) m N 2 + H 2 +C 3 ϕ0) m N H 2 lh) m N + 2γ δ ϕ0)k 0 + ϕ0)k 2 lh) γ H 2 δ 1 + 3lH))]+ H2 } C 3 Cf,0 2 + Nm 4[N m c 0 x κ q+γx ) H] C 1 q+1+γx + C 2 p+γ f ) 2 2m N 2 H 2 From te proof of te followig Teorem 2 it follows tat te umber H satisfies te coditio 7) N m c 0 x κ q+γx ) H > 0. Tus it is atural to put 8) H := H N = N m NC 1 α, were C α = c 0 x κ q+γx 0 α 1/2 ) 1, α > c 0 x κ q+γx ad N ) is a sequece satisfyig Assumptio 5. It sould be oted tat, accordig to te assumptios of te teorem below te umber C α ad te sequece H N ) are assumed kow. Te, deotig l N = lh N ) ad ϕ N = ϕl N ), we ave { VN 0 := V N N, H N ) = 2 [2C 3 Cf, γ x,δ )C 4 )Cα 2 + C 3Cf,0 2 α γ x,δ ) 8 C 3 Cαϕ0) 2 + 2γ δ ϕ0)k 0 Cαl 2 1 N )] γ N m x,δ )[ϕ0)k 2 C 2 l N αγ δ N N 1 + 3l N)+ m N } )2 +2γ δ K 0 KCαϕ 2 N m N ] + 2C2 αc 1 q+1+γx N + C 2 p+γ f N ) 2. Te mai result of tis sectio is te followig 0 ) 2 ), Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

10 10 Teorem 2. Uder model 1), let te Assumptios 1-4 be fulfilled, were te umber C f,0 i Assumptio 1 is supposed to be kow. Te: 1) for every positive umbers, H satisfyig te coditio 7) te trucated sequetial estimator fn, H) as te MSE: Ef N, H) f) 2 V N, H); 2) for H = H N ) defied i 8) ad = N ) satisfyig 7) it olds Ef N N, H N ) f) 2 V 0 N; 3) assume tere exists a umber r > 2, suc tat te badwidt = N ) N 1 from Assumptio 5 satisfies te additioal coditio 9) N 1 1 mr 1) N N < r/2 ad H = H N ) is defied i 8). Te lim N τ N N, H N ) N < 1 a.s. ad te estimator f N N, H N ) is o-degeerate as N i te followig sese: P f N N, H N ) = ˆf N N, H N )) = 1 for N large eoug. Remark 3. From te tird assertio of Teorem 2 it follows, tat similarly to sequetial estimators, te trucated sequetial estimators ave te asymptotic properties of Nadaraya Watso estimators 2). However, i distictio from te sequetial estimators cosidered i Sectio 3., te trucated estimators ave kow variace for fixed sample sizes. Remark 4. If te umber C f,0 i Assumptio 1 is ukow, te te Defiitio 2.2 of sequetial estimator is modified as follows: fn, H) = ˆf N ) ) x 0 x i N, H) χ K H. All assertios of Teorem 2 will be fulfilled wit sligtly caged but avig similar structure) fuctios V N, H) ad VN 0. More precisely, te umber 4 i te deomiator of te last summad i te defiitio of V N, H) sould be omitted ad te umber 8 i te deomiator of te tird summad i te rigt ad side i te defiitio of VN 0 sould be caged to te umber Badwidt coice for trucated sequetial estimators From Teorem 2 it follows, tat by te defiitio of VN 0 ad assertio 2) of Teorem 2, te trucated sequetial estimator fn N, H N ) as similar to te sequetial estimator 3) rate of covergece as N ). Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

11 11 I te case of i.i.d. oises δ i ) te MSE of te costructed sequetial estimator as followig decreasig rate EfN N, H N ) f) 2 = O 2ϱ N + 1 ) as N. N m N Te te badwidt wit te optimal rate is proportioal to N N 1 2ϱ+m) ad te coditio 9) is fulfilled for ϱ > m/2 ad r > 4ϱ 2ϱ m. I particular, for te case p = 0, γ f = 1, m = 1, EfN N, H N ) f) 2 = O 2 N + 1 ) N N ad r > 4. Remark 5. Teorems 1 ad 2 give kow upper bouds for te bias ad te MSE of preseted estimators as well as for te mea of observatio time i Teorem 1 if te parameters of classes of fuctios, itroduced i Assumptios 1 ad 2 are kow C f,k, p, γ f, etc). I tis case optimal badwidt ca be foud from miimizatio of te upper bouds for te MSE s. At te same time te assertios of Teorems 1 ad 2 are fulfilled eve if some of tese parameters are ukow. Estimators wit suc properties ca be successfully used i various adaptive procedures as pilot estimators see, e.g., [28]- [32]). 5. Examples I tis sectio examples of various depedece types of regressors ad oises of te model 1) are give Case of idepedet regressors X ad Cosider examples of iputs of te model i tis case. We cosider a scalar case m = 1 for simplificatio oly. Te extesio to te multivariate case is immediate Example of regressors X. Assume tat regressors x i ) satisfy te followig equatio 10) x i = Ψx i 1,..., x i r ) + ε i, i 1, were Ψ ) is a bouded fuctio Ψy) Ψ <, y R r, r 1. Assume tat te iput oises ε i i te model 10) are i.i.d. wit te desity fuctio f ε ). I tis case f i 1 t) = f ε t Ψx i 1,..., x i r ) F i 1 ). Tus Assumptio 2 olds true if te desity fuctio f ε ) is q-t differetiable, q 0 ad for some positive umbers c 0 x, C x,0, L x ad γ x 0, 1] te followig relatios if f ε t) c 0 x, t x 0 Ψ Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

12 12 ad for every x, y R 1, sup t x 0 Ψ f ε i) t) C x,i, i = 0, q f q) ε x) f q) y) L x x y γx ε are fulfilled Example of oises. Assume tat oises δ i ) satisfy te followig autoregressive equatio 11) δ i = λδ i 1 + η i, i 1, wic is supposed to be stable, λ < 1, ad η i are i.i.d., Eη i = 0, Eη 2 i σ 2 η, i 0; η 0 = δ 0. Te ϕk) = λ k σ 2 η 1 λ 2. I tis case we ca put after optimizatio of te upper boud of te MSE) ad te summads l log, l 1 + µ) log λ 1, µ > 0 ϕ 1 ) 1 1 = o. 1+µ Te te coditio 6) wic is ecessary for trucated estimators as well) is fulfilled if log = o1) as. It is clear, tat tis example ca be easy geeralized for te stable autoregressive process 11) of a arbitrary order Case of depedet regressors X ad Cosider te estimatio problem i te oliear autoregressive model Y i = fy i 1 ) + δ i, i 1, were δ i, i 1 form te sequece of zero mea i.i.d.r.v s wit a desity fuctio f δ ) ad fiite variace Eδi 2 = σ 2, i 1; Y 0 is a zero mea radom umber wit fiite variace ad idepedet from δ i, i 1. Te x i = Y i 1, i 1 ad ϕk) = σ 2 χk = 0), k 0, as well as i tis case f i t) = f δ t fy i 1 )) Fi 1), Y Fi Y = σ{y 0, δ 1,..., δ i }. Tus Assumptio 2 olds true if te fuctio f ) is uiformly bouded sup ft) C f, te desity fuctio f δ ) is q-t differetiable, q 0 ad for t R 1 some positive umbers c 0 x, C x,0, L x ad γ x 0, 1] te followig relatios if f δ t) c 0 x, t x 0 C f Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

13 13 ad for every x, y R 1, sup f i) δ t) C x,i, i = 0, q t x 0 C f f q) δ x) f q) y) L x x y γx δ are fulfilled. 6. Coclusio Te sequetial approac for te problem of estimatio of a multivariate regressio fuctio from depedet observatios is developed. It is supposed, tat te fuctio to be estimated belogs to te Hölder class ad iput processes of te model ca be ubouded wit a positive probability. Two types estimators are preseted. Sequetial estimators give te possibility to get estimators wit a arbitrary mea square accuracy by fiite stoppig time. Trucated sequetial estimators ave kow variace ad costructed by a sample of a fiite fixed) size. Bot estimatio procedures work uder geeral depedecy types of model iputs. Asymptotic rate of covergece of bot estimators coicides wit te optimal rate of Nadaraya Watso estimators costructed from idepedet observatios. At tat we cosider te mea Eτ avig te rate 4) as a duratio of observatios i sequetial approac for compariso wit Nadaraya Watso estimators calculated by te sample of te size. Preseted estimators ca be used directly ad as pilot estimators i various statistical problems. Refereces 1. Er-Wei Bai, Yu Liu, Recursive Direct Weigt Optimizatio i Noliear System Idetificatio: A Miimal Probability Approac // Automatic Cotrol, IEEE Trasactios o Issue Date: July 2007 Volume: 52 Issue: 7 P Borisov V.Z., Koev V.V. O sequetial estimatio of parameters i discrete-time processes // Automat. ad Remote Cotrol Vol. 38. No. 10 P i Russia). 3. Cramér H. Matematical metods of statistics. Priceto Uiv. Press, Darmasea, S, Zeepogsekul, P ad De Silva. Two stage sequetial procedure for oparametric regressio estimatio // Australia ad New Zealad Idustrial ad Applied Matematics Joural Vol. 49. P. C699-C Efroimovic S.Yu. Noparametric curve estimatio. Metods, teory ad applicatios. Berli, N. Y.: Spriger-Verlag, Efroimovic Sam. Sequetial Desig ad Estimatio i Heteroscedastic Noparametric Regressio // Sequetial Aalysis: Desig Metods ad Applicatios Vol. 26. No. 1. P Autor: Sam Efromovic DOI: / Fourdriier, D., Koev, V. ad Pergamescikov, S. Trucated sequetial estimatio of te parameter of a first order autoregressive process wit depedet oises // Matematical Metods of Statistics Vol. 18. No. 1. P L.Györfi, M.Koler, A.Krzyżak, H.Walk A distributio-free teory of oparametric regressio. Berli, N. Y.: Spriger-Verlag, T. Hodaa. Sequetial estimatio of te gargial desity fuctio for a strogly mixig process // Sequetial Aalysis: Desig Metods ad Applicatios Vol. 17. No. 3,4. P DOI: / Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

14 Koev, V.V. Sequetial parameter estimatio of stocastic dyamical systems. Tomsk: Tomsk Uiv. Press, 1985 i Russia). 11. Koev, V.V. ad Pergamescikov, S.M. Trucated sequetial estimatio of te parameters i radom regressio // Sequetial aalysis Vol. 9. No. 1. P Koev, V.V. ad Pergamescikov, S.M. O trucated sequetial estimatio of te driftig parameter mea i te first order autoregressive model // Sequetial aalysis Vol. 9. No. 2. P Koev, V. V. ad Pergamescikov, S. M. O Trucated Sequetial Estimatio of te Parameters of Diffusio Processes // Metods of Ecoomical Aalysis. Cetral Ecoomical ad Matematical Istitute of Russia Academy of Sciece, Moscow P U. Kücler ad Vasiliev, V. O guarateed parameter estimatio of a multiparameter liear regressio process // Automatica, Joural of IFAC. Elsevier No. 46. P Liptser R.S., Siryaev A.N. Statistics of radom processes. I: Geeral teory. N. Y.: Spriger-Verlag, : Applicatios. N. Y.: Spriger-Verlag, T. McMurry ad D. N. Politis. Noparametric regressio wit ifiite order flat-top kerels // J. Noparam. Statist Vol. 16. No P A.V. Nazi ad L. Ljug. Asymptotically Optimal SMootig of Averaged LMS for Regressio // Proceedegs of te 15t IFAC World Cogress, Barceloa, July, Novikov A.A. Sequetial estimatio of te parameters of te diffusio processes // Teory Probab. Appl Vol. 16. No. 2. P i Russia). 19. D.N.Politis. O oparametric fuctio estimatio wit ifiite-order flat-top kerels // Probability ad Statistical Models wit applicatios, C. Caralambides et al. Eds.), Capma ad Hall/CRC, Boca Rato P D.N.Politis, J.P.Romao. O a Family of Smootig Kerels of Ifiite Order i Computig Sciece ad Statistics // Proceedigs of te 25t Symposium o te Iterface, Sa Diego, Califoria, April 14-17, M. Tarter ad M. Lock, eds., Te Iterface Foudatio of Nort America, P Dimitris N. Politis ad Josep P. Romao, Multivariate desity estimatio wit geeral flat-top kerels of ifiite order // Joural of Multivariate Aalysis Vol. 68. P D. Politis, J.Romao ad M. Wolf. Subsumplig. Spriger Verlag, Berli, Heidelberg, New York, Prakasa Rao, B.L.S. Noparametric Fuctioal Estimatio. New York: Academic Press, J. Roll, A. Nazi, ad L. Ljug. A No-Asymptotic Approac to Local Modellig // Te 41st IEEE CDC, Las Vegas, Nevada, USA, Dec Regular paper.) Predocumetatio is available at ttp:// 25. J. Roll, A. Nazi, ad L. Ljug. No-liear System Idetificatio Via DirectWeigt Optimizatio // Automatica, Special Issue o Data-Based Modellig ad System Idetificatio, V.41, No.3, 2005, pp IFAC Joural. Predocumetatio is available at ttp:// 26. M. Roseblatt, Stocastic curve estimatio // NSF-CBMS Regioal Coferece Series, Istitute of Matematical Statistics, Hayward, 1991, Vol D. W. Scott, Multivariate desity estimatio: teory, practice ad visualizatio. Wiley, New York, Vasiliev V.A. O Idetificatio of Dyamic Systems of Autoregressive Type // Automat. ad Remote Cotrol Vol. 58. No 12, P Vasiliev V.A. O adaptive cotrol problems of discrete-time stocastic systems // Proceedigs of te 18t World Cogress Te It. Fed. Autom. Cotr., Mila, Italy, August 27 - September 2. 6 pages to be publised). 30. V.A.Vasiliev, A.V.Dobrovidov ad G.M.Koski. Noparametric estimatio of fuctioals of statioary sequeces distributios. Moscow.: Nauka, 2004, 508 p. i Russia) 31. Vasiliev V.A., Koev V.V. O Optimal Adaptive Cotrol of a Liear Stocastic Process // Proc.15t IMACS World Cogress o Scietific Computatio, Modellig ad Applied Matematics, August 24-29, 1997, Berli, Germay. Vol. 5. P Vasiliev V.A., Koski G.M. Noparametric Idetificatio of Autoregressio // Proba- Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012

15 15 bility Teory ad Its Applicatios Vol. 43. No. 3. P Wald A. Sequetial aalysis. N. Y.: Wiley, Proceedigs of teix Iteratioal Coferece System Idetificatio ad Cotrol Problems SICPRO 12, Moscow,30Jauary - 2February, 2012