No-asyptotic sequetial cofidece regios with fixed sizes for the ultivariate oliear paraeters of regressio Adrey V Tiofeev Abstract I this paper we cosider a sequetial desig for estiatio of o-liear paraeters of regressio with guarateed accuracy No-asyptotic cofidece regios with fixed sizes for the least squares estiate are used The obtaied cofidece regio is valid for a fiite ubers of data poits whe the distributios of the observatios are ukow Key words: o-asyptotic oliear estiatio, sequetial aalysis, cofidece regios, guarateed accuracy AMS 99 subect classificatio: 6H, 6, 6F5 Itroductio I this paper we will ivestigate soe o-asyptotic properties of the least squares estiates for the o-liear paraeters of regressio Of late, there has bee cosiderable iterest i o-liear odels i differet practical fields, especially i ecooics The ai reaso for this iterest is that liear odels, while havig a wide variety of practical applicatios, did ot quite eet the expectatios The asyptotic properties of oliear least squares estiates are well ivestigated ad discussed (Jerich[], ug [], ai [3], Aderso ad Taylor [4], Wu [5], Hu [6], Skouras [7]) At the sae tie, a few results addressig the fiite saple properties exist, whereas the o-asyptotic solutio for the proble of the paraeter estiatio for regressio is practically iportat because the saple volue is always liited fro above No-asyptotic estiatio of scalar paraeter of o-liear regressio by eas of cofidece regios was exaied by Tiofeev [3] Siilar estiatio of ultivariate paraeter was researched by Tiofeev [4] I this paper a sequetial desig is suggested that will ake it possible to solve the proble of o-liear estiatio of ultivariate paraeter for regressio by eas of cofidece regios i the o-asyptotic settig As opposed to the ethod suggested by Tiofeev [4], the solutio preseted i this paper does ot use explicit expressios for the gradiets of the loss fuctio The solutio was obtaied uder coditio of partial a priori defiiteess as regard to the stochastic distributio of the observatios The ea observatio tie was estiated i the suggested sequetial desig Stateet of the proble A stochastic process X ( k) k X is adheres to the followig equatio X ( k) A( k, ) ( k), k, ()
where X( k ), Ak (, ), ( k) R The oliear fuctios A( k, ) k sequece ( k) k with ukow distributio is such that T k : E ( k ), E ( k ) ( k ) ( k ), are defied A vector where ( k) k is a kow sequece of o-rado diagoal atrixes that ca be described as follows: k : ( k) diag l ( k),, l ( k), K : l ( k) K A ukow paraeter belogs to a closed ball ebedded i a -diesioal Euclidea space R et =,, R, :, cofidece parallelepiped P, c be a set of required diesios of the is the required value of the cofidece coefficiet We eed to develop a sequetial desig for cofidece estiatio of the paraeter that would deterie the stochastic stoppig tie, a rule of buildig a cofidece rectagular parallelepiped ( ) i the copact, that eet the followig coditios: P ( ) P c, P ( ),, ( ) :, Fro ow o stads for a or of the space i which the copact is ebedded a fro i ow o stads for the i-th copoet of the vector a The brackets will be oitted whe o abiguity arises For the sake of clarity, the followig shorthad otatio will be used throughout the rest of the paper: P( istead of P ( ad E( istead of E ( 3 Solutio ethod et us assue that stochastic vector fuctios A( k, ) k o eet the followig coditio:,, : k, () k R
3 k T, (, ) (, ) ( ) (, ) (, ), R, R et R A k A k k A k A k be a sequece of estiators of the paraeter which is defied as follows: k : Arg If I(, ) (3) t( ) T k where I(, ) X ( k) A( k, ) ( k) X ( k) A( k, ) ( t( ) ) Here {t()} N, N ={,,,}, li t( ), ad a closed set sequece ( ),,,, I(, ) I(, ), I(, ) so that, ( ) :,, c( ) R For each defie a fuctioal Here c( ) is a kow sequece of o-stochastic fuctios For each defie such eleets (), () that, ( ) : ( ) ( ) The sequece of sets ( ) is such that : ( ), ( ) ( ) The sequetial desig for cofidece estiatio of the paraeter ((), ) where, if ( ) : ( ) ( ) ( ) will be regarded as a pair The properties of the sequetial desig are described by the followig theore: Theore Assue that the followig stateets are true: ), ; ) the sequece A( k, ) k eets the coditio (); 3) with probability k ], [ :, :
4 A( k, ) A( k, ) l ( k), is a or i 4 4 4) P k E k l k k ( ) ( ) ; 5) the sequece of stochastic fuctios Rk (, )( ) k R ; coverges to a liitig fuctio R(, ) o whe is fixed, i additio R(, ) is a cotiuous fuctio o ad R(, ) > whe ; 6) t( ) r,, a - itegral part of the а, r > Further, if the sequece {c() } is such that 5 4q r / : ( ), 5 ( Pc ) 6 c the followig assertios hold true: ) : P ( ) Pc ; ) : P The proof of the Theore appears i Appedix It is ecessary to say that whe t ( ), the obtaied cofidece regio will have a a priori fixed size which is deteried by set The coditios 3), 4) ad 5) are used i the proof of the secod stateet of the Theore oly 4 Evaluatio of the average observatio tie i the cosidered sequetial desig For practical purposes it is iportat to evaluate E - the average observatio tie for the sequetial desig ( ), et us evaluate If, where E fro above et us set i i =,, R For soe z ], [ defie the followig value: where the costats sg, ], [ are such that where if s g( / ) c( ) z,, ', s : G,, ' g ' as (4) G,, ' t( ) Rk, ' t( ), k Theore Assue that the coditio )-4) of the theore are et, besides there are kow values П >, p >, W> ad
5 E p sup ( (,, )), 4, t( ) p W, where T, (,, ) ( k) ( k) A( k, ) A( k, ) / t( ) k I this case, if the coditio (4) holds true, it is safe to assue that P ) ) 4 E Wz The proof of the Theore appears i Appedix (,, ) R 5 Discussio The ai idea of the ethod is based o research of the behavior of a loss fuctio I (, ) i the iiu eighbourhood For each t ( ) the size of the cofidece set for the oliear paraeter is deteried by the value c() ad by the sesitivity of the loss fuctio I (, ) to the variatios of this paraeter The assuptio of existece o the copact of the ipschitz costat for fuctios Ak (, ) k is ot a strog coditio, ad it is et for ost o-liear fuctios that are used i regressio aalysis Siilar assuptios were used by Wu [5] ad Skouras [7] i a proof of the strog cosistecy of least squares estiates i oliear regressio odels The bigger the followig values are (q - size of the copact, K - the upper boud of the oise dispersio values, P c - cofidece coefficiet ad - the ipschitz costat), the bigger the value c ( ) is, ad therefore, the bigger the size of the cofidece set ( ) foud for a particular t ( ) is The coditio (4) is a requireet of a strogly covex fuctio 5 G,, ', o the copact with a covexity paraeter (gq) 5, which follows fro the strog covex variat of the Weierstrass Theore paraeter g is, the ore the sesitivity of the fuctio The ore the absolute value of the G,, ' to the variatios of the paraeter is, ad the less saple volue t ( ) ca be used i order to fid a cofidece set of give diesios et us cosider a very siple exaple Skouras [7] cosiders the followig odel y( t) exp( x( t)) ( t), where [, ],, { xt ( )} is a sequetial of bouded positive regressors I additio, we assue that Px( t) [A,B], 4 t : ( t), ( t), ( t) 4 E E E,{ xt ( )} are idepedet rado variables ad there exists a costat Z such that, [, ]: E exp( x( k)) exp( x( k)) Z I this case, R (, ) exp( x( k)) exp( x( k)), k
6 t( ) (,, ) ( k) exp( x( t)) exp( x( t)) t ( ) k Accordig to Skouras [7], for every, [, ] there exist costats c, c such that c x( t) exp( x( t)) exp( x( t)) c x( t) c B It is easy to see that E exp( x( k)) exp( x( k)) Z Z k k kk Thus usig Kologorov's strog law of large ubers (Feller [5]) we have x k x k E x k x k R k k Here fuctio R(, ) has o [, ] the ipschitz costat c B li exp( ( )) exp( ( )) li exp( ( )) exp( ( )) (, ) cotiuous fuctio o Besides, R(, ) > whe Further, 6, ad therefore is a P k k E ( k) / 6 ; 4 4 ad the assuptio 4) of Theore are et Further, the ext iequality is obvious t( ) k k G,, ' R, ' t( ) c x ( t) c A ad the coditio (4) for the process yt () holds true The, [, ]: E (,, ), ad we ca use the theore of Dharadhikari ad Jogdeo [5] to get the iequality t( ) 4 3 4 4 4 4 4 4 4 4 k E (,, ) R(4) t ( ) c B R(4) t ( ) c B Here p3 ( p)/ r R( p) p( p )ax(, ) K, K The itegral value is p r ( r)! such that p I this case we have: W / 6 4 4 4 4c R(4) B ad Thus we have got the costats,w fro Theore coditios Now we have checked all the assuptios of the theores ad for the cosidered saple odel 6 Siulatio study The followig odel was used for the siulatio study: y( t) H exp( x( t)) ( t) Here () t is a sequece of the idepedet rado variables distributed by N(,), ad () xt is a sequece of the idepedet rado variables evely distributed o the iterval [,], ad
7 P ad the required size of the cofidece iterval are fixed at 95 ad 5, respectively I this case if r=, the followig is [,] Cofidece coefficiet c true: : c( ) 4 H 3 5 The observatio was stopped at rado oets if ( ) ( ), where ( ) is the upper boud of the cofidece iterval, ( ) is the lower boud of the cofidece iterval Thus, at the oet whe observatio was stopped, the required value of the cofidece iterval for the paraeter was achieved The estiate for the average observatio tie ( ) i the sequetial desig has bee defied as the ea tie value i the realizatios of the siulatio experiet groups Every siulatio experiet group has bee carried out for a fixed uique pair of the odel paraeters H, ad cosisted of the series For every experiet group the lower boud of the cofidece iterval ( ( )) ad upper boud of the cofidece iterval ( ( )) were deteried The results are preseted i the table As show, ( ) depeds o values of the paraeters H ad to a sigificat degree Paraeter H deteries the sigal to oise ratio for the observed saple Therefore, if is fixed, bigger values of the H correspod to saller values of the ( ) ad vice versa: saller values of the H correspod to bigger values of the average observatio tie ( ) This proves a assuptio that ight see obvious: with fixed P c ad to get the required size of the cofidece iterval with bigger sigal to oise ratio we eed a saller volue of the saple as opposed to a saple with a saller sigal to oise ratio O the other had, with fixed P и H the depedece betwee values ad ( ) is deteried by c paraetric sesitivity of the fuctioal I (, ) i the evelope of the poit Followig [6] we have: the paraetric sesitivity of the fuctioal I (, ) is characterized with a coefficiet sesitivity k (, ) which is defied as a derivative of the fuctioal I (, ) with respect to Reasoig iforally, we ca see that the depedece value ( ) o is obvious eough: the bigger is the value of the fuctio k (, ) i the evelope of the, the saller is the value of the average observatio tie ( ) I other words, the bigger sesitivity of the fuctioal I (, ) to variatios of the paraeter i the evelope of the is, the saller saple volue will be eeded to desig of the cofidece iterval with a give size ad fixed value of the cofidece coefficiet P c 7 Coclusio rearks I this paper we derived o-asyptotic cofidece regios with fixed sizes for the S estiatio of the ultivariate paraeter The solutio is based o sequetial aalysis with o iforatio about the distributio fuctio of observatios The cosidered ethod ca be used for oasyptotic estiatio of the ultivariate paraeters of the wide class of the oliear regressios The results of the siulatio study proved that the suggested ethod for cofidece estiatio works
8 Table Cofidece itervals of the paraeter H ( ) Average lower boud Average upper boud 8-8 7 5 8 6 5 7 5 56 6 55-3 65 6 5 5 5 7 5 8 3 8 Appedix Proof of Theore The proof uses the followig facts: ea [7] et {I ()} be a sequece of stochastic fuctios coplyig to the coditios: ) for each the sequece {I ()} is cosistet with a odecreasig flow of - subalgebras {F } ; ) for each whe is fixed the followig relatio holds true: P I li ( ) (, ), where (, ), is a real valued fuctio which is cotiuous o ad such that (, ) (, ), 3) whe is fixed, the fuctio I ( ), is cotiuous o ;
9 4) for ay > there exists > ad a fuctio с(), >, с(),, such that for ay ad ay < < the followig relatio holds true: P li sup I( ) I( ) c ( ), Further, if is a copact the where P, li arg If I ( ), is a or of the space i which the copact is ebedded ea Assue that the coditios -5 of the cosidered theore are et, the Proof of ea The represetatio P P li ( ), li ( ) t( ) T (, ) k (, ) ( ) ( ) (, ) (, ) k I R k k A k A k T t( ) k k ( k) ( k) ( k) t( ) R (, ) t( ) t T T ( k) ( k) ( k) t( ) ( ) ( k) ( k) A( k, ) A( k, ) t( ) t( ) k k is obvious The properties of the atrixes { } ake it easy to observe that ad t( ) E Cosider a process z li ( ( k) l ( k)) (t() ) k t( ) ( ) t( ) T li ( k) ( k) ( k)( t( ) ) as k t( ) (5), where t( ) T t( ) ( ) ( ) ( ) (, ( ), ) (, ( ), ) k z k k A k x k A k x k
Due to the fiiteess of the set, there exists a fiite value q sup, It is obvious that for ay, the followig represetatio is possible: E E ( t( ),, ) : ( t( )) l ( A( t( ), ) A( t( ), ) ) A( t( ), ) A( t( ), ) ( t( )) l q It is easy to observe that k q k Therefore, for ay, ( k) k E A( k, ) A( k, ) k l ( k) By strog law of large ubers, we have:, : P li t( ) z t ( )( ) (6) Takig ito accout (5), (6) ad the coditio 5 of the theore beig proved,, : P I R li (, ) (, ) Hece, the coditio of ea is et It follows fro the coditio 3 ad Radeachers Theore [9] that the fuctio Ak, k,, (, ) is cotiuous o with probability Therefore, usig the theore o cotiuity of a coposite fuctio, we coclude that for ay fiite the fuctio I (, ) is cotiuous o ad the coditio 3 of ea is et et us set t( ) G,, Rk, t( ), k Further, takig ito accout the coditio 3 ofthe cosidered theore, A( k, Supl ( k) K ad ( ) t G(,, ) A( k, A( k, A( k, k l ( k) t( ) l ( k) t( ) t( ) t( ) A( k, A( k, A( k, k l ( k) l ( k)
( k, ( k, ( k, t( ) A A A K K q t( ) k l ( k) l ( k) For ay, whe is fixed, applyig Taylor s expasio to the fuctio G(,, ) about the poit Where,, D is a reaider ter ad,,,,,, D,, G G (7) Kq D Further, (7) allows us to coclude that Applyig Jese s iequality, :,,,, K q G G ad therefore, The fial coclusio is Cosider the followig evets: 5,,, : G,, G,, K q (8) 5 t( ) T k F : B(,,, ) ( k) ( k) A( k, ) A( k, ) ( t( ) ), F B, : (,,, ) where B(,,, ),,,, G G, K q 5 Takig (8) ad (6) ito accout for ay,,, P li li, ie for ay > we ca use the followig the followig represetatio:
P li sup I(, ) I(, ) li sup (,,, ) B t( ) T ( k) ( k)( A( k, ) A( k, )) /( t( ) ) k Give (9), the coditio 4 of the lea is et with the fuctio c() = Therefore, for the sequece I(, ) all the coditios of the lea are et, so Further, P li = : P li c ( ) Cosider a arbitrary sequece of the eleets of the copact such that : ( ), For the sequece there exists a real positive sequece which ca be described as follows: : ( ) arg If (,, ) (9) () () It is obvious that : ( ): ( ) P c () et us check whether the coditios ad for of the lea are et for the sequece of fuctioals (,, ) (it is obvious that the rest of the coditios are et) Takig ito accout () ad (), Besides, P, : li P li (,, ) R(, ) R(, ) R(, ), where the fuctio R(, ) has the oly iiu o at = ad is cotiuous Therefore, the coditio of the lea is et For ay, (,, ) (,, ) I(, ) I(, ) Takig (8) ito accout, for ay,
3, : P li sup (,, ) (,, ) I this case for the sequece (,, ) fuctio c ( ) based o the lea the coditio 4 of the lea is et with the P li ( ) Due to the arbitrary choice of the sequece ( ) P li ( ), P li ( ) Hece, the lea is prove Fro the strog cosistecy of the sequeces ( ) ad ( ) it follows that is a ed poit which eas that the secod assertio of the cosidered theore holds true Whe a assertio x iplies a assertio y, we will use the followig otatio: x y If a evet () leads to a evet (), we will write () () Further, for ay,, cosider a value It is obvious that t( ) T (,, ) ( k) ( k) A( k, ) A( k, ) t( ) k t( ) ( k) A( k, ) A( k, ) E k l ( k) l ( k) t( ) E (,, ) 4 t( ) t( ) ( k) A( k, A( k, 4E 4 E Rk (, ) ( t( ) ) k l ( k) l ( k) t( ) E k Further, takig ito accout the coditio 3 of the cosidered theore, t( ) k 4 E Rk (, ) ( t( ) ) 4 q ( t( ) ) For the sake of brevity, the followig shorthad otatios will be used: (, ) E (,, ), (, ) (,, ) (, ), Further, for ay (,, ) ( ) (, ) (, ) ( ) P c P c (, ) E (,, ) ( ) (, ) ( ) E (,, ) P c P c
4 Further, : E (,, ) 4 q ( t( ) ) However, accordig to yapuov s iequality : E (,, ) E (,, ) so 5 5 : E (,, ) 4 q ( t( ) ) q t( ) Further, 5 5 r r 5 r / t( ), That is why 5 5 c : E (,, ) q t( ) c( ) P / 6 / It is obvious that P 5 < и 5 c /6 <, so ad for t c( )/ E (,, ) we have Further, : E (,, ) c( )/ 5 (,, ) ( ) (, ) ( ) (,, ) P c P c E = P (, ) c( ) / t P (, ) c( ) / : P (,, ) c( ) P (, ) c( ) / Chebyshev s iequality allows us to say that E (, ) ( ) / 4 ( ) (, ) 4 ( ) E (,, ) 6 ( ) ( ( ) ) P c c c c q t ad : P (,, ) c( ) 6 c ( ) q ( t( ) ) (3) The followig represetatio is possible: (,, ) G,, G,, (,, ),, If = the oly if G,, G,, ad the paraeter does ot belog to the set () is such that (,, ) I I G G c (4) (, ) (, ),,,, (,, ) ( )
5 Cosider a situatio whe I I c (, ) (, ) ( ) I this case it is obvious that (,, ) c( ) because sig I I sig G G I other words, if coclusios hold true: sig (5), Takig (3) ito accout, (,, ) c( ) (, ) (, ),,,, the ( ) ad o the set ( ) the followig : (,, ) c( ) ( ) ( ) (5) : P (,, ) c( ) P ( ), : P ( ) P (,, ) c( ) 6 c ( ) q ( t( ) ) Cosider ow the probability of the evet : ( ) ot happeig P ( ) P ( ), P ( ) 6 c ( ) q ( t( ) ) r r r 6 q c ( ) t ( ) ( Pc) t ( ) ( Pc) 6 6 r r ( Pc) Pc 6 Here we have take ito accout that 6 Hece, P ( ) P c, ad we have proved the first stateet of the Theore It is obvious that the rectagular parallelepiped () is cofidetial for the paraeter It also adheres to the stateet of the proble ad is fully deteried by the vectors ( ), ( ) fidig which costitutes the realizatio of the suggested sequetial desig ( ), Proof of Theore Takig ito accout that G,,,, : I(, ()) I(, ) G,, () G,, (,, ()) G,, () (,, ()) I a siilar aer, : I(, ()) I(, ) G,, () (,, ())
6 et us set Here Sice we have H () () Cosider the followig evets: (, ) : I(, ) I(, ()) g( / ) (, ) : G,, () c( ) (,, ) s(,, ()) g( / ) (, ) : I(, ) I(, ()) g( / ) (, ) : G,, () c( ) (, (), ) s(,, ()) g( / ) (4, ) : () ( / ) (4, ) : () ( / ) (, ) : H ( ) / s(,, ())=c( ) G,, () (,, ()), s(,, ())=c( ) G,, () (,, ()) G,, () (,, ()) ad G,,, () (,, ()) s(,, ())<c( ),s(,, ())<c( ) (6) Because of true: () () () () H the followig iplicatios are () ( / ) & () ( / ) () () ( ) /, () () () () H () () ( ) / () () ( ) / () () ( ) / ( ) / H ( ) / () () ( ) () () Further takig (4) ito accout, (, ) (4, ), (, ) (4, ) (7) sig the triagle iequality ad takig (7) ito accout,
7 (, ) (, ) (4, ) (4, ) (, ) (5, ) : () () ( ) / (6, ) : () () (7, ) : Assue the followig otatio for the evets: (8, ) : (,, ()) s(,, ()) ( / ) ( ) g c (8, ) : (,, ()) s(,, ()) ( / ) ( ) g c (9, ) : (,, ()) c() ( / ) ( ) g c (9, ) : (,, ()) c() ( / ) ( ) g c (9, ) : (,, ()) > ( / ) ( ) g c (9, ) : (,, ()) > ( / ) ( ) g c sig Boole s iequality ad (6), : P (, ) (, ) P (8, ) (8, ) P (9, ) (9, ) P P P P (8, ) (8, ) (,, ()) >z (,, ()) >z (8) Takig the coditio of the cosidered theore ito accout: 4 4 p ;, ' : (,, ') sup ( (,, E E ')), Further, usig Chebyshev s iequality P z E z z (9) It follows fro (8) that for ay ], [ 4 4 p 4 ;, ' : (,, ') (,, ') P (, ) (, ) P P, I this case, usig (8) ad (9) whe P P (,, ()) >z P (,, ()) >z z p 4 Ad takig the coditio of the theore ito accout we ca say that E P z Wz 4 4 p Hece, the secod stateet of the theore is proved It also follows that P
8 sig the Borel-Catelli lea, we ay coclude that P ad the first stateet of the theore is prove Refereces R I Jerich, Asyptotic properties of o-liear least squares estiators, A Math Statist 4 (969) 633-643 ug, Syste Idetificatio Theory for ser Secod editio Pretice Hall, pper Saddle River, NJ d editio, 999 3 T ai, Asyptotic Properties of Noliear east Squares Estiates i Stochastic Regressio Models The Aals of Statistics, (4) (994) 97-93 4 TW Aderso ad J Taylor, Strog cosistecy of least squares estiatio i dyaic odels The Aals of Statistics, 7 (979) 484-489 5 CF Wu Asyptotic theory of oliear least squares estiatio The Aals of Statistics, 9 (98) 5-53 6 I Hu Strog cosistecy of Bayes estiates i stochastic regressio odels Joural Multivariate Aalysis, 57 (996) 5-7 7 K Skouras, Strog Cosistecy i Noliear Stochastic Regressio Models The Aals of Statistics, 8(3) () 87-879 8 AV Tiofeev, No-asyptotic solutio of cofidece-estiatio paraeter task of a o-liear regressio by eas of sequetial aalysis, Proble of Cotrol ad Iforatio Theory, (5) (99) 34-35 9 AV Tiofeev, No-asyptotic cofidece estiatio of oliear regressio paraeters: a sequetial aalysis approach, Autoatio ad reote cotrol (Auto reote cotrol), vol 58 (),o, (997) 6-66 W Feller, The Strog aw of arge Nubers, 7 i A Itroductio to Probability Theory ad Its Applicatios, Vol, 3rd ed New York: Wiley, 968, pp 43-45 SW Dharadhikari, K Jogdeo, Bouds o oets of sus of rado variables A Math Statist 4, (969), 56 59 A Ya, Dorogovtsev, The Theory of Estiates of the Paraeters of Rado Processes, Vyshcha shkola, Kiev (Russia), 98 3 AN Shiryaev, Probability, secod ed, Spriger, New York, 996 4 H Radeacher, ber partielle ud totale Dierezierbarkeit vo Fuktioe ehrerer Variabel ud uber die Trasforatio der Doppelitegrale Math A 79 (99) 34-359 5 Dharadhikari, SW, Jogdeo K Bouds o oets of sus of rado variables// A Math Statist969 V 4 P56 59 6 Rosewasser, E, Yusupov, R, Sesitivity of autoatic cotrol systes, CRC Press,
9