Common Trends in European School Populations

Transcription

1 Common rends in European Shool Populations P. Sebastiani 1 (1) and M. Ramoni (2) (1) Department of Mathematis and Statistis, University of Massahusetts. (2) Children's Hospital Informatis Program, Harvard University Medial Shool. Keywords: Autoregressive models, model-based lustering, Bayesian model seletion, temporal data. Abstrat. his paper uses a novel Bayesian lustering method to ategorize the temporal evolution of the share of population partiipating in tertiary/higher eduation in 14 European nations. he method represents time series as autoregressive models and applies an agglomerative lustering proedure to disover the most probable set of lusters desribing the essential dynamis of these time series. o inrease effiieny, the method uses a distane-based heuristi searh strategy. his lustering method partitions the evolution of shool population into three groups, thus revealing signifiant differenes among tertiary/higher eduation in the 14 European nations. 1. Introdution he time series in Figure 1 desribe the evolution of the share of population enrolled in higher eduation in 14 nations of the European ommunity between 1970 and Our tas is to group the 14 time series on the basis of their similarity in order to detet signifiant differenes among European higher eduation trends. Data were provided by UNESCO and Eurostat, via the r ade data ban, available at the URL (Uneso, 1997). he method to solve this problem depends on the meaning we attah to similar time series. hroughout this paper, we will assume that time series are the realization of stohasti proesses and two or more time series are similar when the same proess generates them. hus, deiding whether two time series are similar is equivalent to deiding whether they are observations of the same proess. Put in this way, the tas of grouping of the time series an be solved as a lustering problem: given a bath of time series, we wish to luster them so that eah luster ontains time series generated by the same proess. Partiularly, we wish to solve this problem without speifying, a priori, the number of lusters. We solve this problem by using a novel Bayesian for method lustering of ontributions. 1 Address for orrespondene: Department of Mathematis and Statistis, Lederle Graduate Researh ower, University of Massahusetts, Amherst MA sebas@math.umass.edu, elephone:

2 We model the stohasti proess generating eah time series as an autoregressive model of order p, say AR(p), and then we luster those time series that have a high posterior probability of being generated by the same AR(p) model. he distinguished feature of this method is to desribe a lustering of time series as a statistial model so that the lustering tas an be solved as a Bayesian model seletion problem. hus, the lustering model we loo for is the most liely partition of the time series, given the data at hand and prior information about the problem. In priniple, we just need to evaluate the posterior probability of all possible lustering models of time series and selet that one with maximum posterior probability. However, the number of lustering models grows exponentially with the number of time series and a heuristi searh is needed to mae the method feasible. he method we adopt uses a measure of similarity between AR(p) models to drive the searh proess in a subspae of all possible lustering models. An important feature of this heuristi searh is to provide a stopping rule, so that lustering an be done without assuming a given number of lusters as traditional lustering methods do. he lustering method we use is fully desribed and evaluated in Sebastiani and Ramoni (2001). In the next setion we briefly desribe the method and the searh algorithm. he analysis of the higher eduation data set is desribed in Setion 3 and a disussion is in Setion AU BE DK GR FR ES P I IE FI NL UK LU SW Years Figure 1. Share of population enrolled in higher eduation, between 1970 and 1996, in the14 European ountries: AU: Austria; BE: Belgium; DK: Denmar; GR: Greee; FR: Frane; ES: Spain; P: Portugal; I: Italy; IE: Ireland; FI: Finland; NL: he Netherland; UK: United Kingdom; Lu: Luxemburg; SW: Sweden.

3 2. Bayesian lustering by dynamis he lustering method we desribe here has three omponents: a model for the time series, the posterior probability of a lustering model and a heuristi searh strategy. hese three elements are desribed very briefly. More details are provided in Sebastiani and Ramoni (2001) Autoregressive models Let S = { y p, L, y 1, y1, L, yt, L, yn} be a time series of values observed for a ontinuous variable Y. he series follows an AR(p) model if y β = Xβ + ε where y is the n-dimension vetor y = ( y 1, L, y n ), X is the n n matrix with tth row given by the vetor of p observations yt, L, y, (,, ) 1 t p β = β1 L β p is a vetor of autoregressive oeffiients, and ε is a vetor of unorrelated errors. We assume that the errors are normally distributed, with 2 expetation E( ε t ) = 0, and variane V ( ε t ) = σ for any t. We shall denote by τ the preision, so that σ 2 = 1/ τ. he value p is alled the order of the autoregression, and speifies the Marov order of the series: namely that yt ( y p, L, yt p 1 ) ( yt 1, L, yt p ), where we use the symbol to denote stohasti independene. he series follows a stationary proess if the roots of the polynomial p j f ( u) = 1 j = β 1 ju have moduli greater than unity. When some of the roots have moduli smaller than unity, the proess is non-stationary, but typially some transformations of the data are suffiient to ahieve stationarity. he model above desribes the evolution of the proess around a zero mean. By adding an interept term β 0, the model an be extended to inlude a non-zero mean µ, for eah y t, so that β = β, β, L, β ) and the matrix X is augmented of a olumn of ones. he proess mean and the ( 0 1 p autoregressive oeffiients are related by: µ = β 0 /( 1 = β 1 j ). We wish to ompute Bayesian estimates of the parameters β and τ. o ompute the Bayesian estimates of β and τ, we need to up-date their joint prior density f ( β, τ ) into the posterior density f ( β, τ y), by using Bayes' heorem: p j f ( β, τ ) f ( y β, τ ) f ( β, τ y) =, f ( y)

4 Where f ( y β, τ ) is the lielihood funtion and f ( y) is the marginal lielihood, whih is omputed as f ( y) = f ( β, τ ) f ( y β, τ ) dβdτ. For a given autoregressive order p, we ompute the lielihood funtion, onditional on the first p values of the time series, as f ( y β, τ ) = n τ (2π ) n τ ( y Xβ ) ( y Xβ ) exp. 2 2 We assume as prior density for β and τ the improper prior f ( β, τ ) Xτ, with τ > 0 (see Jeffreys, 1946). Suppose the matrix X is of full ran, and let βˆ and RSS denote respetively the ordinary least squares estimate of β and the residual sum of squares respetively: ˆ β = ( X X ) 1 X y RSS = y ( I n X ( X X ) 1 X ) y where I n is the identity matrix. hen, one an show that the marginal lielihood is f ( y) = RSS 2 n (2π ) q+ 2 n q n q 2 Γ 2 1/ 2 det( X X ) where q is the dimension of the vetor β. Furthermore, the posterior distribution of β and τ, is normal-gamma, with β y, τ ~ N( ˆ,( β τ ( X X )) RSS n q 2 τ y ~ Gamma,, ) where Gamma(a,b) is a gamma distribution with expeted value a/b and variane a/b 2. Both distributions are proper whenever as the matrix X is of full ran, and n > q + 2. he Bayesian posterior point estimates of β and τ are and ( n q 2) / RSS.

5 2.2. Clustering Suppose we have a bath of time series S = { S 1, L, S m }, whih are generated by an unnown number of stationary AR(p) models with a ommon autoregressive order p, and different autoregressive oeffiients. We wish to luster the time series in S aording to their dynamis. Our goal is two-fold: o find the set of lusters that gives the best partition of the data; o assign eah time series to one and only one luster. Contrary to ommon pratie, we do not want to speify, a priori, a preset number of lusters. Formally, the lustering method regards a partition as an unobserved disrete variable C with states C, L 1,C. Eah state C of the variable C labels, in the same way, the time series generated by the same AR(p) model and, hene, it represents a luster of time series. he number of states of the variable C is unnown, but it is bounded above by the total number of time series in the data set S. he lustering algorithm tries to re-label those time series that are liely to have been generated by the same AR(p) model and thus merges the initial states C 1, L,Cm of the variable C into a subset C 1, L,C, with < m. he speifiation of the number of states of the variable C and the assignment of one of its states to eah time series S i define a statistial model M. his allows us to regard the lustering tas as a Bayesian model seletion problem, in whih the model we see is the most probable way of relabeling time series, given the data. If P ( M ) is the prior probability of eah model M, by Bayes' heorem its posterior probability is P( M S) P( M ) f ( S M ), where f ( S M ) is the marginal lielihood, now written as expliit funtion of the lustering model. A model-based Bayesian solution to the lustering problem onsists of seleting the lustering model with maximum posterior probability. It is shown in Sebastiani and Ramoni (2001) that, under some assumptions on the sample spae, the adoption of a partiular parameterization for the model M and the speifiation of an improper-uniform prior lead to a simple, losed-form expression for the marginal lielihood f S M ). ( Conditional on the model M and, hene, on a speifiation of the number of states of the variable C and of the labeling of the original time series, we suppose that the marginal distribution of the variable C is multinomial, with ell probabilities θ = P ( C = C θ ). Furthermore, we suppose that, onditional on C = C, the bath of m time series { S j } assigned to luster C are independent of the bath of time series { S lj } assigned to any other luster C l, and that the time series generated by the same AR(p) model in luster C are mutually independent. We denote by β the vetor of auto-regression oeffiients and by τ the preision of the AR(p) model generating the time series in luster C. We suppose that eah of these series an be represented as

6 y j β, τ = X β + ε. j j he index indiates luster membership, and ε j is a vetor of unorrelated errors, whih we 1 assume to be normally distributed, with E( ε jt ) = 0 and V ( ε jt ) = τ, for any t. he fat that series assigned to the same luster C are haraterized by the same vetor of auto-regression 2 1 oeffiients β, and by the same variane σ = τ, allows us to represent the whole bath of series { S j } in luster C as y β, τ = X β + ε where the vetor y and the matrix X are defined as y y = M y 1 m X X = M X 1 m Let β denote the set of parameter vetors β = ( β ), where eah β is a random vetor, and let τ denote the set of parameters τ = ( τ ), for = 1, L,. hen, by the independene of series assigned to different lusters, the overall lielihood funtion is f ( y θ, β, τ ) = θ f ( y X, β, τ ) = 1 m where m is the number of time series that are assigned to luster C. Here, the overall lielihood is onditional on the set of ( p + 2) values upon whih the lielihood funtion of eah series is onditioned. We tae as our prior distribution for θ a Dirihlet D( α 1, L, α ), and assign the improper prior with density 2 f ( β, τ ) τ to β and τ. hen, using standard results on Dirihlet integration, it is easy to show that the marginal lielihood is

7 f ( y M ) = f ( y θ, β, τ ) f ( θ, β, τ ) dθdβdτ Γ( α) = Γ( α + m) Γ( α + m ) ( RSS / 2) Γ(( n q 2) / 2) q+ 2 n ( n q) / 2 1/ 2 = 1 Γ( α ) (2π ) det( X X ) 1 = y ( I n X ( X X ) X y is the residual sum of squares in luster where α = α is the overall luster prior preision, n is the dimension of the vetor y, and RSS ) lielihood is well defined as long as eah matrix X is of full ran. C. he marginal One the most liely partition has been seleted a posteriori, eah luster C is assoiated with theparameters β, whih model the auto-regression equation, and the preision τ. he posterior distribution of β τ, y is ˆ 1 N( β,( ( τ X X )) ) with ˆ 1 β = ( X X ) X y, while the posterior distribution of τ y is Gamma( ( RSS / 2),( n q 2) / 2). he marginal posterior distribution of the auto-regression oeffiients β y is a non-entral Student's t, with expetation βˆ, whih provides a point-estimate of (( n q 2) / ) β. he estimate of the within luster preision RSS. he probability that C = C is estimated by ˆ θ = ( α + m ) /( α + m). τ is In our appliation, we use a symmetri prior distribution for the parameter vetor θ, with a ommon prior preision α. he initial m hyper-parameters α are set equal to α / m and, when two time series are assigned to the same luster C, their hyper-parameters are summed up. hus, the hyper-parameters of a luster merging m time series will be m ( α / m). In this way, the speifiation of the prior hyper-parameters requires only the global prior preision α, whih measures the onfidene in the prior model. he urrent implementation of the algorithm assumes that the series follow stationary autoregressive models of a given order p, and then hes that the stationarity onditions are met at the end of the lustering proess Searh In priniple, the lustering method desribed in the previous setion requires one to ompute the posterior probability of eah lustering model and then hoose the lustering model with maximum posterior probability. Sine the number of possible partitions grows exponentially with the number of series, a heuristi method is required to mae the searh feasible. Our method uses a measure of similarity between AR(p) models to effiiently guide the searh proess in a subset of all possible lustering models. Sine all AR(p) models have the same order, this similarity measure is an estimate of the symmetri Kullba-Liebler divergene (Jeffreys, 1946) between marginal posterior distributions of the auto-regressive oeffiients β y assoiated with the lusters. he estimate is given by omputing the symmetri Kullba-Liebler

8 divergene for every pair of parameters β, β j, assuming a normal distribution onditional on the within-luster preisions τ, τ. he preisions are then replaed by their posterior estimates. j Initially, the algorithm transforms the time series in S into a set of m AR(p) models, using the proedure desribed in the previous setion, and omputes the set of m ( m 1) / 2 pair-wise distanes between posterior distributions of the parameters. hen, the algorithm sorts the generated distanes, labels in the same way the two losest AR(p) models and evaluates whether the resulting lustering model M, in whih the two losest AR(p) models are assigned to the same luster, is more probable than the model M s in whih they are distint. If the probability P( M y) is larger than P( M s y), the algorithm updates the set of series by replaing the two series with the luster resulting from their merging. Consequently, the algorithm updates the set of ordered distanes by removing all the ordered pairs involving the merged time series, and by adding the distanes between the parameters of the new AR(p) model and the remaining models in the set. he proedure is then iterated on the new set. If the probability P( M y) is not larger than P( M s y), the algorithm tries to merge the seond best, the third best, and so on, until the set of pairs is empty and, in this ase, returns the most probable partition found thus far. he rationale behind this heuristi is that merging losest AR(p) models first should speed up the searh for lustering models with large posterior probability. Empirial evaluations of the methods on simulated data appear to support this intuition (see Sebastiani and Ramoni, 2001). 3. Analysis We apply the lustering algorithm desribed in setion 2 to the analysis of the fourteen time series reporting the temporal evolution of the share of the population engaged in tertiary/higher eduation in 14 European ountries depited in Figure 1. Sine the average length of a university degree aross European nations is three-four years, we applied the lustering algorithm under the assumption that all time series were generated by stationary AR(3) models with a non-zero mean. We assumed α = 1, the improper prior with density 2 f ( β, τ ) τ, and uniform prior on all lustering models. Stationarity of the autoregressive models was heed at the end of the lustering proess. Figure 2, 3 and 4 show the three lusters of time series found by the algorithm. Cluster C 1 groups the evolutions of the proportion of the population enrolled in higher eduation institutes in Portugal and Luxembourg, see Figure 2. he estimates of the auto-regression oeffiients are ˆ0 β , ˆ1 β , ˆ2 β and ˆ3 β hus, the model is stationary --- the roots of the polynomial f (u) are -2.38, 1.28±0.11i --- with a mean ˆ µ 8.532%. Note that the time series desribing the evolution of shool population in Luxemburg has a slight inreasing trend during the 1970s, and then beomes stationary, with a mean slightly above 8%.

9 P LU C1 Figure 2. Cluster C 1 groups the evolution of shool population in Portugal and Luxemburg. he evolutions of the proportion of the population enrolled in higher eduation institutes in Austria, Denmar, Greee, Spain and Ireland are merged into luster C 2 in Figure 3. he estimates of the auto-regression oeffiients are ˆ0 β , ˆ1 β , ˆ2 β and ˆ3 β , with a mean ˆ µ he AR(3) model is stationary, with roots of the polynomial f (u) equal to 6.09 and 1.02±0.1i AU DK GR ES IE C2 Figure 3 Cluster of time series desribing the evolution of shool population in Austria, Denmar, Greee, Spain and Ireland.

10 Of the series assigned to this luster, those desribing the evolution of shool population in Austria, Denmar and Greee are evidently stationary, while the time series desribing the evolution of shool population in Spain and, partiularly, Ireland exhibit some trend. he assignment of the two series to this luster ould indiate that the inreasing trend is only temporary, and that the proportion of the population enrolled in higher eduation institutes beomes stable during the 1990s BE FR I FI NL UK SW C3 Figure 4 Cluster merging the evolution of shool population in Belgium, Frane, Italy, Finland, he Netherlands, the United Kingdom and Sweden. Cluster C3 in Figure 4 groups the evolutions of the proportion of the population enrolled in higher eduation institutes in Belgium, Frane, Italy, he Netherlands, Finland, United Kingdom and Sweden. he estimates of the auto-regression oeffiients are ˆ0 β , ˆ1 β , ˆ2 β 2.283, and ˆ3 β , thus defining a stationary auto-regression equation, with roots of the polynomial f (u) equal to and 1. ±0.32i. he mean of the proess is ˆ µ his luster groups the European nations that have been onsistently stronger from an eonomi point of view in the past thirty years. All these nations have a solid higher eduation tradition, and university urriula lasting, on the average, four years. All series assigned to this luster are inreasing up to the 1980s, and then derease. his fat would be onsistent with the large demand for highly silled labors and for higher eduation reated by the pae of eonomi development in Europe in the 1960s. he ontration of the population together with the eonomi reession in the 1980s, ould be responsible for the derease of the proportion of population enrolled in higher eduation in the late 1980s and the 1990s.

11 he means of the proesses generating the time series assigned to lusters C 2 and C 3 are essentially the same. However, the autoregressive equation for luster C 3 desribes a more stable proess around the mean, with smaller flutuations. hus, the results would suggest a more stable higher eduation enrollment in Italy, Frane, he Netherlands, United Kingdom, Belgium, Finland and Sweden, ompared to Austria, Denmar, Greee, Spain and Ireland. he fat that the time series desribing the evolution of the population in higher eduation of he Netherlands is assigned to the third luster is slightly disappointing: the dynami of this series is similar to that of the other series in the luster, but this series has a different mean. o evaluate the influene of this time series on the results, we run the lustering algorithm exluding the time series of he Netherlands. he algorithm found the same three lusters, thus showing that this series is not influential. Figure5.Observed (ontinuous line) and fitted (dash line) time series in the lusters in Figure 2. During the analysis we assumed the time series were generated by AR(3) models. Plots of the observed and fitted values within lusters provide an overall assessment of the robustness of the result with respet to this assumption. Figure 5 plots the time series of observed values in the three lusters and values fitted using the AR(3) models assoiated with eah luster. he lose math supports the assumption that AR(3) models are a good approximation of the proesses generating the original fourteen series. Finally, we note that the searh algorithm found the three lusters of time series in just eighteen steps. his number is muh smaller than the total number of lusters to be onsidered without the heuristi searh. Figure 6 shows the inrease of the log-marginal lielihood --- up to a onstant --- at eah step of the agglomerative searh proedure. In the first seven steps, there is a linear inrease of the marginal lielihood. hus, merging the time series that belong to the lusters with nearest

12 autoregressive oeffiients inreases the marginal lielihood. In the next eight steps, merging the losest lusters does not always inrease the marginal lielihood, so that the merging of the seond best is evaluated and aepted. his is so until step 15, when the algorithm has merged the fourteen time series into three lusters. At this point, the three possible merging of two lusters at a time are evaluated and, sine they all result in a derease of the marginal lielihood, the algorithm stops and returns the three lusters so found. Figure 6.Change of the marginal lielihood, in logarithmi sale, at eah step of the agglomerative searh proedure. 4. Disussion and related wor Auto-regressive models have reeived great attention, (see Box and Jenins, 1976, for a systemati exposition and West and Harrison, 1997, for a Bayesian analysis). Bayesian model-based lustering was originally proposed by Banfield and Raftery, (1993), to luster stati data. Ramoni et al., (2000, 2001) proposed a Bayesian lustering by dynamis algorithm, alled BCD, to luster disrete time series. BCD lusters time series modeled as Marov hains and, ontrary to popular methods, finds also the number of lusters. Notwithstanding the, somewhat restritive, Marov hain assumption, BCD has been applied suessfully to luster robot experienes based on sensory inputs (see Sebastiani et al., 2001), simulated war games (Sebastiani et al., 1999), as well as the behavior of stos in the finanial maret and automated learning and generation of Bah's ounterpoint. Unlie BCD, the algorithm used in this paper lusters time series of ontinuous variables. he different type of data requires different modeling assumptions thus produing an algorithm whih is

13 similar to BCD, in being Bayesian and model-based, but its methodology is novel. he heuristi searh used by the lustering method is similar to that implemented in BCD although, here, the searh is driven by a distane between posterior distributions of parameters haraterizing the AR(p) models of different lusters, while in BCD the searh uses the distane between preditive distributions of estimated Marov hains. he model seletion strategy of our algorithm sees the lustering model with maximum posterior probability. Other hoies here would be possible suh as seleting the median posterior probability model (Barbieri and Berger, 2000). One would need to ompare these different model hoies and see whether a similar heuristi searh an be developed when the algorithm sees for the median posterior probability model. At first glane, modeling time series with auto-regression equations of the same order may appear to be a severe restrition. We have investigated the limitation of this assumption in simulated data (see Sebastiani and Ramoni, 2001) and the emerging result is that the results of our lustering method are robust to misspeifiation of the autoregressive order. 5. Anowledgements his researh was supported by Eurostat, under ontrat EP he authors than Ed George and an anonymous referee for their invaluable help to improve the paper. 6. Referenes Banfield, J. D., and Raftery, A. E. (1993), Model-based Gaussian and non-gaussian lustering, Biometris, Vol. 49, pp Barbieri, M., and Berger, J.O. (2000), Optimal preditive variable seletion, ISDS Disussion paper, Due University. Box, G. E. P., and Jenins, G. M. (1976, ime Series Analysis: Foreasting and Control. Holden- Day, San Franiso, CA. Jeffreys, H. (1946), An invariant form for the prior probability in estimation proedures, Proeedings of the Royal Soiety, London, A, Vol. 186, pp Ramoni, M., Sebastiani, P., and Cohen, P. (2000), Multivariate lustering by dynamis, in Proeedings of the Seventeenth National Conferene on Artifiial Intelligene, San Franiso, CA. Morgan Kaufmann, pp Ramoni, M., Sebastiani, P., and Cohen, P. (2001), Bayesian lustering by dynamis, Mahine Learning, to appear.

14 Sebastiani, P., and Ramoni, M. (2000), Bayesian model-based lustering of time series Submitted. Sebastiani, P., Ramoni, M., and Cohen, P. (2001), Bayesian analysis of sensory inputs of a mobile robot, In Proeedings of the 5th Worshop on Case Studies in Bayesian Statistis, pp Sebastiani, P., Ramoni, M., Cohen, P., Warwi, J., and Davis, J. (1999), Disovering Dynamis Using Bayesian Clustering, In Proeedings of the hird International Symposium on Intelligent Data Analysis. Leture Notes in Computer Siene, Springer, New Yor, pp Uneso (1997), Shooling population [omputer file], Paris:UNESCO (produer), r ade online servie (distributor), Universities of Durham and Essex. West, M., and Harrison, J. (1997), Bayesian Foreasting and Dynami Models (2nd Ed), Springer, New Yor, NY.