arxiv: v1 [cs.lg] 18 Feb 2015 ABSTRACT

Transcription

1 Tempora Embedding in Convoutiona Neura Networks for Robust Learning of Abstract Snippets Jiajun Liu Kun Zhao Brano Kusy Ji-rong Wen Raja Jurdak AS Program, CSIRO, Puenvae, Austraia {jiajun.iu, kun.zhao, brano.kusy, Renmin University of China, Beijing, China arxiv: v1 [cs.lg] 18 Feb 2015 ABSTRACT The prediction of periodica time-series remains chaenging due to various types of data distortions and misaignments. Here, we propose a nove mode caed Tempora embeddingenhanced convoutiona neura Network (TeNet) to earn repeatedy-occurring-yet-hidden structura eements in periodica time-series, caed abstract snippets, for predicting future changes. Our mode uses convoutiona neura networks and embeds a time-series with its potentia neighbors in the tempora domain for aigning it to the dominant patterns in the dataset. The mode is robust to distortions and misaignments in the tempora domain and demonstrates strong prediction power for periodica time-series. We conduct extensive experiments and discover that the proposed mode shows significant and consistent advantages over existing methods on a variety of data modaities ranging from human mobiity to househod power consumption records. Empirica resuts indicate that the mode is robust to various factors such as number of sampes, variance of data, numerica ranges of data etc. The experiments aso verify that the intuition behind the mode can be generaized to mutipe data types and appications and promises significant improvement in prediction performances across the datasets studied. 1. INTRODUCTION The behaviors of many of the word s inhabitants are fundamentay bound by the cyce of the sun and the moon which creates day and night. It is the reason why across the days of an average person, there often exist periodica patterns for their mobiity or more generay, their behavior [26, 27]. Utiizing such re-occurring patterns coud drasticay benefit various modern ubiquitous appications. For exampe, the abiity to predict a day s power consumption of many individua houses at midday wi be profoundy beneficia for the smart grid to manage dynamicay its power Permission to make digita or hard copies of a or part of this work for persona or cassroom use is granted without fee provided that copies are not made or distributed for profit or commercia advantage and that copies bear this notice and the fu citation on the first page. To copy otherwise, to repubish, to post on servers or to redistribute to ists, requires prior specific permission and/or a fee. Copyright 20XX ACM X-XXXXX-XX-X/XX/XX...$ suppy resources. Whie in the scenario of smart ocation tracking [14, 34], with a repenish-abe energy budget the system either aims to minimize the energy efficiency of ocation tracking, or attempts to maximize the tracking accuracy given a fixed energy budget. A crucia chaenge invoved in such a smart tracking system is to estimate at any time of day how much further the moving entities wi move for the remainder of the day. Ideay, with a greater estimated vaue of the tota trave distance, the system wi empoy a more conservative samping strategy (ower samping frequencies) to cover as much as possibe of the whoe trip using the restricted energy budget, whereas a more aggressive strategy (higher samping frequencies) wi be favored on the presence of a smaer estimated tota trave distance, so that better tracking precision wi be achieved. Ceary the estimation of the entity s daiy trave distance using partia information is a chaenging yet crucia ingredient for the system s success. Approaches have been proposed to predict generic timeseries and many of them have capitaized on the phenomenon that for each individua there often exist re-occurring sma fragments of time (which we ca snippets ) in their histories. By detecting and reusing such snippets, we are abe to reconstruct a day with the eements from previous reevant days. We show an exampe of snippet earning for daiy traveing time prediction and the difficuties it faces by using a commuter s daiy routines. It is worth noting that throughout the entire paper, we assume that besides the time-series itsef, no other support information such as ocations are avaiabe to the prediction agorithm. For exampe, to predict a day s trave distance, the agorithm s ony input is a partia time-series of the distances traveed in each interva. With a 30-minutes interva, the whoe day wi have 48 time-series entries, and we aim to use the first haf of them to predict the accumuated trave distance for the whoe day. Imagine that a person in our exampe has two usua routines: 1) on workdays the person goes to work by a particuar bus ine that stops outside the apartment every 8 a.m., and arrives at the workpace around 9 a.m. The person gets unch around 12 p.m. at somepace near the workpace everyday, and finishes work around 5 p.m., 2) on weekends the person prefers going to the beach in the morning and coming home in the evening. In the idea case, the person begins and finishes the same activity at the exact same time on every workday, and the resuting time-series for trave distances woud be identica across days. With snippets, a time-series for a workday woud then be transformed into a series of snippets ike < d waking to bus stop A, 1

2 d 10km ride on bus ine 1, d waking into office, d working,...>. Now to predict how much further the moving object wi move for the remainder of the day at a certain time on the day (e.g. midday), we are eft with a simpe task. For every interva of the snippet sequence in the exampe, if the current day shows an identica partia time-series for that interva, the person is ikey to be working that day and is ikey to yied the same tota trave distance as any other workday. The same method works for the weekends too. In reaity, such patterns do repeat themseves, ony not in such a perfecty aigned way but instead often on a shifted timeine and at a differing pace. Instead of having high coherences at a times between two working days of a person, in reaity a day s time-series may often be partiay simiar to and partiay divergent from another day s, posing a serious chaenge for the aforementioned prediction method. There are many possibe causes which prevent a perfect resember for a snippet sequence from happening. For exampe, the bus in the morning may be 20 minutes ate, or the person may wait for a coffee to miss the bus he/she is supposed to take. Then, the person may have a ater than usua unch at work. Finay, the person on one day decides to do usua item A/B in the order of B/A. Couped with the huge number of non-work-reated ocations a person coud go to and the numerous possibe sequences of visiting them, the resuting time-series coud have a huge variety of distortions to the reguar time-series. In such cases, how to effectivey earn representative snippets and how to use them effectivey remains a major chaenge. To sove this compex probem, we adopt the concept of snippets but take a step forward and propose a robust earning and time-series prediction mode to systematicay reduce the effect of such distortions. Specificay, we make the foowing contributions in this paper: We propose a nove regression mode, which is based on convoutiona neura networks, to sove the robust snippets earning and periodica time-series prediction probem. We propose a nove technique caed tempora embedding to improve the cassica convoutiona neura networks capabiity for earning robust snippets and for predicting accuratey. We design a network ayer based on this concept, devise a compete four ayer network (TeNet) for regression, and sove the corresponding backpropagation probem. We aso offer a detaied case study to iustrate the effect of tempora embedding. We conduct extensive experiments on 15 individua datasets representing three data modaities and one synthetic dataset to evauate the advantages and characteristics of the proposed mode. The rest of the paper is organized as foows. Next in Section 2 we present the background and reevant iterature of the probem studied. In Section 3 we give the intuition behind TeNet, describe in detai the technique of tempora embedding and other ayers of TeNet, and offer soutions to the backpropagation of TeNet. We then enter Section 4 and evauate the proposed mode. Finay we concude our work in Section BACKGROUND AND RELATED WORK Learning abstract features (with neura networks in many cases) has been extensivey studied in recent years and has proved effective in many appications. For instance, numerous studies [3, 2, 15, 18, 9] have shown that deep neura networks perform we for compex computer vision cassification tasks, whie many demonstrate that success can be achieved with deep earning architectures for audio cassification tasks as we[19, 22]. These we-performing deep neura networks have a variety of core ideas, ranging from restricted botzmann machines that utiize an energy mode [13, 17, 13], to sparse autoencoders that introduce an unsupervised denoising mechanism to remove insignificant, noisy signas from data [29, 3, 30], to using convoution as an effective way to earn representative features robust to geometric ocations of images [18, 5]. The main advantage of such methods is that they have a strong capabiity of unraveing the hidden hierarchica structure of data to derive representative features. Moving from a shaower architecture to a deeper architecture, these modes progressivey detect essentia components of the data from oca parts ike strokes in human handwriting, to goba compositions such as digits or objects. Among the variations of neura networks, inspired by bioogica processes [20], convoutiona networks in particuar exce in finding such abstract features that are robust to geometric variations in images [18]. Interestingy, such advantages of convoutiona neura networks are present not ony in vision tasks, but aso in speech recognition [1, 8, 12] and natura anguage processing [6, 7]. Now we consider the periodica time-series prediction probem for data such as daiy traveing distances or daiy househod power consumptions. To tacke this probem, conventionay statistica modes such as autoregression and its variants are strongy favored. Whie in the past decade, reaizing there is abstract and structura information beneath the raw numeric vaues in the time-series, researchers have experimented to discover such patterns by custering or motif discovery [23, 26, 27]. Though conceptuay simiar, these motifs usuay are concrete subsequences that are restricted by specific mathematica definitions, which differentiate themseves from the concept of abstract, representative snippets in our paper. However, how to design a method that can find abstract patterns as we as predict future vaues, that meanwhie is robust to various tempora distortions and misaignment, is yet to be answered. Inspired by the success of convoutiona neura networks, we investigate using convoution-based neura networks to address this probem. 3. THE MODEL 3.1 Intuition The two main chaenges for the periodica time-series prediction are: 1) how to find representative snippets for the prediction of future changes; and 2) how to minimize the effect of distortions in the tempora domain and get accurate regression resuts. Here we examine the two chaenges separatey and propose soutions to them from a neura networks perspective. The first chaenge, i.e. snippet earning, invoves finding abstract sequences in the training time-series. Naturay there is an assumption that the snippets shoud ony be of moderate ength. For exampe, if we were to predict daiy 2

3 x t input tempora embedding convoution max-pooing tanh sigmoid 1-reguarized east-squares t output W (4) (3x1), b (4) (1) W (3) (4x3), b (3) (3) b (2) (2) W (1) (3x6), b (1) (6) W (2) (2x3) input size 1x6 1x6 2x1x2 1x3 output size 1x6 2x1x2 1x3 1x1 Figure 1: An instance of TeNet for 6-d input. It is composed of a sparsey connected tempora embedding ayer, a convoution/pooing ayer with two fiters of size 1 3 and poo size 1 2 (foowing the conventions in constructing convoutiona neura networks, the convoution ayer and max-pooing ayer are iustrated as a singe ayer), a fuy-connected sigmoid ayer that transforms the feature map from size into 1 3, and finay an 1-reguarized east-squares regression ayer that yieds the predicted vaue. W (), b () are the weights and bias of the connections between ayers and + 1. Connections with the same coors in the convoution ayer indicate that those connections share the same weights, and the two shaded areas represent the two feature maps from the fiters. The dimensionaities of the weights, the input and the output for each ayer are provided at the bottom. Biases are not iustrated in this figure. human mobiity, a time window of from one haf-hour to a few hours woud be a reasonabe setting, as intuitivey such a period of time shoud be enough to cover most of the common trips in daiy ife. Hence in the prediction mode, we examine such periods of time using a convoutiona approach. We create randomy initiaized fiters that have a given, moderate ength as the ength of the target snippets. In 2D image cassification tasks, fiters in convoutiona neura networks are often used as edge detectors, whie in ours, the fiters wi act as snippet detectors. In the training phase, the weights for the fiters wi be adjusted during the backpropagation so that they respond maximay to the reoccurring and significant components in the training data. We then sove the second chaenge by adding a tempora embedding operation in the neura network. The tempora embedding process provides a supervised way of denoising subspace earning. When deaing with time-series, a naïve technique is to shift the training data forward and backward aong the timeine. For exampe, a shifting routine with windows size 1 woud transform a training sampe x < x 1, x 2,..., x d > y into three training sampes x < 0, x 1, x 2,..., x d 1 > y, x < x 2, x 3,...x d, 0 > y, x < x 1, x 2,..., x d > y. Though usefu sometimes, this naïve approach introduces heavy noise by incuding artificia training sampes that may never actuay happen in the rea word. Aso it is unabe to benefit case where the order of the subsequence is changed. We argue that the naïve technique can evove to a much more effective approach caed tempora embedding that integrates into the earning process mechanisms for removing distortions. With tempora embedding, two temporay-shifted copies are created for each sampe during the earning process, and then the origina sampe and the two shifted copies are encoded into a singe sampe so that the processed sampe wi not ony carry its own information, but aso bear a piece of information for each of its shifted neighbors. Again, the weights for the encoding are earned in a supervised way during backpropagation. Next we present an overview of the TeNet mode. 3.2 Mode Overview We propose a convoutiona neura network to earn the snippets from the periodic time-series as iustrated in Figure 1. The mode has three invisibe ayers, namey the tempora embedding ayer, the convoution/max-pooing ayer, and the sigmoid ayer. The output ayer is an 1-reguarized east squares regression ayer. The iustrated mode is an exampe instantiation of the proposed mode, with the input size, embedding window size, number of snippets, snippet size, max-pooing and sigmoid ayer sizes to be 6, 1, 2, (1,3) and (1,2) and 3 respectivey. The mode impements the foowing work fow: 1. It takes an input sampe, and appies the tempora embedding. This ayer transforms the sampe into a denser representation with not ony the sampe itsef but aso information of its potentia tempora neighbors. The weights of the transformation are iterativey updated during the training process. 2. The embedded input is sent into a convoution ayer where a set of fiters, or snippet detectors, scan through the sampe using the convoution operator. Each snippet wi be convoved against the sampe, resuting in a feature map considered as the snippet s response to that sampe. 3. The snippets responses to the sampe, being supposedy sparse and representative, are input into a sigmoid ayer to combine some of the responses into higher-eve and more abstract representations in ower dimensions. 3

4 This transformation aso invoves a set of weights that is earned over the training process. 4. Finay the abstract representation of the sampe is used to perform an 1-reguarized east-squares regression to obtain the predicted vaue. The intuition behind the 1 reguarization is that if we consider the previous ayer s output, ie. the high-eve neuron s responses to the sampe, as high-eve pattern recognizers responses to the signa, a sparse soution wi utiize the most significant responses and hence wi be ess sensitive to noise [21, 25]. In the foowing subsections we discuss the ayers separatey in detai. In the rest of the paper, the technica detais of the neura network wi be described mosty in vector forms, and we wi use the assumptions and notations isted in Tabe 1. Tabe 1: Tabe of Notations Notation Description x R d the input time-series of ength d the ayer number W () the weights for the th ayer b () the bias for the th ayer a () the input of neurons in the th ayer z () the intermediate vaues for the th ayer g () () the activation function for the th ayer δ the intermediate error (cost) of the th ayer J(W, b; x, y) the network s cost given W, b; x, y T the transpose operator the dot product operator the eement-wise product operator the convoution operator g () the derivative of function g () 3.3 Tempora Embedding The tempora embedding ayer aims to aign ess dominant sampes to the dominant patterns by reducing the tempora distortions and misaignment (e.g. shifting or skewed sequence of events), corresponding to two cases in our previous exampe: 1) the commuter starts the day 30 minutes earier than usua, so every event in the morning rush hour is shifted ahead equay by 30 minutes, 2) for some reason the commuter does not take the usua bus ine which directy stops at his workpace, instead he/she takes a train and waks 1km to work from the station. In the resuting time-series we wi see two distinct effects as a resut of 1) and 2). For exampe, assume that on norma day the trave distance time-series segment in the morning wi be v < 0, 1, 2, 4, 1, 0 >, then for case 1 we wi have u < 1, 2, 4, 1, 0, 0 >, and in case 2 it wi be u < 0, 1, 4, 2, 1, 0 >. Now we assume both cases happen on the same day, giving us u < 1, 4, 2, 1, 0 >, which is heaviy distorted from v. It is a significant chaenge for a prediction agorithm to reaize that for the two days the trave distances shoud be very simiar despite the sequences and the vaues of their time-series are so different. Tempora embedding addresses this issue, by optimay embedding a vaue s tempora neighbors into itsef, so that for the whoe dataset the dominant pattern remains unchanged but the distorted patterns are reaigned. The ayer is configured by one hyperparameter d te that contros how many neighbors of an eement in each direction shoud be embedded to the eement itsef (the embedding size). This ayer has 2 d te +1 sets of parameters, represented by matrices W (1) j, W m (1) and W r (1) j R d(1) d (1), and the same number of constant sparse matrices W j, W m and W rj R d(1) d (1). The subscriptions and r represent the direction of the neighbors on the timeine, and j here means the weights for the j th neighbor in the fina embedding. In the case of d te 1, there are three W matrices and three W matrices in this ayer. The six matrices together impement the embedding operators. Here we use the input dimensions in Figure 1 (where d (1) 6) as an exampe for how this ayer works. The constant matrices, are defined as: W (1) r 1 W (1) (1) (1)T (1) W 1, W m I eye(d (1) ) (2) Weights in W (1) 1,W m (1) W r (1) 1 that correspond to the 1s in W 1, W m and W r1 represent the weights for the embedding of the sampe s eft neighbor (forward), the sampe itsef and its right neighbor (backward) respectivey, and they are initiaized with corresponding constant matrices respectivey. The ayer s output is subsequenty defined as foows: z (1) a (1) (W (1) 1 W 1 + W r (1) 1 W r1 + W m (1) W m) + b () (3) g (1) (z (1) ) z (1) (4) W (1) enforces a constraint that the connections between this ayer and its input are restricted, and ony the weights at the desired neighboring positions for each eement are used in the fina embedding for that eement. The ayer yieds the tempora embedded output g (1) (z (1) ) R d( 1), or 6 in this exampe, as the output of the ayer. One can aso use the sigmoid function as the activation function in the tempora embedding ayer, though our experiments show that the difference it makes on the prediction accuracy is insignificant (most of the times adding the sigmoid activation wi sighty decrease the prediction accuracy). The ayer s output is a vector of the same size as the input, however the embedded sampe is now significanty more robust to tempora distortions. With tempora embedding, the mode detects dominant patterns in the training time-series, and tries to correct the systematica distortions within the specified time window. Using the commuter exampe, the mode wi find that the person s reguar time for the bus to work, and wi try to reaign the systematica misaignment on those unusua days. Some readers may argue that a simpe moving average agorithm might be abe to sove the distortion probem; however tempora embedding is far more effective, as the concrete exampe beow shows. Discussion and Case Study. Reca our exampe with v and u, where v represents the dominant pattern in the dataset, whie u represents a day that in fact wi yied a simiar end-of-day resut but shows very distorted patterns in its time-series. Now given the parameter matrices W, W m, W r and the constant matrices 4

5 W, W m, W r initiaized as in Equation 2, our objective is to reaign u with v by eiminating the distortion, and meanwhie keeping v as unchanged as possibe, which is effectivey equivaent to soving the foowing minimization probem in Equation 7: v t v (W W + W m W m + W r W r) + b (5) u t u (W W + W m W m + W r W r) + b (6) arg min W,W r v t v 2 + u t v 2 (7) where v t and u t are the embedded new time-series. By soving the optimization, the non-zero weights in W W, W m W m and W r W r are determined as < 0, 0.61, 0.24, 0.44, 1 >, < 0, 0.22, 0.4, 0.15 > and < 0.66, 0.24, 2.1, 1 > respectivey. Now v t and u t can be cacuated according to Equations 5 and 6, and we subsequenty investigate how tempora embedding performs in terms of preserving v and reaigning u to v, compared with the moving average approach, with v s and u s being the output of v and u of a moving average of window size 3 (v i < v i 1, v i, v i+1 >). Tabe 2: Tempora Embedding vs. Moving Average v < 0, 1, 2, 4, 1, 0 > u < 1, 4, 2, 1, 0, 0 > v t < 0, 1, 2, 4, 1, 0 > u t < 0, 1, 2, 4, 1, 0 > v s < 0.5, 1, 2.3, 2.3, 1.7, 0.5 > u s < 2.5, 2.3, 2.3, 1, 0.3, 0 > Squared Error Intersection Pearson s v, u v, v t v t, u t v, v s v s, u s Tabe 3 measures the reations between the vectors before and after the transformations with three metrics, namey squared error, intersection and Pearson s correation. First we note that u is so distorted that the correation between v and u is merey 0.11, which can be considered uncorreated. Now we examine the differences between the effects of tempora embedding and moving average. Ideay, the transformation shoud show the foowing properties: 1) since v represents the reoccurring pattern in the training set, we want v t to be as unchanged as possibe after the transformation 2) after the transformation, u t shoud be as simiar to v t as possibe, indicating that the misaignments in u has been minimized and u is reaigned to the representative sampe v. We verify the two aspects by examining the reations between v and v t, and that between u t and v t, and observe that tempora embedding has achieved both goas. First we observe that v t is identica to v (with 0 squared error), whie u t has been transformed to a form that is perfecty identica to v and v t now, with the dominant vaues at the second and third positions swapped and reaigned to the third and forth position to be more inine with v. However, we can see moving average resuted in a squared error of 2 between v and v s, showing that v has not been preserved successfuy in the transformation. Second, though moving average does strengthen the reation between v and u by reducing the squared error ( ) and by increasing the simiarity by intersection, it has even resuted in a drop in the correation ( compared with the origina v and u). We concude its resut is ceary ess successfu compared to tempora embedding (4.5 0 in squared error, 4 8 in intersection, and ). It is worth noting that athough the tempora embedding ayer in the proposed neura network is not exacty the same as in Equation 7 as it does not have knowedge initiay about which sampes hod the representative patterns, as the training proceeds, the weights wi progressivey favor the reoccurring patterns, and eventuay approach the soution of Equation 7. Next we describe the convoution, the max-pooing and the sigmoid ayers. 3.4 Convoution, Max-pooing and Sigmoid The convoution/pooing ayer performs a series of discrete 1-d convoutions W (2) i a (2) with a specified number of fiters n f of a specified ength d f. Each of the fiters sweeps through the entire input signa and takes the input signa segment at the corresponding position as input. With a fiter kerne W (2) i < W (2), W (2) (2),..., W id f id f 1 i1 > (taking the convention of reversey-ordered weights for convoution kernes and outputs), the i th fiter s output has the k th eement: (W (2) i a (2) )[k] d f p1 W (2) ip a(2) k+p 1 (8) In the exampe in Figure 1 we have set two fiters with size 1x3, hence in the convoution ayer, each neuron wi ony be connected to three neurons from the tempora embedding ayer. Such sparse connectivity between the fiters to their inputs enforces that the convoution ayer wi be focusing on finding the oca snippets with moderate engths. Though the convoution traverses the entire time-series in a siding-window stye and seemingy has a positive effect in reducing the tempora distortions, it is very different from tempora embedding. The main factor differentiating them is in the weight-sharing scheme (see Figure 1). A fiter in the convoution ayer has its weights shared among a its output neurons (meaning a fiter is siding through the data, trying to match the same particuar pattern), whie in tempora embedding each neuron has individuaized weights to enabe optima oca embedding for each position. Such fexibiity enabes it to identify and reaign much more compex distortions and misaignments. For exampe, given v < 0, 1, 2, 4, 1, 0 >, convoution wi not be abe to recognize the cose reation between u < 1, 4, 2, 1, 0, 0 > and v because of the heavy distortions in both the positions and the sequences. In the experiments we wi aso show that without the tempora embedding ayer, convoutiona neura network does not work we on such time-series. The output of the convoution wi be of the size n f d ( df +1). In Figure 1 s exampe where n f 1, d 6, d f 3, we have the 8 neurons in the convoution ayer. The output is then received by the max-pooing ayer, where ony the maxima vaue is kept from any poo of 1 2. The fiter s output wi hence be down samped and transformed by an eement-wise hyperboic tangent function, reducing the output to 4-dimensiona. Then as the ast hidden ayer, the sigmoid ayer wi perform a projection from the convoution/pooing s output to a further reduced dimension as a means of both earning non-inear features and dimension reduction. Finay, the input is transformed into a dense, robust and representative feature representation of 1 3. Intuitivey we can consider the sigmoid ayer as a higher-eve 5

6 feature earner, after the convoution ayer has discovered those reativey more oca snippets reguarized Least-squares The output ayer of the proposed mode is a 1-reguarized east-squares regression ayer, defined as: g (4) (a (4) ) a (4) W (4)T + b (4) (9) with the cost function in the from of: J(W, b; x, y) 1 2 a(4) W (4)T + b (4) y 2 + λ W (4) 1 (10) where λ is a hyperparameter for the weight of the reguarization term. The advantage of using the 1 reguarizer over 2 is that the 1 reguarizer forces the optimization to find a sparse soution that ony uses the most distinctive high-eve features to conjure the fina prediction [21, 25]. With the 2 reguarizer the weights tend to have smaer variance, often making the mode spread the energy thiny across a features, hence making the mode ess distinctive and ess accurate. 3.6 Backpropagation The parameters in the network are updated by stochastic gradient descent. In particuar, W (4) can be earned by: J(W, b; x, y) W (4) a (4)T (a (4) W (4)T + b (4) y) + λ sign(w (4)T ) Where sign() is the sign of a vector. One can speed up this optimization process using the methods proposed in [28]. To update the parameters in the tempora embedding ayer, taking W (1) as an exampe, we appy the chain rue and arrive at: J(W, b; x, y) J(W, b; x, y) W (1) g (1) 1 J(W, b; x, y) z (1) g(1) z z(1) δ (1) W (1) z(1) (1) W (1) 1 z(1) W (1) Since the eement-wise product has the property: (11) a (1) (W (1) (1) W ) (a (1) (1) W ) W (1) (12) we have the partia derivative of z () w.r.t. W (1) z (1) W (1) as: a (1) (1) W (13) We cacuate the error propagates from ayer 2 to ayer 1 as: d f i1 δ (1) δ (2) i J(W, b; x, y) z (2) W (2) i a (2) a (2) z(2) g(1) g (1) z (1) d f i1 δ (2) i fip(w (2) i ) (14) where fip() returns the input vector in reversed order. With the convoution ayer s back propagated error being δ (2) (which can be cacuated by the method described in [16]), J(W,b;x,y) W can therefore be updated with the gradient: (1) J(W, b; x, y) W (1) d f i1 [ ] δ (2) i fip(w (2) i ) (a (1) (1) W ) (15) W (1) and W m (1) can be updated using simiar procedures. Meanwhie, b (1) is updated with the gradient: J(W, b; x, y) b (1) d f i1 δ (2) i (16) Next we present the experimenta resuts and offer indepth anaysis and discussion. 4. EXPERIMENTS In the experiments, we conduct extensive tests on the proposed mode, with 15 individua datasets and 4 competitive methods. The goas of the experimenta studies are fourfod: 1) to evauate the prediction performance of the proposed mode, in terms of prediction accuracy, and compare it with the competitive modes; 2) to evauate the mode s behavior and sensitivity to features of diverse datasets; 3) to investigate the isoated effects of tempora embedding; and 4) to visuaize the snippets and show how they work with intermediate vaues from the earning process. 4.1 Datasets To support the comprehensive evauation, we use a variety of univariate, periodica time-series datasets that represent three modaities, ranging from human mobiity patterns to househod power consumption. The reason we choose these modaities is that the behaviors they represent are expected to exhibit compex periodica patterns in daiy cyces, which is an idea testbed for the proposed mode to demonstrate its capabiity of discovering and capturing such abstract features and to test its robustness to various factors. The first modaity is Human Mobiity - daiy traveing Distance (HMD) in kiometers, and the second is Human Mobiity - daiy traveing Time (HMT) in minutes. Both modaities are extracted from the LifeMap [4]) that contains human mobiity traces coected from eight individuas, spanning from a few months to around two years. In tota there are 52,819 position fixations, most of which are from reguar samping every two to five minutes. HMD is the tota dispacement for an individua in a day, and HMT is accumuated from short-term movements cacuated as foows: for each five minute interva, if the individua s dispacement is higher than 500 meters 1, then the five-minute period is counted as traveing and is accumuated to the daiy tota traveing time. The third modaity is daiy Househod Power Consumption (HPC). Two datasets are used for this modaity, i.e. househod power consumption datasets from France 2 and Austraia 3 (HPC-FR, HPC-AU). HPC-FR consists of 2,075,259 active power consumption in watt samped every minute for 48 months from a singe househod. HPC-AU consists of 618, 189 househod power meter readings in kwatt hour samped every 30 minutes from 31 househods for up to 29 months. To prepare the data, we deveoped a program to extract ony the sampes that have compete (or neary compete) day cyces, meaning that every data sampe used must have reguar readings in each period of time in a compete day. To obtain meaningfu resuts, ony individuas with more than 150 days of records are used in the experiments. 1 median errors of ocaization with assisted GPS, WiFi positioning and ceuar network positioning are reported to be 8, 74 and 600 m [33]

7 ID n [ ] σ Tabe 3: Performance comparison Error (MRE/MSE) TeNet SVLN SVSIG SVPOLY MKR TeNet SVLN SVSIG SVPOLY MKR Househod Power Consumption - Austraia (HPC-AU) Househod Power Consumption - France (HPC-FR) Human Mobiity - Traveing Distance (HMD) Human Mobiity - Traveing Time (HMT) For the human mobiity datasets, we use the two individuas datasets with the highest quaity of data in terms of timespan (>150 days) and samping frequency. We extract the traveing distances and traveing times for each interva (e.g. a 30 minutes interva creates 48-d time-series for a day), and use the resuting time-series for the experiments. Simiar preprocessing is appied on the power consumption datasets. After preprocessing, each time-series sampe has d eements as x < x 1,..., x d >, each x i is the occurred vaue in the corresponding time interva (non-cumuative). For each individua dataset, we randomy divide the sampes equay into three fods: the training set, the vaidation set and the test set. The mode is trained using the training set, and is then tested on the vaidation set. Such crossvaidation is performed on the same individua dataset for five times with random spits, and the reported performance is the averaged vaue cross the five iterations. The settings of hyperparameters with the best vaidation performance are kept as the hyperparamters of the mode. Finay we test the mode on the test set and report the performance. 4.2 Evauation Settings For evauation we consider the periodica accumuation prediction probem, where each input x R d (d < d) is a head segment of a compete x and corresponds to a target vaue y d i0 xi representing the periodica accumuation. Ceary the mode can be used to perform other types of prediction such as time-series forecast or k-ahead prediction. Due to space imit here we use periodica accumuation prediction as a showcase for TeNet s performance advantages. TeNet is impemented using Python with the Theano framework 4. For comparison, we consider four competitive methods, namey Support Vector regression with Linear kerne (SVLN), Support Vector regression with Radia Basis kerne (SVSIG), Support Vector regression with Poynomia kerne (SVPOLY), and Mutipe Kerne Regression (MKR) [24]. The parameter seection criterion for the SV-famiy is that we carefuy tune the parameters ɛ (error margin), d (degree of kerne function), and γ (kerne coefficient) for kernes. Each parameter s vaue is seected from the sets ɛ {10 5, 10 4,...1,..., 10 4, 10 5 }, d {1, 2, 3}, γ {10 5, 10 4, , 10 5 } respectivey, so in tota there are 363 combinations for each mode. For each test run, during training we iterate through every combination of ɛ,d and γ s candidate vaues, and keep the vaues that generate the highest accuracy on the vaidation set, then use these parameters on the test set and report the resuts. For comparabe evauation against MKR, we use an offine impementation where test sampes are not used to update the parameters, and the number of support vectors is set to 120 for matching the parameter size of TeNet. The hyperparameter seection of TeNet foows the same procedure. We provide more detais in Section For most of the experiments d is set to 28, meaning for each day, the time-series up to 2pm is known to the mode. Seecting this particuar number is because considering humans rarey remain active from 12am to 4am and the vaues in that period are amost a zeros, the first 28d represent information from exacty haf of the active period from 4am to 12am of the next day. Such setting is chaenging in the sense that the gap between 2pm to 12am next day is substantia and it eaves numerous possibe outcomes for the daiy accumuation. The compexity invoved hence provides insight about how we the proposed and the competitive modes can capture an individua s daiy patterns and make prediction from imited information. Next we present the experimenta resuts for the proposed method and the competitive methods, and aso offer indepth discussion about hyperparameter tuning and about the effect of tempora embedding. 4.3 Prediction for Periodica Accumuation Tabe 3 studies the prediction performances of the proposed method and four competitive methods on 15 individua datasets of three different modaities, evauated by average HitRate(HR)@20% and 30%, Mean Squared Error (MSE) and Mean absoute Reative Error (MRE). Using four metrics is due to that for datasets with ong-taied vaues (which human behaviors can often be characterized to be [11]), as an absoute measurement, MSE aone is not an idea metric to evauate a regression method s performance because it is heaviy biased by sampes in the ong tai [31, 32]. Therefore we mainy use reative measures for the evauation whie keeping MSE as a reference. 7

8 The highighted numbers in red, back, magenta and bue indicate the winning performance on that dataset under the corresponding metric ( magenta HR@20%, bue HR@30%, red RE, back MSE). Mutipe highighted numbers with the same coor in a row indicate mutipe winners under that metric on that dataset. We aso report some of the properties, i.e. the tota number of sampes n, the numeric range [ ], and standard deviation σ, for each individua dataset. A coser ook at these dataset statistics suggests arge varieties in terms of number of sampes (from 156 to 874), numerica ranges (0.2 to 345) and variances ( σ from 2.8 to 47). To present the reader with more intuitive and meaningfu resuts, the numbers shown are unnormaized. TeCNN adv. ratio scores TeNet SVLN SVSIG SVPOLY MKR HR@20% HR@30% MRE MSE-normed Figure 2: Mean Average Performance correation no.sampes (a) n vs. adv. MRE TeCNN adv. ratio correation entropy (b) entropy vs. adv. MRE Figure 3: The correation between TeNet s performance advantage and sampe size/data compexity Generay, the distribution of the highighted and winning performances shows that TeNet achieved best resuts in most of the cases, with a few but non-systematica exceptions spread across the competitive methods. Out of the 15 individua datasets, TeNet has won 14 entries in HR@20%, 15 entries in HR@30%, 13 entries in MRE, and 7 entries in MSE, showing a superior performance among the evauated modes. SVLN and SVSIG show east competitive resuts by having 1, 0, 2, 1 and 1, 0, 1, 0 winning performances respectivey. SVPOLY obtains sighty better resuts with 3, 2, 3, 0 wins. MKR on the other hand, has shown comparabe resuts in MSE but far ess competitive resuts in other metrics, by having 0, 0, 1, 8 wins. In addition, we find that MKR is ess robust to arger numerica ranges such as in HMD-8, HMD-12, HMT-8, and HMT-12, whie TeNet demonstrates consistent performances cross a datasets. To compare the methods quantatitivey, we pot Figure 2 and show each method s mean average scores cross a individua datasets (MSE is normaized with the maximum MSE among the methods in each entry). On the 15 individua dataset, TeNet achieved best average performance under a four metrics. Taking a TeNet vs. a approach, we find TeNet s performance and the average of other methods MRE performance under HR@20%, HR@30%, MRE and MSE are 69 vs. 60, 84 vs. 78, 0.22 vs and 34 vs. 51 respectivey, showing that TeNet makes a reative improvement of 15%, 8%, 19% and 33% respectivey under the corresponding metric. Then if we investigate TeNet vs. the best among the rest, with HR@20% 69 and HR@30% 84, TeNet beats the second best HR@20% 61 (SVLN, SVPOLY) by 8, the second best HR@30% 78 (SVLN, SVPOLY) by 6; on MRE and MSE, TeNet s average errors are 0.22 and 34, whie the second bests are 0.24 and 40 (MKR). Hence for a 15 individua dataset, in average TeNet marks an 13% increase in HR@20%, an 8% increase in HR@30%, a 9.1% decrease in MRE and a 15% decrease in MRE to the second best method under each corresponding metric. We aso observe that though in a 15 individua datasets TeNet obtained the best performance under HR@30%, the average winning margin is the smaest than those under other metrics. This is because HR@30% is a reative ooser measurement than other metrics, which eads to the resut that ess accurate prediction tends to have simiar performances. However, the consistent advantage of TeNet in not ony HR@30% but a four metrics sti suggests that it has the best prediction accuracy. We hence concude that TeNet has shown consistent advantages which are robust to variations in the data modaity as we as the statistics characteristics of the data. MSE-normed RE d (a) d vs. error HR HR@20% HR@30% d (b) d vs. HR Figure 4: The effect of the kerne function degree d We further examine TeNet s abiity to scae up its earning effectiveness with a growing sampe size or an increasing compexity of the data. Taking MRE for exampe, we measure two correations using Pearson s correation coefficient: 1) the correation between the averaged performance advantage ( 1 n n i min({resv,mkr i }) re T enet i ) and the sampe size, 2) the correation between the averaged performance advantage and the entropy, for each individua dataset. The measurements yied correation coefficients 0.7 and 0.79 respectivey, suggesting a strong correation between each set of the variabes. Such patterns mean that as the sampe size or the compexity of the data grows, TeNet is abe to earn more effective than other methods to achieve better performance. The correations are aso visuay identifiabe as we pot the the performance advantage ratios in Figure The Effect of d Figure 4 iustrates the effect of the feature dimensionaity d on the prediction accuracy. Here we use HPC-AU-8 as a case study. Figure 4(a) shows the changes of MRE and normaized MSE to a growing d. Unsurprisingy, both errors decrease monotonicay as d increases, from 1, 0.35 at d 8 to 0.08, 0.07 at d 44. Figure 4(b) depicts how the HR responds to a growing d. Again, we see monotonic growths (amost, except for d 16) in HR@20% and HR@30%. These resuts confirm that TeNet can effectivey use the additiona 8

9 information and in the mean time has received itte impact from the noise in the additiona dimensions. (a) Random Snippets (b) Learned Snippets Figure 5: Visuaization of a set of earned snippets by TeNet (correspond to the convoution fiters) 4.5 The Effect of Tempora Embedding In Section 3.3 we discussed how hypotheticay tempora embedding woud boost the performance of the mode by automaticay reaigning the distorted time-series to the dominating patterns in a dataset, and verified it with a case study on a synthetic exampe. To further vaidate this hypothesis on rea data, we create a designated dataset from HPC-AU-8 by performing the foowing procedure: 1. We run a custering with the affinity propagation method in [10], and find the top 10 exempars. 2. We take the exempars and generate 300 synthetic sampes (30 for each exempar) by distorting the exempars with randomy seected operations such as swapping two neighboring segments or shifting the data forward and backward. They are equay spit into training, vaidation and test set. 3. We train a mode with a modified cassica convoutiona neura network fore regression (CNN, input convoution/pooing sigmoid 1-inear regression) without tempora embedding, and a mode with TeNet, and examine the performance differences. The resuts are reported in Tabe 4. We observe that with the tempora embedding ayer, the prediction accuracy has been improved by more than a haf (15.5 to 6.4, 0.34 to 0.12) for MSE and MRE, and for about 100%/40% in HitRate@20% and 30%. This shows that tempora embedding is abe to earn the weights which are conceptuay equivaent to a reverse operation for the distortions and misaignments. Tabe 4: Performance w/&w/o tempora embedding HR@20% HR@30% MRE MSE CNN TeNet Discussion Distinctive Snippets We present a visuaization of the random snippets and earned snippets for the first cross-vaidation iteration on HPC-AU-8 in Figure 5. Each ce is a snippet, a segment of time-series the mode deems representative. The figures show some noteworthy properties. Firsty the random snippets are fairy dense, whie the earned ones are much more sparse, meaning that in most of cases there are ony a smaer number of spikes and vaeys in each earned snippets. Secondy, the sparsity of the earned snippets is aso accompanied by a visuay identifiabe high distinctiveness across the earned snippets, which means snippets earned tend to be different from one another because they effectivey capture different patterns in the training data. Both properties suggest that the snippets are truy earning from the patterns in the dataset and both properties have a positive effect on the mode s prediction accuracy Seection of Hyperparameters As an issue often posed to compex earning modes incuding neura networks, how to seect the hyperparameters is an open question studied by many [15]. There are six hyperparameters in the proposed mode: Tabe 5: Hyperparameters and seection candidates Notation Description Candidates d f fiter size {3,5,7} n f no.kernes in conv. ayer {20,30,40,60} r earning rate {0.01,0.02} d te tempora embedding step {1,2} n (3) no.output in sigmoid ayer {12,16} λ weight for the 1-term {0.1,0.01,0.001} In this paper, since the sizes of the datasets are moderate, we use an intuitive approach to find the hyperparameters for the testing. The seection and testing processes foows that described in the third paragraph of Section 4.2. One can aso use the greedy hyperparameter seection processed described in [15]. We aso used two optiona data preprocessing, i.e. high pass fitering to denoise, and data shifting to synthesize more training data. The activation of each technique is subject to a contro parameter which is tuned using the same process. Note that since a the hidden nodes in ayers 2, 3 output sma vaues ony, with the settings we used for experiments, the regression ayer s abiity to predict arger numbers (e.g. >1000) is imited. To predict arger numbers, one can consider either rescaing the data or setting smaer λ to adjust to the numerica range of the specific dataset Network Depth and Number of Parameters The proposed mode has a moderate number of ayers (four if we count the convoution/pooing as one), and hence a moderate number of parameters to estimate. For exampe, with d 28, d te 1 (one W and one W r), n f and d f set to 20 and 5, n (3) 12, we have: { W, b } ( ) + ( ) +( ) + (12 + 1) 3197 (17) It is possibe to add more ayers to construct a deeper architecture based on tempora embedding and convoution. However, the data itsef must be compex enough to provide more potentia for the mode to expoit. Given the granuarity of daiy human behaviors, for the task of predicting modaities such as traveing distance/time and power consumption, a deeper architecture has ony imited effect. 5. CONCLUSION Motivated by the observation that reguarities in periodica time-series sometimes manifest at different moments and at varied paces, in this paper we propose a technique 9