Multi-Robot Transfer Learning: A Dynamical System Perspective

Transcription

1 2017 IEEE/RSJ International Conerence on Intelligent Robots and Systems (IROS) September 24 28, 2017, Vancouver, BC, Canada Multi-Robot Transer Learning: A Dynamical System Perspective Mohamed K. Helwa and Angela P. Schoellig Abstract Multi-robot transer learning allows a robot to use data generated by a second, similar robot to improve its own behavior. The potential advantages are reducing the time o training and the unavoidable risks that exist during the training phase. Transer learning algorithms aim to ind an optimal transer map between dierent robots. In this paper, we investigate, through a theoretical study o single-input singleoutput (SISO) systems, the properties o such optimal transer maps. We irst show that the optimal transer learning map is, in general, a dynamic system. The main contribution o the paper is to provide an algorithm or determining the properties o this optimal dynamic map including its order and regressors (i.e., the variables it depends on). The proposed algorithm does not require detailed knowledge o the robots dynamics, but relies on basic system properties easily obtainable through simple experimental tests. We validate the proposed algorithm experimentally through an example o transer learning between two dierent quadrotor platorms. Experimental results show that an optimal dynamic map, with correct properties obtained rom our proposed algorithm, achieves 60-70% reduction o transer learning error compared to the cases when the data is directly transerred or transerred using an optimal static map. I. INTRODUCTION Machine learning approaches have been successully applied to a wide range o robotic applications. This includes the use o regression models, e.g., Gaussian processes and deep neural networks, to approximate kinematic/dynamic models [1], inverse dynamic models [2], and unknown disturbance models [3], [4] o robots. It also includes the use o reinorcement learning (RL) methods to automate a variety o human-like tasks such as screwing bottle caps onto bottles and arranging lego blocks [5]. Nevertheless, machine learning methods typically require collecting a considerable amount o data rom real-world operation or simulations o the robots, or a combination o both [6]. Transer learning (TL) reduces the burden o a robot to collect real-world data by enabling it to use the data generated by a second, similar robot [7], [9]. This is typically carried out in two phases [9]. In the irst phase, both robots generate data, and an optimal mapping between the generated data sets is learned. In the second phase, the learned map is used to transer subsequent learning data collected by the second robotic system, called the source system, to the irst robotic system, called the target system (see Figure 1). The goal o transer learning is to reduce the time needed or teaching robots new skills and to reduce the unavoidable risks The authors are with the Dynamic Systems Lab ( Institute or Aerospace Studies, University o Toronto, Canada. M. K. Helwa is also with the Electrical Power and Machines Department, Cairo University, Giza, Egypt. mohamed.helwa@robotics.utias.utoronto.ca, schoellig@utias.utoronto.ca. This research was supported by NSERC grant RGPIN and OCE/SOSCIP TalentEdge Project # d(t) System 2 + e 2(t) u 2(t) x 2(t) Controller 2 Plant 2 System 1 Learning 2 Learning 1? Transer Learning Controller 1 Plant 1 + e 1(t) u 1(t) x 1(t) Fig. 1. Multi-robot transer learning ramework; transer learning allows System 1 (target system) to use data rom System 2 (source system). In this paper, we provide an algorithm or determining the properties o the optimal transer learning map between robotic systems. Figure adopted rom [9]. that usually exist in the training phase, particularly or the cases where the target robotic platorm is more expensive or more hazardous to operate than the source robotic platorm. The use o transer learning in robotics can be classiied into (i) multi-task transer learning, in which the data gathered by a robot when learning a particular task is utilized to speed up the learning o the same robot in other similar tasks [2], [5], [7], [10] [12], and (ii) multirobot transer learning, where the data gathered by a robot is used by other similar robots [7] [9], [13] [17]. The latter is the main ocus o this paper. Multi-robot transer learning has received less attention in the literature, c. [18]. In [13], task-dependent disturbance estimates are shared among similar robots to speed up learning in an iterative learning control (ILC) ramework, while in [14], [15], polices and rules are transerred between simple, inite-state systems in an RL ramework to accelerate robot learning. For a similar coniguration, skills learned by two dierent agents in [8] are used to train invariant eature spaces instead o transerring policies. One typical approach or transer learning, used in many applications including robotics, is maniold alignment, which aims to ind an optimal, static transormation that aligns datasets [16], [17], [19], [20]. In [16], [17], maniold alignment is used to transer input-output data o a robotic arm to another arm to improve the learning o a model o the second arm. As partially stated in [9], although multi-robot transer learning has been successully applied in some robotic examples, there is still an urgent need or a general, theoretical study o when multi-robot transer learning is beneicial, how the dynamics o the considered robots aect the quality o transer learning, what orm the optimal transer map /17/$ IEEE 4702

2 takes, and how to eiciently identiy the transer map rom a ew experiments. To ill this gap, the authors o [9] recently initiated a study along these lines or two irst-order, linear time-invariant (LTI), single-input single-output (SISO) systems. In particular, in [9], a simple, constant scalar is applied to align the output o the source system with the output o the target system, and then an upper bound on the Euclidean norm o the transormation error is derived and minimized with respect to (w.r.t.) the transormation parameter. The paper [9] also utilizes the derived, minimized upper bound to analyze the eect o the dynamics o the source and target systems on the quality o transer learning. In this paper, we study how the dynamical properties o the two robotic systems aect the choice o the optimal transer map. This paper generalizes [9], as we consider higherorder, possibly nonlinear dynamical systems and remove the restriction that the transormation map is a static gain. The contributions o this paper may be summarized as ollows. First, while many transer learning methods in the literature depend on inding an optimal, static map between multi-robot data sets [9], [16], [17], we show through our theoretical study that the optimal transer map is, in general, a dynamic system. Recall that in the time domain, static maps are represented by algebraic equations, while dynamic maps are represented by dierential or dierence equations. Second, we utilize our theoretical study to provide insights into the correct eatures or properties o this optimal, dynamic map, including its order and regressors (i.e., the variables it depends on). Third, based on these insights, we provide an algorithm or selecting the correct eatures o this transormation map rom basic properties o the source and target systems that can be obtained rom ew, easy-to-execute experimental tests. Knowing these eatures greatly acilitates learning the map eiciently and rom little data. Fourth, we veriy the soundness o the proposed algorithm experimentally or transer learning between two dierent quadrotor platorms. Experimental results show that an optimal, dynamic map, with correct eatures obtained rom our proposed algorithm, achieves 60-70% reduction o transer learning error, compared to the cases when the data is directly transerred or transerred through a static map. This paper is organized as ollows. Section II provides preliminary, dynamic-systems deinitions. In Section III, we deine the transer learning problem studied in this paper. In Section IV, we provide theoretical results on transormation maps that achieve perect transer learning, and then utilize these results to provide insights into the correct eatures o optimal transer maps. In Section V, we present our proposed, practical algorithm. Section VI includes a robotic application, and Section VII concludes the paper. II. BACKGROUND In this section, we review basic deinitions rom control systems theory needed in later sections, see [21], [22]. We irst introduce these deinitions or linear systems, and then generalize them to an important class o nonlinear systems, namely control aine systems. To that end, consider irst the LTI, SISO, n-dimensional state space model, ẋ(t) = Ax(t) + Bu(t) y(t) = Cx(t), where x(t) R n is the system state vector, u(t) R is its input, and y(t) R is its output. It is well known that the input-output representation o (1) is the transer unction (1) G(s) = Y (s) U(s) = C(sI A) 1 B =: N(s) D(s), (2) where N(s) and D(s) are polynomials in s, and we assume without loss o generality (w.l.o.g.) that they do not have common actors. Evidently, the system (2) is bounded-inputbounded-output (BIBO) stable i and only i all the roots o D(s) are in the open let hal plane (OLHP). The relative degree o the system (2) is deg(d(s)) deg(n(s)), that is the order o the denominator polynomial D(s) minus the order o the numerator polynomial N(s). The deinition o relative degree remains the same or discrete-time linear systems Y (z) U(z) = N(z) D(z), where z is the orward shit operator. For (1), it can be shown using the series expansion o the transer unction (2) that the relative degree is the smallest integer r or which CA r 1 B 0, and consequently, the relative degree is also the lowest-order derivative o the output that explicitly depends on the input recalling y (r) (t) = CA r x(t) + CA r 1 Bu(t), where y (r) (t) represents the r-th derivative o y(t) w.r.t. t. The relative degree r can be calculated rom the step response o the system. For continuous-time systems, it is the lowest-order derivative o the step response y that changes suddenly when the input u is suddenly changed. For discretetime systems, it is the number o sample delays between changing the input and seeing the change in the output. We now extend the relative degree deinition to nonlinear systems. Let C denote the class o smooth unctions whose partial derivatives o any order exist and are continuous. The Lie derivative o a smooth unction λ(x) w.r.t. a smooth vector ield (x), denoted L λ, is the derivative o λ in the direction o ; that is, L λ := λ x (x). The notation L 2 λ is used or the repeated Lie derivative; that is, L2 λ = L (L λ(x)) = L λ(x) x (x). Similarly, one can derive an expression or L k λ, where k > 1. Now consider the SISO control aine system, ẋ(t) = (x(t)) + g(x(t))u(t) y(t) = h(x(t)), where x(t) D R n, u(t) R, y(t) R, and, g, h are C, nonlinear unctions. Analogous to linear systems, the relative degree o the system (3) is the smallest integer r or which L g L r 1 h(x) 0, or all x in the neighborhood o the operating point x 0. By successive derivatives o the output y, it can be shown that y (r) = L r h(x) + L gl r 1 h(x)u. Hence, the relative degree again represents the lowest-order derivative o the output that explicitly depends on the input. For example, the nonlinear dynamics θ = cos(θ) + u, θ ( π, π), with output θ and input u, have relative degree 2 or all θ in the operating range. (3) 4703

3 We next review the let inverse o the dynamics (3), which is used in the literature to reconstruct input u rom the output y, see [21], and which we utilize in our discussion in Section IV. Note that the inverse dynamics o linear systems can be easily derived rom the transer unction, and its stability is determined by the zeros o the original transer unction (the roots o the polynomial N(s)). Suppose that (3) has a well-deined relative degree r in the operating range. Recall that y (r) = L r h(x)+l gl r 1 h(x)u, where by deinition L g L r 1 h(x) 0. By reordering this equation and rom (3), we obtain the inverse dynamics ( ) L r ẋ = (x) g(x) h(x) L g L r 1 + h(x) u = Lr h(x) L g L r 1 h(x) + 1 L g L r 1 h(x) y(r), g(x) L g L r 1 h(x) y(r) with input y (r) and output u. A necessary condition or the stability o (4) is that the dynamics o (4) when y(t) = 0 uniormly (consequently, y (r) (t) = 0) are stable in the Lyapunov sense; this is called the zero dynamics o the system (3). While it appears rom (4) that the inverse dynamics have n states, this is not the minimum realization o the inverse dynamics. Instead, or dynamic systems (3) with well-deined relative degree, one can always ind a nonlinear coordinate transormation to convert (3) into a special orm, called the Byrnes-Isidori normal orm. Using this orm, a minimum realization o (4) can be derived with (n r) states, inputs y, ẏ,, y (r), and output u, reer to [21], [22]. III. PROBLEM STATEMENT In this paper, we study transer learning between two robotic systems rom a dynamical system perspective. We use our theoretical results to provide insights into the properties o optimal transer maps. These insights acilitate the identiication o this optimal map rom data using, or instance, system identiication algorithms. In particular, as shown in Figure 2, we consider a source SISO dynamical system D S with input reerence signal d and output y s, representing the source robot, and a target SISO dynamical system D T with the same input d and output y t, representing the target robot. Assuming that d is an arbitrary bounded signal, the transer learning problem is to ind a transer map D T L with input y s and output y T L such that the error e between y t and y T L is minimized. To make the transer learning problem tractable, we assume that both the source system D S and the target system D T are input-output stable (this is typically characterized by the BIBO stability notion or LTI systems and by the Input to Output Stability (IOS) notion or nonlinear systems [23]). This is a reasonable assumption, given that input-output stability is necessary or the sae operation o the robot and transer learning is only eicient or stable systems [9]. IV. MAIN RESULTS In this section, we assume that the source system dynamics D S and the target system dynamics D T are known, and (4) Fig. 2. Illustrative igure o the transer learning problem: the objective is to identiy a transer learning map D T L to minimize the error between the transerred output y T L and the target system s output y t. then provide a theoretical study on when it is possible to identiy a dynamic map D T L that achieves perect transer learning rom D S to D T, i.e., it perectly aligns y t and y T L resulting in zero transer learning error (e(t) = 0). From this theoretical study, we provide insights into the correct properties o the dynamic map D T L, including its order, relative degree, and input-output variables. We then show that these properties can be determined rom basic properties o the source and target systems, which can be identiied through short, simple experiments. There is no need to know the source/target system dynamics a priori. Knowing the properties o the optimal transer map greatly acilitates the identiication o this map rom data using standard system identiication tools, as we will show in Section VI. For simplicity, we irst present our theoretical study and insights or linear systems, and then show that these insights remain valid or nonlinear systems. To that end, in this paper, we say that an LTI system is minimum-phase i the dynamics o the system and its inverse dynamics are BIBO stable. Theorem 4.1: Consider two continuous-time, BIBO stable, SISO, LTI systems, with rational transer unctions G S (s) and G T (s), and suppose that G S (s) is minimumphase. Then, there exists a causal, BIBO stable map rom the source system G S (s) to the target system G T (s) that achieves perect transer learning i and only i the relative degree o G S (s) the relative degree o G T (s). Proo: ( ) Let G S (s) := N S(s) D S (s) and G T (s) := N T (s) D T (s). By assumption, there exists a causal unction G α (s) such that or any bounded input u, (G α (s)g S (s) G T (s))u(s) = 0. Since u is arbitrary, then clearly G α (s)g S (s) G T (s) = 0. Equivalently, G α (s)g S (s) = G T (s). Let G α (s) := Nα(s) N S (s) D S (s) = N T (s) D α(s). Then, we have Nα(s) D α(s) D T (s). Even in the presence o pole-zero cancellations, it can be shown that the above equation implies (deg(d α (s)) + deg(d S (s))) (deg(n α (s)) + deg(n S (s))) = (deg(d T (s)) deg(n T (s))). By reordering the terms on the let hand side (LHS), the summation o the relative degrees o G α (s) and G S (s) is equal to the relative degree o G T (s). Since G α (s) is causal, the relative degree o G α (s) 0, and the result ollows. ( ) Suppose that the relative degree o G S (s) the relative degree o G T (s). We construct a causal, stable map G α (s) that achieves perect transer learning. Let G α (s) := D S(s) N T (s) N S (s) D T (s). (5) 4704

4 Since the relative degree o G S (s) the relative degree o G T (s), we have deg(d S (s)) deg(n S (s)) deg(d T (s)) deg(n T (s)). Equivalently, deg(d S (s)) + deg(n T (s)) deg(d T (s)) + deg(n S (s)). This implies G α (s) is a causal unction. Notice that the poles o G α (s) are a subset o the roots o D T (s) and N S (s). Then, since G T (s) is BIBO stable and G S (s) is minimum-phase by assumption, the roots o D T (s) and N S (s) are all in the OLHP, and G α (s) is a BIBO stable transer unction. Next, one can veriy that or the selected G α (s), we have G α (s)g S (s) G T (s) = 0, and consequently G α (s) achieves perect transer learning. Equation (5) and its associated discussion are similar to standard methods in linear control synthesis. Similar results can be derived or discrete-time LTI systems. Theorem 4.2: Consider two discrete-time, BIBO stable, SISO, LTI systems, with rational transer unctions G S (z), G T (z), and suppose that G S (z) is minimum-phase. Then, there exists a causal, BIBO stable map rom the source system G S (z) to the target system G T (z) that achieves perect transer learning i and only i the relative degree o G S (z) the relative degree o G T (z). Theorems 4.1 and 4.2 provide the ollowing insights into the properties o the optimal transer maps between systems. Insight 1: From (5), one can see that the optimal transer map is, in general, a dynamic system. Thereore, limiting the transer map to be static [9], [16] may be restrictive; see also Section VI. Insight 2: To be able to identiy the optimal transer learning map rom data using system identiication algorithms, it is important to decide on the right order o the dynamic map. From (5), the order o the optimal map that achieves zero transer learning error is in general deg(n S (s)) + deg(d T (s)). Equivalently, the correct order o the map is n s r s + n t, where n s is the order o the source system, n t is the order o the target system, and r s is the relative degree o the source system, which can be identiied experimentally rom the system step response as stated in Section II. Insight 3: From (5), the relative degree o the optimal transer learning map is r t r s, where r s, r t are the relative degrees o the source and target systems, respectively. The relative degree o the transer map is also needed or standard system identiication algorithms. By knowing the order and the relative degree o the transer learning map, the regressors o the map are determined. For instance, or a discretetime transer learning map o order 3 and relative degree 1, the map relates the output y T L (k) to the inputs y T L (k 1), y T L (k 2), y T L (k 3), y s (k 1), y s (k 2), y s (k 3). Insight 4: From Theorems 4.1 and 4.2, i the relative degree o the source system r s is greater than the relative degree o the target system r t, then we cannot ind a causal map satisying perect transer learning (zero transer learning error). Nevertheless, since we have the complete input-output data o the source robot available beore carrying out the transer learning, the causality requirement can be relaxed. For instance, although a discrete-time transer learning map rom {y s (k 1), y s (k), y s (k + 1), y T L (k 1)} to y T L (k) is non-causal, it can be implemented since all the uture values o y s are saved beore using the transer learning map or transerring the source data to the target system. However, system identiication computer tools such as MATLAB s identiication toolbox are typically used or identiying causal models such as causal transer unctions (MATLAB: test), nonlinear autoregressive exogenous (NARX) models (MATLAB: nlarx), and recurrent neural networks, among others. One possible trick to solve this problem is to tailor the input o the dynamic transer learning map as ollows. First, or continuous-time systems, we know rom (5) that the optimal transer map is Y T L(s) Y s(s) = Nα(s) D α(s), where deg(d α ) = n s + n t r s, deg(n α ) = n s + n t r t, and or this case deg(n α ) > deg(d α ) (non-causal map). Instead o identiying this non-causal map, we use standard system identiication computer tools to identiy the causal map Y T L (s) s (rs r t ) Y s(s) = N α(s) s (rs r t ) D α(s), which represents the Laplace transorm o the map rom y s (rs rt) (t), the (r s r t )- th derivative o the source robot s output y s (t), to y T L (t). Hence, rom the y s (t) response, we calculate y s (rs rt) (t) (and possibly low-pass ilter y s (t) to avoid noise ampliication). We then use y s (rs rt) (t) as the input to the dynamic transer map to be identiied. Notice that this is not the only choice. One can, or example, use the system identiication tools to identiy the causal map Y T L(s) P (s)y = Nα(s) s(s) P (s)d α(s), where P (s) is a known (r s r t )-th order polynomial in s, with all its roots in the OLHP. Since both y s and the polynomial P (s) are known, one can deine the data column or the tailored input o the dynamic map to be identiied. For instance, i or r s r t = 1, one selects the polynomial P (s) = s + 1, then the tailored input to the dynamic map, to be identiied, is y s (t)+ẏ s (t), and so on. Similarly, or discrete-time systems, we use system identiication tools to identiy the causal Y T L (z) z (rs r t ) Y = s(z) N α(z) z (rs r t ) D α(z) map. For this tailored, causal map, the input is y s,mod, which is obtained by shiting each element in the data column or y s orward in time by (r s r t ) samples, and the output is y T L. We now show that these insights remain valid or nonlinear systems. Theorems 4.1, 4.2 and the related insights mainly depend on the deinition o relative degree, which is also deined or control aine nonlinear systems as discussed in Section II. Hence, suppose that we have two smooth, control aine nonlinear systems o the orm (3): a source system with order n s and a well-deined relative degree r s in the operating range, and a target system with order n t and a well-deined relative degree r t in the operating range. Also, suppose that the source system dynamics and its inverse dynamics are both input-output stable, and that the target dynamics are input-output stable [23]. From (5), one can see that the optimal transer map, that achieves zero transer learning error, is composed o two cascaded systems: the inverse o the source system dynamics and the target system dynamics. Intuitively, the inverse o the source dynamics is utilized to successully reconstruct the input d rom the source output response y s, and then the target system dynamics are applied to exactly obtain the target output response y t rom d. A similar approach can 4705

5 Algorithm 1 Finding the properties o transer learning maps Given: (1) Two SISO robotic systems: a source system with order n s and well-deined relative degree r s, and a target system with order n t and well-deined relative degree r t ; (2) output responses o the source and target systems, y s and y t, respectively, or a bounded reerence input d. Objective: Find the properties o the optimal, dynamic transer learning map, particularly its order, relative degree, input and output training data. Steps: 1) I r s r t, proceed. Otherwise, jump to step 3. 2) Input data to the dynamic map: y s ; output data: y t ; order o the dynamic map is n s + n t r s ; relative degree o the dynamic map is r t r s. Stop. 3) Deine tailored input data to the dynamic map y s,mod. For continuous-time transer learning maps, y s,mod =, the (r s r t )-th derivative o the saved time response y s. For discrete-time maps, the y s,mod data column is obtained rom the y s column by shiting each element o y s orward in time by r s r t samples. 4) Input data: y s,mod ; output data: y t ; order o the dynamic map is n s + n t r t ; relative degree o the dynamic map is zero. y (rs rt) s be utilized or nonlinear systems to get zero transer learning error. From the last paragraph in Section II, we know that the minimum realization o the inverse dynamics o the source system has order n s r s, while the order o the target system dynamics is by deinition n t. Thereore, the correct order o the optimal dynamic map, composed o these two cascaded systems, is n s + n t r s, which is the same conclusion we reached in Insight 2 or linear systems. Then, or this optimal map, we have rom the relative degree deinition or the source system with internal state x (subscript S is dropped rom, g, h or notational simplicity) y s (rs) = L rs h(x) + L gl rs 1 h(x)d, where L g L rs 1 h(x) 0, and or the target dynamics with internal state v (subscript T is dropped rom, g, h) y (rt) T L = Lrt h(v) + L gl rt 1 h(v)d, where L g L rt 1 h(v) 0. By getting d rom the irst equation and substituting it in the second one, we have y (rt) h(v) L gl rt 1 h(v) L rt y (rt) T L L rs h(x) L gl rs 1 T L = + LgL r t 1 h(v) h(x) L gl rs 1 h(x) y(rs) s, i.e., explicitly depends on y(rs) s, and consequently, it is reasonable to select the relative degree o the optimal map rom y s to y T L to be r t r s as in Insight 3. V. ALGORITHM Inspired by the insights presented in the previous section, we provide an algorithm or getting the correct properties o the optimal, dynamic transer learning map between two robotic systems rom simple experiments. As discussed beore, we assume that both systems are input-output stable and that the source system has stable inverse dynamics. Once the properties o the map are determined, one can utilize any system identiication tool, such as MATLAB s identiication Fig. 3. The two quadrotor platorms used in our experiments; we learn a transer learning map rom the Parrot AR.Drone 2.0 (let) to the Parrot Bebop 2.0 (right). toolbox, to identiy the map rom collected data as we will show in our practical examples in Section VI. The identiied map can then be used to transer any subsequent learning data rom the source system to the target system. The main steps are summarized in Algorithm 1. Notice that step 2 o the algorithm directly ollows rom Insights 2, 3 in Section IV, while steps 3, 4 directly ollow rom Insight 4. To better understand the algorithm, suppose, as a toy numerical example, that we have two minimum-phase, discretetime systems with zero initial conditions and orders n s = 5 and n t = 3. We assume that this is the only available inormation about the systems. To identiy the relative degrees o the systems, we apply at time step k = 0 a step input to both systems. From the step response o the source system, we ound that the output only changes at k = 4, and consequently, r s = 4. Similarly, we ound that r t = 3. We then ollow the steps o our algorithm: (1) since r s > r t, we jump to step 3; (3) we construct the y s,mod data column by shiting each element in the step response y s orward in time by r s r t = 1 sample; (4) the input training data is y s,mod, the output training data is y t, the map order is 5, and its relative degree is 0. The transer learning map should relate y T L (k) to y T L (k 1),, y T L (k 5), y s,mod (k),, y s,mod (k 5) to best it the output data y t. One advantage o the proposed algorithm is that it does not require precise knowledge o the robots dynamics and/or parameters. Instead, it only requires the knowledge o basic properties o the robotic systems, namely the system order and the relative degree. The order o the robotic system can be determined rom approximate physics models, or even rom general inormation about the robot structure. For instance, an N-link manipulator has a dynamical model o order 2N. Similarly, the relative degree o the system may be determined rom physics models, or experimentally rom the step response o the system as discussed in Section II. Another advantage o the proposed algorithm is that it is generic in the sense that it can be combined with any system identiication model/algorithm. For instance, one can utilize the proposed algorithm to determine the correct properties o both linear and nonlinear dynamic transer maps. VI. APPLICATION In this section, we utilize the proposed algorithm to identiy a dynamic transer learning map between two dierent quadrotor platorms, namely the Parrot AR.Drone 2.0 and the Parrot Bebop 2.0 (see Figure 3), and then veriy through experimental results the eectiveness o our proposed map. Quadrotor vehicles have six degrees o reedom: the translational position o the vehicle s center o mass (x, y, z), measured in an inertial coordinate rame, and the vehicle s 4706

6 x, m Target System Source System Transerred Output Reerence Input Time, s Fig. 4. The training input-output data used to identiy the transer learning map in the x-direction, and the transerred output using the proposed map. The transerred output its the target system s output with 95.79%. attitude, represented by the Euler angles (φ, θ, ψ), namely the roll, pitch, and yaw angles, respectively. The ull state o the vehicle also includes the translational velocities (ẋ, ẏ, ż) and the rotational velocities (p, q, r), resulting in a dynamic model o the vehicle with 12 states. Detailed description o the quadrotor s dynamic model can be ound in [24]. In our experiments, the quadrotor s states are all measured by the overhead motion capture system, which consists o ten 4- mega pixel cameras running at 200 Hz. In our study, the two quadrotor platorms utilize a control strategy that consists o two controllers: (i) an onboard controller that runs at 200 Hz, receives the desired roll φ d, pitch θ d, yaw velocity r d and the z-axis velocity ż d, and outputs the thrusts o the quadrotor s our motors, and (ii) an o-board controller that is implemented using the open-source Robot Operating System (ROS), runs at 70 Hz, receives the desired vehicle s position, and outputs the commands (φ d, θ d, r d, ż d ) to the on-board controller. For the o-board controller, we utilize a nonlinear control strategy to stabilize the z-position o the vehicle to a ixed value and the yaw angle to zero, and then manipulate φ d and θ d to control the vehicle s motion in the x-, y- directions. In particular, we select φ d and θ d to implement a nonlinear transormation that decouples the dynamics in the x-, y-directions into approximate, linearized, second-order dynamics in each direction, and then utilize a proportionalderivative (PD) controller or each direction. More details can be ound in [25]. In this application, we identiy a transer learning map rom the Parrot AR.Drone 2.0 platorm, the source system, to the Parrot Bebop 2.0 platorm, the target system, or each o the x-, y-directions. We irst stabilize the vehicle s y- and z-positions to constant values, and study the motion in the x-direction. For this case, the input to each system is the desired x-value reerence, while the output is the actual x- value o the quadrotor. We start by collecting data or both vehicles in the x-direction. In particular, we apply the same desired reerence x d to both vehicles and detect their outputs (see Figure 4). We then utilize this collected data to identiy a continuous-time transer learning map with the aid o our proposed algorithm. Following the previous paragraph, we know that under the applied control strategy the x-direction dynamics or the quadrotors have approximately order 2, and by analyzing the dynamic equations, we have ound that the relative degree or the quadrotors in this case is 1. We have veriied this value experimentally rom the collected data in x, m Target Output Transerred Output Time, s Fig. 5. The output o the target quadrotor and the transerred output using our proposed, dynamic map or transerring six minutes o collected data o the source quadrotor. The proposed map achieves an RMS error o m, which reduces the direct transer learning error by 70.65%. Fit to Data, % Training Data 330 Second Testing Data Transer Function Order Fig. 6. The igure shows how optimal transer unctions with dierent orders and zero relative degree it the y-direction training and testing data. The NRMSE itness measure is used. The dynamic order 3, proposed by our algorithm, best its the training data, and it achieves the second highest it to the 330-second testing data. Figure 4 as discussed in Section II. To sum up, we have n s = n t = 2 and r s = r t = 1. By ollowing the steps o our algorithm, the correct input to the transer learning map is the x-output o the source system, y s, its output is the x-output o the target system, y t, its dynamic order is 3, and its relative degree is 0. Since the applied control strategy turns the closed loop into an approximately linear behavior in the x-direction as discussed in the previous paragraph, we identiy a linear transer learning map with the desired properties using MATLAB s test or identiying transer unctions. The obtained transer unction its the training data (y t ) with 95.79%, measured based on the well-known normalized root mean square error (NRMSE) itness value (it = 100(1 NRMSE)%), see Figure 4. For comparison, we have also identiied an optimal, static gain rom y s to y t using MATLAB s test with the unction s orders set to (0, 0); the gain is , and it its the data with 27.28%. We next test the identiied transer learning maps or transerring six minutes o collected data rom the Parrot AR.Drone 2.0 to the Parrot Bebop 2.0. Figure 5 shows the actual output o the target quadrotor and the transerred output using our proposed map. The proposed map achieves an RMS error o m, compared to m or direct transer learning (identity map), and m or the identiied, optimal, static map. The proposed map achieves 70.65% reduction in error over the direct transer learning, while the optimal, static map achieves only 4.81% reduction. We similarly identiy a transer learning map rom the Parrot AR.Drone 2.0 to the Parrot Bebop 2.0 in the y- direction. We omit the details or brevity. The proposed, identiied map is a transer unction with order 3 and zero relative degree, and it its the training data with 96.59%. The optimal, static TL gain is 0.518, and it its the data with 21.73%. We then test the proposed, dynamic map or transerring 330-second y-direction data rom the Parrot 4707

7 TABLE I TL ERROR (RMS) FOR DATA FROM TRACKING A CIRCLE Direct TL Proposed Map Reduction x-direction 0.87 m m 49.26% y-direction m m 91.5% Total m m 59.58% AR.Drone 2.0 to the Parrot Bebop 2.0. For this testing data, our proposed map has an RMS error o m, which achieves 68.29% error reduction compared to direct transer learning and 62% error reduction compared to the optimal, static gain. Figure 6 shows how optimal transer unctions with dierent orders and zero relative degree it the training and testing data. The order 3, proposed by our algorithm, best its the training data, and it achieves the second highest it to the 330-second testing data ater the order 2. However, using this testing data as training data or identiying new transer unctions again shows that the third-order transer unction outperorms the second-order one in itting this testing data. While it is expected due to overitting that higher-order transer unctions have lower it on the testing data, this is less obvious or the training data. The explanation is likely that the orders in Figure 6 are not high enough to overit the 25-second training data (5000 data points). Indeed, or order 50, the obtained transer unction its the training data with 98.14%, but it completely ails to transer the testing data. We then test the proposed, identiied TL maps in the x-, y-directions or transerring the x-, y-data rom the Parrot AR.Drone 2.0 to the Parrot Bebop 2.0, or the case where both vehicles are required to track a unit circle in the (x, y)- plane (with requency 0.14 Hz, which is dierent rom the requencies o the reerences used in the training data). Table I summarizes the transer learning errors or both the proposed map and the direct transer learning. Our proposed, dynamic maps achieve signiicant reduction o the direct transer learning errors. However, the total improvement is less than in the previous examples. This is likely due to the unmodeled coupling in the x-, y-directions. The optimal TL map or this case should be a (2 2) matrix o dynamic maps to account or the coupling between the two directions. VII. CONCLUSIONS We have studied multi-robot transer learning (TL) rom a dynamical system perspective or SISO systems. While many existing methods utilize static TL maps, we have shown that the optimal TL map is a dynamic system and provided an algorithm or determining the properties o the dynamic map, including its order and regressors, rom knowing the order and relative degree o the systems. These basic system properties can be obtained rom approximate physics models o the robots or rom simple experiments. Our results show that or the testing data, dynamic maps, with correct eatures rom our proposed algorithm, achieve on average 66% reduction o TL errors compared to direct TL, while optimal, static gains achieve only 15% reduction. For uture research, we consider the generalization o this algorithm to multi-input multi-output systems. REFERENCES [1] D. Nguyen-Tuong, J. Peters. Model learning or robot control: a survey. Cognitive Processing, vol. 12(4), pp , [2] Q. Li, J. Qian, Z. Zhu, X. Bao, M. K. Helwa, A. P. Schoellig. Deep neural networks or improved, impromptu trajectory tracking o quadrotors. IEEE Intl. Con. on Robotics and Automation, 2017, pp [3] C. J. Ostaew, A. P. Schoellig, T. D. Baroot. Robust constrained learning-based NMPC enabling reliable mobile robot path tracking. The International Journal o Robotics Research, vol. 35(13), pp , [4] F. Berkenkamp, A. P. Schoellig. Sae and robust learning control with Gaussian processes. European Control Con., 2015, pp [5] S. Levine, N. Wagener, P. Abbeel. Learning contact-rich manipulation skills with guided policy search. IEEE Intl. Con. on Robotics and Automation, 2015, pp [6] A. Marco, F. Berkenkamp, P. Hennig, A. P. Schoellig, A. Krause, S. Schaal, S. Trimpe. Virtual vs. real: trading o simulations and physical experiments in reinorcement learning with Bayesian optimization. IEEE Intl. Con. on Robotics and Automation, 2017, pp [7] C. Devin, A. Gupta, T. Darrell, P. Abbeel, S. Levine. Learning modular neural network policies or multi-task and multi-robot transer. IEEE Intl. Con. on Robotics and Automation, 2017, pp [8] A. Gupta, C. Devin, Y. Liu, P. Abbeel, S. Levine. Learning invariant eature spaces to transer skills with reinorcement learning. Intl. Con. on Learning Representations, Available at ArXiv, arxiv: [cs.ai]. [9] K. V. Raimalwala, B. A. Francis, A. P. Schoellig. An upper bound on the error o alignment-based transer learning between two linear, time-invariant, scalar systems. IEEE/RSJ Intl. Con. on Intelligent Robots and Systems, 2015, pp [10] P. Janssens, G. Pipeleers, J. Swevers. Initialization o ILC based on a previously learned trajectory. American Control Con., 2012, pp [11] M. Hamer, M. Waibel, R. D Andrea. Knowledge transer or highperormance quadrocopter maneuvers. IEEE/RSJ Intl. Con. on Intelligent Robots and Systems, 2013, pp [12] T. T. Um, M. S. Park, J.-M. Park. Independent joint learning: a novel task-to-task transer learning scheme or robot models. IEEE Intl. Con. on Robotics and Automation, 2014, pp [13] A. P. Schoellig, J. Alonso-Mora, R. D Andrea. Limited beneit o joint estimation in multi-agent iterative learning. Asian Journal o Control, vol. 14(3), pp , [14] B. Lakshmanan, R. Balaraman. Transer learning across heterogeneous robots with action sequence mapping. IEEE/RSJ Intl. Con. on Intelligent Robots and Systems, 2010, pp [15] G. Boutsioukis, I. Partalas, I. Vlahavas. Transer learning in multiagent reinorcement learning domains. Recent Advances in Reinorcement Learning, pp , Springer-Verlag Berlin, [16] B. Bócsi, L. Csató, J. Peters. Alignment-based transer learning or robot models. Intl. Joint Con. on Neural Networks, 2013, pp [17] N. Makondo, B. Rosman, O. Hasegawa. Knowledge transer or learning robot models via local procrustes analysis. IEEE-RAS Intl. Con. on Humanoid Robots, 2015, pp [18] K. Tuyls, G. Weiss. Multiagent learning: basics, challenges, and prospects. AI Magazine, vol. 33(3), pp , [19] C. Wang, S. Mahadevan. Maniold alignment using procrustes analysis. Intl. Con. on Machine Learning, 2008, pp [20] C. Wang, S. Mahadevan. A general ramework or maniold alignment. AAAI Fall Symposium on Maniold Learning and Its Applications, 2009, pp [21] F. J. Doyle, M. A. Henson. Nonlinear Systems Theory. M.A. Henson, D.E. Seborg (Eds), Nonlinear Process Control, Prentice Hall, [22] A. Isidori. Nonlinear Control Systems, 3rd edition. Springer-Verlag London Limited, [23] E. Sontag, Y. Wang. A notion o input to output stability. European Control Con., 1997, pp [24] A. P. Schoellig, M. Hehn, S. Lupashin, R. D Andrea. Feasibility o motion primitives or choreographed quadrocopter light. American Control Con., 2011, pp [25] M. K. Helwa, A. P. Schoellig. On the construction o sae controllable regions or aine systems with applications to robotics. IEEE Con. on Decision and Control, 2016, pp Available at ArXiv, arxiv: [cs.sy]. 4708