NONLINEAR FEEDFORWARD-FEEDBACK CONTROL OF AN UNCERTAIN, TIME-DELAYED MUSCULOSKELETAL ARM MODEL FOR USE IN FUNCTIONAL ELECTRICAL STIMULATION

Transcription

1 NONLINEAR FEEDFORWARD-FEEDBACK CONTROL OF AN UNCERTAIN, TIME-DELAYED MUSCULOSKELETAL ARM MODEL FOR USE IN FUNCTIONAL ELECTRICAL STIMULATION By PETER COOMAN Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Dissertation Advisor: Robert F. Kirsch Department of Biomedical Engineering CASE WESTERN RESERVE UNIVERSITY January 2014

2 CASE WESTERN RESERVE UNIVERSITY SCHOOL OF GRADUATE STUDIES We hereby approve the thesis/dissertation of Peter Cooman candidate for the DOCTOR of PHILOSOPHY degree *. (signed) Robert F. Kirsch (chair of the committee) Patrick E. Crago Antonie J. van den Bogert Frans C.T. van der Helm Wei Lin (date) 03/27/2013 *We also certify that written approval has been obtained for any proprietary material contained therein. 2

3 Contents List of Tables... 6 List of Figures... 7 Abbreviations... 8 Nonlinear Feedforward-Feedback Control of an Uncertain, Time-delayed Musculoskeletal Arm Model for use in Functional Electrical Stimulation... 9 Abstract... 9 I. Introduction Spinal cord injury background Functional electrical stimulation motor neuroprostheses background Current FES neuroprostheses Control of current upper extremity FES neuroprostheses Performance requirements of upper extremity FES motor neuroprostheses II. Model-based evaluation of advanced feedforward-feedback controllers for restoring arm movements via functional electrical stimulation Abstract Introduction Methods Modeling approach Feedforward-Feedback PD Control Sliding Mode Control Adaptive Control Resolving muscle redundancy Simulation protocol Results Discussion III. Nonlinear feedforward-feedback control of human arm FES with time delay compensation Abstract Introduction Methods Model-based approach The controller

4 3.3.3 Resolving muscle redundancy Experimental protocol Results Discussion IV. Feasibility of controlling a realistic 5 Degrees of Freedom human arm using Functional Electrical Stimulation Abstract Introduction Methods General overview The musculoskeletal arm model The controller Control signal sampling Resolving muscle redundancy Performance measurement and requirements Controller tuning Additional perturbations Results Discussion V. Conclusions and future recommendations Overview Limitations and future research Improving robustness against model uncertainty Alternative time delay compensation mechanism Functionally relevant performance requirements, or how good is good enough? Significance Speculation Gain-scheduling: replicating able-bodied human arm movement control Divide-and-conquer: a new control paradigm for upper extremity FES Interacting with the environment: simultaneous position and force control VI. Acknowledgements VII. Appendix

5 Appendix A: Derivation of the equations of motion of a two degree of freedom (2DOF) planar arm Appendix B: Nominal PD-control Appendix C: Sliding Mode Control Appendix D: Adaptive Control Appendix E: Combined feedforward-feedback controller with time delay compensation theoretical stability analysis Introduction Stability Lyapunov s Direct Method Preliminaries Stability analysis Appendix F: 5 Degrees of Freedom arm model (MATLAB code for Robotics Toolbox) VIII. Bibliography

6 List of Tables Table II-1. Segment inertial parameters for the 2DOF arm model Table II-2. Simulated tracking performance for high-gain feedforward-feedback PD, adaptive and sliding mode control Table II-3. Simulated tracking accuracy (RMSEs) for low-gain feedforwardfeedback PD control (PD) and sliding mode control (SMC) Table III-1. Segment inertial parameters for the 2DOF arm model Table IV-1. Contributions of various perturbations on optimal tracking performance

7 List of Figures Figure I-1 Upper extremity functional electrical neuroprosthesis design Figure I-2. Feedforward control block diagram Figure I-3. Feedback control block diagram Figure I-4. Combined feedforward-feedback block diagram Figure I-5. Adaptive control block diagram Figure II-1. Block diagrams for pure feedforward, pure feedback and combined feedforward-feedback control Figure II-2. Two degrees of freedom (2DOF) planar arm model Figure II-3.. Block diagrams for the feedforward-feedback PD, sliding mode and adaptive controllers and muscle redundancy resolution Figure II-4. Simulated tracking performance for high-gain feedforward-feedback PD, adaptive and sliding mode control Figure II-5. Simulated tracking performance for low-gain feedforward-feedback PD, adaptive and sliding mode control Figure III-1. Control system block diagram of combined feedforward-feedback control Figure III-2. Two degrees of freedom (2DOF), planar arm model Figure III-3. Block diagram of the nonlinear feedforward-feedback time delay compensating controller Figure III-4. Simulated tracking performance of the FES-enabled arm for the nonlinear feedforward-feedback PID controller with time delay compensation Figure IV-1. 5DOF arm model DOFs, ranges of motion and inertial parameter values Figure IV-2. Block diagrams of the nonlinear feedforward-feedback PID controller with time delay compensation, muscle redundancy resolution and controller tuning approach Figure IV-3. Optimal RMSEs for 36 combinations of model uncertainty and measurement noise Figure IV-4. Simulated optimal tracking performance in the absence of model uncertainty and measurement noise Figure IV-5. simulated optimal tracking performance in the presence of measurement noise Figure IV-6. Simulated optimal tracking performance in the presence of model uncertainty Figure IV-7. Simulated optimal tracking performance in the presence of model uncertainty and measurement noise Figure V-1. Block diagram of gain-scheduling controller Figure V-2. Layered structure control block diagram Figure VII-1. Skeletal system of the two degrees of freedom, planar arm model

8 Abbreviations AC DOF FES PD PID SCI SMC s.t. RMSE Adaptive control Degree of freedom Functional electrical stimulation Proportional-derivative control Proportional-integral-derivative control Spinal cord injury Sliding mode control subject to Root-mean-square error 8

9 Nonlinear Feedforward-Feedback Control of an Uncertain, Time-delayed Musculoskeletal Arm Model for use in Functional Electrical Stimulation Abstract by PETER COOMAN Using a model-based approach, we designed and evaluated nonlinear combined feedforward-feedback algorithms to control arm movements in the presence of a wide range of perturbations: (1) manipulation of objects of unknown mass, (2) sensor noise, (3) muscle fatigue, (4) model uncertainty with respect to the true inertial properties of the arm and the true muscle dynamics, and (5) input time delays. These algorithms were developed specifically for use in FES neuroprostheses for individuals with paralysis due to spinal cord injury and other neurological disorders. An initial comparison of combined feedforwardfeedback proportional-derivative (PD), adaptive control, and sliding mode control showed that input time delays quickly caused instability for all three controllers if feedback gains were chosen too high. Decreasing the feedback gains (i.e., shifting towards feedforward control) re-established stability, but greatly reduced performance as measured by the root-mean-square error between the specified movement intent and the simulated joint angles. Input time delays are unavoidable in FES applications, arising from the muscle dynamics and the 9

10 relatively low stimulation frequency (i.e. 12Hz) typically used in upper extremity FES neuroprostheses. The destabilizing effects of time delay therefore cannot be ignored. Using a 2DOF musculoskeletal arm model, we designed and evaluated a nonlinear combined feedforward-feedback controller with time delay compensation. In the presence of a typical 80ms time delay, this controller achieved excellent tracking accuracy, both under ideal conditions (shoulder RMSE: 0.27º, elbow RMSE: 0.62º) and in the presence of a wide variety of perturbations expected under normal operating conditions (shoulder RMSE: 2.99º, elbow RMSE: 5.15º). We extended this time delay compensating controller to a more functionally relevant 5DOF arm model. This extended controller achieved stable, accurate tracking, even in the presence of time delay, measurement noise, muscle fatigue and additional loading - but only if the inertial properties of the arm were exactly known. Future research in gravity compensation and adaptation may further improve the robustness of the time delay compensating controller for the 5DOF arm model. This work represents an important step towards a clinically viable controller for upper extremity FES neuroprostheses for human SCI subjects. 10

11 I. Introduction 1.1 Spinal cord injury background A spinal cord injury (SCI) is defined as damage to the spinal cord, often as the result of trauma (e.g., bruising, severing). In terms of voluntary movement control, the spinal cord transmits neural signals between the brain and the rest of the body. When this pathway is interrupted, the individual suffers loss of voluntary motor control and sensation below the injury site. The severity of functionality loss depends on the level of injury and the extent of damage to the spinal cord (i.e. complete or incomplete). As a general rule, functional loss increases with higher levels of injury. When the injury occurs between the spinal cord segments T2 and S2, the individual suffers paralysis of the lower extremities only (paraplegia). While an injury between C1 and T1 results in paralysis of both the lower and upper extremities (tetraplegia). The National Spinal Cord Injury Database has estimated the number of people in the U.S. living with a SCI to be approximately 259,000. The annual incidence rate was estimated at about 12,000 new cases. The average age at injury is 40.2, with the majority of SCI cases occurring in males (80.9%). The four leading causes of SCI are vehicular accidents (42.1%), falls (26.7%), violence (15.1%) and sports (7.6%) [25]. 11

12 1.2 Functional electrical stimulation motor neuroprostheses background Functional Electrical Stimulation (FES) is a rehabilitation technique aimed at restoring lost functionality by applying small electrical currents to neural structures. It has a wide range of applications including pain relief [42] and hastening awakening in long term coma patients [6]. FES is primarily used in the context of neuroprostheses: devices that aim to artificially replace the function of damaged or otherwise malfunctioning structures by electrically stimulating nerves. Some examples of FES neuroprostheses include cochlear implants and pacemakers. Motor neuroprostheses form a subclass of neuroprostheses where the main goal is to restore motor function in individuals who have suffered a SCI, a cerebrovascular accident (stroke), or other neurological disorders. For these cases, FES can be applied to artificially stimulate the paralyzed muscles and/or the peripheral nerves that innervate them, provided that the muscle itself is intact and the peripheral Figure I-1 Upper extremity functional electrical neuroprosthesis design 12

13 nerve has not sustained damage from the SCI. A FES motor neuroprosthesis generally operates as follows: 1) sensors measure signals from a command source (e.g., remaining voluntary EMG, motor cortex). These signals are subsequently decoded to determine the individual s intended movement. 2) Based on the motor command, a controller determines a set of stimulation parameters (amplitude, frequency and timing) for the paralyzed muscles. 3) Stimulating electrodes (implanted or surface) apply the electrical stimulations either to the muscle directly or to the peripheral nerves which will then activate their corresponding muscles. By carefully coordinating the electrical stimulations, the resulting muscle contractions should allow the arm to perform the function as originally intended by the neuroprosthesis user. 1.3 Current FES neuroprostheses The work in our group focuses on SCIs at the mid-to high cervical levels. At this level the arm is partially or completely paralyzed (tetraplegia). For these individuals there is a FES neuroprosthesis available that can restore the hand grasp function [16, 18, 27 and 28]. Our long term goal is to extend the functionality of this neuroprosthesis to also restore elbow and shoulder function. This extension would significantly increase the individual s reachable work space and therefore greatly improve his or her functional independence. 13

14 Many challenges still remain before a FES neuroprosthesis for shoulder and elbow function can be practically implemented. The control problem is an important, if not the most important remaining challenge. The control problem is defined as finding an appropriate set of muscle activation parameters such that, when they are applied to the paralyzed muscles, the arm exhibits a predefined desired behavior (e.g., movement trajectory, hand orientation, force). Solving this problem is very difficult considering the fact that the human arm is a highly complex system exhibiting nonlinear behavior, redundancy (more muscles than degrees of freedom), and time-varying effects (muscle fatigue). 1.4 Control of current upper extremity FES neuroprostheses The proposed upper extremity FES neuroprosthesis controllers follow one (or a combination) of three basic control strategies: 1) feedforward control, 2) feedback control and 3) adaptive control [10]. Most current FES motor neuroprostheses use a feedforward control strategy [28-29]. To visualize the concept of feedforward control, consider the following block diagram: 14

15 Figure I-2. Feedforward control block diagram Here G represents the forward dynamics of the human arm, perturbed by external disturbances d, whose output behavior is denoted by y. The output behavior comprises a wide variety of modalities including joint angles, muscle forces, hand position and orientation. We want y to follow a user-defined arm behavior r as closely as possible. The control problem then becomes one of finding the controller K that maps the desired arm behavior r to a set of muscle stimulation parameters u such that, when these stimulations are applied to the arm, y corresponds to the desired behavior r. In the absence of external disturbances d (d=0), the ideal solution is to let K represent the inverse model GG 1 of the arm dynamics G. If this model GG 1 is a perfect inverse, then the input-output relationship for this interconnection can be written as: yy = KK GG rr + GG dd = GG 1 GG rr + GG 0 = II rr = rr In practice, the external disturbances d are rarely equal to zero and the inverse model will always be an inaccurate simplification of the real-world system. These errors will cause the arm output y to deviate from its intended behavior r. However since the controller K only considers the desired output and not the 15

16 current arm behavior, it is unaware of these deviations and thus incapable of taking corrective actions. To allow for error compensation, a so called feedback controller needs to be implemented [10-21]. The basic feedback control tracking interconnection is shown below: Figure I-3. Feedback control block diagram Here artificial or natural sensors measure the arm behavior y [41] and feed this information back for comparison with the original desired behavior r. The difference between the measured output ym and r is defined as the error e and forms the input to the controller K. Using a model of the forward arm dynamics GG, many algorithms become available to synthesize in a suitable controller. The feedback control strategy has many benefits: it increases the stability of the arm, imposes a much lower cognitive burden on the FES neuroprosthesis user (as opposed to feedforward control), and allows for disturbance rejection. However 16

17 feedback control also has its limitations. It is slower than feedforward control, since the controller needs to wait for an error to build up before taking action. Also, sensors are never perfect, implying that additional noise is injected into the system. This noise generally has a negative effect on the overall system performance. Some researchers have proposed a combination of feedforward and feedback control, combining the benefits of both strategies [4, 40]: Figure I-4. Combined feedforward-feedback block diagram Here the feedforward controller Kff generates an initial estimate of the required muscle activations, while the feedback controller Kfb intervenes whenever there is an error y between the measured and desired output. As mentioned earlier, the human arm exhibits time-varying behavior, for example due to muscle fatigue. Some researchers ([10, 11, 36]) have proposed an adaptive control strategy where the measured and desired outputs are 17

18 tracked and an adaptation mechanism A autonomously adjusts the feedforward and/or feedback controller to maintain an acceptable level of performance. Figure I-5. Adaptive control block diagram 1.5 Performance requirements of upper extremity FES motor neuroprostheses Our long-term goal is to design upper extremity FES neuroprostheses that allow the user to independently perform a wide range of functional tasks (e.g., drinking from a cup, combing hair). Towards this goal, we aim to design a controller that stabilizes the arm and calculates the required electrical stimulation parameters such that the resulting muscle forces drive the arm and hand as closely as possible along some pre-defined intended trajectory. Under typical real-world operating conditions, a wide variety of perturbations act on the arm, significantly complicating the task of the controller. We argue that the most relevant of these perturbations are: (1) changing arm 18

19 dynamics due to interacting with objects of unknown mass, (2) muscle fatigue, (3) measurement noise, (4) uncertainty with respect to the maximum muscle forces (muscle model uncertainty), (5) uncertainty with respect to the inertial properties of the arm (i.e. arm segment mass, length and center of gravity location; skeletal model uncertainty) and (6) input time delays. Before we consider a candidate controller as acceptable for clinical implementation in an upper extremity FES neuroprosthesis, it needs to achieve both stability and high tracking accuracy in the presence of any or all of these perturbations. In the subsequent paragraphs we will discuss each of these six perturbations in greater detail: (1) Changing arm dynamics due to interaction with objects of unknown mass: the majority of functional tasks require FES neuroprosthesis users to interact with objects in their immediate surroundings. These tasks often require the user to lift the object and transport it over some distance (e.g., feeding, retrieving a book off a shelf). Over the lifetime of the neuroprosthesis, the user will interact with a wide variety of objects. It is not possible to know the mass of each of these objects a priori. However, picking up an object does change the inertial properties of the arm and therefore affects the stability and tracking accuracy of the FES-enabled arm. A viable candidate controller should maintain stability and satisfactory tracking performance, not only when the arm remains un-weighted, but also 19

20 when the hand and arm interact with a wide range of objects of varying, and a priori unknown to the controller, mass. (2) Muscle fatigue: as a muscle maintains a contraction, its ability to generate force gradually decreases, so the amount of muscle force generated at a given level of muscle activation decreases as fatigue increases. Unfortunately, we cannot practically measure the level of fatigue in vivo. This implies a certain level of muscle model uncertainty: if the controller predicts a set of electrical muscle stimulations, the respective muscles may generate very different forces depending on their current level of fatigue. (3) Measurement noise: Feedback controllers rely on sensors to measure the current behavior of the arm. Unfortunately these sensors are never perfectly accurate, so to some extent their measurements will always be corrupted. This is problematic, because the feedback controller cannot differentiate between errors caused by genuine trajectory deviations, which require corrective action, and errors caused by measurement noise. Compared to the noise-free case, adding measurement noise complicates the task of the feedback controller, resulting in decreased tracking performance, if not instability of the arm. Note that measurement noise does not affect feedforward control, since feedforward control relies exclusively on its internal model of the arm and the intended movement trajectory, while ignoring current arm behavior. 20

21 (4) Maximum muscle force uncertainty: it is possible in practice to measure in vivo the maximum force that each of the arm muscles can generate. Similar to muscle fatigue, the muscle forces generated for a given set of electrical muscle stimulations may vary significantly depending on the true maximum muscle forces. Even when the muscle forces that a given FES neuroprosthesis user can generate are overor under-estimated, the controller needs to adjust its electrical muscle stimulations accordingly to ensure stability and satisfactory tracking performance. (5) Skeletal model uncertainty: all feedforward controllers, and some advanced feedback controllers require a model of the dynamics of the human arm. The performance of the overall controlled FES neuroprosthesis strongly depends on the accuracy of these models [40]. An accurate arm model is an invaluable tool for FES neuroprostheses designers in general and is especially useful for controller designers [1]. The musculoskeletal arm models used by our group are based on the inverse model of the 3D arm dynamics, including the so-called scapulothoracic gliding plane [39]. The parameters of the model depend on a set of morphological data including muscle attachment sites, joint rotation centers and bone geometry. These parameters have been measured by Klein-Breteler during extensive cadaver studies [20]. The resulting arm model accurately reflects the arm dynamics of these cadavers and many 21

22 research questions can be answered with this model. However the arm morphology of any specific individual will differ from that of the measured cadaver. This means that a controller designed for optimal performance with respect to the cadaver based model, when applied in the FES neuroprosthesis of a SCI individual, will lead to suboptimal performance at best and instability at worst. Ideally the problem of model discrepancy could be resolved by measuring the morphological data of each SCI individual, in other words custom fitting the model to the specific neuroprosthesis user. Unfortunately many important inertial and morphological arm properties are difficult or impossible to accurately measure in vivo (e.g. maximum muscle forces). This means that we will have to accept a certain level of model uncertainty. To date we do not know the quantitative effects of model uncertainties on the stability and performance of the controlled FES neuroprosthesis. (6) Input time delays: corrective actions prescribed by the controller do not take effect instantaneously. There is a delay between the controller prescribing a set of electrical muscle stimulations and the generated muscle forces acting on the skeletal system. These time delays arise primarily from two sources: (1) the excitation-activation dynamics of the muscles (i.e. transportation of electrical signals over nerves, release and reuptake of calcium) and (2) the relatively low 12Hz frequency discrete commands typically used in upper extremity FES 22

23 neuroprostheses [28]. The stimulation frequency acts as a zero-order hold, so the controller can only update the electrical stimulations once every 80ms, after which the stimulation parameters remain constant for the subsequent 80ms. This makes control significantly more difficult because the controller cannot intervene to correct a trajectory deviation for 80ms, during which the arm deviates further away from its intended trajectory. Compared to continuous, instantaneous control, a lower stimulation frequency decreases tracking performance and may even destabilize the arm. While higher stimulation frequencies allow the controller to intervene faster and therefore limit the negative effects of time delays, these higher frequencies would lead to unacceptably higher fatigue rates [28]. While multiple studies have proposed controllers for upper extremity FES neuroprostheses, all of them neglected to evaluate one or more of these six perturbations - especially input time delays. Using a two degrees of freedom (2DOF) and a 5DOF musculoskeletal arm model, we evaluated the strengths and weaknesses of three popular, nonlinear, combined feedforwardfeedback control strategies for controlling FES-generated movements in the presence of the six perturbations described earlier. After identifying input time delays as the most significant threat to arm stability and tracking accuracy, we designed a novel feedforward-feedback controller that maintains stability and performance for all six perturbations, including 23

24 input time delays. We designed and evaluated two versions of this time delay compensating controller, using a 2DOF and a 5DOF arm model respectively. 24

25 II. Model-based evaluation of advanced feedforward-feedback controllers for restoring arm movements via functional electrical stimulation Peter Cooman 1 and Robert F. Kirsch 1,2 1. Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA 2. Center of Excellence in Functional Electrical Stimulation, Louis Stokes Cleveland VA Medical Center, Cleveland, OH,

26 2.1 Abstract We have analyzed the strengths and weaknesses of feedforwardfeedback algorithms for controlling human arm movements generated using functional electrical stimulation (FES). These algorithms were developed specifically for use in FES neuroprostheses for individuals with paralysis due to spinal cord injury and other neurological disorders. For this analysis, we used a computational model that captures the essential nonlinear musculoskeletal properties of the arm, an approach that allowed us to develop controllers that achieve satisfactory performance under a very wide range of conditions and perturbations, including muscle and skeletal model uncertainty, muscle fatigue, sensor noise, additional loads and input time delays. We analyzed and compared three combined feedforward-feedback control strategies: proportional-derivative (PD) control, adaptive control, and sliding mode control. Our results show that (1) the performance differences between the three controllers were very minor, (2) in the absence of time delays and measurement noise, strong feedback control results in near-perfect tracking performance for all three controllers, and (3) strong feedback control in the presence of time delays and measurement noise causes instability for all three controllers. Shifting the control burden towards feedforward control can re-establish stability, but at a significant cost of decreased performance and increased sensitivity to perturbations. 26

27 2.2 Introduction A complete spinal cord injury (SCI) prevents motor information from the brain from reaching the muscles and prevents proprioceptive information from these same muscles from reaching the brain. The extent of sensory and motor control loss depends primarily on the level of injury. When the SCI occurs at a high cervical level (C3-C4), the loss of function extends to the entire upper extremity [2], making the person with SCI dependent on outside help for many activities of daily living such as feeding and grooming. Functional Electrical Stimulation (FES) [28] is a rehabilitation technique aimed at restoring motor function by electrically stimulating paralyzed muscles or the nerves that innervate them. With exception to muscles whose nerves were damaged, muscles innervated below the injury site retain their ability to contract and generate force. By applying artificial electrical stimulation in a coordinated manner we can partially restore function. FES has proved successful in restoring hand grasp after SCI [18]. The long-term goal of our research is to extend this functionality to shoulder and elbow control [4, 19], which would significantly increase the workspace of the arm and hand, allowing the FES-enabled arm to perform a much wider range of functional tasks. The challenge then becomes one of prescribing which muscles to stimulate and at what time and at what intensity, such that the arm is driven along a given intended trajectory. Solving this requirement is rather complex for restoring arm movements to individuals with paralysis, where the mass of the arm is significant, the range of motion of the arm is quite large, the number of joints is far too large to control one at a time or 27

28 to use pre-set movement patterns, and the lack of proprioception and any of the natural neural feedback mechanisms other than vision eliminates significant components of the intact motor Movement intent FEED- FORWARD (Inverse Arm Dynamics) Forward Arm Dynamics Movement output a control system. b Control strategies can Movement intent + - FEED- BACK (K) Forward Arm Dynamics Movement output be subdivided into pure feedforward, pure feedback or combined feedforwardfeedback control [9] (see also Figure 1). Feedforward control predicts the appropriate Movement intent + - FEED- FORWARD (Inverse + Arm Dynamics) + FEED- BACK (K) Forward Arm Dynamics c Movement output muscle stimulations using only the movement intent as the input and typically takes the form of the dynamic inverse of the system being controlled (see Figure 1a). In our application, having perfect Figure 1. Two degrees of freedom (2DOF), planar arm model. (a) Skeletal dynamics model. 2DOF: shoulder horizontal adduction/abduction (q 1) and elbow flexion/extension (q 2). Segment inertial parameters: masses (m 1, m 2), lengths (L 1, L 2), locations of the center of gravity (Lc 1, Lc 2) and moments of inertia (I 1, I 2) for each arm segment. (b) Musculature of the arm model. Anterior and posterior deltoid span the shoulder joint only. Brachialis and triceps (lateral head) span the elbow joint only. Biceps (long head) and triceps (long head) span both joints. knowledge of the true arm dynamics, would lead to an Figure II-1. Block diagrams for pure feedforward, pure feedback and combined feedforward-feedback control. 28

29 inverse model that exactly complemented the true arm dynamics and would result in perfect tracking of the movement intent. Unfortunately, in practice we never have perfect knowledge of the arm dynamics. Also, a wide variety of perturbations, such as external forces and changes in muscle force capacity, affect the arm in unexpected ways. Since the feedforward controller cannot account for the actual movement performance, it cannot compensate when the arm inevitably deviates from its intended trajectory. Feedback control predicts the appropriate muscle stimulations based on the difference between the current and intended trajectory of the arm (see Figure 1b). Feedback works to adjust the stimulation parameters in an attempt to decrease this difference over time until the current trajectory matches the movement intent. It can, to some extent, compensate for the negative effects of perturbations, but by design the controller cannot take action until an error has already developed. Feedback control requires sensors that measure the movement states of the arm (i.e., position, velocity, acceleration), something that increases the complexity of the FES system. Fortunately, emerging FES technologies will allow the inclusion of movement sensors within stimulation and recording devices scattered throughout the arm [35]. Combining feedforward and feedback control brings together the best features of both controller approaches: feedforward control gives a fast but imperfect estimate of the required muscle stimulations, while feedback control compensates for any remaining deviations (see Figure 1c). 29

30 To date, the most advanced controller for upper extremity FES is a combined feedforward-feedback Proportional-Integral-Derivative (PID) controller [3] (see Figure 2C). Here PID means that the controller determines the electrical muscle stimulations based on the current tracking error (P: proportional to the current error), its integral (I: the past behavior of the error) and its derivative (D: how the error is likely to change in the future). Other proposed controllers for upper extremity FES include pure feedforward [18, 22], pure feedback [9, 15] and combined feedforward-feedback control [4, 36]. The main limitation of these works is that they did not assess performance when subjected to perturbations that are expected during clinical practice most notably measurement noise and time delay. Measurement noise corrupts the information that the feedback controller relies on, so it will clearly impact performance. Similarly, there are inherent time delays in FES applications due to the excitation-activation dynamics of the muscles and the discrete nature of FES stimulation. This means that intended control actions will not take effect instantaneously, making the controller less effective and possibly unstable. Specifically in upper extremity FES, stimulation frequency is typically limited to 12Hz-15Hz, with higher frequencies causing rapid muscle fatigue [28]. This relatively low 12Hz frequency effectively imposes a 80ms delay between applying stimulations and the muscles generating force. These time delays are unavoidable and therefore cannot be ignored [36]. 30

31 The present work used a musculoskeletal arm model that incorporated its physiological and mechanical nonlinearities and dynamics to analyze the strengths and weaknesses of three well-established control strategies: feedforward-feedback PD control, adaptive control and sliding mode control. We simulated the behavior of the arm model with each of these controllers under a wide range of conditions, including: different arm loading conditions, muscle model inaccuracies, imperfect knowledge of the arm s inertial properties, muscle fatigue, measurement noise and time delays. Interested readers can find the complete derivations and a theoretical stability analysis for each controller in the associated Appendixes. Methods Modeling approach Our goal was to evaluate the performance of three control strategies for a wide range of conditions. Our ability to experimentally control many of the internal and external conditions is very limited, and the desire to explore the limits of performance would subject human participants to potentially dangerous conditions and require impractically long and tedious experiments. In this initial study, we have therefore chosen to pursue a model-based approach [19]. 31

32 Figure 2. Block diagrams of the three main strategies to control the modeled forward dynamics of the human arm: (a) pure feedforward control strategy, (b) pure feedback control strategy and (c) combined feedforward-feedback control. Figure II-2. Two degrees of freedom (2DOF) planar arm model The arm was modeled as two rigid links (upper arm and forearm) connected by two hinge joints (shoulder and elbow), allowing for two degrees of freedom (2DOF): shoulder horizontal adduction/abduction and elbow flexion/extension (see Figure 2a). The hand was modeled as a point-mass affixed to the end of the forearm. We used the Euler-Lagrange formulation to derive the equations of motion for this system (see Appendix A): ( ( )) h ( ( )) 2 2 ( ( )) ( ( )) 2 c2 1 c2 2 2 h ( 2 1 c2 ( 2) h 1 2 ( 2) )( 1 2 2) τ 1 = m1lc1 + I1 + m2 L1 + Lc2 + 2L1Lc2 cos q2 + I2 + m L1 + L2 + 2L1L2 cos q2 + I 2 q m L + L L cos q + I + m L + L L cos q q m L L sin q + m L L sin q 2q q + q 2 2 ( ( )) + ( ) ( ) τ 2 = m2 Lc2 + 2L1Lc2 cos q2 + I2 mh L2 + L1L2 cos q 2 q ( ) ( )... + m L + I + m L q + m L L + m L L sin q q c2 2 h c2 h (1) 32

33 where the subscripts 1,2 and h denote the upper arm, forearm and hand respectively (or 1: shoulder and 2: elbow joint), q, q and q denote the joint angular positions, velocities and accelerations respectively, τ denotes the joint torque, m denotes the link s mass, L denotes the link s length, Lc denotes the distance from the proximal joint of the link to its center of gravity and I. denotes the moment of inertia of the link about its center of gravity calculated as: for i = 1, Ii = ml i i (2) 12 We can group the inertial parameters as follows: ( ) ( ) a : = ml + I + m L + L + m L + L + I (1) c c2 h ( ) a := 2 mll + m LL (2) c2 h 1 2 a : = ml + I + ml (3) c2 2 h 2 resulting in the classic Lagrangian formulation: ( ) ( ) M q q+ C q,q q =τ (4) with ( ) M q a 2 a1+ a 2cos( q2) a3+ cos( q2) 2 = a 2 a3+ cos( q2) a 3 2 (5) 33

34 ( ) C q,q a2 a2 sin ( q2) q 2 sin ( q2)( q 1+ q 2) 2 2 = a 2 sin ( q 2) q (6) where M( q) is the inertia matrix and C( q,q ) describes the Coriolis/centripetal effects. Note that there were no gravitational effects, given that arm movement was restricted to the horizontal plane. Let ϑ be defined as: a 1 ϑ= : a 2 a 3 (7) Then the equations of motion (1) of Parameter Mean Standard deviation mm kg 0.321kg LL m 0.015m LL cccc m 0.017m mm kg 0.162kg LL m 0.013m LL cccc 0.129m 0.012m mm hh 0.447kg 0.072kg Table 1.Segment inertial parameters. Estimated mean values and standard deviations for the upper arm mass (m1), length (L1) and center of gravity location (Lc1), for the forearm mass (m2), length (L2) and center of gravity location (Lc2) and for the (unloaded) hand mass (mh) for the population studied in [44] (modified according to [23]). Table II-1. Segment inertial parameters for the 2DOF arm model the arm can be written such that the system is linear in ϑ : where ( ) ( ) ( ) M q q + C q,q q = Y q,q,q ϑ=τ (8) 2 q2 q 2 q1 cos( q2) q1+ sin( q2) q 1q2 + q2 2 2 Y( q,q,q ) = (9) 2 q2 q 1 0 cos( q2) + sin ( q2) q1+ q

35 The inertial parameter values were derived from the work of Zatsiorsky and Seluyanov [44], who measured the body segment inertial parameters of 100 subjects. Table 1 lists their mean values and standard deviations, following the modifications suggested in [23]. The inertial parameters of the arm model used in this study were initially set to the mean values listed in Table 1. The joint torques were generated by six muscles, organized in three antagonistic pairs (see Figure 2b): one pair spanning the shoulder joint (anterior and posterior deltoid), another pair spanning the elbow only (brachialis and lateral triceps) and a pair spanning both joints (biceps and triceps long head). The dynamics of each muscle were described by a second-order Hill-type model [24], with parameter values obtained from [20] Feedforward-Feedback PD Control We define a new set of variables: q: = q q (10) d q: = q (11) q d q : = q +Λq (12) r r d q : = q + Λq (13) r d q d σ : = q q = q +Λq q = +Λq (14) 35

36 where q d, q d and q d are the intended joint angles, angular velocities and angular accelerations respectively and Λ is a tunable, positive definite, diagonal matrix. The feedforward-feedback PD controller is given as: ( ) Cq ˆ ( ) τ := Mqq ˆ +,q q + Kσ (15) c r r where ˆM ( q ) and Ĉ( q,q ) represent the modeled inertia matrix M( q) and modeled Coriolis/centripetal effects C( q,q ) respectively and K represents a tunable, positive definite, diagonal matrix. Figure 3a shows a block diagram of the arm model with feedforward-feedback PD control. 36

37 37

38 Figure 3. Control system block diagrams. (a) Feedforward-Feedback PD control: the arm model under feedforward-feedback PD control (b) Adaptive control: the arm model under adaptive control (c) Sliding model control: the arm model under sliding mode control, (d) Muscle redundancy resolution: we distributed the control torque over the muscle activations using an inverse optimization assuming static muscle models. We then applied the resulting activations to dynamic muscle models, which, given their respective moment arms, resulted in the true torques acting on the arm joints. Figure II-3.. Block diagrams for the feedforward-feedback PD, sliding mode and adaptive controllers and muscle redundancy resolution. Note that Kσ describes a basic PD-controller: where K P and respectively. ( ) D P Kσ= K q +Λ q = Kq+ KΛq K q + K q (16) K D denote the proportional and derivative gain matrices The feedforward part of the controller M( q) q Cqq ˆ ( ) it is linear with respect to its parameters: where ( ) ˆ ( ) ( ) r r r r ˆ +, q can be written such that Mˆ q q + C q,q q = Y q,q,q,q ϑˆ (17) r r ( ) Y q,q,q,q r r qr2 q r1 q r2 q r2 qr1 cos( q2 ) qr1 + sin( q2 ) q 2 + q 1 + q 2 qr = qr1 q r1 0 cos( q2 ) + sin ( q2 ) q 1 q r1 + qr2 2 2 (18) and ˆϑ is the modeled version of ϑ defined earlier: 38

39 ( ) ( ) ( ) mˆ 1L ˆ c1 + ˆI1 + mˆ ˆ ˆ ˆ ˆ ˆ 2 L1 + Lc2 + mˆ h L1 + L2 + I 2 aˆ 1 ϑ= ˆ : aˆ = 2 mll ˆ ˆ ˆ + mˆ LL ˆ ˆ c2 h 1 2 a ˆ ml ˆ ˆ ˆ 2 c2 + I2 + ml ˆ ˆ h 2 (19) where the ˆ operator signifies that the parameter is not perfectly known but has instead been measured or estimated. A theoretical proof (see electronic Supplement B) shows that the feedforward-feedback PD-controller maintains stability and achieves global asymptotic tracking (error converges to zero) if the measured or estimated inertial parameter values match the true values (i.e. ˆϑ=ϑ). Unfortunately, this is never the case, since we can only measure or estimate the inertial parameters at a finite accuracy. This inevitably creates a mismatch between the model and the true arm dynamics. This mismatch means that the feedforward controller will not perfectly cancel out the nonlinear dynamics, giving rise to a destabilizing residual term. A control system is said to be robust if the effects of model mismatch on stability and tracking performance remain small. Even though the feedforward-feedback PD controller has some inherent robustness due to its feedback part, stability cannot be guaranteed. The two main approaches to increase robustness are sliding mode control and adaptive control. These approaches will be discussed next Sliding Mode Control As a measure of the mismatch between the model and the true arm dynamics we define ϑ as: 39

40 ϑ : =ϑ ϑˆ (20) This difference was unknown, but was assumed to remain bounded: ρ ϑ (21) Sliding mode control extends the feedforward-feedback PD controller with an additional term specifically designed to counteract the residual term originating from the imperfect cancelation of the true arm dynamics by the inaccurate model embedded in the feedforward controller: ( ) ˆ ( ) ( )( ˆ ) τ : = Y q,q,q, q ϑ+ Kσ+ Y q,q,q, q u = Y q,q,q, q ϑ+ u + Kσ (22) c r r r r r r where u has been chosen as: u = ξ i ρ i if ξ >ε i ξi ξ ε ρ i i if ξ i ε (23) where ξ i denotes the i th component of the vector T Y σ, ρ i is a bound on the mismatch of each element in ϑ (i.e. ρ ϑ i i ) and ε denotes the thickness of the boundary layer of the sliding surface (here we chose ε =0.01). The latter was introduced to decrease the rapid switching of the control signal u, known as chattering, as ξ i approaches zero. Figure 3b shows a block diagram of the arm model with sliding mode control. Provided that the true model mismatch remains within the assumed bounds, we can prove theoretically (see Appendix C) that the sliding mode controller will drive the joint angles towards a manifold (called the sliding surface) described by: T Y σ= 0 (24) 40

41 This implies that σ converges to zero, which implies that the joint angles and angular velocities track the intended angles and velocities Adaptive Control Adaptive control attempts to solve the problem of model mismatch by trying to learn the true inertial parameters. This requires an adaptation law to prescribe how the estimated inertial parameters should change based on the performance of the arm. Ideally, adaptation recovers the true inertial parameter values, eliminating the mismatch between the model and the true arm dynamics. The control law is identical to the feedforward-feedback PD control law: ( ) ˆ ( ) ( ) τ : = Mˆ q q + C q,q q + Kσ= Y q,q,q,q ϑˆ + K σ (25) c r r r r However, as opposed to feedforward-feedback PD control, ˆϑ is no longer assumed to be fixed, but is allowed to vary. ˆϑ is continuously updated based on new information, according to the following adaptation law: ˆ ϑ= K Y σ 1 T ϑ (26) where K ϑ was chosen to be a positive definite, square, symmetric matrix, typically the identity matrix. Figure 3c shows a block diagram of the arm model under adaptive control. For a more detailed theoretical discussion of the adaptive controller we refer to Appendix D. 41

42 2.2.5 Resolving muscle redundancy The three proposed controllers calculate the joint torques required to drive the FES enabled arm along its intended trajectory. However the FES neuroprosthesis can only influence the joint torques indirectly by electrically stimulating the paralyzed muscles. An important characteristic of the human arm is that there are more muscles (i.e. six in our example) than there are degrees of freedom (i.e. two in our example). This redundancy, and the antagonistic organization of the muscles, typically results in an infinite number of possible combinations of muscle forces that generate the same joint torques. To find a unique solution, we used an optimization approach (see also Figure 3d): find the set of muscle activations that minimized muscle effort approximated by the sum of squares of the muscle stimulations while still generating the required joint torques. Mathematically, the optimization program can be written as: i= 6 min J = a (27) i= 1 2 i i= 6 subject to r F ( a,q) τ = for j 1, 2 c, j ij i i i= 1 = (28) Where a i is the activation of the i th muscle, τ c is the vector of torques to be achieved, r ij is the moment arm of the i th muscle with respect to the j th DOF and i ( ) F a,q is the force of the i th muscle. The muscle force depends on the activation i a i, while the current joint angles q are used to determine the muscle length, necessary for the muscle length-tension relationship. During each optimization, 42

43 we assumed the arm to be static (i.e. no excitation-activation dynamics, infinitely stiff tendon, zero muscle contraction velocity and fixed arm position) Simulation protocol Tracking performance was evaluated using the root-mean-square error (RMSE) between the intended and simulated joint angle trajectories: RMSE ( q q ) T ( q q ) where n denotes the number of data points. d d = (29) n We used two different movement intents in our simulations. Both were constructed to minimize the jerk of the movement (i.e. the derivative of acceleration) to produce a maximally smooth motion [12]. The first movement intent represented a periodic series of reaches, mimicking a feeding movement, with the joint angles moving through their ranges of motion. Each reach started with a five second moment phase from the rest position (i.e. shoulder fully abducted at 5º and the elbow fully flexed at 170º) to the final position (i.e. shoulder fully adducted at 130º and the elbow fully extended at 5º), followed by a five second plateau to simulate the need to maintain the endpoint position during object manipulation, a five second retraction phase back to the original starting position, and ending with another five second plateau to allow for functional task completion. The second movement intent was constructed as a series of ten minimum-jerk point-to-point reaches each to a different randomized target 43

44 sampled uniformly over the ranges of motion of the joint angles. Each of these randomized reaches consisted of a three second moment phase followed by a two second plateau phase. These random trajectories allowed us to evaluate whether the controller could maintain stability and good tracking performance for different intended trajectories (amplitude and velocity). Besides tracking accuracy, control effort is an important criterion: higher control effort causes muscles to fatigue faster and therefore performance to degrade sooner. We quantified control effort as the sum of squares of the control torques predicted by the controller. The feedback gain matrices K and Λ of the feedback-feedforward PD controller were tuned by increasing the feedback gains until satisfactory stability and tracking performance were obtained (i.e. no overshoot, no oscillations and small tracking errors). This resulted in K matrix), which is equivalent to Kp = 9I and Kd = 6I and Λ= 1.5I ( I denoting the identity = 6I. We used the same feedback gain matrices for the adaptive and sliding mode controller to maintain a fair comparison between the three controllers. We evaluated the performance of each controller under eight simulated conditions, covering a wide range of perturbations: 44

45 1. Baseline case: (movement intent: feeding pattern) the ideal case where no perturbations acted on the arm and where the modeled inertial parameter values were assumed to match those of the true arm. 2. Different endpoint loading conditions: (movement intent: feeding pattern) we increased the hand mass during the retraction phase of the reach to simulate the act of grasping and retrieving an object during a functional task. For four subsequent reaches, we increased the additional load from 0kg (baseline) to 3kg in 1kg increments. 3. Muscle model uncertainty: (movement intent: feeding pattern) evaluated the effect of the controller overestimating or underestimating the true maximum muscle forces. For five subsequent reaches, we increased the mismatch from -50% (50% overestimation) to +50% (50% underestimation) in 25% increments. 4. Muscle fatigue: (movement intent: feeding pattern) the force produced by each muscle at a given activation level was decreased at a linear rate of 1%/s. After 100s the muscles were fully fatigued and unable to produce force. 5. Measurement noise: (movement intent: feeding pattern) to simulate realistic sensors, we added zero-mean white noise to the joint angle and the angular velocity feedback measurements. For successive reaches, we increased the amplitude of the noise signals from 0⁰ to 20⁰ (in 5⁰ increments) and from 0⁰/s to 10⁰/s (in 2.5⁰/s increments), for the joint angle and angular velocity measurements respectively. 45

46 6. Skeletal model mismatch: (movement intent: random reaches) while maintaining the internal arm model embedded in the controller at its initial values, the parameters of the plant (i.e. the simulated arm being controlled) were varied over wide ranges. Specifically, the inertial properties of the controlled arm were varied in 459 different ways by sampling the parameters from normal parameter distributions with means and standard deviations listed in Table 1. This is a probabilistic approach aimed at estimating the worst case performance ([17], [37]). 7. Combination case: (movement intent: random reaches) we added a constant 1kg load acting at the end-point, assumed 25% maximum muscle force over-estimation, linear fatigue from 0%-75% over 75s, 5⁰ and 2.5⁰/s amplitude zero mean white noise acting on the joint angle and angular velocity measurements respectively and set the true arm inertial parameters to the worst case as determined in scenario Time delay: (movement intent: feeding pattern) the electrical stimulations predicted by the controller were sampled at a frequency of 12Hz (i.e. 80ms time delay), before acting on the muscles of the arm model. 2.3 Results Figure 4 and Table 2 respectively show the qualitative and quantitative results from our comparisons of the high-gain feedforward-feedback PD controller, an adaptive controller and a sliding mode controller. Panels a-h in Figure 4 plot the shoulder angle and elbow angle tracking performance of the 46

47 three different controllers for each of the eight simulated conditions described above. The thick gray lines in each panel denote the intended joint angle trajectories, while the controller-produced joint angles are drawn using thin black lines with * markers (feedforward-feedback PD control), o markers (sliding mode control) and markers (adaptive control). Two important trends are immediately apparent: 1. All three controllers provide stable tracking of the arm model for the same set of perturbations: endpoint load (Figure 4b), muscle model (Figure 4c) and skeletal model uncertainty (Figure 4e) and muscle fatigue (Figure 4d). Moreover, the arm model became unstable for all three controllers when subjected to the same set of perturbations: measurement noise (Figure 4f) and input time delays (Figure 4h). 2. When stable, the performance of the three controllers was excellent as indicated by the very low (typically less than 1 degree) RMSE values (see Table 2). In most scenarios the sliding mode controller outperformed the adaptive controller and both did better than the PD controller, however the RMSE differences are very small this is also apparent in Figure 4 where the traces for the three different controllers nearly perfectly overlap. 47

48 48

49 Figure. 4 Simulated tracking performance for high-gain feedforward-feedback PD, adaptive and sliding mode control. The top two plots in each panel show the tracking performance for shoulder horizontal adduction/abduction (top plot) and elbow flexion/extension (second plot). Thick gray lines denote intended joint angle trajectories, while the simulated joint angles are drawn using thin black lines with * markers (feedforward-feedback PD control), o markers (sliding mode control) and markers (adaptive control). Figure II-4. Simulated tracking performance for high-gain feedforward-feedback PD, adaptive and sliding mode control. PD (High Gain) AD (High Gain) SMC (High Gain) Shoulde Elbow Shoulde Elbow Shoulde Elbow BASELINE kg LOAD 1kg kg kg % MUSCLE -25% % % % FATIGUE ROBUSTNESS , 0 /s NOISE 5, 2.5 /s unstable unstable ,5 /s unstable unstable unstable unstable unstable unstable 15, 7.5 /s unstable unstable unstable unstable unstable unstable 20, 10 /s unstable unstable unstable unstable unstable unstable COMBINATION DELAY unstable unstable unstable unstable unstable unstable Table 2 Simulated tracking accuracy (RMSEs) for high-gain feedforward-feedback PD control (PD), adaptive control (AC) and sliding mode control (SMC). Table II-2. Simulated tracking performance for high-gain feedforward-feedback PD, adaptive and sliding mode control. Input time delays resulted in rapid instability for all three controllers (illustrated in Figure 4h). It is possible to re-establish stability by decreasing the feedback gains, effectively shifting the control burden away from feedback 49

50 control towards feedforward control, which is indifferent to time delay. However, this severely deteriorated tracking accuracy and robustness against perturbations. Figure 5a illustrates the tradeoff between time delay (inversely related to stimulation frequency) and the upper limit to the feedback gains for the controllers: as time delay increased or as stimulation frequency decreased we were forced to keep lowering the feedback gains (shown in blue) in order to avoid instability. Similarly, the RMS tracking errors (shown in green) increased as time delay increased. For zero time delay (i.e. infinite stimulation frequency) the feedforward-feedback PD controller (indicated with * markers) achieved RMSE values as low as 0.06 for the shoulder and 0.06 for the elbow joint with maximum feedback gains K = 26.4I and Λ = 6.6 (equivalent to Kp = I and Kd = 26.4I ), and the sliding mode controller (indicated with o markers) achieved RMSE values of 0.16 for the shoulder and 0.35 for the elbow joint with maximum gains K = 9.8I and Λ = 2.45 (equivalent to Kp = 24.01I and Kd = 9.8I ). For the tracking RMSEs, solid lines denote shoulder horizontal adduction/abduction, while dashed lines correspond to elbow flexion/extension. 50

51 51

52 Figure 5. (a) Tradeoff between time delay (stimulation frequency), maximum stabilizing feedback gains (equivalent Kp, drawn in blue) and tracking RMS errors (drawn in green). The * markers denote the feedforward-feedback PD controller and the o markers denote the sliding mode controller. For the tracking RMSEs, solid lines denote shoulder horizontal adduction/abduction, while dashed lines correspond to elbow flexion/extension. Panels b through h show the tracking performance (top plot: shoulder DOF, second plot: elbow DOF), for the low-gain feedforward-feedback PD controller ( * markers) and sliding mode controller ( o markers) for different simulation scenarios each with 12Hz stimulation frequency. Figure II-5. Simulated tracking performance for low-gain feedforward-feedback PD, adaptive and sliding mode control. PD (Low Gain) SMC (Low Gain) Shoulder Elbow Shoulder Elbow BASELINE LOAD 0kg kg kg kg % MUSCLE -25% % % % FATIGUE ROBUSTNESS , 0/s NOISE 5, 2.5/s , 5/s , 7.5/s , 10/s COMBINATION Table 3 Simulated tracking accuracy (RMSEs) for low-gain feedforward-feedback PD control (PD) and sliding mode control (SMC). Table II-3. Simulated tracking accuracy (RMSEs) for low-gain feedforward-feedback PD control (PD) and sliding mode control (SMC). For a 12Hz stimulation frequency typical for upper extremity FES we had to decrease the feedback gains to K = 0.6I and Λ= 0.15I (equivalent to KP = 0.09I and KD = 0.6I ). For the sliding mode controller the arm model 52

53 remained unstable until we decreased the gains to K = 0.1I and KP = 0.4I (equivalent to KP = 0.04I and KD = 0.4I ). For the adaptive controller, decreasing the feedback gains failed to re-establish stability for the time delay associated with 12Hz stimulation. We again evaluated the stabilized but low-gain feedforward-feedback PD controller and sliding mode controller for the same perturbations (see figure 5 panels b through h), each with 12Hz stimulation frequency (80ms time delay). Panels b through h of Figure 5 and Table 3 respectively show the qualitative and quantitative results for the low-gain feedforward-feedback PD and sliding mode controller with the realistic time delay (i.e. 80ms, corresponding to 12Hz stimulation). The tracking performance of the low-gain controllers was significantly worse than their high-gain counterparts: tracking was slow and sluggish with much larger residual errors (see Figure 5). This decrease in tracking accuracy and robustness against perturbations can also be observed by comparing the RMSE values for the low-gain controllers (Table 3) with those achieved by the high-gain controllers (Table 2). In almost all cases, the high-gain controllers achieved much higher tracking accuracy compared to the low-gain controllers, with measurement noise as the only notable exception. Finally, we quantified the control effort by the square root of the sum of squares of the joint torques predicted by the controller. For the combination 53

54 scenario, the high-gain sliding mode controller required the highest effort, with a root sum of squares of [85.11Nm, 58.64Nm] (shoulder and elbow DOF), followed by the high-gain feedforward-feedback PD controller ([76.32Nm, 48.93Nm]), the high-gain adaptive controller ([76.31Nm, 48.16Nm]), the low-gain feedforwardfeedback PD controller ([17.02Nm, 11.45Nm]) and the low-gain sliding mode controller ([16.64Nm, 8.64Nm]). Clearly, high-gain controllers required a higher control effort than the low-gain controllers. This can also be seen graphically by comparing the amplitudes of the required muscle activations of the high-gain and low-gain controllers in the case of fatigue (Figure 4d and Figure 5e). For both high-gain and low-gain controllers, the required muscle activations increase as muscle fatigue increases, however these increases are greater for the high-gain controllers than for the low-gain controllers. 2.4 Discussion We have evaluated three control strategies (i.e. feedforward-feedback PD, adaptive and sliding mode control) for controlling the motions of a twodimensional model of the human arm that included the dynamics of the arm, the single and multi-joint actions of muscles at the shoulder and elbow, and the nonlinearities of muscle contractile properties. We evaluated the strengths and weaknesses of these three controllers for a wide range of conditions: different endpoint loading conditions, muscle and skeletal model mismatches, muscle fatigue, measurement noise and time-delays. 54

55 In the absence of perturbations and time delays, all three controllers (feedforward-feedback PD, adaptive and sliding mode control) achieved nearperfect tracking performance since high feedback gains (strong feedback action) could be used. Excellent tracking was maintained when the system was subjected to perturbations that feedback control was assumed to be effective against: different endpoint loading conditions, muscle and skeletal model mismatches (robustness), and muscle fatigue. Sliding mode control performed marginally better than feedforwardfeedback PD control and adaptive control. However, of all three controllers, sliding mode control required the highest control effort. This is likely due to the additional control term introduced to dominate the effects of model mismatch. Adaptive control also performed better than feedforward-feedback PD control, but, as for sliding mode control, the marginal improvements are unlikely to be functionally significant. Control efforts remained low, comparable to those required by the feedback-feedforward PD controller. This is likely due to the fact that in adaptive control, after a learning phase, the arm model embedded in the controller should more closely resembles the true arm dynamics, resulting in better a priori cancelation of the dynamics of the true arm, and thus a smaller residual term for the feedback controller to compensate. 55

56 In those scenarios where the feedforward-feedback PD controller achieved stable tracking, its excellent performance left little room for improvement. Given how both adaptive and sliding mode control are negatively affected by measurement noise and time delays, their marginal performance gains do not justify their higher computational complexity and, for the sliding mode controller, higher control effort. We therefore recommend feedforwardfeedback PD control for upper extremity FES; at least until performance requirements become significantly more demanding. Our results confirmed some potential weaknesses of feedback control: measurement noise and time delays. Feedback control relies on accurate measurements of the system outputs. When these measurements were corrupted by noise, all three controllers were significantly negatively affected even to the point of becoming unstable. Unfortunately, little is known about the noise properties of implantable sensors. To provide some context, we refer to the work of Roetenberg et al. [31] ], which describes a portable sensor suite to measure arm kinematics. Compared to the Vicon 470 (Oxford Metrix, UK) motion capture system, they achieved a measurement accuracy of 3º RMSE. All three controllers failed to maintain stability even at this relatively low noise level. To our knowledge the issue of measurement noise remains an open problem in the field of nonlinear control theory. However, we can, to some extent, influence measurement noise by developing more accurate sensor suites. The second weakness of feedback control is its sensitivity to time delays. We found that 56

57 maintaining strong feedback control in the presence of time delays led to instability for all three controllers, even for relatively small delays. Unfortunately, time delays are inevitable in FES: the muscle excitation-activation dynamics are fixed and stimulating at the higher frequencies needed to reduce samplingrelated delays would result in unacceptably higher muscle fatigue rates. We succeeded in re-establishing stability by significantly decreasing the feedback gains. This approach effectively shifts the control burden from the feedback path towards the feedforward path, which is entirely indifferent to time delays since it does not rely on feedback measurements. For the same reason, decreasing the feedback gain reduced the system s sensitivity to measurement noise. However, these benefits come at the cost of decreased tracking performance and higher sensitivity to muscle and skeletal model uncertainty (lower robustness), different endpoint loading conditions, and muscle fatigue the very perturbations that strong feedback control was successfully compensating. 57

58 III. Nonlinear feedforward-feedback control of human arm FES with time delay compensation Peter Cooman 1 and Robert F. Kirsch 1,2 1. Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA 2. Center of Excellence in Functional Electrical Stimulation, Louis Stokes Cleveland VA Medical Center, Cleveland, OH,

59 3.1 Abstract A high-level cervical (C1-C4) spinal cord injury (SCI) paralyzes most, if not all, arm muscles, severely limiting the individual s ability to independently perform activities of daily living. Functional Electrical Stimulation (FES) can partially restore movement by stimulating paralyzed muscles to contract. This work focuses on designing the FES controller: a mathematical algorithm that coordinates the electrical stimulations for each muscle using sensor-based feedback such that the FES-enabled arm follows a user-commanded trajectory. Time delays inherent to the musculoskeletal system and to the use of FES can result in instability, however. Using a two degrees-of-freedom (2DOF) arm model, we designed and evaluated a nonlinear feedforward-feedback Proportional- Integral-Derivative (PID) controller with time delay compensation. We evaluated the tracking performance of the controlled arm model for several relevant perturbations including: time delay, different end-point loads, muscle fatigue, measurement noise, and model mismatch. Root-mean-square error (RMSE) tracking errors remained extremely low in ideal conditions (shoulder angle RMSE: 0.27º, elbow angle RMSE: 0.62º) and still quite low even when all perturbations were active (shoulder RMSE: 2.99º, elbow RMSE: 5.15º). The proposed controller thus shows great promise for clinical implementation in upper extremity FES. 59

60 3.2 Introduction A complete spinal cord injury (SCI) paralyzes the muscles innervated by nerves originating at spinal cord sections at and below the injury site. For a high cervical level (C1-C4) injury, this paralysis includes all muscles of the arm, significantly limiting the individual s ability to independently perform everyday functional tasks such as feeding or grooming [2]. However, a paralyzed muscle retains its ability to contract, provided that its innervating nerve is intact. By applying small electrical currents either to the paralyzed muscles directly or to their innervating nerves, muscle contractions can be evoked [28] a rehabilitation technique known as Functional Electrical Stimulation (FES). Among other applications [28], FES has proven successful in restoring hand grasp after SCI [18]. Our long-term goal is to extend the functionality of the hand grasp FES neuroprosthesis to control the shoulder and elbow joints, thereby significantly increasing the reachable workspace of the hand and arm. Within this context, this work focuses on the controller design problem: design a mathematical algorithm that, given a desired arm movement trajectory, autonomously predicts which muscles should be stimulated, at what time and at what intensity, such that the resulting muscle forces drive the arm along the desired trajectory. 60

61 Previously proposed controllers can be categorized into three groups [9]: pure feedforward control (e.g. [18, 22]), pure feedback control (e.g. [9, 15]) and combined feedforward-feedback control (e.g. [4, 36]). using only the intended trajectory Pure feedforward control predicts the appropriate muscle stimulations using only the intended trajectory as the input. The feedforward controller typically takes the form of the inverse of the arm dynamics. However, feedforward controllers are typically simplified descriptions of the actual system and use parameter estimates with finite accuracy, and do not account for the actual movement performance. Pure feedback control automatically adjusts the appropriate muscle stimulations based on the difference between the current (measured) and intended arm trajectories until the current trajectory matches the intended trajectory. As opposed to feedforward control, it can to some extent, compensate for the negative effects of perturbations, but by design the controller cannot take action until an error has already developed. Combining feedforward and feedback control (see Figure 1) brings together the best features of both controller approaches: the feedforward controller calculates an initial but Figure 1. Control system block diagram of combined feedforward-feedback control. The feedforward controller relies on an internal model of the inverse arm dynamics, while the behavior of the feedback controller depends on the choice of feedback gains K. Figure III-1. Control system block diagram of combined feedforward-feedback control. 61

62 imperfect estimate of the required muscle stimulations, while the feedback controller compensates for any remaining deviations due to perturbations and imperfections in the feedforward model. Feedback control can thus significantly improve performance and can do this in an automatic manner. However, the performance of feedback systems can be significantly degraded by time delays between the commanded and actual actions of the system (as shown in Chapter II). Such delays are inherent to FES neuroprostheses [28]. The two primary sources for time delays are (1) the excitation-activation dynamics of the muscles and (2) the discrete-time nature of FES. Specifically, in upper extremity FES the stimulation frequency is typically limited to 12Hz-15Hz because higher stimulation frequencies cause rapid muscle fatigue [28]. However, a 12Hz stimulation frequency means that the controller can adjust the stimulation parameters only at 80ms intervals, so it takes a minimum of 80ms before any corrective action initiated by the controller will have an effect on the arm. In Chapter II we compared three advanced feedforwardfeedback control strategies (i.e. feedforward-feedback PD, adaptive and sliding mode control), and showed that three types of feedback controllers were highly sensitive to time delays. Even small time delays (i.e. 80ms) quickly resulted in unstable behavior. It was shown that stability could be re-established by greatly reducing the contribution of the feedback controller and instead relying more on feedforward control, which is indifferent to time delays. However, in doing so the benefits of feedback control were greatly diminished, resulting in decreased 62

63 performance and higher sensitivity to perturbations. Recently, Sharma et al. [33, 34] proposed a pure feedback controller with time delay compensation. Computer simulations and experimental trials proved that their controller achieved robust tracking performance when controlling a 1DOF knee joint [33] and a 2DOF robot manipulator [34]. The controller presented here builds on their work by adding a feedforward component to their pure feedback controller and applies this new controller to a planar 2DOF human arm model that captures its nonlinear and dynamic properties. We evaluated the performance and robustness of the controller for a wide selection of perturbations including: time delay, changing loading conditions, muscle fatigue, measurement noise, and model mismatch. 3.3 Methods Model-based approach As in Chapter II, we have taken a model-based approach to controller design and evaluation. This allowed us to easily evaluate the controller s behavior for a wide variety of scenarios, some of which would have been difficult or even impossible to replicate in a clinical setting, or would have posed unacceptable safety risks to human research subjects [19]. 63

64 Figure 2. 2DOF planar arm model. (a) Skeletal dynamics modeled as two rigid links. Shoulder horizontal adduction/abduction (q1) and elbow flexion/extension (q2), both in the horizontal plane, are allowed. Segment inertial parameters are shown: segment masses (m1, m2), lengths (L1, L2), locations of the center of gravity for each link (Lc1, Lc2) and moments of inertia (I1, I2). (b) Musculature of the arm model. Six muscles are included. Anterior and posterior deltoid span the shoulder joint only. Brachialis and triceps (lateral head) span the elbow joint only. Biceps (long head) and triceps (long head) span both joints. Figure III-2. Two degrees of freedom (2DOF), planar arm model. The model used in this study is identical to the one detailed in Chapter II. Briefly, the arm was modeled as two rigid links and a point mass, representing the upper arm, forearm and hand respectively. The shoulder and elbow joints were modeled as frictionless hinges. Movement was restricted to the horizontal plane, resulting in two degrees of freedom: shoulder horizontal adduction/abduction and elbow flexion/extension (see Figure 2a). The simulated range of motion of the shoulder joint was 5º to 130º, and that of the elbow joint was 5º to 170º. The equations of motion for this linkage system were derived using the Euler-Lagrange formulation (see Appendix A): ( ( )) ( ) ( ) ( ) ( ) ( ) ( ) ( ) M q t q t + C q t,q t q t + d t =τ t T (1) 64

65 where q( t ), q ( t) and q ( t) are vectors of the joint angular positions, velocities and accelerations, M( q) is the inertia matrix, C( q,q ) are the Coriolis/centripetal effects, d is the torque vector due to any unexpected and/or unmeasurable influences, and τ is the vector of joint torques delayed by a time T. Note that there were no gravity effects due to the arm being constrained to move in the horizontal plane. The inertia and Coriolis/centripetal matrix depend on the inertial parameters of the arm segments: mass ( m ), length ( L ) and center of gravity location ( L ). The subscripts 1,2, and h denote the upper arm, forearm and hand c respectively. The values for these inertial parameters were chosen based on the work of Zatsiorsky and Seluyanov, who measured body segment masses, lengths, centers of mass and radii of gyration of 100 young adult Caucasians [44]. These measurements used bony landmarks as reference points. We therefore, in accordance to deleva [23], calculated the corresponding inertial parameter values with respect to the joint rotation centers, which are more appropriate for our purpose of biomechanical modeling. The resulting mean values and standard deviations are given in Parameter Mean Standard deviation mm kg 0.321kg LL m 0.015m LL cccc m 0.017m mm kg 0.162kg LL m 0.013m LL cccc 0.129m 0.012m mm hh 0.447kg 0.072kg Table 1.Segment inertial parameters. Estimated mean values and standard deviations for the upper arm mass (m1), length (L1) and center of gravity location (Lc1), for the forearm mass (m2), length (L2) and center of gravity location (Lc2) and for the (unloaded) hand mass (mh) for the population studied in [44] (modified according to [23]). Table III-1. Segment inertial parameters for the 2DOF arm model Table 1. 65

66 The modeled joints were actuated by three antagonistic muscle pairs, as shown in Figure 2b. The first pair (anterior and posterior deltoid) spanned only the shoulder joint. The second pair (brachialis and lateral triceps) spanned only the elbow joint. The third pair (long head of biceps and long head of triceps) spanned both joints. The dynamics of each muscle were described by a secondorder Hill-type model developed by McLean et al. [24]. The values for the muscle model parameters were measured in cadaver studies [20] The controller We propose the following controller (see also Figure 3): c ( ) ( ) ( ) ( ) ( ( )) ( ) ( ( )) ( ) ˆ ( ( ) d ( )) ( ) τ t = Mˆ q t q t +α Mˆ q t e t +βmˆ q t σ t + C q t,q t q t + where t ( ) ( ) ˆ 1 + k σ t +βσθ+ M ( θ) τθ ( T) τθ ( ) dθ kσ0 0 where qd ( t ), q d ( t) and d ( t) ( ) ( ) ( ) ( ) ( ) e t = q t q t (3) d ( ) = ( ) ( ) e t q t q t (4) d ( t) e( t) e( t) σ = +α (5) q are the desired joint angular positions, velocities and accelerations, and α, β and k are tunable, positive, real-valued controller parameters. The ˆ -operator indicates that we cannot perfectly know the true arm dynamics and therefore need to rely on the modeled inertia matrix ˆM, Coriolis/centrifugal effects Ĉ. 66 (2)

67 This controller can be subdivided into: (a) a feedforward component: τ ff ( t) = Mˆ ( q( t) ) qd ( t) + Cˆ ( q( t ),q ( t) ) q ( t) (b) a feedback component: t ( ) ˆ τ fb t =α M q( t) e t +βm q t σ t + k σ t + βσ θ dθ kσ 0 0 ( ) ( ) ˆ ( ( )) ( ) ( ) ( ) ( ) which, after substituting (5), can be written as a PID-controller: P-action: αβ ˆM ( q ( t )) e ( t ) + k ( α + β) ( e( t) e( 0 )) t I-action: ( ) kαβ e θ dθ D-action: ( ) 0 ( ) ( ) ( ( )) ( ) k ( ( ) ( )) α M q t e t +β M q t e t + e t e 0 (c) a time delay compensating term: ( ) ˆ 1 delay ( )( ( ) ( )) t τ t = km θ τθ T τθ dθ 0 67

68 Figure 3. (a) The arm model in a closed loop with the proposed controller. Perturbations ( d ) act on the arm and noise ( n ) corrupts the measurements. (b) The inverse optimization approach to resolve the muscle redundancy problem. The stimulation-related time delay (T) is also shown explicitly in panel b. Figure III-3. Block diagram of the nonlinear feedforward-feedback time delay compensating controller. The proposed controller builds on the work of Sharma et al. [33, 34], by adding the feedforward component and also adding two terms to the feedback ( ) ( ) ( ) ( t) component (i.e. αm q( t) e t and M q( t) out nonlinearities. β σ ), for the purpose of canceling It can be proven (see Appendix E) that under the proposed controller (Eq. 2) and for ideal operating conditions, the tracking error will, after a finite time, converge to and remain within a bounded region around zero (i.e. Semi-globally 68

69 Uniformly Ultimately Bounded tracking performance). This implies that the actual joint angular positions and velocities will approach and remain close to their desired values. The tunable controller parameters (i.e. α, β and k) should be chosen such that the following sufficient, but not necessary, conditions (see Appendix E) are satisfied: 1 α>, ζγ β> 1+, 4 2 K ω Tk > 0, k > K, 2 2T T ω ωt > γ γ (6) where γ, ω and K are positive, real numbers and ζ is the, also positive, upper bound of the infinity norm of the inverse inertia matrix: ( ( )) 1 M q t ζ (7) which, based on the initial values for the segment inertial parameters, can be calculated as: ( ( )) 1 M q t = 2 ζ kg.m (8) The controller parameters were tuned by hand to satisfy the conditions (Eq. 6) for a time delay T of 80ms (e.g., choose γ= 0.042, ω= 105 and K = 0.09 ), resulting in the following parameters: α= 6 1 s, β= 6 1 s, k 0.1 kg.m s = 2 (9) 69

70 3.3.3 Resolving muscle redundancy The proposed controller (Eq. 2) predicts the joint torques such that the arm model tracks a given desired movement trajectory. Unfortunately, the joint torques are not under direct control in a FES system. Rather, the muscle stimulation parameters are controlled, which then lead to the generation of the various muscle forces. Since the arm model had more muscles (i.e. six) than degrees of freedom (i.e. two), however, there was typically an infinite number of possible combinations of muscle forces that could generate a given torque vector. To find a unique solution for distributing the torque vector over the available muscles, we used an optimization approach that minimized control effort indicated by the sum-of-squares of the muscle activations -- while requiring that the resulting muscle forces generate the torque vector as predicted by the controller. This optimization problem can be written as: i= 6 min J = a (10) i= 1 2 i i= 6 Subject to r F ( a,q) τ = for j 1, 2 c, j ij i i i= 1 = (11) where a i is the activation of the i th muscle, τc is the vector of torques to be achieved, r ij is the moment arm of the i th muscle with respect to the j th joint and i ( ) F a,q is the force of the i th muscle. The muscle force depends on the activation i a i, while the current joint angles q were used to determine the muscle length, necessary for the muscle length-tension relationship. The activations obtained from the inverse optimization were then applied to the forward muscle models, 70

71 resulting in the true joint torques ( τ ) (see Figure 3b). Ideally τ should equal the joint torques predicted by the controller ( τ c ). However, this was rarely the case. Note that the inverse optimization (Eq ) ignored the muscle excitationactivation dynamics and the muscle force-velocity relationship (i.e., it used a static muscle model). Since there is no practical way to measure muscle excitation levels and contraction velocities, the inverse optimization could not take these contributions into account. The forward muscle models used in simulations, however, did include both excitation-activations dynamics and the force-velocity relationship. As long as the intended movement trajectories remained slow, the effect of this mismatch was negligible. However, for fast movements, this mismatch will degrade the tracking performance of the FESenabled arm and stability cannot be guaranteed Experimental protocol Various perturbations affect the controller s ability to maintain stability and good tracking performance. These perturbations include time delays, changes in the end-point load ( m ), imperfect knowledge of the muscle and segment inertial h parameters, measurement noise, and muscle fatigue. The proposed controller was evaluated in simulation over seven different scenarios: one ideal case where no perturbations were active (baseline performance), five scenarios that evaluated the effects of a single perturbation, and a final scenario where all perturbations were active (i.e., close to expected operating conditions). All simulation scenarios included a 80ms time delay associated with a 12Hz 71

72 stimulation frequency. A more detailed discussion of the six non-ideal scenarios follows: (1) Changes in end-point load: Many functional tasks involve manipulating objects. Picking up an object changes the inertial properties of the arm, creating a mismatch between the true arm dynamics and the model embedded in the feedforward part of the controller, which assumes that the arm remains unloaded. We simulated a series of reaches where a mass was picked up and retrieved to the mouth area. The additional mass was present only in the retraction phases (i.e., hand to mouth movement components but not the initial reaching out to acquire the object) and was varied in different trials from 0kg to 3kg in increments of 1kg. This range covers the majority of relevant activities of daily living. (2) Muscle model mismatch: The inverse optimization (Eq ) that calculates muscle activations for a given vector of joint torques requires an accurate muscle model. If the muscle model differs from the true muscle dynamics, then the actual forces, and thus joint torques ( τ ), will be lower or higher than expected by the controller ( τ c ). There is no practical way to measure the maximum muscle forces in vivo, meaning that the true muscle forces will often be over- or underestimated. We introduced the following relationship between the joint torques predicted by the controller ( τ c ) and the joint torques applied to the forward arm model ( τ ): τ : = η τ (12) c 72

73 For a series of reaches this factor η was increased from 0.5 (50% overestimation of the true maximum muscle forces) to 1.50 (50% underestimation of the true maximum muscle forces) in increments of (3) Measurement noise: Feedback control relies on accurate measurements of the current joint angles and angular velocities. However, measurements are always corrupted by noise. We performed simulations where zero mean white noise (ZMWN) with amplitudes of 0º to 20º, in increments of 5º, were added to the joint angle measurements and amplitudes of 0º/s to 10º/s, in increments of 2.5º/s, to the joint angular velocity measurements. (4) Muscle fatigue: As a muscle maintains a tetanic contraction, its ability to generate force gradually decreases. To maintain steady force (and joint torque) levels, the controller needs to increase muscle activations to compensate. This is possible until activations reach their maximum (i.e. 1). When activations saturate, any additional fatigue can no longer be compensated by the controller and will cause muscle forces to decrease. At that point, the system effectively becomes open-loop, and poor tracking performance and/or instability usually follows. To simulate the effects of fatigue, a multiplicative term was introduced similar to (12): τ : = fatigue η τ (13) c where η= 1 and fatigue was decreased linearly from 1 (no fatigue) to 0 (fully fatigued) at an accelerated rate (relative to a real muscle) of 0.01/s. (5) Skeletal model mismatch: The segment inertial parameters (i.e. mass, length, center of gravity), can only be measured or estimated at a finite accuracy, 73

74 creating a mismatch between the model embedded in the feedforward controller and the true arm dynamics. This model mismatch means that the feedforward controller will not perfectly cancel out the nonlinear dynamics, giving rise to a residual, destabilizing, term. We used a probabilistic approach to analyze the robustness (sensitivity to model mismatch) of the controller [17, 37]. While the inertial parameters of the controller s internal feedforward arm model remained fixed at their initial values, those of the forward arm dynamics were sampled from normal parameter distributions. The mean and standard deviation of each parameter (see Table 1) were obtained by combining data from [44] and [23]. These ranges do not cover all human arms, but represent a reasonably broad population. We performed a total of 459 simulations, each with a different set of inertial parameter values for the forward arm dynamics, and determined the worst-case performance over all simulations, as indicated by the highest RMSEs between the desired and simulated shoulder and elbow angles. According to [17, 37], 459 simulations allows us to state, with 99% certainty, that 99% of all arms sampled from the normal distributions described above will have better tracking performance than the worst performance evaluated over these 459 simulations. (6) Combination scenario: A final simulation scenario evaluated the tracking performance when all perturbations were in effect: an additional end-point load of 0.35kg (weight of a soda can, active for the entire duration of the movement), worst-case estimates for the segment inertial parameters, 25% overestimation of all maximum muscle forces ( η= 0.75 ), linear muscle fatigue from 0% to 50% over 74

75 the 50s time interval, and ZMWN with amplitudes of 5º on joint angle measurements and 2.5º/s on angular velocity measurements. For all simulations, the muscle stimulation frequency was set to 12Hz, meaning that the muscle stimulations were adjusted only once every 80ms and that these stimulations were then kept constant over each 80ms interval (i.e. 80ms zero-order hold). Tracking performance was measured using the Root- Mean-Square Error (RMSE) between the desired joint angles ( q d ) and the simulated joint angles (q). The RMSE was calculated as: RMSE where n is the number of data points. ( q q ) T ( q q ) d d = (14) n Two different desired joint angle trajectories were considered. The first consisted of a series of reaching movements mimicking a feeding pattern: starting with the hand at the mouth (30º shoulder angle, 160º elbow angle) the joint angles were moved along a maximally smooth (i.e., "minimum jerk") trajectory ([12]) until the arm was fully extended forward (i.e. 90 º shoulder angle, 5º elbow angle). The time allotted for this extension was 2s. After the extension phase, the arm was required to hold its position for 3s to allow for the manipulation of objects (e.g., food, cup). The arm was then retracted for another 2s to its original position, followed by an additional 3s hold to allow time for task completion (e.g., feeding, grooming). These reaching movements were used to 75

76 evaluate the effects of changes in end-point load, muscle model inaccuracies, measurement noise and muscle fatigue. The second set of desired joint angle trajectories was constructed as a series of twenty step functions for both joints, where the target joint angles were sampled uniformly over their respective ranges of motion. The time to reach each target was again set to 2s, with a subsequent plateau of 3s before a new reaching movement was initiated. Again, the trajectories were chosen such that the target joint angles were reached within the allotted time while maximizing smoothness. These trajectories were used to evaluate the effects of imperfect knowledge with respect to the segment inertial parameters (i.e. robustness analysis) and the combination scenario where all perturbations were active. All Simulations were performed using MATLAB and its Simulink environment (R2009b, The Mathworks, Natick, MA). [38]. 3.4 Results Figure 4 shows the simulated tracking performance achieved by the proposed nonlinear feedforward-feedback PID controller with time delay compensation. The figure contains seven panels showing the results for each of the seven simulation scenarios: (a) ideal scenario (no active perturbation, no model mismatch), (b) changing loading conditions, (c) muscle model mismatch, (d) muscle fatigue, (e) skeletal model mismatch (robustness), (f) measurement noise and (g) combination scenario (all perturbations active with worst-case model mismatch). Each panel contains multiple plots. The top two plots of each panel show the simulated (thin black line) and intended joint angle trajectories 76

77 (thick grey line) for the shoulder (top plot) and elbow (second plot from top). Depending on the simulation scenario, additional plots may be included. This is the case in panels b, c, d, f and g. In panel b (changing loading conditions), a third plot shows the addition of different end-point loads over time.. In panel c (muscle model mismatch), a third plot shows the under- or overestimation of the maximum muscle forces over time. In panel d (muscle fatigue) a third plot shows the joint torques as predicted by the controller. In panel f (measurement noise), a third and fourth plot show the noise added to the angular position and angular velocities, respectively, at different time points. Finally in panel g (combination scenario), a third plot shows the muscle fatigue and a fourth and fifth plot again show the noise added to the angular position and angular velocities respectively. The RMSE between the simulated and intended joint angle trajectories was used to quantify the tracking performance of the controller. The RMSE between the simulated and intended joint angle trajectories was used to quantify the tracking performance of the controller and are included in the top two plots of each panel either for each individual reach as in panels b, c and e or for the entire movement trajectory as in panels a, e and g. An exception to this rule is panel d (muscle fatigue), where only the data for the first 50s (0%-50% fatigue) were considered when calculating the RMSEs. Baseline scenario (panel a): in the absence of any perturbations (but with stimulation frequency limited to 12Hz) the controller achieved RMSE values of 0.27º for the shoulder and 0.62º for the elbow joint. 77

78 Changing loading conditions (panel b): as the mass of the manipulated object was increased the system response became more oscillatory and the tracking performance degraded from its baseline performance (i.e. 0kg load) to 1.65º RMSE for the shoulder and 9.88º RMSE for the elbow (i.e. 3kg load),. Muscle model mismatch (panel c): the RMSEs decreased as the arm s maximum muscle forces increased, even though the mismatch between the muscle models of the arm and the controller became larger. RMSE values dropped from 1.26º for the shoulder and 3.44º for the elbow at 50% maximum muscle overestimation to 0.28º for the shoulder and 0.48º for the elbow at 50% maximum muscle underestimation. Muscle fatigue (panel d): Note that the joint torques as predicted by the controller increased as the fatigue rate increased. The RMSE values calculated over the first 50s (0%-50% muscle fatigue), were 0.64 º for the shoulder and 1.70º for the elbow. Note that the system became unstable at a 72% muscle fatigue level because insufficient force could be generated. Skeletal model mismatch (panel e): evaluated over 459 sampled arm models, the highest RMSE values were 0.86º for the shoulder and 1.11º for the shoulder. However, these maxima were found for different arm models. Panel E shows the simulated responses for the worst-case arm model that resulted in a RMSE of 1.11º for the shoulder, which then resulted in a RMSE of 0.53º for the elbow. Measurement noise (panel f): RMSE values for both the shoulder and elbow joint increased as the amplitude of the measurement noise increased, with the arm becoming unstable at noise levels of 20º amplitude for the joint angle 78

79 measurements and 10º/s amplitude for the angular velocity measurements. For noise levels expected under normal operating conditions (i.e. 5º and 2.5º/s amplitude ZMWN) we found RMSE values of 1.39º for the shoulder and 2.86º for the elbow. Combination scenario (panel e): when all perturbations were active, the controller achieved RMSE values of 2.99º and 5.15º for the shoulder and elbow respectively. 79

80 80

81 Figure 4. Simulated tracking performance of the FES-enabled arm for the nonlinear feedforward-feedback PID controller with time delay compensation. The top two plots of each panel show the tracking performance for the shoulder (top plot) and elbow (second plot) joints. The intended trajectories are indicated by a thick gray line, while the simulated trajectories are indicated by a thin black line. Where additional plots are included, they show the evolution of perturbations acting on the arm for that particular simulation scenario, with the exception of the third plot in panel d, which shows the joint torques predicted by the controller. Figure III-4. Simulated tracking performance of the FES-enabled arm for the nonlinear feedforwardfeedback PID controller with time delay compensation. 3.5 Discussion Using a 2DOF arm model we evaluated the stability and tracking performance of the proposed nonlinear feedforward-feedback controller with time delay compensation (Eq. 2) for a wide variety of internal and external perturbations: changing end-point loading, muscle and skeletal model mismatches, muscle fatigue, and measurement noise. In all of the simulations, the electrical stimulation frequency was limited to 12Hz to be consistent with physiological limitations and clinical practice, imposing a 80ms time delay. The results show that the proposed controller achieved excellent tracking performance in the presence of this substantial time delay, even when subjected to a wide variety of perturbations. Changing end-point loading: excellent tracking was maintained when picking up objects of up to 3kg, which is sufficiently large to cover the majority of activities of daily living. 81

82 Robustness to muscle and skeletal model mismatch: The controller proved capable of compensating for muscle model mismatch due to incorrect estimates of the maximum muscle forces. RMSE values remained very low, even when the controller overestimated the true maximum muscle forces by 50%. A peculiar result is that RMSE values decreased when the model s maximum muscle forces increased with respect to those of the controller s internal muscle model. A possible explanation is that higher than expected muscle forces imply that the controller has access to more resources (muscle force generating capabilities) making it relatively easier to drive the arm along the desired trajectory and compensate for any deviations. The controller also showed excellent robustness to skeletal model mismatch. We can state with 99% certainty that there is a 99% probability that an arm whose inertial parameters are drawn from normal distributions with mean and standard deviations listed in Table 1, will have less than 0.86º RMSE for the shoulder and less than 1.11º RMSE for the elbow joint. These inertial parameter ranges do not cover all possible human arms, but do represent a significantly large population. If a particular arm does not belong to the studied population, then the inertial parameters can be measured in a personalized fashion. The measurement errors of these personalized measurements are likely to be smaller than the standard deviations of the entire population studied in this work, leading us to assume that robust stability and performance should be maintained, provided the controller is adjusted accordingly. 82

83 Measurement noise: compared to the other perturbations, measurement noise had the largest effect on tracking performance, although this may have been due to our choices of the amplitudes of each perturbation. Measurement noise was incremented until instability was observed at amplitudes of 20º for joint angle measurement noise and 10º/s for joint angular velocity measurement noise. These noise amplitudes are quite high and are expected to be much lower in a realistic clinical setting. Unfortunately, it is difficult to estimate the noise properties of a sensor suite, since these properties depend on the number of sensors, their type (e.g., inertial, magnetic) and their location (e.g., skin surface, implanted). For an indication of the expected noise amplitudes, we refer to Roetenberg et al. [30, 31] whose portable inertial-magnetic sensor suite measured the shoulder angle with a RMSE of 3º compared to the Vicon 470 motion capture system (Oxford Metrics Group, U.K.). Our results show excellent performance for these noise levels. One caveat is that we only considered zeromean white noise. It is likely that during clinical practice some form of signaldependent noise will be encountered. Muscle fatigue: The controller provided appropriate compensation by increasing the muscle activations as fatigue increased. Only when fatigue had surpassed 72% did the system become unstable. However, during normal every-day use, it is very unlikely for muscle fatigue to surpass 50% [4]. Combination scenario: A combination case showed that good tracking performance was maintained (shoulder RMSE: 2.99º, elbow RMSE: 5.15º) even when all perturbations were combined. 83

84 The allotted time to complete each reaching movement in this study was 2s. For controller design in general, as the desired joint angle trajectories become faster, maintaining good tracking performance becomes increasingly difficult as overshoot becomes larger and oscillations take longer to subside. Although the 2s reaching time we used is relatively slow for able-bodied subjects, we are interested in developing a control system for an FES system where the ability of the user to command rapid movements is much more limited. Thus, system performance (and the accompanying controller parameter values) at these slower movement speeds are far more relevant than for faster movements. Our long-term goal is to restore arm function after SCI. The simulation results shown here are very promising, although several practical challenges remain before the controller can be implemented in a clinical setting with a human SCI subject. Stimulated muscles forces may be quite low (<50% of able-bodied maximum) in some muscles in some individuals. Indeed, some muscles may be denervated, rendering them unexcitable even by FES. Depending on the cervical level of the SCI, denervation can affect function-critical muscles like the Deltoid (gravity compensation) and Biceps (elbow flexion). SCI may cause muscle spasticity and abnormal muscle synergies (e.g., synergistic binding of Biceps and Triceps), making the arm more difficult to control. The proposed feedforwardfeedback controller may mitigate some of these physiological limitations. Since good tracking performance was maintained at 50% muscle fatigue and at 50% overestimation of maximum muscle forces, we expect the controller to perform 84

85 adequately even with severely weakened muscles. If muscle weakness is particularly prominent in a particular subject, good performance may be reestablished by slowing down the desired joint angle trajectories, which would require lower joint torques. In case of significant denervation and/or muscle spasticity, personalized surgical intervention may be required (e.g., surgical nerve transfers and/or tendon transfer surgery). Robotic or exoskeleton assistants could also be used to augment the FES-generated muscle forces, and the controller could be extended to include these capabilities. 85

86 IV. Feasibility of controlling a realistic 5 Degrees of Freedom human arm using Functional Electrical Stimulation Peter Cooman 1 and Robert F. Kirsch 1,2 1. Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA 2. Center of Excellence in Functional Electrical Stimulation, Louis Stokes Cleveland VA Medical Center, Cleveland, OH,

87 4.1 Abstract We have designed feedforward-feedback controller with time delay compensation for a realistic (i.e., 5 mechanical degrees of freedom (5DOF), 24 muscles, nonlinear, dynamic) musculoskeletal model of the human arm. Our purpose was to evaluate the feasibility of controlling a real human arm via functional electrical stimulation (FES) to restore arm function to individuals with arm paralysis due to high-level cervical (C1-C4) spinal cord injury (SCI). The controller developed here would coordinate the small electrical currents applied by the FES system to evoke muscle contractions such that the FES-enabled arm tracks movement commands provided by the user. Input time delays inherent to FES systems significantly complicate this task: control actions take effect later than intended, compromising tracking accuracy and stability. Previously, we successfully demonstrated a time delay compensating controller for a planar 2DOF arm model. Here we extend this proof-of-concept controller to a more functionally relevant 5DOF arm model. The 5DOF controller achieved stable, accurate tracking, even in the presence of time delay, measurement noise, muscle fatigue and additional loading - if the inertial properties of the arm were accurate. This represents an important step towards a clinically viable controller for upper extremity FES neuroprostheses for human SCI users. 4.2 Introduction A complete spinal cord injury (SCI) blocks motor commands traveling from the brain to the muscles and prevents sensory information from these same muscles 87

88 from reaching the brain. Following a complete high cervical level (C1-C4) SCI, the sensory and motor control loss extends to the entire upper extremity [2], making the individual dependent on outside help for many activities of daily living such as feeding and grooming. Functional electrical stimulation (FES) [28] is a rehabilitation technique aimed at restoring motor function by electrically stimulating paralyzed muscles or the nerves that innervate them. As long as their nerves remain intact, muscles innervated below the injury site retain their ability to contract and generate force. By carefully coordinating artificial electrical stimulations, FES can partially restore arm function. In a FES neuroprosthesis, a mathematical algorithm (called the control law or controller) is responsible for coordinating these electrical stimulations. The controller guides the arm along the intended trajectory, which can be a challenging task due to a wide variety of perturbations that affect the arm in unpredictable ways: 1) muscle fatigue: muscles grow weaker after extended use, 2) measurement noise: sensors (e.g., to monitor the current arm trajectory) introduce noise, 3) environmental interaction: many functional tasks (e.g., feeding, grooming) involve interacting with objects of unknown mass or mechanical impedance, 4) model uncertainty: the arm s inertial properties (e.g., segment masses, lengths) can only be measured with finite accuracy and practical ways to measure maximum muscle forces in vivo do not exist, and 5) input time delays: intended control actions do not take effect instantaneously. Delays arise from the muscle dynamics (e.g., calcium release and reuptake, 88

89 about 10-30ms) and the discrete nature of FES stimulation. Specifically, in upper extremity FES, stimulation frequency is typically capped at 12Hz-15Hz, because higher frequencies cause rapid muscle fatigue [28]. This relatively low frequency effectively imposes an additional 80ms delay between applying stimulations and the muscles generating force. A number of controllers have been proposed ([4], [9], [10], [10], [15], [22], [36]), but they all omitted one or more of the five main perturbations. In particular, time delays were typically ignored. Previous work, described in Chapter II, evaluated three popular control strategies (proportional-derivative control, adaptive control and sliding mode control) using a simplified 2 degrees-offreedom (2DOF) arm model. Simulation results showed that each controller achieved excellent tracking accuracy, even in the presence of muscle fatigue, measurement noise, environmental interaction and model uncertainty. However, introducing input time delays quickly caused instability. In Chapter III we designed and evaluated a controller with time delay compensation, again using the planar arm model for a 2DOF arm that included one shoulder rotation (horizontal abduction-adduction) and one elbow rotation (flexion-extension), constraining the arm to the horizontal plane. Our simulation results showed excellent tracking performance for all perturbations described earlier, including time delays. 89

90 While the 2DOF planar arm model offers an interesting test case that would be relevant for individuals using an external arm support, most functional tasks require additional degrees of freedom. Therefore, this work aims to extend the time delay compensating controller to a much more realistic 5DOF arm model, allowing three shoulder and two elbow rotations. This would allow FES neuroprosthesis users to perform a much wider selection of functional tasks. 4.3 Methods General overview Our goal was to design and evaluate a controller for a 5DOF upper extremity FES neuroprosthesis in the presence of input time delays. In this study, we used a 5DOF musculoskeletal arm model that describes the salient mechanical and muscle contractile properties of the arm, and allowed us to evaluate the control system challenges for this system prior to actually implementing such an invasive and speculative system in human subjects. The controller coordinated the muscle activations (inputs) to the arm model such that simulated joint angles (outputs) closely tracked a given set of intended trajectories. To quantify tracking accuracy, we calculated the root-mean-square error (RMSE) between the simulated and intended joint angles across all five movements tested (see below). We tuned 36 different sets of parameters for the controller for different situations: combinations of six measurement noise amplitudes and six levels of arm model uncertainty (robustness). For each 90

91 combination scenario, we used a grid search to tune the controller parameters to maximize tracking accuracy (minimize RMSE). Subsequently, we simulated the tracking accuracy for each of the 36 optimal controllers for two additional perturbations: muscle fatigue and a 0.35kg load acting at the wrist. A block diagram of this process is included in Figure 2c. The following sections provide further details of each step The musculoskeletal arm model We followed a model-based approach to controller design and evaluation, because it allowed us to easily evaluate the controller s behavior for a wide variety of scenarios, some of which would have been difficult or even impossible to replicate in a clinical setting (e.g., increasing arm length) [19]. The 5DOF musculoskeletal arm model has previously been described in [5]. Figure 1a shows the degrees of freedom of the model along with the coordinate reference frame convention. The upper arm and the forearm were modeled as rigid links. The shoulder joint was modeled as a series of three revolute frictionless joints according to a Y-Z -Y Euler angle convention [23], corresponding to plane of elevation (Y), angle of elevation (Z ) and internal/external rotation (Y ). Similarly, the elbow joint was modeled as a series of two revolute frictionless joints, according to an X-Y convention [43], 91

92 corresponding to elbow flexion-extension (X) and forearm pronosupination (Y ) respectively. The ranges of motion allowed for each DOF are given in Figure 1b. Figure 1. (a) Coordinate frame definition for the five degrees of freedom (5DOF) musculoskeletal arm model (posterior view of the right arm). (b) Simulated ranges of motion of the five DOFs. (c) Model parameter mean values and standard deviations. Figure IV-1. 5DOF arm model DOFs, ranges of motion and inertial parameter values. The equations of motion for the 5DOF arm are: ( ( )) ( ) ( ) ( ) ( ) ( ) ( ( )) ( ) ( ) M q t q t + C q t,q t q t + G q t + d t =τ t T (1) where q( t ), q ( t) and q ( t) are vectors of the joint angular positions, velocities and accelerations, M ( q ) is the joint-space inertia matrix, C( q,q ) are the 92

93 Coriolis/centrifugal effects, G( q ) denotes the effects of gravity, d is the torque vector due to any perturbations, and τ is the vector of joint torques delayed by a time T. The equations of motion are parameterized by the body segment inertial parameters: mass ( m 1 and m 2 ), length ( L 1 and L 2 ) and center of gravity location with respect to the proximal joint ( L and c1 L c2 ). The subscripts 1 and 2 designate the upper arm and forearm respectively. The hand was modeled as a point mass located at the end of the forearm. Its mass is denoted by m. Initially, the inertial parameters were set to the mean values found by [44] and [23] (see Table 2). (see Figure 1c). We used the Robotics Toolbox for MATLAB [7] to construct and solve the equations of motion (see Appendix F for MATLAB code). h Even with FES assistance, SCI subjects typically have insufficient strength to support their arm against gravity, necessitating the use of a mechanical arm support system. For gravity compensation, we assumed a constant upward force of 9.9N acting at the wrist the force required to statically balance the arm in a resting position similar to that of a person using the armrest of a wheelchair: upper arm vertically oriented along the side, 90 elbow flexion, and the palm of the hand facing downward. 93

94 The model joints were actuated by 24 muscles, each described by its own second-order Hill-type muscle model [24] and incorporating a combination of a first-order excitation-activation dynamics model and a first-order activationcontraction dynamics model. The parameter values for the muscle models were obtained from cadaver studies [20] The controller The proposed control law is given as (see also Figure 2a): c ( ) ( ) ( ) ( ) ( ( )) ( ) ( ( )) ( ) ˆ ( ( ) ( )) d ( ) ˆ ( ( )) τ t = Mˆ q t q t +α Mˆ q t e t +βmˆ q t σ t + C q t,q t q t + G q t t ( ) ( ) ˆ 1 + k σ t +βσθ+ M ( θ) τθ ( T) τθ ( ) dθ kσ0 0 (2) ( ) ( ) where ( ) ( ) ( ) e t = q t q t (3) d ( ) = ( ) ( ) e t q t q t (4) d ( t) e( t) e( t) σ = +α (5) where q( t ) and q ( t) are the current joint angular positions and velocities, ( ) q ( t) and q ( t) d are the intended joint angular positions, velocities and d q t, accelerations, and α, β and k are positive, real-valued, tunable parameters. The ˆ -operator indicates that we cannot perfectly know the true arm dynamics and d 94

95 therefore need to rely on the modeled inertia matrix ˆM, Coriolis/centrifugal effects Ĉ and gravitational effects Ĝ. This controller can be subdivided into: (a) a feedforward component: ff ( t) Mˆ ( q( t) ) qd ( t) Cˆ ( q( t ),q ( t) ) q ( t) Gˆ ( q( t) ) τ = + + (b) a feedback (proportional-integral-derivative or PID) component: t ( ) ˆ τ fb t =α M q( t) e t +βm q t σ t + k σ t + βσ θ dθ kσ 0 0 ( ) ( ) ˆ ( ( )) ( ) ( ) ( ) ( ) (c) a time delay compensating term: ( ) ˆ 1 delay ( )( ( ) ( )) t τ t = km θ τθ T τθ dθ 0 95

96 96

97 Figure 2. (a) Block diagram of the arm model in the loop with the proposed controller. The controller takes as inputs the desired arm kinematics ( q d, q d and q d ) and the measured joint angles and angular velocities ( q and q ) corrupted by noise ( n ). The controller outputs torques that are to be distributed over the muscles of the arm model (see panel b). The muscle torques generate arm movements that may be perturbed. ( d ). (b) Block diagram for the muscle redundancy resolution. The control torques ( τ c ) are distributed over the 24 muscles of the arm model using an optimization approach (Eq, 6-7). 12Hz electrical stimulation introduces 80ms time delay ( T ). (c) Controller parameter tuning using grid search. Given measurement noise amplitudes X and model uncertainty Y, we simulated the tracking accuracies of the arm model under 8000 candidate controllers. The optimal controller parameters ( α*, β *, k * ) were those that maximized tracking accuracy (i.e. minimized RMSE). This grid search was repeated 36 times for different combinations of measurement noise amplitudes and model uncertainty. Each optimal controller was subsequently evaluated under four different conditions: (1) no additional perturbations (baseline), (2) muscle fatigue, (3) 0.35kg at the wrist and (4) muscle fatigue and 0.35kg load. Figure IV-2. Block diagrams of the nonlinear feedforward-feedback PID controller with time delay compensation, muscle redundancy resolution and controller tuning approach. This controller was based on the pure feedback time delay compensating controller designed by Sharma et al. ([33-34]), to which we added a feedforward controller. A more detailed discussion of the proposed combined feedforwardfeedback time delay compensating controller (Eq. 2) can be found in Chapter III Control signal sampling The muscle stimulations were discretely generated at a frequency of 12Hz, the stimulation frequency most typically used in clinical FES systems to avoid muscle fatigue. Thus, the controller calculated the stimulation parameters only once every 80ms and held them constant during each interval. This reflects the actual operation of a FES system and effectively introduced a 80ms delay between stimulations and the muscle forces acting on the skeleton, in addition to any delays incurred by the muscle excitation-activation dynamics (10-30ms). 97

98 4.3.5 Resolving muscle redundancy The proposed controller (Eq. 2) determines the joint torques required to steer the arm along the intended trajectory and to compensate for any deviations. However, it does not directly modulate joint torques, but rather specifies muscle stimulation levels such that the several resulting muscle forces together generate the required joint torques. This problem is dynamically redundant because the arm model includes significantly more muscles (i.e. 24) than degrees of freedom (i.e. 5). This, and the antagonistic organization of many of the muscles, implies that a given set of joint torques can usually be accomplished by an infinite number of different muscle force combinations. To find a unique solution, we used an optimization approach (see also Figure 2b) to find the set of muscle activations that minimizes muscle effort approximated by the sum-of-squares of the muscle stimulations - while still generating the required joint torques. Mathematically, the optimization algorithm can be written as: i= 24 2 ai i= 1 min J = (6) i= 24 Subject to r F ( a,q) τ = for j=1,,5 (7) c, j ij i i i= 1 where a i is the activation of the i th muscle, τ c is the vector of torques to be achieved, r ij is the moment arm of the i th muscle with respect to the j th DOF and i ( ) F a,q is the force of the i th muscle. The muscle force depends on the activation i a i, while the current joint angles q are used to determine the muscle length, 98

99 necessary for the length-tension relationship. Since there is no practical way to measure the muscle contraction velocities in vivo, the optimization ignores forcevelocity contributions. The same holds for the excitation-activation dynamics. Please note, however, that, the forward muscle dynamics in the model used to evaluate tracking performance did include excitation-activation dynamics and force-velocity effects Performance measurement and requirements We quantified tracking accuracy, using the root-mean-square error (RMSE) between the intended joint angles ( q d ) and simulated joint angles ( q ), calculated as follows: RMSE ( q q ) T ( q q ) d d = (8) n where n is the number of data points. The design of a given controller thus entails finding the particular values for parameters α, β and k that minimize the RMSE. The movement intent q d was chosen to resemble the five functional tasks proposed by Cornwell et al. [8] as representative for a broad range of activities of daily living: (1) grasping and drinking from a cup, (2) pressing an elevator button, (3) touching the face, (4) picking a book off a shelf and placing it in the lap, and (5) picking food off a plate. We modified the speed and duration of these 99

100 motions, which were obtained from recordings of able-bodied subjects, to be more appropriate (i.e., slower) for FES-restored movements. Specifically, each task started and ended at the same (resting) position and the duration of each task was increased to 15s. Thus, the entire series of five movements had a total duration of 75s Controller tuning The proposed controller (Eq. 2) has three positive, real-valued tunable parameters: α, β and k. The optimal values for the controller parameters depend on the expected type and magnitude of the perturbations acting on the FES enabled arm. Two perturbations in particular are important: model uncertainty and measurement noise. Model uncertainty reflects the fact that we only have limited information about the true inertial properties of the arm model, so the feedforward portion of the controller would command somewhat inaccurate muscle actions. In controller design, robustness (maintaining stable and accurate performance even with modeling errors) and measurement noise sensitivity are typically mutually exclusive objectives: the former benefits from strong feedback control, while the latter performs better under weak feedback control (see Chapters I and II for a more detailed discussion). Unfortunately, the model uncertainty and measurement noise for an implanted FES neuroprosthesis remains largely unknown. Therefore, we explored the performance of 36 different versions of the controller, each optimized for a different combination of model uncertainty and measurement noise amplitudes (see also Figure 2c). 100

101 Noise corrupting the joint angle and angular velocity measurements was modeled as zero-mean white noise (ZMWN) with six possible choices of amplitudes: (1) 0º and 0º/s, (2) 2º and 1º/s, (3) 4º and 2º/s, (4) 6º and 3º/s, (5) 8º and 4º/s, and (6) 10º and 5º/s. Model uncertainty was also varied over 6 levels: 0% to 10% in 2% increments. Here, 0% represents the ideal case where the controller s internal arm model perfectly matches the forward arm model. X% model uncertainty means that the body segment inertial parameters were measured with an error of up to X% with respect to the true value. Allowing the inertial parameters to take up any value within the specified region (+/- X% with respect to the mean values), would result in an infinite set of perturbed arm models. To guarantee stability with absolute certainty would require us to evaluate the tracking performance for this infinite number of perturbed models. Instead, as in earlier work (see Chapters II and III), we used a probabilistic approach to robustness testing ([17], [37]). This approach estimated the worst-case performance (highest RMSE) with high levels of certainty by simulating the tracking accuracy for a large number of perturbed arm models. Specifically, 459 perturbed arm models were generated with inertial parameters sampled from normal distributions with mean values equal to the initial values and standard deviations equal to the mean values times X/2, where X is the level of model uncertainty (i.e. 0%, 2%, 4%, 6%, 8% or 10%). Using a stable, but not necessarily optimal, controller, we 101

102 simulated the tracking accuracy for these 459 perturbed arm models. The model with the highest RMSE (worst accuracy) was selected as the worst-case and subsequently used for optimizing the controller parameters. The six possible measurement noise amplitude choices and six choices for model uncertainty result in 36 combination scenarios. For each combination we performed a grid search to tune the three controller parameters α, β and k. The candidate controller parameters values were: α = 1/s to 20/s in increments of 1/s, β = 1/s to 20/s in increments of 1/s and k = 0.1 kkkk. mm2 ss to 2 kkkk. mm2 ss in increments of 0.1 kkkk. mm2 ss. This thus resulted in 8000 candidate controllers. We simulated the tracking accuracy for each controller and selected the one that achieved the best performance (i.e. lowest RMSE) Additional perturbations Once we had an optimal controller for each of the 36 combinations of measurement noise amplitude and model uncertainty, this controller was evaluated for two additional (and functionally important) perturbations: muscle fatigue and changing loading conditions. Prolonged contraction causes muscles to fatigue and gradually lose their ability to generate force. We assumed a linear fatigue rate increasing from 0% to 50% over 75s. Functional tasks often involve objects (e.g., cup, comb). Grasping an object changes the inertial properties of the arm, so we selected a constant, additional mass of 0.35kg (i.e. soda can) for the 75s movement intent duration. 102

103 Thus, for each of the 36 combinations of measurement noise and model uncertainty, we evaluated performance by performing four simulations using the controller that was found to be optimal for that combination: (1) baseline: no additional perturbations, (2) muscle fatigue: increasing linearly from 0% to 50% over 75, (3) additional 0.35kg load acting at the wrist, and (4) combined muscle fatigue and additional load. 4.4 Results Figure 3 summarizes the optimal tracking performance, for 36 different combinations of model uncertainty and measurement noise amplitude, in the absence of any additional perturbations (i.e. muscle fatigue and additional wrist loading). Each panel in figure 3 corresponds to a different joint degree of freedom: shoulder plane of elevation (panel a), shoulder elevation (panel b), shoulder internal rotation (panel c), elbow flexion/extension (panel d) and forearm pronosupination (panel e). Each panel contains a 3D plot and a table showing the optimal RMSE value for each combination of expected model uncertainty and measurement noise amplitudes, with lower RMSE values indicating better tracking performance. The RMSE values were calculated over all five archetypal functional tasks described above. Note that we were able to find stabilizing controllers for all 36 combination scenarios. Also, for all five DOFs, RMSEs followed a common trend, specifically increasing (i.e. tracking performance decreased) as either model uncertainty or measurement noise amplitudes 103

104 increased. 104

105 Figure 3. Summary of simulated optimal tracking performance as indicated by the RMS tracking errors (Z-axis on plots) for different combinations of measurement noise amplitude (X-axis) and arm model uncertainty (Y-axis). Each of the five panels corresponds to one of the five joint DOFs of the arm model. The data in the tables corresponds to the 3D plots in each panel. Figure IV-3. Optimal RMSEs for 36 combinations of model uncertainty and measurement noise. Figure 4 illustrates the tracking performance when the controller (Eq. 2) was optimized assuming a perfect arm model (0% uncertainty) and noise-free measurements (optimal controller parameters: 2 α= 20, β= 11, k = 0.4 kg.m s s s ). Each of the four panels shows the tracking performance for a different set of perturbations: no additional perturbations (panel a), an additional 0.35kg load acting at the wrist (panel b), muscle fatigue (panel c), and both the 0.35kg load and muscle fatigue combined (panel d). Each panel contains five plots corresponding to each joint DOF of the arm model. Thick gray lines indicate the movement intent to be tracked by the controlled arm model, while thin black lines indicate the simulated joint angles achieved by the controller. The RMSE value for each DOF, can be found in the upper left hand corner of each plot. Note that in all four cases, the proposed controller achieved stable tracking. With the exception of shoulder internal rotation (DOF3) for the case of load plus noise (panel d),there is no overshoot or oscillation, and it is very minor even in this case. Also, for all DOFs in all four simulated scenarios, tracking accuracy 105

106 remained high, as indicated by the low RMSE values. 106

107 Figure 4. Simulated tracking performance for the controller optimized assuming noisefree measurements and a perfect arm model (no measurement noise, no model uncertainty). The X-axes denote simulation time, while the Y-axes denote joint angles for each of the five DOFs: shoulder plane of elevation (DOF1), shoulder elevation (DOF2), shoulder internal rotation (DOF3), elbow flexion/extension (DOF4) and forearm pronosupination (DOF5). Thick gray lines denote the intended joint angles, while thin black lines denote the simulated joint angles. Panel a illustrates tracking performance for no added load or fatigue, panel b for the added 0.35 kg load, panel c for fatigue, and panel d for both added load and fatigue. The optimal tracking performance for each DOF, indicted by the RMSE between the intended and simulated joint angle trajectories, is shown in the top-left corner of each sub-plot. Figure IV-4. Simulated optimal tracking performance in the absence of model uncertainty and measurement noise. Figure 5 shows the same variables and conditions as Figure 4, except here measurement noise (i.e. 4º and 2º/s amplitude ZMWN) was added and the controller was optimized specifically for these measurement noise amplitudes (optimal controller parameters: 2 α= 8, β= 11, k = 0.1 kg.m ). Note that the s s s controller again maintained stability for all DOFs, even in the presence of additional loading and/or muscle fatigue. However, compared to the noise-free case (see Figure 4), RMSE values increased for all five DOFs in all four scenarios. Also, note the modest increase in oscillations, specifically for shoulder internal rotation (DOF3), in all four scenarios. 107

108 108

109 Fig. 5 Simulated tracking performance for the controller optimized assuming measurement noise.. In the illustrated cases, noise: joint angle noise of 4º amplitude ZMWN and joint angular velocities noise of2º/s amplitude ZMWN were added.. The X-axes denote simulation time, while the Y-axes denote joint angles for each of the five DOFs. Thick gray lines denote the intended joint angles, while thin black lines denote the simulated joint angles. Panel a illustrates tracking performance for no added load or fatigue, panel b for the added 0.35 kg load, panel c for fatigue, and panel d for both added load and fatigue. The optimal tracking performance for each DOF, indicted by the RMSE between the intended and simulated joint angle trajectories, is shown in the top-left corner of each sub-plot. Figure IV-5. simulated optimal tracking performance in the presence of measurement noise. Figure 6 shows the same variables and conditions as Figures 4, except here the arm model was also assumed to be uncertain (up to 10%) and the controller was the one that achieved the best tracking performance for the estimated worst-case arm model (optimal controller parameters: 2 α= 10, β= 19,k = 1.9 kg.m ). In the absence of additional loading or s s s fatigue (panel a), the controller achieved stable performance, albeit with modest oscillations for shoulder internal rotation (DOF3), and tracking performance (as indicated by the increase in RSME for all five DOF) decreased compared to the when the model was assumed to be perfectly known (Figure 4a). The most obvious impact of adding model uncertainty, however, was that adding extra 109

110 loading and/or muscle fatigue rapidly caused instability (panels b, c and d). 110

111 Figure 6. Simulated tracking performance for the controller optimized assuming 10% arm model uncertainty. The X-axes denote simulation time, while the Y-axes denote joint angles for each of the five DOFs. Thick gray lines denote the intended joint angles, while thin black lines denote the simulated joint angles. Panel a illustrates tracking performance for no added load or fatigue, panel b for the added 0.35 kg load, panel c for fatigue, and panel d for both added load and fatigue. The optimal tracking performance for each DOF, indicted by the RMSE between the intended and simulated joint angle trajectories, is shown in the top-left corner of each sub-plot. Figure IV-6. Simulated optimal tracking performance in the presence of model uncertainty. Figure 7 is similar to Figure 6 in including 10% model uncertainty, except that measurement noise (4º and 2º/s amplitude ZMWN) has also been included. The controller was optimized for this combination of model uncertainty and noise amplitudes, resulting in controller parameters of 2 α= 10, β= 16,k = 1.1 kg.m. Similar to the noise-free, 10% model s s s uncertainty case (Figure 6), the controller achieved stable tracking only when no other perturbations were present. Although stable with no addition perturbations, several DOFs exhibited significant oscillations, specifically shoulder internal rotation (DOF3) and to a lesser extent shoulder angle of elevation (DOF2), elbow flexion/extension (DOF4) and forearm pronosupination (DOF5). When both model uncertainty and measurement noise were included (panel a), the tracking accuracy for all five DOFs decreased (higher RMSE values) compared to when measurement noise acted alone (Figure 5a), when model uncertainty acted alone (Figure 6a) and when both perturbations were absent (Figure 4a). Also, note that introducing the extra 0.35kg load (panel b), muscle fatigue (panel c) or both the 111

112 load and fatigue (panel d) immediately resulted in instability. 112

113 Figure 7. Simulated tracking performance for the controller optimized for both model uncertainty and measurement noise. Model uncertainty was 10%, added angular noise was 4º amplitude ZMWN and added angular velocity noise was 2º/s amplitude ZMWN. The X-axes denote simulation time, while the Y-axes denote joint angles for each of the five DOFs. Thick gray lines denote the intended joint angles, while thin black lines denote the simulated joint angles. Panel a illustrates tracking performance for no added load or fatigue, panel b for the added 0.35 kg load, panel c for fatigue, and panel d for both added load and fatigue. The optimal tracking performance for each DOF, indicted by the RMSE between the intended and simulated joint angle trajectories, is shown in the top-left corner of each sub-plot. Figure IV-7. Simulated optimal tracking performance in the presence of model uncertainty and measurement noise. Table 1 summarizes the effects of measurement noise, extra loading and muscle fatigue on tracking accuracy in the absence of model uncertainty. Compared to the noise-free scenario, adding only measurement noise (4º and 2º/s amplitude ZMWN), increased the RMSE values for all five DOFs (compare scenarios 2 and 1), by an average of 2.73º. When only muscle fatigue was included, only some of the DOFs were negatively affected, specifically shoulder elevation (DOF2) and internal rotation (DOF3), while the RMSEs for the other DOFs decreased (compare scenarios 3 and 1). Still, averaged over all five DOFs, adding muscle fatigue increased RMSEs by 0.35º. Adding 0.35kg at the wrist, but no measurement noise or muscle fatigue, increased RMSEs for all five DOFs (compare scenarios 4 and 1) by an average of 0.80º. Including all three perturbations resulted in the highest RMSEs for all five DOFs, with an average RMSE increase of 8.32º compared to the perturbation-free case (compare scenarios 8 and 1). Again, please note that Table 1 excludes all conditions with 113

114 model uncertainty, as this produced instability when combined with any of the other perturbations. Table IV-1. Contributions of various perturbations on optimal tracking performance. Table 1. Summary of the relative contributions of the each studied perturbation on optimal tracking performance. For each of the eight simulated scenarios, a different set of perturbations was active (indicated by check marks). The corresponding optimal tracking RMSEs (in degrees) for each of the five degrees of freedom are listed on the right side of the table. 4.5 Discussion Our goal was to extend the time delay compensating controller previously developed for a 2DOF, planar arm model [4] to a controller for a 5DOF arm model, motivated by the fact that most functional tasks require the arm to move outside of the horizontal plane. Extending the controller to the 5DOF arm model would significantly expand the reachable arm space and allow the FES neuroprosthesis user to perform a much wider variety of functional tasks. 114

115 We successfully designed controllers that achieved stable tracking performance for a 5DOF arm model, even in the presence of a realistic input time delay (i.e. 80ms), a wide range of model uncertainty levels (0% to 10%) and measurement noise amplitudes (0º to 10º amplitude ZMWN for joint angle positions, and 0º/s to 5º/s amplitude ZMWN for joint angular velocity measurements). In earlier work, described in Chapter II, we have shown that time delays rapidly cause arm instability even for advanced, well-established control strategies (i.e. feedforward-feedback PD control, adaptive control and sliding mode control). Therefore the results of the current study represent an important step towards clinically implementing the controller in an upper-extremity FES neuroprosthesis for high-level SCI individuals. As model uncertainty increased, the highest attainable tracking accuracy decreased (minimum RMSE increased), as was expected. We allowed for model uncertainty levels of up to 10%, reflecting our assumption that inertial parameter measurement and estimation errors deviate at most 10% from their true values. Even allowing for 10% model uncertainty, the proposed controller maintained stability, with reasonably good tracking RMSEs (< 10.18º). However, we also found that controllers tuned to be optimal for systems with model uncertainty, did not generalize to additional perturbations not taken into account in the optimization (i.e. muscle fatigue or 0.35kg load), resulting in rapid instability. Since we cannot a priori measure the weight of each object an FES-user may interact with or measure muscle fatigue, we cannot include these variables in the 115

116 controller optimization. We recommend two approaches that may alleviate these issues. First, adding an adaptation component would allow the controller to learn the properties of its changing environment (e.g., object weights) and change the controller settings accordingly. A second approach involves augmenting the FES neuroprosthesis with a more advanced gravity compensation system (e.g., robot manipulator). The earlier work on the 2DOF arm model without gravity (see Chapter III) showed that the proposed controller easily achieved excellent tracking performance for additional loading of up to 3kg, far more than the 0.35kg used here. By reducing, or eliminating gravity, the results obtained for the 5DOF arm model may more closely resemble those obtained for the 2DOF arm model. As measurement noise amplitudes increased, tracking performance decreased. The noise amplitudes ranged from 0º to 10º for joint angular position measurements and 0º/s to 5º/s for joint angular velocity measurements. The joint angle measurement noise amplitude was based on the positional accuracy of an external, portable upper extremity sensor suite designed by Roetenberg et al.[30-31]. We set the angular velocity measurement noise amplitude to half of the angular position noise amplitude, based on the amplitudes of the respective noise-free signals. FES neuroprostheses will use implantable sensors, potentially attached to the bones. This means that kinematic measurements would not suffer from tissue-related artifacts and are therefore likely to be even more accurate (< 4º). 116

117 Adding a 0.35kg load at the wrist had little effect on the arm stability and tracking performance, except for the simulated scenarios where the arm model was uncertain, in which case it immediately caused instability. We expect that when the issue of model uncertainty is resolved, the controlled arm should achieve adequate performance, even for loads heavier than the 0.35kg used in this study. The effects of adding muscle fatigue remained limited as well, again with the exception of the scenarios where the arm model was uncertain, in which case it also caused instability. In the absence of model uncertainty, muscle fatigue in some case decreased the tracking RMSEs for some joint DOFs. Here, muscle fatigue may have acted as a blunting mechanism, effectively reducing the gains of the optimized controller. Similar to the additional 0.35kg load, we expect that resolving the model uncertainty issue should result in a controller that adequately compensates for muscle fatigue. Of the five joint DOFs, shoulder internal rotation (DOF3) typically had the most severe tracking errors and oscillations, followed by forearm pronosupination (DOF5) and elbow flexion/extension (DOF4). Shoulder plane of elevation (DOF1) and angle of elevation (DOF2) proved far less sensitive to perturbations. This phenomenon may be explained by the relative differences in the moments of inertia of the various joint DOFs, where lower moments of inertia would correspond to higher sensitivity to perturbations. This is an interesting result that 117

118 is not immediately apparent from studying a planar 2DOF arm model, where both links typically have similar moments of inertia. A final limitation relates to the intended movement velocity and muscle redundancy resolution. When distributing the torques (prescribed by the controller) over the muscles, the inverse muscle models neglected force-velocity effects. However, these effects were included in the forward muscle models. For slow muscle contractions (negligible force-velocity effects), the difference between the joint torques acting on the joints and those predicted by the controller should remain small. To a large extent, feedback control should successfully compensate for these small differences, but as muscle contraction velocities increase, the torque discrepancies grow and the controller can no longer guarantee good tracking performance. Therefore, we limited the intended movement to slowly changing trajectories. At the current stage of FES neuroprosthesis development, replicating even these slow movements in SCI subjects would be a significant improvement. However, as performance requirements increase, this limitation may become a bigger concern. 118

119 V. Conclusions and future recommendations 5.1 Overview We studied the problem of controlling arm movements generated by functional electrical stimulation to restore function in individuals who have suffered paralysis due to a spinal cord injury or other neurological disorders. The controller design problem was defined as design a mathematical algorithm that coordinates the electrical muscle stimulation parameters (which muscles, at what time and at what intensity), such that the resulting muscle forces drive the arm along a pre-defined intended trajectory. The standard control strategy in upper extremity FES combines feedforward and feedback control. The feedforward controller gives a fast, but inaccurate, estimate of the required muscle stimulations, based on an internal inverse model of the human arm dynamics. The feedback controller monitors the current performance of the arm and, if necessary, adjusts the stimulation parameters in an attempt to compensate for any deviations caused by external perturbations or inaccuracies in the inverse arm model embedded in the feedforward controller. Using a musculoskeletal arm model, we analyzed the strengths and weaknesses of three popular feedforward-feedback control strategies: proportional-derivative (PD) control, adaptive control and sliding mode control. The first controller represents the current standard, while adaptive and sliding 119

120 mode control are more advanced controllers designed specifically to guarantee stability and satisfactory performance in case of model uncertainty, i.e., differences between the true arm dynamics and the arm model embedded in the feedforward controller. Model uncertainty arises from our inability to perfectly measure or calculate the true arm dynamics. For example, there is currently no practical way to measure the maximum muscle forces in vivo. Feedforwardfeedback control does offer some robustness against this model uncertainty by virtue of its feedback controller, but, in a theoretical sense, stability and satisfactory performance cannot be guaranteed. With adaptive and sliding mode control we can theoretically prove that stability and performance will be maintained, provided the differences between our arm model and the true arm dynamics remains bounded. Using a two degrees of freedom (2DOF), planar, two-link arm model, we evaluated the feedforward-feedback PD, adaptive, and sliding mode controller for a wide range of perturbations that we expect neuroprosthesis users will encounter during normal operating conditions: changing loading conditions as users interact with their environment, muscle fatigue, muscle and skeletal model uncertainty, measurement noise, and input time delays. We found that all three controllers performed very similarly: they remained stable for the same set of perturbations (i.e., changing loading conditions, muscle, and model uncertainty, muscle fatigue) and became unstable for the same set of perturbations (i.e., measurement noise and input time delays). For those perturbations where the 120

121 controllers achieved stable tracking, performance was excellent as indicated by very low values (< 1 ) of the low root-mean square errors (RMSEs) between the various simulated and intended joint angle trajectories. In terms of tracking accuracy, the sliding mode controller performed best, followed by the adaptive controller and the PD controller. However, the performance differences were very small, and likely not functionally significant. The tracking accuracy achieved by the PD controller was already so high that it left little room for improvement by the other controllers. Also, adaptive and sliding mode control introduce higher numerical complexity and, in case of the sliding mode controller, higher control effort (roughly estimated by the sum of squares of the muscle activations predicted by the controller). In our opinion, the marginal tracking accuracy improvements of the adaptive and sliding mode controller fail to justify their higher complexity and control effort. Therefore, we recommend the feedforwardfeedback PD controller for use in upper extremity FES neuroprostheses, at least until performance requirements become much more demanding and even the marginal performance improvements of adaptive and sliding mode control become functionally significant. Our simulation results pointed towards measurement noise and input time delays as the two factors with the most negative impact on the stability and performance of the FES: for all three controllers the arm model quickly became unstable in the presence of measurement noise and time delays. Of these two factors, we have the most influence over measurement noise, by designing and 121

122 implementing more accurate sensors and better noise filters. Input time delays, however, are unavoidable in upper extremity FES neuroprostheses because they are caused by the muscle dynamics (excitation-activation dynamics) and the relatively low stimulation frequency (i.e. 12Hz) that injects an estimated 80ms of delay between the controller prescribing a set of stimulations and those stimulations actually affecting the arm. Higher stimulation frequencies would reduce these time delays and therefore their negative effects on stability and performance. Unfortunately, muscles fatigue at much higher rates when stimulation frequencies are high, which effectively imposes an upper limit of 12Hz to 15Hz [28]. Our initial attempt to solve the issue of input time delays involved reducing the gains of the feedback controller. By decreasing the feedback gains, we would effectively shift the control burden away from the feedback controller towards the feedforward controller. Since feedforward control relies exclusively on the movement intent and an inverse arm model, it is entirely insensitive to time delays, so using control signals that rely more on feedforward control may decrease the sensitivity to time delays for the entire controlled system. We performed many simulations at varying stimulation frequencies, to verify whether 1) decreasing the feedback gains would in fact re-establish stability, 2) how much we would need to decrease the feedback gains in order to re-establish stability and 3) what, if any, would be the effects on tracking performance and robustness against the remaining perturbations (e.g. muscle fatigue, measurement noise). 122

123 We found that decreasing the feedback gains did succeed in reestablishing stability in the presence of input time delays. Longer time delays (i.e. lower stimulation frequencies), required a larger decrease in the feedback gains. Also, as time delays became longer, optimal tracking performance deteriorated. Simulated movements became slower to respond to changes in the movement intent (sluggish), had significant tracking errors, and proved far more sensitive to other perturbations (e.g. muscle and skeletal model uncertainty). Thus, decreasing the feedback gains succeeded in re-establishing stability, but the benefits of strong feedback control (i.e., fast and efficient tracking error compensation and robustness against perturbations) were lost. We therefore set out to develop a novel controller that could retain the benefits of high-gain feedback control even in the presence of input time delays. Based on the work of Sharma et al. [33, 34], we designed a feedforwardfeedback time delay compensating controller for use in upper extremity FES neuroprostheses. Sharma et al. proposed a pure feedback controller that used a special prediction term to solve the problem of time delay. Using an auxiliary signal based on finite integrals of the difference between the current and delayed control signals multiplied by the modeled inertia matrix, the time delayed system could be transformed into a time delay free system. Sharma et al. proved theoretically that their proposed controller provides robustness against time delays for Lagrangian systems (e.g. the human arm) and showed its 123

124 effectiveness in practice for a two-link robot arm and 1DOF FES-assisted knee flexion/extension. Given the success and proven effectiveness of feed-forward-feedback controllers, we extended the time delay compensating controller proposed by Sharma et al a pure feedback control through the inclusion of a feedforward controller. Using the aforementioned 2DOF planar arm model, we then evaluated the stability and tracking performance of this new controller for the same set of perturbations: changing loading conditions, muscle and skeletal model uncertainty, muscle fatigue, and measurement noise. In all conditions, we assumed a stimulation frequency of 12Hz. The results of these simulations showed that the feedforward-feedback controller achieved excellent tracking accuracy in all cases. For the baseline case, assuming perfect knowledge of the true arm dynamics and no external perturbations, the controller achieved tracking RMSE values as low as 0.27º for shoulder horizontal adduction/abduction and 0.62º for elbow flexion/extension. When all perturbations were combined a simulation scenario intended to represent expected clinical operating conditions the tracking RMSEs remained low: 2.99º for the shoulder DOF and 5.15º for the elbow DOF. These results were excellent and proved the feasibility of a controller maintaining the benefits of strong feedback action (i.e., tracking error compensation and robustness against perturbations) even in the presence of input time delays. However, while the 2DOF arm model provides an interesting, easy to interpret platform for evaluating the controller, its low dimensionality limits 124

125 the arm to move in a horizontal plane, whereas the majority of functional tasks require the arm and hand to move in a three dimensional space. Building on the success of the feedforward-feedback controller for the 2DOF planar arm model, we extended the controller to control of a 5DOF arm model. This 5DOF arm model allowed for three shoulder rotations (i.e. plane of elevation, angle of elevation, and internal rotation) and two elbow rotations (elbow flexion/extension and forearm pronation/supination). Compared to the 2DOF model, the 5DOF model greatly expands the reachable workspace of the hand and arm and thus the number of functional tasks that could be performed. Besides the increase in DOFs and an increase in muscle elements, the 5DOF also accounts for gravity, which was omitted in the 2DOF arm model (where movement was limited to the horizontal plane). These extensions mean that the 5DOF arm model more closely resembles the actual human arm and the simulation results of this model should provide us a better idea of what performance we can expect if we were to implement the controller in an upper extremity FES neuroprosthesis. Using a grid search we tuned the gains of the feedback-feedforward time delay compensating controller for 36 separate scenarios, each a combination of one of six measurement noise amplitudes and one of six levels of model uncertainty. This exploration was intended as a tool for FES neuroprosthesis 125

126 designers to help them answers questions such as given the accuracy of our sensors and our knowledge of the true arm dynamics, what is the best possible tracking accuracy we could achieve? and, vice versa, if we want to achieve a pre-defined desired level of tracking accuracy, how accurate should our sensors and our arm model be?. In all 36 simulated conditions, we successfully found gains for which the feedforward-feedback time delay compensating controller stabilized the 5DOF arm model. In the baseline scenario (i.e. no measurement noise and a perfect arm model) the controller achieved tracking RMSE values as low as 2.12, 0.72, 1.69, 4.70 and 3.30 for the five DOFs respectively. For our most extreme simulated scenario (i.e. 10 amplitude zero-mean white measurement noise on joint angle measurements, 5 /s amplitude zero-mean white noise on joint angular velocities and up to 10% deviation between the arm model assumed by the feedforward controller and the controlled arm model), the controller achieved tracking RMSE values of 7.07, 8.65, 26.19, and for the five DOFs respectively. After optimizing the gains of the controller for given amplitudes of measurement noise and model uncertainty, we evaluated whether the resulting optimal controllers would generalize to any additional perturbations. Therefore, for each of the 36 noise-model uncertainty combinations, we tested the controller that was found to be optimal for that combination for two additional perturbations: 126

127 changing loading conditions (an additional 0.35 kg load acting at the wrist) and muscle fatigue (0% to 50% over 50 at a linear rate of 1%/s). We also simulated the controller s stability and performance when both of these additional perturbations were combined. From these simulations we found that adding an additional load or muscle fatigue ALWAYS destabilized the arm unless the true arm dynamics were perfectly known (no model uncertainty). The feedforward-feedback time delay compensating controller represents an important step towards controlling arm movements generated by FES. Our simulation results showed that for given noise amplitudes and expected model uncertainty, we were able to tune the controller such that it achieved stable tracking performance for that combination of noise amplitudes and model uncertainty level, in addition to the presence of input time delays of 80ms (i.e. 12Hz stimulation frequency). This was the case for a wide range of measurement noise amplitudes (up to 10-5 /s) and a high level of model uncertainty (up to 10% deviation from the true arm inertial properties). However, for any nonzero level of model uncertainty, adding an additional load or muscle fatigue (but not measurement noise) caused the 5DOF arm model to become unstable. While the proposed feedforward-feedback time delay compensating controller shows significant promise, the robustness against model uncertainty needs to improve before we can implement it clinically in an upper extremity FES neuroprosthesis to be used by SCI individuals. 127

128 5.2 Limitations and future research Improving robustness against model uncertainty While the proposed feedforward-feedback time delay compensating controller for the 5DOF arm model achieved very good tracking performance in most simulation scenarios, it failed to generalize to all scenarios. Specifically whenever there was model uncertainty combined with an additional perturbation (i.e. changing loading conditions, muscle fatigue), the 5DOF arm model became unstable. Due to safety concerns, we cannot yet recommend implementing this controller in an upper extremity FES neuroprosthesis with a SCI individual in the loop until this issue has been resolved. To better understand the problem of model uncertainty, we will first go into more detail about its effects on stability and performance, before suggesting two ways to increase the robustness of the proposed controller. We can broadly define the control problem as making a decision based on available information. This information can take many forms. Specifically in the case of controlling human arm movements this information includes, but is not limited to, the inertial properties of the arm, its current performance, the movement intent, the admittance and inertial properties of the environment and the nature and intensity of any expected perturbations. Feedforward control can rely on information from all these sources, with the exception of the current performance; we base our control decision on all the information that is available before executing the task (a priori). For example, if we intend to grasp and lift an 128

129 object of known mass, we can adjust our estimate of the required muscle activations to reflect this information before performing the lift. To achieve the best performance in terms of stability, tracking accuracy and control effort (here muscle activations), we want to use as many sources of information as possible and make sure that the information is actually reliable. In the previous example, if we had not known the mass of the object, or if we had chosen to ignore this information, or if the true mass was significantly different, then we would likely have observed an over- or an undershoot. In any case, the actual performance would have been inferior compared to the case where the feedforward controller knewand used the correct mass information. Any discrepancy in the feedforward controller will result in the arm deviating from its intended trajectory. These deviations then need to be compensated by the feedback controller. While the feedback controller is often very effective at this job, its resources are limited. It needs to compensate deviations from many different sources, soif we can reduce the impact of one perturbation, then the feedback controller can devote more of its resources to compensating the deviations due to the remaining perturbations, which would likely result in better overall stability and tracking accuracy at a lower muscular effort. In the proposed feedforward-feedback time delay compensating term, the feedforward controller only relies on information concerning the movement intent and a model of the arm dynamics. Therefore there is a clear incentive to minimize the difference between the modeled arm and the true arm dynamics. We would like to offer suggestions for future 129

130 research to reduce this difference and thus improve the overall performance and robustness of the proposed controller: 1. A better musculoskeletal arm model (a priori): the quality of the internal musculoskeletal arm model can be improved in two ways: (1) relaxing modeling assumptions (e.g., adding stiction and friction dynamics, which we omitted in this study) and (2) developing more accurate methods to measure and/or estimate the inertial parameters of the arm and tailor the FES neuroprosthesis for each individual user. 2. Adaptive control is an on-line method that monitors the current tracking performance of the arm and adjusts its internal arm model such that it more closely resembles the true arm dynamics. 3. Better gravity compensation: considering the scenario of manipulating objects of unknown mass, the feedforward-feedback time delay compensating controller for the 2DOF arm achieved excellent tracking for loads of up to 3kg, while the controller for the 5DOF arm model seemed to struggle with the relatively smaller load of 0.35kg. What differences between the arm models could explain this discrepancy? Besides the increase in DOFs, the main difference between the 2DOF and 5DOF arm model was that the latter included gravitational effects. In the case of the 2DOF arm model, gravity could be ignored, because arm movement was constricted to the horizontal plane. This means that objects of unknown mass will only cause model mismatches due to their effects on the inertia on the arm, but will have no effects in terms of gravity. On the other hand, 130

131 for the 5DOF arm model, objects of unknown mass will cause model mismatches both in terms of inertia and gravity, with the latter typically larger in size. Assuming we could somehow perfectly compensate these gravity effects as was, by definition, the case in the 2DOF arm model -- we hypothesize that the results obtained for manipulating objects of unknown mass for the 5DOF arm model would be closer to those obtained for the 2DOF arm model. When designing the controller for the 5DOF arm model, we did include some gravity compensation in the form of a fixed upward force of 9.9N acting at the wrist. This force was dimensioned such that the arm remained in equilibrium at its resting position. However, for the majority of arm poses, 9.9N would be insufficient to fully compensate for gravity. Any of these uncompensated gravitational forces, need to be supported by the arm muscles. In this study, we chose the fixed normal force because it is the easiest to implement practically and almost all SCI individuals have access to such simple gravity compensation devices. While more complex and better gravity compensation systems exist (e.g., active robotic arms), these may not necessarily be available for every SCI individual. By choosing a fixed normal force, we set a performance baseline: using more complex gravity compensation schemes should result in better stability and gravity performance. 4. Improve muscle models: the proposed feedforward-feedback time delay compensating controller calculates a set of appropriate joint torques and uses an inverse optimization scheme to distribute these torques over the 131

132 various muscle elements, thereby avoiding the muscle redundancy problem (the arm model has more muscles than degrees of freedom). This inverse optimization relies heavily on internal models of the muscle elements. Therefore, the overall stability and tracking performance of the controlled arm model depends on the quality of the muscle models and specifically how closely they resemble the true arm muscles. We modeled the muscle elements using a first order Hill type model [24]. Compared to the real world human arm muscles this model is by definition a simplification: higher order dynamics have been omitted. Also, some of the parameters of these muscle models are impractical, if not impossible, to measure (e.g., to date there is no practical method to measure maximum muscle forces and contraction velocities in vivo), which inevitably introduces a significant amount of muscle model uncertainty. Again, any mismatch between the modeled and true arm dynamics adds to the control burden of the feedback controller and means that stability and tracking performance will deteriorate compared to the case where the modeled and true muscles are identical. Switching to a more complex model based on the molecular cross-bridge dynamics of the model would improve the quality of the muscle model, but the associated numerical cost may be too high to allow for timely simulations or real-time FES applications. Another way to improve the muscle model would be to develop novel techniques for in vivo measurement of the muscle dynamics (e.g., excitations, activations, muscle lengths, contractile velocities). 132

133 Especially the muscle contractile velocities are, in our opinion, of significant importance. Since they cannot be measured in vivo, the controller had to assume static muscle models (zero contractile velocities) when distributing the control torques over the muscles. For slow movements, the true muscle contractile velocities were also expected to be low, so the mismatch between the slowly contracting true muscles and the static muscle models remained low. Such small muscle model mismatches likely were compensated by the feedback controller and therefore had little effect on the overall stability and performance of the system. However, faster movement intents would have required faster muscle contractions, effectively increasing the mismatch between the true and the modeled muscles. While our results show that the proposed controller achieved excellent stability and tracking accuracy for our chosen movement intents, performance is likely to decrease when attempting to track faster movement intents and stability cannot be guaranteed Alternative time delay compensation mechanism An alternative solution to the input time delay problem would be to remove its root cause: the time delay itself. Besides the muscle excitation-activation dynamics which we cannot control -- the main source of inputs time delays is the relatively low stimulation frequency typically used in upper extremity FES neuroprostheses: 12Hz, injecting close to 80ms between the controller prescribing a set of stimulations and the muscles effectively generating the 133

134 corresponding forces. The obvious solution would be to simply increase the stimulation frequency, thereby decreasing the time delay. Unfortunately, higher stimulation frequencies result in much high muscle fatigue rates [28]. Still, even these high fatigue rates may prove to be acceptable provided they are only applied for short durations: instead of choosing a fixed, constant high stimulation frequency, we may instead choose to keep the stimulation frequency at 12Hz during rest and only increase it during fast, short movements, after which we can decrease the frequency back to the original 12Hz. Similarly, we could limit the effects of the heightened fatigue rates by effective cycling of synergistic muscles, allowing each muscle some time to rest Functionally relevant performance requirements, or how good is good enough? As a final limitation we would like to address the question of How good is good enough? In our study, we quantified the tracking accuracy of the controller by calculating the root-mean square error (RMSE) between the intended and simulated joint angle trajectories. We then interpreted lower RMSE values as higher tracking accuracy and evaluated a controller that achieved a lower RMSE as better than one that achieved a higher RMSE value. While such a decision is often valid, it remains unclear exactly what a given RMSE value means on a functional level. It may be tempting to derive (necessary and/or sufficient) functional performance requirements like: to successfully execute a functional task X (e.g. drinking from a cup), the controller needs to achieve a tracking 134

135 RMSE of <Y (e.g., 5 ). We would argue that for drafting such performance requirements, RMSE may be misleading and an unreliable basis. For example, suppose a FES neuroprosthesis user attempts to drink from a glass. Suppose perfect tracking is maintained during the entire movement, with the exception of a rapid, short duration 180 forearm rotation. In this example, the RMSE would be low given that, for most of the time, the arm followed the intended trajectory perfectly. However, the task failed: the short 180 rotation caused the water to spill out of the cup. This thought experiment suggests that failure modes may form a better basis for performance requirements. That is, in what ways can a given functional task fail and how likely is each such failure mode likely to occur (failure rates). For the example of drinking from a cup, these failure modes include failing to grasp the cup, knocking over the cup, spilling water, dropping the cup. Deriving FES neuroprosthesis performance requirements in terms of these failure modes and failure rates still constitutes a challenging problem, for the following five reasons: (1) Each functional task has its own set of unique failure modes that need to be identified, (2) failure modes have different levels of severity. For example, failing to grasp the cup can generally be considered as less problematic than spilling its contents, (3) ranking the severity of these failure modes is inherently subjective, depending on the tolerance of each individual FES neuroprosthesis user, (4) acceptable failure rates vary depending on the functional task. For example, dropping one out of every hundred cups is less problematic than falling once for every hundred steps, (5) there is much variability in terms of the FES neuroprosthesis user, the functional task, and the 135

136 environment. Again for the example of drinking from a cup, different users will have different hand shapes and sizes, the cup may be full or only half-full, the cup may come in different shapes and sizes (e.g. wine glass). This variability means that we would need to derive performance requirements for each possible combination of operational parameters, resulting in a near-infinite set of possible scenarios. To aid controller and prosthesis design, it would be very helpful if the rehabilitation community drafted a consensus set of functional tasks including detailed specifications (e.g. cup size and shape, fill level) and corresponding failure modes and minimum acceptable failure rates. Such a set of tasks and performance requirements would in turn allow the rehabilitation community to compare controllers and (neuro)prosthesis designs proposed from different research groups, objectively and in terms of the more meaningful functional benefits. As an alternative approach, we could attempt to derive sufficient, but not necessary, performance requirements. Such requirements can be derived relatively easily by recording the kinematics of able-bodied subjects performing functional tasks. After performing the same functional task many times, we can draw manifolds that define a closed subspace such that all recorded joint angles remain within the boundaries of this subspace. Assuming the able-bodied subjects completed the task successfully each time, these bounds effectively make up sufficient performance requirements: the task is guaranteed to be successful as long as the joint angles remain within these boundaries. These 136

137 sufficient performance requirements are easier to construct than necessary performance requirements, however they lead to conservatism: even if the arm strays outsides of the subspace defined by the sufficient performance requirements, it is still possible that the user completes the task successfully. 5.3 Significance Using musculoskeletal arm models, we designed and evaluated the strengths and weaknesses of three nonlinear combined feedforward-feedback control strategies (i.e. PD, adaptive, and sliding-model control) for controlling arm movement generated by an FES neuroprosthesis aimed to restore arm function after high-level (C3-C4) spinal cord injury. Our results show that, in the absence of measurement noise and input time delays, all three controllers achieved excellent performance as indicated by very low root-mean-square errors between the intended and simulated joint angle trajectories. In our opinion, the marginal performance improvements achieved by adaptive and sliding-mode control compared to the PD controller are unlikely to do be functionally significant and therefore do not justify their higher numerical complexity and, in the case of sliding-mode control, higher control effort (here: muscle activations). For the foreseeable future, we recommend the combined feedforward-feedback PD controller for use in upper extremity FES neuroprostheses, at least until performance requirements become clearer and/or more demanding. 137

138 This study provides new insights into the respective strengths and weaknesses of feedforward and feedback control, and specifically identified measurement noise and input time delays as the two factors with the greatest negative impact on the performance and stability of the FESenabled arm. Introducing either measurement noise or input time delays quickly resulted in instability of the arm model for all three candidate controllers (PD, adaptive, and sliding-mode). Of these two factors, we can influence measurement noise in a meaningful way by implanting more sensors and/or designing more accurate sensors and better noise filters, we could improve the signal-to-noise ratio of our kinematic measurements and therefore decrease the negative impact of measurement noise. Unfortunately, input time delays are inherent to upper extremity FES neuroprostheses and we cannot meaningfully change the sources of these delays: the muscle excitation-activation dynamics (10-30ms delay) are fixed and the relatively low 12Hz electrical stimulation frequency (80ms delay) used in upper extremity FES cannot be raised high enough without increasing the fatigue rate of the muscles to unacceptable levels. Input time delays have severe negative consequences for the stability and performance of upper extremity FES neuroprostheses and their inevitability means that, when designing new controllers and control strategies, they cannot be ignored. Our results show that studies on controller design that fail to take measurement noise and input time delays into effect overestimate the benefits of 138

139 high-gain feedback control. Feedback control has many important advantages including compensating tracking errors and improving robustness against many perturbations such as muscle fatigue and model uncertainty. However, feedback control also has its weaknesses, namely it is sensitive to measurement noise and input time delays. It would be tempting to simply exclude these two factors and choose high-gain (aggressive) feedback control, which would result in excellent tracking performance and high robustness against perturbations. However, in doing so, we would reach incorrect conclusions, exaggerating the benefits of feedback control. Simply adding feedback control will not automatically solve all control problems; good controller design requires a careful balance of both feedforward and feedback control where the feedback gains are set as high as the measurement noise and input time delays will allow. We designed and evaluated a nonlinear feedforward feedback time delay compensating controller for a 2DOF musculoskeletal arm model (arm movement constrained to the horizontal plane) and for a 5DOF arm model (more DOFs and included gravity). Our results show that the controllers achieved stable, robust tracking performance in the presence of a wide range of perturbations. The controller for the 2DOF arm model (which omitted gravity effects) especially showed excellent performance over a very wide range of operating conditions. While the primary intended application for these controllers was upper extremity FES, the time delay compensating controller should prove useful 139

140 in other applications where time delays may be an issue, such as stroke rehabilitation and prostheses that replace missing or lost limbs. 5.4 Speculation Gain-scheduling: replicating able-bodied human arm movement control Our long term goal is to restore arm function in individuals who have suffered a high-level spinal cord injury. In that regard, we need to ask ourselves the following questions: What functionalities is the able-bodied human arm capable of? How do able-bodied individuals control arm movement? Can we artificially replicate these able-bodied control strategies such that an upper extremity FES neuroprosthesis would enable a paralyzed arm to perform the full set of able-bodied arm functionalities? To control their arm, able-bodied individuals have both feedforward and feedback mechanisms mechanism at their disposal. The brain serves as a feedforward controller, formulating a movement intent and a corresponding initial motor plan. The spinal cord serves as a feedback controller, adjusting the muscle activations based on proprioceptive information (i.e. position, force) provided by the muscle spindles and Golgi tendon organs. Proprioceptive information from the muscles needs to travel along the peripheral nerves to the spinal cord and back to the muscles, a process that introduces time delays. These delays do not become problematic until we attempt to execute very fast, ballistic movements. 140

141 For ballistic movements, proprioception may not be fast enough to keep up with the movement and may in fact deteriorate performance. It has been suggested that able-bodied individuals may modulate the contribution of their reflexes, effectively shifting the control burden towards primarily feedforward control for fast, ballistic movements (e.g. throwing a baseball) or towards feedback control for slow movements that require high precision (e.g. threading a needle) [32]. To date, the controllers proposed for upper extremity FES, including the controller proposed in this study, have assumed fixed feedback gains, implying a fixed division of the control burden between the feedforward and feedback controllers. This implies that the combined feedforward-feedback controller with fixed feedback gains will achieve good performance for a part of the spectrum of movements, but will likely provide poor, if not unstable, performance for the rest of the spectrum: if we pick low feedback gains (primarily feedforward control), we could generate fast, ballistic movements, at the expense of accuracy and robustness against perturbations. Vice versa, if we pick high feedback gains, we may get high accuracy and high robustness against perturbations, but only for slow intended movements. A better approach, inspired by able-bodied movement control, would be to allow the feedback gains to vary. A simple inverse relationship between the velocity of the intended movement and the feedback gains could suffice. For slow movements, we can pick high feedback gains and achieve the corresponding high levels of accuracy, whereas for ballistic movements, we can pick slow feedback gains and sacrifice some accuracy to gain increased speed. Figure 1 shows a block diagram of how such an approach 141

142 also known as gain-scheduling may be implemented for the case of a combined feedforward-feedback controller. Figure 1. Gain scheduling based on the intended arm behavior. A combined feedforward-feedback controller, where the gains of the feedback controller are set by a gain scheduler. Here, the gain scheduler chooses high feedback gains when the velocity of the intended arm trajectory remains low (i.e. strong feedback action to allow for high tracking accuracy and robustness against perturbations), and chooses low feedback gains when the velocity of the intended arm trajectory is high (i.e. rely more on feedforward control to allow for higher movement velocity at the cost of accuracy). Figure V-1. Block diagram of gain-scheduling controller Divide-and-conquer: a new control paradigm for upper extremity FES We can extend this line of thinking further. Similar to modulating the feedback gains (i.e. gain-scheduling), we can switch between entirely different control strategies (e.g. PD, adaptive, sliding-mode control; controller-scheduling) depending on the available information concerning the required functional task and the state of the workspace (i.e., a layered control architecture; see also Figure 2). After decades of research, the field of control theory has developed 142

143 many different control strategies, each with its own strengths and weaknesses. So far the paradigm has been to design a single controller capable of controlling a particular system for all its expected operating conditions and perturbations. Instead, we suggest a divide-and-conquer approach: first designate a finite number of discrete operating regions for the system, then design a controller (that is in some sense optimal) for each of these operating regions and specify rules to select the appropriate feedback control strategy depending on the current operating region of the system. We illustrate this idea using the example of controlling arm movement for the task of picking up an object of unknown mass (e.g. milk carton). Initially, we may rely on adaptive control to learn the dynamics of the arm as it moves around in free-space (improving the feedforward controller by learning a better internal model of the arm dynamics). After this initial learning phase, we may switch to simple PD control and start to reach for the milk carton. As we make contact with the milk carton, our primary concern would be to deduce its weight as this information would have affect our performance (and thus our motor planning) the most. Suppose we have three options going forward: maintain PD control, switch to adaptive control, or switch to sliding-mode control. If we decide to continue with PD control, we will need to make a guess about the weight of the milk carton (e.g. 0kg, 1kg). If by chance we happen to guess the weight of the milk carton guess correctly (or in case we know the exact weight a priori), then our performance should remain optimal. However, there is no guarantee our guess will be correct and, as discussed earlier, any mismatch will lead to deteriorated performance, if not instability. If we 143

144 instead switch to adaptive control, we can rely on this adaptation mechanism to autonomously learn the correct weight of the milk carton, after which the performance should again be optimal. This seems to be a preferable strategy compared to using only PD control. However, adaptive control has an important weakness: it is sensitive to measurement noise. Adaptive control cannot differentiate whether a given error was caused by measurement noise or a genuine discrepancy in the internal arm model. This means that the adaptive controller may adjust the internal model unnecessarily and/or incorrectly. If we expect a significant amount of measurement noise, we may instead choose to rely on sliding mode control. In our example, we would be required to provide a lower and upper bound of the weight of the milk carton, and the sliding mode controller then picks feedback gains such that arm remains stable as long as this weight remains within the provided bounds. Finally, after releasing the milk carton, we again know the exact weight of our arm (i.e. un-weighted) and we can switch back to our original PD control with feedback gains optimized for the unweighted arm. To recap: we could use PD as our default feedback controller and if we need to resolve model uncertainty (e.g. picking up an object of unknown weight), switch to either adaptive control or sliding mode control depending on how certain we are of our initial guess and how much measurement noise we expect. Note that this is only one of many different possibilities. 144

145 Figure 2. Layered control architecture block diagram. A combined feedforward-feedback controller (shown in blue), with two adaptation mechanisms: shown in green, the adaptation of the internal arm model used by the feedforward controller, and shown in red, adaptation of the feedback controller weights (WW jj ) and their corresponding gains (KK jj ) for j=1,,m, depending on the current operating region of the arm (OO ii ). The Artificial Intelligence block gathers information from the environment (e.g., presence of perturbations, obstacles), the intended arm behavior (e.g., trajectory, stiffness) and current arm behavior, to determine which region the arm is operating in and what the corresponding feedback controller weights and gains should be. Figure V-2. Layered structure control block diagram. The key to the divide-and-conquer control paradigm is to subdivide the complete set of possible operating states and conditions into smaller operating 145

146 regions, designing a specific feedback controller that is optimal for each of these smaller operating regions, and then switching to the appropriate feedback controller (i.e. set of weights of the linear weighted sum of all feedback controllers) depending on which region the arm currently operates. Implementing this divide-and-conquer control strategy raises a number of challenging questions. How many different distinct operating regions do we need? How does the master controller recognize each operating region and therefore when it needs to switch to a different feedback controller? When switching to a new control strategy, how does the master controller know which specific feedback control strategy it should switch to? To facilitate this discussion we start by defining the entire operating space as the aggregate of all possible states of the arm (e.g. joint angles) and its environment (e.g. presence of measurement noise, weight of manipulated objects), and we denote this operating space by the symbol Ω. Then we can define different operating regions as subspaces of the entire operating space Ω. Here, we denote the i th operating regions by the symbol OO ii (e.g., moving in freespace in the absence of external perturbations can be considered as an operating region, forming a subspace of the entire operating space Ω). We subdivide the operating space Ω into n distinct operating regions OO ii (for i=1,,n), such that the union of these operating regions covers Ω: Ω = OO 1 OO 2.. OO nn (1) 146

147 To allow the us to easily switch between different feedback control strategy, let us define the feedback control law (denoted by CC) as a weighted linear sum of a large number mm of different feedback control laws (e.g., PD, PID, adaptive, sliding-mode, H 2, H ), each denoted by CC jj (for j=1,,m): CC = WW 1 CC 1 + WW 2 CC WW mm CC mm (2) By choosing different values for the weights, we can easily select a single feedback control strategy CC ii or even some combination of multiple strategies. The behavior of each controller depends on its feedback gains KK. We can therefore write (2) as: CC = WW 1 CC 1 (KK 1 ) + WW 2 CC 2 (KK 2 ) + + WW mm CC mm (KK mm ) Note that our choice of the controller weights depends on the current operating region of the arm. Therefore our goal is to choose the n-dimensional vector WW(OO ii ) and the feedback gains of the corresponding controllers KK jj (OO ii ) (for j=1,,m), for each operating region OO ii (for i=1,,m). The first, and simplest, approach would be to let the system and controller designer define a fixed set of operating regions OO ii a priori and prescribe the appropriate feedback controller for each region (WW(OO ii ) and KK jj (OO ii )). In the example of picking up a milk carton of unknown weight, we may specify two distinct operating regions: free-space (the un-weighted arm reaching out towards the milk carton) and the weighted arm case, when the milk carton is held in the hand. We would need additional sensors to enable the FES neuroprosthesis to differentiate between these two operating regions. A possible solution could 147

148 involve pressure sensors on the finger tips that inform the neuroprosthesis when the hand makes contact with an object. As for which specific feedback controller is most appropriate for each operating region, that too can be prescribed a priori by the designer (e.g. PD and sliding mode control for the un-weighted and weighted arm respectively). This solution requires the least amount of intelligence on the part of the controller. The implementation is simple, but the designer will need to specify everything in advance. This means that the designer would need to break down every single task into operating regions and train optimal controllers for each region. This could prove to be a challenging task considering the large amount of functional tasks and movements that can be performed by the human arm. An alternative solution would be for the designer to only specify the different operating regions (i.e., when the master controller might benefit from switching to a different feedback control strategy or gain settings) and the preferred feedback control strategy WW(OO ii ) for each operating region OO ii, and then letting the controller itself determine which gain settings KK jj (OO ii ) would be most appropriate for each operating region. This could be accomplished using an optimization approach: the controller would determine the feedback gains that minimize some cost function JJ as specified by the designer (e.g. related to tracking error, control effort). We can write this optimization problem as follows: 148

149 Given the different operating region OO ii (i = 1,,n) and their corresponding preferred feedback control strategies WW jj OO jj (j=1,,m): min JJ (pppppppppppppppppppppp(cc), eeeeeeeeeeee(cc), ) KK jj (OO ii ) Where CC = WW 1 CC 1 (KK 1 ) + WW 2 CC 2 (KK 2 ) + + WW mm CC mm (KK mm ) If successful, this approach could be expanded to let the controller also determine which feedback control strategy WW(OO ii ) to use in each operating region. Specifically, the optimization would simultaneously optimize the control strategy (i.e. the weights of the weighted sum of feedback control strategies) and the gains settings of the different controllers, effectively working on two different levels. This can be written mathematically as: Given the different operating region OO ii (i = 1,,n): min JJ (pppppppppppppppppppppp(cc), eeeeeeeeeeee(cc), ) WW(OO ii ), KK jj (OO ii ) Where CC = WW 1 CC 1 (KK 1 ) + WW 2 CC 2 (KK 2 ) + + WW mm CC mm (KK mm ) However, simultaneously optimizing the feedback control strategy weights and feedback gains may result in convergence problems due the possible large number of parameters that need to be optimized, depending on how many different feedback control strategies we choose to include in the weighted sum of 149

150 controllers and the total sum of their respective feedback gains. The optimization is likely to get stuck in local minima. To increase our confidence that the controller will in fact find the best selection of feedback controllers and their respective optimal weights, we could use more advanced optimization algorithms, such as simulated annealing or genetic algorithms. Such optimization algorithms increase the probability of finding the global minimum, but typically need more time to converge. We could perform these optimizations for a limited number of representative tasks and then train an ANN to map the defining characteristics of the different operating regions (e.g., measurement noise, estimated model uncertainty, finger pressure sensor input) and the optimal feedback controller corresponding to each operating region. Such an ANN may learn simplified rules-of-thumb that could be applied when considering new tasks and would give us a closer initial guess of the optimal control strategy and gains, without having to perform the full optimization. So far we have assumed that the designer prescribes for the master controller which operating regions may be of interest (i.e., may warrant a feedback control strategy switch). Instead, the master controller could grow the number of operating regions organically. Suppose we initially have a single operating region, with good performance during the first half of the task and poor performance during the second half. Suppose we, as designers, specify a performance threshold JJ mmmmmm and instruct the controller to split off operating regions whenever the optimal performance for a particular operating region drops 150

151 below this threshold ( min JJ (OO ii) > JJ mmmmmm ). In our example, the controller WW(OO ii ), KK jj (OO ii ) would designate the second half of the task as a separate operating region and would then determine whether switching to a different feedback control strategy and/or gain setting this performance could be improved, by optimizing the feedback controller weights and gains for this new operating region. A clustering algorithm may prove useful in determining specifically which features differentiate the new operating regions from the original set (e.g., manipulated objects heavier than 1kg, the presence of measurement noise with amplitude > 5º). The entire process can be summarized as follows: 151

152 1. Initialization: OO 1 = Ω 2. Optimize feedback controller weights and gains for the single operating region (= entire operating space) min JJ (pppppppppppppppppppppp(cc), eeeeeeeeeeee(cc), ) WW(OO ii ), KK jj (OO ii ) Where CC = WW 1 CC 1 (KK 1 ) + WW 2 CC 2 (KK 2 ) + + WW mm CC mm (KK mm ) 3. Set JJ mmmmmm * 4. If the optimized cost from step 2 exceeds JJ mmmmmm, add a new operating regionoo nn+1 and find the feature sets that distinguish each operating region (using heuristics or a clustering algorithm) 5. Repeat the optimization to find feedback controller weights and gains for each operating region in the new set: min JJ (pppppppppppppppppppppp(cc), eeeeeeeeeeee(cc), ) WW(OO ii ), KK jj (OO ii ) Where CC = WW 1 CC 1 (KK 1 ) + WW 2 CC 2 (KK 2 ) + + WW mm CC mm (KK mm ) 6. If desired, incrementally decrease JJ mmmmmm and repeat step 5. * * Caution: Choosing low values for JJJJJJJJ may lead to very large sets of operating regions and therefore numerically expensive optimizations Interacting with the environment: simultaneous position and force control So far we have limited our discussion to controlling the arm and hand position, while neglecting another very important control variable: force. Most 152

153 functional tasks require the arm and hand to interact with the various objects and the environment in general. This interaction takes the form of trading forces the hand and arm exert forces on objects and, vice versa, the environment exerts forces on the hand. In extreme cases where the environment has zero compliance (e.g. pushing against a brick wall), the movement of the arm and hand become constrained, further complicating the control problem. In the case of nonzero compliance environments, we could control arm and hand position and force simultaneously. However, to do so effectively is a very difficult control problem. Simultaneous position and force control has been achieved successfully for industrial robot arms, but only by carefully prescribing in detail the compliance of the robot arm and its environment. This approach works for industrial settings where we can design the robot arm ourselves (and thus prescribe it some known, desired compliance) and the environment is predictable. However, the human arm needs to perform a much wider variety of functional tasks in a much wider range of environments with unpredictable compliance properties. Consider, for example, manipulating a tomato and an egg. Both have different weights and compliances, and we can observe that able-bodied humans use different strategies to interact with these objects. We hypothesize that humans have an internal database of common objects that contains information relevant to position/force control such as typical weight, compliance and surface smoothness. Able-bodied humans then use this information to a priori adjust their motor planning with regards to the type of grip, hand opening and grip force (e.g., gripping a tomato with high enough force to 153

154 prevent it from slipping, but not too much force so that the tomato would get crushed). If we had a similar database, we could replicate such a mechanism artificially for use in an upper extremity neuroprosthesis. When a certain task requires the user to interact with a particular object, we would only need to look up the inertial and compliance properties of that object and adjust the internal model used by the feedforward controller accordingly. Constructing such a database poses a significant challenge, however. While we could measure the inertial and compliance properties of a limited set of common objects, it is impossible to catalogue the properties of every object an FES user could conceivably interact with. Able-bodied individuals build up their internal world model based on a lifetime s worth of experiences and interactions. An alternative solution may be to pre-program only a small set of common objects (e.g., egg, tomato, soda can) and use an adaptation scheme to learn the properties of new objects not yet catalogued in the internal database. To speed up this process, new objects that have been catalogued this way could be uploaded to a single database shared by all upper extremity FES neuroprosthesis users. This way each FES user can benefit from the experiences of all the other users, which would significantly speed up the learning process. Or estimates pertaining to some object, provided by multiple users, could be averaged to give more accurate information (i.e., mean and standard deviation). The ultimate goal of this effort is to improve the accuracy of our internal world model with respect to the real human arm, the task at hand, and the user s environment. The quality of this internal world model is of vital importance to the feedforward controller. Anything 154

155 that brings the feedforward controller s internal arm and environmental model closer to the true arm and environment will result in a more appropriate motor plan and a smaller mismatch, which in turn requires less effort on the part of the feedback controller to compensate position and force deviations caused by this model mismatch. 155

156 VI. Acknowledgements The project described was supported by Contract Number N01-NS from the National Institute of Neurological Disorders and Stroke. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Neurological Disorders and Stroke or the National Institutes of Health. 156

157 VII. Appendix Appendix A: Derivation of the equations of motion of a two degree of freedom (2DOF) planar arm. This section provides a detailed mathematical derivation of equations of motion for the two-link arm model as shown schematically in Figure II-2A of the paper and repeated here for convenience: Figure VII-1. Skeletal system of the two degrees of freedom, planar arm model. The variables of the arm model are: q 1 (t) q 2 (t) Shoulder joint angle [rad] Elbow joint angle [rad] The parameters of the model are: m 1 m 2 Upper arm mass [kg] Forearm mass [kg] 157

158 m h Hand mass (including any objects it may be carrying) [kg] I 1 Inertia of the upper arm segment [kg m 2 ] I 2 Inertia of the forearm segment [kg m 2 ] L 1 L 2 L c1 Length of the upper arm segment [m] Length of the forearm segment [m] Distance from the shoulder joint axis to the upper arm s center of gravity [m] L c2 Distance from the elbow joint axis to the forearm s center of gravity [m] Note that the moment of inertia is not an independent variable and can be calculated as: I i = 1 m 3 il 2 ci for i = 1, 2 Also note that since the hand (and any object it carries) is modeled as a point mass, it has no length, center of gravity or moment of inertia. Now, let the Lagrangian L be defined as: L = T U where TT denotes the system s total kinetic energy and UU denotes the total potential energy. Then the equations of motion can be calculated as a set of ordinary differential equations from: ττ ii = dd δδδδ + δδδδ dddd δδqq ii δδqq ii ffffff ii = 1,2 158

159 where ττ ii is the joint torque about the i th joint (ii = 1for the shoulder and ii = 2 for the elbow joint). Assuming that the arm is supported against gravity by a Mobile Arm Support (MAS) or a Haptic Master and that the arm is constricted to move in the horizontal plane only, the total potential energy in the system becomes UU = 0. The kinetic energy is made up of two contributions: energy due to translational velocity and energy due to rotational velocity. TT = 1 2 mmvv IIωω2 where mm vv II ωω Mass [kg] Velocity of the center of gravity Moment of inertia Rotational velocity The total kinetic energy is made up of the kinetic energy of both links and the end-point load: Link 1: TT 1 = 1 mm 2 1LL 2 cc1 qq II 2 1qq 1 2 TT = TT 1 + TT 2 + TT h Link 2: TT 2 = 1 2 mm 2cc 2 TT cc II 2(qq qq 2 2 ) End-point load: TT h = 1 2 mm hcc ḣ TT cc ḣ + 0 (point mass, has no rotational energy) 159

160 Here the vector cc 2 denotes the position of the center of gravity of the second link: cc 2 = LL 1cccccc(qq 1 ) + LL cc2 cccccc(qq 1 + qq 2 ) LL 1 ssssss(qq 1 ) + LL cc2 ssssss(qq 1 + qq 2 ) Differentiating this vector with respect to time yields: cc 2 = LL 1ssssss(qq 1 )qq 1 LL cc2 ssssss(qq 1 + qq 2 )(qq 1 + qq 2) LL 1 cccccc(qq 1 )qq 1 + LL cc2 cccccc(qq 1 + qq 2 )(qq 1 + qq 2) The product cc 2TT cc 2 can now be calculated as: cc 2 TT cc 2 LL 1 2 ssssss 2 (qq 1 )qq 1 2 = + 2LL 1 LL cc2 ssssss(qq 1 )ssssss(qq 1 + qq 2 )qq 1(qq 1 + qq 2) +LL 2 cc2 ssssss 2 (qq 1 + qq 2 )(qq 1 + qq 2) 2 +LL 1 2 cccccc 2 (qq 1 )qq LL 1 LL cc2 cccccc(qq 1 )cccccc(qq 1 + qq 2 )qq 1(qq 1 + qq 2) +LL 2 cc2 cccccc 2 (qq 1 + qq 2 )(qq 1 + qq 2) 2 = LL 1 2 qq LL 1 LL cc2 cccccc(qq 1 + qq 2 qq 1 )qq 1(qq 1 + qq 2) +LL 2 cc2 (qq 1 + qq 2) 2 = LL 1 2 qq LL 1 LL cc2 cccccc(qq 2 )qq 1(qq 1 + qq 2) + LL 2 cc2 (qq 1 + qq 2) 2 Similarly, the vector cc h denotes the position of the hand and is defined as: cc h = LL 1cccccc(qq 1 ) + LL 2 cccccc(qq 1 + qq 2 ) LL 1 ssssss(qq 1 ) + LL 2 ssssss(qq 1 + qq 2 ) Which, following similar derivations, leads to: 160

161 cc ḣ TT cc ḣ = LL 1 2 qq LL 1 LL 2 cccccc(qq 2 )qq 1(qq 1 + qq 2) + LL 2 2 (qq 1 + qq 2) 2 Next, we calculate the partial derivatives δδδδ δδδδ δδqq 1 δδtt 1 δδqq 1 δδtt 2 δδqq 1 = δδtt 1 + δδtt 2 + δδtt ll δδqq 1 δδqq 1 δδqq 1 = mm 1 LL 2 cc1 qq 1 + II 1 qq 1 δδqq 1 for the first DOF. = mm 2 LL 2 1 qq 1 + mm 2 LL 1 LL cc2 cccccc(qq 2 )(2qq 1 + qq 2) + mm 2 LL 2 cc2 (qq 1 + qq 2) + II 2 (qq 1 + qq 2) δδtt h = mm h LL 2 1 qq 1 + mm h LL 1 LL 2 cccccc(qq 2 )(2qq 1 + qq 2) + mm h LL 2 2 (qq 1 + qq 2) δδqq 1 Next, this expression needs to be differentiated with respect to time: dd δδδδ = mm dddd 1 LL 2 cc1 qq 1 + II 1 qq 1 δδqq 1 +mm 2 LL 1 LL cc2 cccccc(qq 2 )(2qq 1 + qq 2) + mm 2 LL 2 cc2 (qq 1 + qq 2) +mm 2 LL 1 2 qq 1 mm 2 LL 1 LL cc2 ssssss(qq 2 )(2qq 1qq 2 + qq 2 2 ) +II 2 (qq 1 + qq 2) + mm h LL 1 2 qq 1 mm h LL 1 LL 2 ssssss(qq 2 )(2qq 1qq 2 + qq 2 2 ) Finally, calculate δδδδ δδqq 1 (note that LL depends only on qq 2 ) +mm h LL 1 LL 2 cccccc(qq 2 )(2qq 1 + qq 2) + mm h LL 2 2 (qq 1 + qq 2) δδδδ δδqq 1 = 0 Putting all terms together gives, after some regrouping, the equation of motion for the first link / first DOF: 161

162 ττ 1 = mm 1 LL 2 cc1 + II 1 + mm 2 LL LL 2 cc2 + 2LL 1 LL cc2 cccccc(qq 2 ) + II 2 +mm h LL LL LL 1 LL 2 cccccc(qq 2 ) + II 2 qq 1 + mm 2 LL 2 cc2 + LL 1 LL cc2 cccccc(qq 2 ) + II 2 + mm h LL LL 1 LL 2 cccccc(qq 2 ) qq 2 mm 2 LL 1 LL cc2 ssssss(qq 2 ) + mm h LL 1 LL 2 ssssss(qq 2 ) (2qq 1qq 2 + qq 2 2 ) Repeating this process for the second link / second DOF: δδδδ δδqq 2 δδtt 1 = 0 δδqq 2 δδtt 2 δδqq 2 = δδtt 1 + δδtt 2 + δδtt ll δδqq 2 δδqq 2 δδqq 2 = mm 2 LL 1 LL cc2 cccccc(qq 2 )qq 1 + mm 2 LL 2 cc2 (qq 1 + qq 2) + II 2 (qq 1 + qq 2) δδtt h = mm h LL 1 LL 2 cccccc(qq 2 )qq 1 + mm h LL 2 2 (qq 1 + qq 2) δδqq 2 Again, differentiate with respect to time: dd δδδδ = mm dddd 2 LL 1 LL cc2 ssssss(qq 2 )qq 1qq 2 δδqq 2 +mm 2 LL 1 LL cc2 cccccc(qq 2 )qq 1 +mm 2 LL 2 cc2 (qq 1 + qq 2) +II 2 (qq 1 + qq 2) mm h LL 1 LL 2 ssssss(qq 2 )qq 1qq 2 +mm h LL 1 LL 2 cccccc(qq 2 )qq 1 +mm h LL 2 2 (qq 1 + qq 2) 162

163 Finally, δδδδ δδqq 2 : δδδδ δδqq 2 = mm 2 LL 1 LL cc2 ssssss(qq 2 )qq 1(qq 1 + qq 2) mm h LL 1 LL 2 ssssss(qq 2 )qq 1(qq 1 + qq 2) Combining and regrouping all terms yields the following equation of motion for the second link / second DOF: ττ 2 = mm 2 LL 2 cc2 + 2LL 1 LL cc2 cccccc(qq 2 ) + II 2 + mm h LL LL 1 LL 2 cccccc(qq 2 ) qq 1 +[mm 2 LL 2 cc2 + mm h LL 2 2 ]qq 2 +(mm 2 LL 1 LL cc2 + II 2 + mm h LL 1 LL 2 )ssssss(qq 2 )qq 2 2 To simplify these expressions, we can group the constants together as follows: aa 1 = mm 1 LL 2 cc1 + II 1 + mm 2 (LL LL 2 cc2 ) + mm h (LL LL 2 2 ) + II 2 aa 2 = 2(mm 2 LL 1 LL cc2 + mm h LL 1 LL 2 ) aa 3 = mm 2 LL 2 2 cc2 + II 2 + mm h LL 2 Then the equations of motion can be written as: ττ 1 = [aa 1 + aa 2 cccccc(qq 2 )]qq 1 + aa 3 + aa 2 2 cccccc(qq 2) qq 2 aa 2 2 ssssss(qq 2)(2qq 1qq 2 + qq 2 2 ) ττ 2 = aa 3 + aa 2 2 cccccc(qq 2) qq 1 + [aa 3 ]qq 2 + aa 2 2 ssssss(qq 2)qq

164 These equations can then be written in the more commonly used Euler-Lagrange form: Where MM(qq)qq + CC(qq, qq )qq = ττ qq = qq 1 qq 2 qq = qq 1 qq 2 ττ = ττ 1 ττ 2 aa 1 + aa 2 cccccc(qq 2 ) aa 3 + aa 2 MM(qq) = 2 cccccc(qq 2) aa 3 + aa 2 2 cccccc(qq 2) aa 3 aa 2 CC(qq, qq ) = 2 ssssss(qq 2)qq 2 aa 2 2 ssssss(qq 2)(qq 1 + qq 2) aa 2 2 ssssss(qq 2)qq 1 0 Note that multiple choices are possible for the structure of CC(qq, qq ). The structure chosen above ensures that NN MM (qq) 2CC(qq, qq ) is a skew-symmetric matrix. A quick calculation verifies that: NN MM (qq) 2CC(qq, qq ) aa 2 ssssss(qq 2 )qq 2 aa 2 = 2 ssssss(qq 2)qq 2 aa 2 2 ssssss(qq 2)qq 2 0 aa 2ssssss(qq 2 )qq 2 aa 2 ssssss(qq 2 )(qq 1 + qq 2) aa 2 ssssss(qq 2 )qq = 2 ssssss(qq 2)(2qq 1 + qq 2) aa 2 2 ssssss(qq 2)(2qq 1 + qq 2) aa 2

165 NN is indeed a skew-symmetric matrix, since it satisfies the property NN TT = NN. Skew-symmetric matrices have the following useful property: For any xx R 2, it holds that xx TT NNNN =

166 Appendix B: Nominal PD-control To derive the error dynamics, a new set of variables needs to be defined: qq = qq dd qq qq = qq dd qq qq rr = qq dd + ΛΛqq qq rr = qq dd + ΛΛqq σσ = qq rr qq = qq dd + ΛΛqq qq = qq + ΛΛqq Where qq dd qq dd ΛΛ Vector of desired joint angles Vector of desired joint angular velocities A symmetric positive definite, usually diagonal, square matrix Let the control law be defined as: ττ = MM(qq)qq rr + CC(qq, qq )qq rr + KKKK Which can also be written as: ττ = YY(qq, qq, qq rr, qq rr)θθ + KKKK Where YY(qq, qq, qq rr, qq rr) qq rr1 = cccccc(qq 2 ) qq rr1 + qq rr2 2 ssssss(qq qq rr1 2) qq qq rr2 qq qq 2 0 cccccc(qq 2 ) qq rr1 2 + ssssss(qq qq rr1 2)qq 1 2 Also, note that the term KKKK, represents a basic PD-controller: KKKK = KK qq + ΛΛqq = KKqq + KKΛΛqq KK DD qq + KK PP qq qq rr2 qq rr2 2 qq rr1 + qq rr2 166

167 Substituting this control law in the equations of motion yields: MM(qq)qq + CC(qq, qq )qq = ττ = MM(qq)qq rr + CC(qq, qq )qq rr + KKKK MM(qq)(qq rr qq ) + CC(qq, qq )(qq rr qq ) + KKKK = 0 MM(qq)σσ + CC(qq, qq )σσ + KKKK = 0 Lyapunov s direct method can be invoked to prove the global stability of the origin of the controlled system. This method is based on Lyapunov s second theorem which states that: IF there exists a scalar function VV(xx), xx being the state of the system, for which holds that: VV(0) = 0 VV(xx) > 0, for all xx UU \{0}, with UU being a neighborhood region around the origin xx = 0 VV (xx) 0, for all xx THEN the origin of the system is globally, asymptotically stable, meaning that for any initial xx, the state will follow a trajectory that eventually converges onto the origin of the system. Using this method, the difficulty is often to find a suitable Lyapunov function candidate. If a particular candidate fails to prove stability, it cannot be concluded that the system is unstable. To prove that the control law proposed earlier does stabilize the system, consider the following Lyapunov function candidate: 167

168 VV(σσ, qq ) = 1 2 σσtt MM(qq)σσ + qq TT ΛΛΛΛqq The first stability condition is readily verified: VV(0) = 0 As for the second condition: since all terms are quadratic, and since both MM(qq) and ΛΛΛΛ are positive definite matrices we can conclude that VV(σσ, qq ) > 0 for all (σσ, qq ) (0,0). To test the third condition, the Lyapunov function candidate is differentiated with respect to time: VV (σσ, qq ) = σσ TT MM(qq)σσ σσtt MM (qq)σσ + 2qq TT ΛΛΛΛqq From MM(qq)σσ + CC(qq, qq )σσ + KKKK = 0, it follows that MM(qq)σσ = CC(qq, qq )σσ KKKK, which can be substituted into VV (σσ, qq ): VV (σσ, qq ) = σσ TT CC(qq, qq )σσ σσ TT KKKK σσtt MM (qq)σσ + 2qq TT ΛΛΛΛqq Next, the skew-symmetric property is invoked. Note that: σσ TT CC(qq, qq )σσ σσtt MM (qq)σσ = 1 2 σσtt MM (qq) 2σσ TT CC(qq, qq ) σσ Since MM (qq) 2σσ TT CC(qq, qq ), is a skew-symmetric matrix, it holds that 1 2 σσtt MM (qq) 2σσ TT CC(qq, qq ) σσ = 0, for all possible values of σσ. This reduces the expression for VV (σσ, qq ) to: VV (σσ, qq ) = σσ TT KKKK + 2qq TT ΛΛΛΛqq Earlier σσ was defined as σσ = qq + ΛΛqq. Substituting this in the expression for VV (σσ, qq ) yields: VV (σσ, qq ) = qq + ΛΛqq TT KK qq + ΛΛqq + 2qq TT ΛΛΛΛqq 168

169 = qq TT KKqq qq TT ΛΛ TT KKΛΛqq qq TT KKKKqq qq TT ΛΛ TT KKqq + 2qq TT ΛΛΛΛqq Since VV(σσ, qq ) is a scalar function, qq TT KKKKqq = qq TT ΛΛ TT KKqq and thus qq TT KKKKqq qq TT ΛΛ TT KKqq + 2qq TT ΛΛΛΛqq = 0, resulting in: VV (σσ, qq ) = qq TT KKqq qq TT ΛΛ TT KKΛΛqq Since all terms are quadratic and since both KK and ΛΛ TT KKΛΛ are positive definite matrices, it can be concluded that VV (σσ, qq ) 0 for all (σσ, qq ). Since all three conditions are met, it holds that the control law (ττ = MM(qq)qq rr + CC(qq, qq )qq rr + KKKK) globally, asymptotically stabilizes the origin of the closed-loop system (MM(qq)σσ + CC(qq, qq )σσ + KKKK = 0). This origin is located at σσ = 0, σσ = 0, which from the definitions of σσ and σσ implies that qq = qq dd qq = 0, qq = qq dd qq = 0 and qq = qq dd qq = 0, meaning that under this control law the actual joint angles, angular velocities and accelerations will asymptotically converge to their desired values. The block diagram of the feedback control system is shown in the Figure below 169

170 170

171 Appendix C: Sliding Mode Control In the nominal PD-control case detailed in Appendix B, it was assumed that the system is perfectly known. Unfortunately, this is never the case. Instead of the true matrices MM(qq) and CC(qq, qq ), only their respective estimates MM (qq) and CC (qq, qq ) are available, with the respective differences defined as follows: MM (qq) = MM (qq) MM(qq) CC (qq, qq ) = CC (qq, qq ) CC(qq, qq ) For the control law this means that, since the true MM(qq) and CC(qq, qq ) cannot be known exactly, their estimated values need to be used instead: ττ = MM (qq)qq rr + CC (qq, qq )qq rr + KKKK Substituting this control into the system yields: MM(qq)qq + CC(qq, qq )qq = ττ = MM (qq)qq rr + CC (qq, qq )qq rr + KKKK MM(qq)qq + CC(qq, qq )qq = MM (qq)qq rr + MM(qq)qq rr + CC (qq, qq )qq rr + CC(qq, qq )qq rr + KKKK MM (qq)qq rr CC (qq, qq )qq rr = MM(qq)(qq rr qq ) + CC(qq, qq )(qq rr qq ) + KKKK MM (qq)qq rr CC (qq, qq )qq rr = MM(qq)σσ + CC(qq, qq )σσ + KKKK The left side of this equation can be rewritten as follows: MM (qq)qq rr CC (qq, qq )qq rr = MM (qq)qq rr CC (qq, qq )qq rr + MM(qq)qq rr + CC(qq, qq )qq rr 171

172 Assuming that the estimated matrices MM (qq) and CC (qq, qq ) have the same structure as the true matrices MM(qq) and CC(qq, qq ), both can be parameterized linearly as: MM (qq)qq rr CC (qq, qq )qq rr = MM (qq)qq rr CC (qq, qq )qq rr + MM(qq)qq rr + CC(qq, qq )qq rr = YY(qq, qq, qq rr, qq rr)θθ + YY(qq, qq, qq rr, qq rr)θθ = YY(qq, qq, qq rr, qq rr) θθ θθ = YY(qq, qq, qq rr, qq rr)θθ Where θθ θθ θθ It is further assumed that the parametric uncertainty is bounded, meaning that there is a value ρρ such that: θθ ρρ The closed-loop system can thus be described by the following model: MM(qq)σσ + CC(qq, qq )σσ + KKKK YY(qq, qq, qq rr, qq rr)θθ = 0 Lyapunov s direct method can be invoked to prove the global stability of the origin of the controlled system. This method is based on Lyapunov s second theorem which states that: IF there exists a scalar function VV(xx), xx being the state of the system, for which holds that: VV(0) = 0 VV(xx) > 0, for all xx UU \{0}, with UU being a neighborhood region around the origin xx = 0 VV (xx) 0, for all xx 172

173 THEN the origin of the system is globally, asymptotically stable, meaning that for any initial xx, the state will follow a trajectory that eventually converges onto the origin of the system. Consider the following candidate Lyapunov function: VV(σσ, qq ) = 1 2 σσtt MM(qq)σσ + qq TT ΛΛΛΛqq The first stability condition is readily verified: VV(0) = 0 As for the second condition: since all terms are quadratic, and since both MM(qq) and ΛΛΛΛ are positive definite matrices we can conclude that VV(σσ, qq ) > 0 for all (σσ, qq ) (0,0). To test the third condition, the Lyapunov function candidate is differentiated with respect to time: VV (σσ, qq ) = σσ TT MM(qq)σσ σσtt MM (qq)σσ + 2qq TT ΛΛΛΛqq From MM(qq)σσ + CC(qq, qq )σσ + KKKK = YY(qq, qq, qq rr, qq rr)θθ, it follows that MM(qq)σσ = CC(qq, qq )σσ KKKK + YY(qq, qq, qq rr, qq rr)θθ, which can be substituted into VV (σσ, qq ): VV (σσ, qq ) = σσ TT CC(qq, qq )σσ σσ TT KKKK + σσ TT YY(qq, qq, qq rr, qq rr)θθ σσtt MM (qq)σσ + 2qq TT ΛΛΛΛqq Next, the skew-symmetric property is invoked. Note that: σσ TT CC(qq, qq )σσ σσtt MM (qq)σσ = 1 2 σσtt MM (qq) 2σσ TT CC(qq, qq ) σσ Since MM (qq) 2σσ TT CC(qq, qq ), is a skew-symmetric matrix, it holds that 1 2 σσtt MM (qq) 2σσ TT CC(qq, qq ) σσ = 0, for all possible values of σσ. This reduces the expression for VV (σσ, qq ) to: 173

174 VV (σσ, qq ) = σσ TT KKKK + 2qq TT ΛΛΛΛqq + σσ TT YY(qq, qq, qq rr, qq rr)θθ Earlier σσ was defined as σσ = qq + ΛΛqq. Substituting this in the expression for VV (σσ, qq ) yields: VV (σσ, qq ) = qq + ΛΛqq TT KK qq + ΛΛqq + 2qq TT ΛΛΛΛqq + σσ TT YY(qq, qq, qq rr, qq rr)θθ = qq TT KKqq qq TT ΛΛ TT KKΛΛqq qq TT KKKKqq qq TT ΛΛ TT KKqq + 2qq TT ΛΛΛΛqq + σσ TT YY(qq, qq, qq rr, qq rr)θθ Since VV(σσ, qq ) is a scalar function, qq TT KKKKqq = qq TT ΛΛ TT KKqq and thus qq TT KKKKqq qq TT ΛΛ TT KKqq + 2qq TT ΛΛΛΛqq = 0, resulting in: VV (σσ, qq ) = qq TT KKqq qq TT ΛΛ TT KKΛΛqq + σσ TT YY(qq, qq, qq rr, qq rr)θθ The third term in the expression of VV (σσ, qq ) represents the effect of the model parametric uncertainty. It is now no longer possible to guarantee that VV (σσ, qq ) 0, and thus stability cannot be guaranteed. Sliding mode control resolves this issue by adding an additional term to the control law specifically geared towards counteracting this uncertainty. The control law is given as: ττ = MM (qq)qq rr + CC (qq, qq )qq rr + KKKK + YY(qq, qq, qq rr, qq rr)uu where the additional control variable uu remains to be determined. Applying this control law to the open-loop system yields: MM(qq)qq + CC(qq, qq )qq = ττ = MM (qq)qq rr + CC (qq, qq )qq rr + KKKK + YY(qq, qq, qq rr, qq rr)uu MM(qq)qq + CC(qq, qq )qq = MM (qq)qq rr + MM(qq)qq rr + CC (qq, qq )qq rr + CC(qq, qq )qq rr + KKKK + YY(qq, qq, qq rr, qq rr)uu MM (qq)qq rr CC (qq, qq )qq rr = MM(qq)(qq rr qq ) + CC(qq, qq )(qq rr qq ) + KKKK + YY(qq, qq, qq rr, qq rr)uu MM (qq)qq rr CC (qq, qq )qq rr = MM(qq)σσ + CC(qq, qq )σσ + KKKK + YY(qq, qq, qq rr, qq rr)uu 174

175 MM(qq)σσ + CC(qq, qq )σσ + KKKK + YY(qq, qq, qq rr, qq rr)uu YY(qq, qq, qq rr, qq rr)θθ = 0 MM(qq)σσ + CC(qq, qq )σσ + KKKK YY(qq, qq, qq rr, qq rr) θθ uu = 0 With this new control law, the expression for VV (σσ, qq ) becomes: VV (σσ, qq ) = qq TT KKqq qq TT ΛΛ TT KKΛΛqq + σσ TT YY(qq, qq, qq rr, qq rr) θθ uu Or written in a more compact form: VV (σσ, qq ) = qq TT KKqq qq TT ΛΛ TT KKΛΛqq + σσ TT YY θθ uu The challenge is now to choose the additional control term uu, such that σσ TT YY θθ uu 0. This is accomplished by choosing: which leads to: uu = ρρ YYTT σσ YY TT σσ σσ TT YY θθ uu = σσ TT YY θθ ρρ YYTT σσ YY TT σσ The Laypunov function is a scalar function. Therefore: σσ TT YY θθ ρρ YYTT σσ YY TT σσ = YYTT σσ θθ ρρ YYTT σσ YY TT σσ Applying the Cauchy-Schwartz inequality we get: YY TT σσ θθ ρρ YYTT σσ YY TT σσ YYTT σσ θθ ρρ As defined above, ρρ θθ, which leads to the conclusion that YY TT σσ θθ ρρ 0, which in turn implies that: VV (σσ, qq ) = qq TT KKqq qq TT ΛΛ TT KKΛΛqq + σσ TT YY(qq, qq, qq rr, qq rr) θθ uu 0 175

176 This result shows that the SMC control law guarantees global asymptotic stability of the origin of the system MM(qq)σσ + CC(qq, qq )σσ + KKKK YY(qq, qq, qq rr, qq rr) θθ uu = 0, which includes the model parametric uncertainty θθ. Again, this means that σσ = 0 and σσ = 0, which, as discussed earlier, implies asymptotic tracking of the desired joint angles, angular velocities and angular accelerations. A drawback of SMC, is that as σσ converges to zero, uu = ρρ YYTT σσ starts oscillating YY TT σσ at high frequencies. This phenomenon, also known as chattering, may have negative effects on the performance of the controlled system since it may excite higher order dynamics. To reduce chattering, the choice of uu is modified to: uu = ρρ YYTT σσ YY TT σσ ρρ YYTT σσ εε iiii YYTT σσ > εε iiii YY TT σσ εε For some small value εε. Using this controller reduces chattering, however the system will no longer converge to σσ = 0. Instead it will converge to and stay within a layer around σσ = 0, with thickness ± εε. The system is then said to no longer be globally asymptotically stable, but ultimately uniformly bounded. Finally, to reduce conservatism, uu can also be chosen as: uu = ρρ ii ξξ ii ξξ ii ξξ ii ρρ ii εε iiii ξξ ii > εε iiii ξξ ii εε Where ξξ ii denotes the ii th component of the vector YY TT σσ and where ρρ ii is a bound on the uncertainty of each parameter in θθ (ρρ ii > θθ ii ). 176

177 The block diagram for the closed-loop system under SMC is shown in the Figure below: 177

178 Appendix D: Adaptive Control An alternative strategy for dealing with parametric uncertainty, besides sliding mode control (SMC), is adaptive control. Unlike SMC, where the control law is static (not changing with time), adaptive control allows the controller to change over time. Let the control law be given as: ττ = MM (qq)qq rr + CC (qq, qq )qq rr + KKKK In this case, it was shown (see Appendix C) that the closed-loop system is described as: MM(qq)σσ + CC(qq, qq )σσ + KKKK YY(qq, qq, qq rr, qq rr)θθ = 0 where KK is chosen to be a positive definite, square, symmetric matrix. Lyapunov s direct method can be invoked to prove the global stability of the origin of the controlled system. This method is based on Lyapunov s second theorem which states that: IF there exists a scalar function VV(xx), xx being the state of the system, for which holds that: VV(0) = 0 VV(xx) > 0, for all xx UU \{0}, with UU being a neighborhood region around the origin xx = 0 VV (xx) 0, for all xx THEN the origin of the system is globally, asymptotically stable, meaning that for any initial xx, the state will follow a trajectory that eventually converges onto the origin of the system. 178

179 Consider the following candidate Lyapunov function: VV σσ, qq, θθ = 1 2 σσtt MM(qq)σσ + qq TT ΛΛΛΛqq θθ TT KK θθ θθ The first stability condition is readily verified: VV(0) = 0 As for the second condition: since all terms are quadratic, and since MM(qq), ΛΛΛΛ and KK θθ are positive definite matrices we can conclude that VV σσ, qq, θθ > 0 for all σσ, qq, θθ (0,0,0). To test the third condition, the Lyapunov function candidate is differentiated with respect to time: VV (σσ, qq ) = σσ TT MM(qq)σσ σσtt MM (qq)σσ + 2qq TT ΛΛΛΛqq + θθ TT KK θθ θθ From MM(qq)σσ + CC(qq, qq )σσ + KKKK = YY(qq, qq, qq rr, qq rr)θθ, it follows that MM(qq)σσ = CC(qq, qq )σσ KKKK + YY(qq, qq, qq rr, qq rr)θθ, which can be substituted into VV (σσ, qq ): VV (σσ, qq ) = σσ TT CC(qq, qq )σσ σσ TT KKσσ + σσ TT YY(qq, qq, qq rr, qq rr)θθ σσtt MM (qq)σσ + 2qq TT ΛΛΛΛqq + θθ TT KK θθ θθ Next, the skew-symmetric property is invoked. Note that: σσ TT CC(qq, qq )σσ σσtt MM (qq)σσ = 1 2 σσtt MM (qq) 2CC(qq, qq ) σσ + θθ TT KK θθ θθ Since MM (qq) 2CC(qq, qq ), is a skew-symmetric matrix, it holds that 1 2 σσtt MM (qq) 2CC(qq, qq ) σσ = 0, for all possible values of σσ. This reduces the expression for VV (σσ, qq ) to: VV (σσ, qq ) = σσ TT KKKK + 2qq TT ΛΛΛΛqq + σσ TT YY(qq, qq, qq rr, qq rr)θθ + θθ TT KK θθ θθ 179

180 Earlier σσ was defined as σσ = qq + ΛΛqq. Substituting this in the expression for VV (σσ, qq ) yields: VV (σσ, qq ) = qq + ΛΛqq TT KK qq + ΛΛqq + 2qq TT ΛΛΛΛqq + σσ TT YY(qq, qq, qq rr, qq rr)θθ + θθ TT KK θθ θθ = qq TT KKqq qq TT ΛΛ TT KKΛΛqq qq TT KKKKqq qq TT ΛΛ TT KKqq + 2qq TT ΛΛΛΛqq + σσ TT YY(qq, qq, qq rr, qq rr)θθ + θθ TT KK θθ θθ Since VV(σσ, qq ) is a scalar function, qq TT KKKKqq = qq TT ΛΛ TT KKqq and thus qq TT KKKKqq qq TT ΛΛ TT KKqq + 2qq TT ΛΛΛΛqq = 0, resulting in: VV (σσ, qq ) = qq TT KKqq qq TT ΛΛ TT KKΛΛqq + σσ TT YY(qq, qq, qq rr, qq rr)θθ + θθ TT KK θθ θθ Again, it is not guaranteed that VV σσ, qq, θθ 0, so global stability cannot be guaranteed. Since VV σσ, qq, θθ is a scalar function, σσ TT YYθθ + θθ TT KK θθ θθ can be written as: σσ TT YYθθ + θθ TT KK θθ θθ = θθ TT YY TT σσ+θθ TT KK θθ θθ = θθ TT YY TT σσ + KK θθ θθ Let the parameter estimate θθ be updated according to the following rule: θθ = KK 1 θθ YY TT σσ (since KK θθ was chosen to be positive definite, its inverse exists!) YY TT σσ = KK θθ θθ Substituting this into the expression for VV σσ, qq, θθ yields: VV σσ, qq, θθ = qq TT KKqq qq TT ΛΛ TT KKΛΛqq + θθ TT KK θθ θθ + θθ Remember that θθ was defined as θθ = θθ θθ. Then θθ = θθ θθ. Assuming that the true model parameters remain constant (θθ = 0), this results in θθ = θθ and thus: VV σσ, qq, θθ = qq TT KKqq qq TT ΛΛ TT KKΛΛqq + θθ TT KK θθ θθ θθ = qq TT KKqq qq TT ΛΛ TT KKΛΛqq 0 180

181 It has thus been shown that the control law in conjunction with the parameter estimate update law, achieve global asymptotic stability of the origin of the system, again guaranteeing that the system variables track their respective desired values. Note however that this does not imply that the parameter estimate θθ converges to the true parameter values θθ! The block diagram of the closed-loop system under adaptive control is shown in the Figure below: 181

182 Appendix E: Combined feedforward-feedback controller with time delay compensation theoretical stability analysis 1. Introduction In this document we present a theoretical proof that the proposed controller provides Semi-global Uniformly Ultimately Bounded (SUUB) tracking control for the time delayed arm model. By SUUB tracking control we mean that the joint angles, after a finite amount of time, will converge and remain within a bounded region around their desired values. This document is subdivided into four sections: Section 1 reviews the concept of stability, how it is defined, specifically for nonlinear systems such as the arm. Section 2 discusses Lyapunov s Direct Method, the main tool for testing stability of nonlinear systems. Section 3 presents the various assumptions and supporting mathematical theorems that are used in the main proof, which can be found in Section Stability To help our discussion we define a generic nonlinear system: xx (tt) = ff xx(tt), uu(tt) (1) Where xx(tt) is a vector containing the state variables, uu(tt) is a vector containing the control inputs and ff(xx(tt), uu) is a nonlinear function. An equilibrium point is defined as any point xx ee such that: xx (tt) = ff(xx ee (tt), 0) = 0 (2) 182

183 When the system is located at such an equilibrium point, it will remain at this equilibrium point, provided that it does not get disturbed (i.e. uu(tt) = 0). As an example, consider the pendulum system shown in the following figure: The pendulum consists of a mass MM located at the end of a rigid link. It is allowed to rotate freely around a pivot point. This system has two equilibrium point: the first equilibrium is reached when the mass hangs straight down (i.e. qq = 0º), while the second is reached when the pendulum stands perfectly upright (i.e. qq = 180º). An equilibrium point is considered stable if the system, following a perturbation, returns to the equilibrium position. According to this definition, the pendulum s first equilibrium point (i.e. hanging down) is stable: after perturbing the pendulum, it will eventually converge back to its hanging position. On the other hand, the pendulum s second equilibrium point (i.e. standing upright) is unstable, since any perturbation will cause the pendulum to diverge further away from this equilibrium point. 183