Model-Free Stochastic Perturbative Adaptation and Optimization

Transcription

1 Model-Free Stochastic Perturbative Adaptation and Optimization Gert Cauwenberghs Johns Hopkins University Learning on Silicon

2 Model-Free Stochastic Perturbative Adaptation and Optimization OUTLINE Model-Free Learning Model Complexity Compensation of Analog VLSI Mismatch Stochastic Parallel Gradient Descent Algorithmic Properties Mixed-Signal Architecture VLSI Implementation Extensions Learning of Continuous-Time Dynamics Reinforcement Learning Model-Free Adaptive Optics AdOpt VLSI Controller Adaptive Optics Quality Metrics Applications to Laser Communication and Imaging

3 The Analog Computing Paradigm Local functions are efficiently implemented with minimal circuitry, exploiting the physics of the devices. Excessive global interconnects are avoided: Currents or charges are accumulated along a single wire. Voltage is distributed along a single wire. Pros: Massive Parallellism Low Power Dissipation Real-Time, Real-World Interface Continuous-Time Dynamics Cons: Limited Dynamic Range Mismatches and Nonlinearities (WYDINWYG)

4 Effect of Implementation Mismatches INPUTS SYSTEM { p i } OUTPUTS ε(p) REFERENCE Associative Element: Mismatches can be properly compensated by adjusting the parameters p i accordingly, provided sufficient degrees of freedom are available to do so. Adaptive Element: Requires precise implementation The accuracy of implemented polarity (rather than amplitude) of parameter update increments p i is the performance limiting factor.

5 Example: LMS Rule A linear perceptron under supervised learning: y i (k) = Σ j p ij x j (k) ε = 1 2 Σ k Σ j ( ) y i target ( k) - y i (k) 2 with gradient descent: p ij (k) = -η ε(k) p ij = -η x j (k) y i target ( k) - y i (k) reduces to an incremental outer-product update rule, with scalable, modular implementation in analog VLSI.

6 Incremental Outer-Product Learning in Neural Nets j x j p ij i x i e j e i Multi-Layer Perceptron: x i = f( Σ j p ij x j ) Outer-Product Learning Update: p ij = η x j e i Hebbian (Hebb, 1949): LMS Rule (Widrow-Hoff, 1960): e i = x i e i = f ' i x i target - x i Backpropagation (Werbos, Rumelhart, LeCun): e j = f ' j p ij e i G. Cauwenberghs Learning on Silicon Σ i

7 Gradient Descent Learning Minimize ε(p) by iterating: from calculation of the gradient: Implementation Problems: p i (k + 1) = p i (k) - η ε ε p i = Σ l Σ m ε y l p i Requires an explicit model of the internal network dynamics. Sensitive to model mismatches and noise in the implemented network and learning system. Amount of computation typically scales strongly with the number of parameters. (k) y l x m x m p i

8 Gradient-Free Approach to Error-Descent Learning Avoid the model sensitivity of gradient descent, by observing the parameter dependence of the performance error on the network directly, rather than calculating gradient information from a preassumed model of the network. Stochastic Approximation: Multi-dimensional Kiefer-Wolfowitz (Kushner & Clark 1978) Function Smoothing Global Optimization (Styblinski & Tang 1990) Simultaneous Perturbation Stochastic Approximation (Spall 1992) Hardware-Related Variants: Model-Free Distributed Learning (Dembo & Kailath 1990) Noise Injection and Correlation (Anderson & Kerns; Kirk & al ) Stochastic Error Descent (Cauwenberghs 1993) Constant Perturbation, Random Sign (Alspector & al. 1993) Summed Weight Neuron Perturbation (Flower & Jabri 1993)

9 Stochastic Error-Descent Learning Minimize ε(p) by iterating: p (k +1) = p (k ) µε (k ) π (k ) from observation of the gradient in the direction of π (k) : ε (k ) = 1 2 ε(p(k ) +π (k ) ) ε(p (k ) π (k ) ) with random uncorrelated binary components of the perturbation vector π (k) : π i (k ) = ±σ ; E(π i (k ) π j (l ) ) σ 2 δ ij δ kl Advantages: No explicit model knowledge is required. Robust in the presence of noise and model mismatches. Computational load is significantly reduced. Allows simple, modular, and scalable implementation. Convergence properties similar to exact gradient descent.

10 Stochastic Perturbative Learning Cell Architecture ^ η ε(t) φ(t) φ(t) π i (t) NETWORK ^ η ε(t) Σ z 1 p i (t) Σ p i (t) + φ(t) π i (t) ε (p(t) + φ(t) π(t)) LOCAL GLOBAL p (k +1) = p (k ) µε (k ) π (k ) ε (k ) = 1 2 ε(p(k ) +π (k ) ) ε(p (k ) π (k ) )

11 Stochastic Perturbative Learning Circuit Cell V σ + V σ π i EN p π i π i V bp sign( ε) ^ POL C perturb p i (t) + φ(t) π i (t) V bn C store π i EN n

12 Charge Pump Characteristics (b) EN p V bp POL I adapt Q adapt V stored C (a) V bn EN n Voltage Decrement ²V stored (V) t = 40 msec 1 msec 23 µsec t = 0 Voltage Increment ²V stored (V) t = 40 msec 1 msec 23 µsec Gate Voltage V bn (V) (a) Gate Voltage V bp (V) (b)

13 Supervised Learning of Recurrent Neural Dynamics BINARY QUANTIZATION UPDATE ACTIVATION AND PROBE MULTIPLEXING H π 1 H π 2 H π 3 H π 4 H π 5 H π 6 Q(.) Q(.) Q(.) Q(.) Q(.) Q(.) W 11 W 12 W 13 W 14 W 15 W 16 W 21 W 22 W 31 W 41 W 51 W 56 W 61 W 65 W 66 x 1 x 2 x 3 x 4 x 5 x 6 TEACHER FORCING x 1 (t) x 1 T (t) x 2 (t) x 2 T (t) π 0 H θ 1 θ 2 θ 3 θ 4 θ 5 θ 6 W off W off W off W off W off W off + I ref I ref DYNAMICAL SYSTEM π 1 V π 2 V π 3 V π 4 V π 5 V π 6 V y(t) dx x(t) F p, x, y z = G () x dt = ( ) z(t)

14 The Credit Assignment Problem or How to Learn from Delayed Rewards INPUTS SYSTEM { } p i OUTPUTS r*(t) ADAPTIVE CRITIC r(t) External, discontinuous reinforcement signal r(t). Adaptive Critics: Discrimination Learning (Grossberg, 1975) Heuristic Dynamic Programming (Werbos, 1977) Reinforcement Learning (Sutton and Barto, 1983) TD(λ) (Sutton, 1988) Q-Learning (Watkins, 1989)

15 Reinforcement Learning (Barto and Sutton, 1983) Locally tuned, address encoded neurons: χ(t) {0,... 2 n 1} : n bit address encoding of state space y(t) = y χ(t) : classifier output q (t) = q χ(t) : adaptive critic Adaptation of classifier and adaptive critic: y k (t+1) = y k (t) + α r(t) e k (t) y k (t) q k (t+1) = q k (t) + β r(t) e k (t) eligibilities: e k (t+1) = λ e k (t) + (1 λ) δ k χ(t) internal reinforcement: r(t) = r(t) + γ q (t) q (t 1)

16 Reinforcement Learning Classifier for Binary Control State Eligibility SEL hor V bp UPD q vert SEL vert e k V αp q k V δ Neuron Select (State) r^ V bn UPD V bn Adaptive Critic SEL hor HYST V bp V bp UPD V αn V αp UPD LOCK HYST LOCK y vert 64 Reinforcement Learning Neurons x 2 (t) State (Quantized) y k Action Network V bn SEL hor x 1 (t) Action (Binary) y = 1 y(t) y = 1 u(t)

17 A Biological Adaptive Optics System brain iris retina lens cornea zonule fibers optic nerve

18 Wavefront Distortion and Adaptive Optics Imaging -defocus -motion Laser beam - beam wander/spread - intensity fluctuations

19 Adaptive Optics Conventional Approach Performs phase conjugation assumes intensity is unaffected Complex requires accurate wavefront phase sensor (Shack-Hartman; Zernike nonlinear filter; etc.) computationally intensive control system

20 Adaptive Optics Model-Free Integrated Approach Incoming wavefront Wavefront corrector with N elements: u 1,,u n,,u N Optimizes a direct measure J of optical performance ( quality metric ) No (explicit) model information is required any type of quality metric J, wavefront corrector (MEMS, LC, ) no need for wavefront phase sensor Tolerates imprecision in the implementation of the updates system level precision limited by accuracy of the measured J

21 Adaptive Optics Controller Chip Optimization by Parallel Perturbative Stochastic Gradient Descent Φ(u) image wavefront corrector u performance metric sensor J(u) J(u) AdOpt VLSI wavefront controller G. Cauwenberghs Learning on Silicon

22 Adaptive Optics Controller Chip Optimization by Parallel Perturbative Stochastic Gradient Descent Φ(u) Φ(u+δu) image wavefront corrector u+δu performance metric sensor J(u+δu) J(u+δu) J(u) AdOpt VLSI wavefront controller G. Cauwenberghs Learning on Silicon

23 Adaptive Optics Controller Chip Optimization by Parallel Perturbative Stochastic Gradient Descent Φ(u) Φ(u+δu) image Φ(u-δu) J(u+δu) wavefront corrector performance metric sensor J(u) J(u-δu) δj u-δu J(u-δu) AdOpt VLSI wavefront controller u j (k+1) = u j (k) + γδj (k) δu j (k) G. Cauwenberghs Learning on Silicon

24 Parallel Perturbative Stochastic Gradient Descent Architecture γ δj δu j {-1,0,+1} <δu i δu j > = σ 2 δ ij uncorrelated perturbations z 1 u j J(u) u j (k+1) = u j (k) + γ δj (k) δu j (k) ( ( k) ( k) ) ( k) ( k) 1 δu1 N δun 1 δ u1... u N δun ( ) 2 δ J (k) = J u (k) +,...,u (k) + J u (k),, (k)

25 Parallel Perturbative Stochastic Gradient Descent Mixed-Signal Architecture analog digital sgn(δj) γ δj δu j {-1,0,+1} δu j = ±1 Bernoulli z 1 u j J(u) u j (k+1) = u j (k) + γ δj (k) sgn(δj (k) ) δu j (k) Amplitude Polarity

26 Wavefront Controller VLSI Implementation Edwards, Cohen, Cauwenberghs, Vorontsov & Carhart (1999) Generate Bernoulli distributed ( ) ( ) { ( k) δ u } j ( ) ( ) k k k j sgn j j with δu = σ and δu = π = ± 1 ( ) ( ) Decompose δj into sgn δj δj (k) (k) (k) ( ) Update u = u γ xor sgn δj π (k+ 1) (k) (k) (k) j j j AdOpt mixed-mode chip 2.2 mm 2, 1.2µm CMOS Controls 19 channels Interfaces with LC SLM or MEMS mirrors

27 Wavefront Controller Chip-level Characterization G. Cauwenberghs Learning on Silicon

28 Wavefront Controller System-level Characterization (LC SLM) HEX-127 LC SLM

29 Wavefront Controller System-level Characterization (SLM) Maximized and minimized J(u) 100 times max min

30 Wavefront Controller System-Level Characterization (MEMS) Weyrauch, Vorontsov, Bifano, Hammer, Cohen & Cauwenberghs Response function MEMS mirror (OKO)

31 Wavefront Controller System-level Characterization (over 500 trials) G. Cauwenberghs Learning on Silicon

32 Quality Metrics imaging: J image r = I( r,t) υ d 2 r Horn(1968), Delbruck (1999) laser beam focusing: { ( r )} J = F I,t beam Muller et al.(1974) Vorontsov et al.(1996) laser comm: J comm = bit-error rate

33 N,M i,j i,j Image Quality Metric Chip Cohen, Cauwenberghs, Vorontsov & Carhart (2001) N,M IQM = I K I K = i,j i,j image Iij edge image Iij K pixel 2mm 22 x µm pixel array 120µm

34 Architecture (3 x 3) Edge and Corner Kernels

35 Image Quality Metric Chip Characterization Experimental Setup G. Cauwenberghs Learning on Silicon

36 Image Quality Metric Chip Characterization Experimental Results dynamic range ~ 20 uniformly illuminated

37 Image Quality Metric Chip Characterization Experimental Results G. Cauwenberghs Learning on Silicon

38 Beam Variance Metric Chip Cohen, Cauwenberghs, Vorontsov & Carhart (2001) 2 N,M N,M 2 = i,j i,j i,j i,j BVM N M I I perimeter of dummy pixels 22 x 22 22x22 20 x 20 array Pixel Pixel array array 70µm

39 I N,M ( 1+κ ) MNI I I 2 κ u i,j u BVM i,j N,M 2 i,j I i,j

40 Beam Variance Metric Chip Characterization Experimental Setup G. Cauwenberghs Learning on Silicon

41 Beam Variance Metric Chip Characterization Experimental Results dynamic range ~ 5

42 Beam Variance Metric Chip Characterization Experimental Results G. Cauwenberghs Learning on Silicon

43 Beam Variance Metric Sensor in the Loop Laser Receiver Setup G. Cauwenberghs Learning on Silicon

44 Beam Variance Metric Sensor in the Loop G. Cauwenberghs Learning on Silicon

45 Conclusions Computational primitives of adaptation and learning are naturally implemented in analog VLSI, and allow to compensate for inaccuracies in the physical implementation of the system under adaptation. Care should still be taken to avoid inaccuracies in the implementation of the adaptive element. Nevertheless, this can easily be achieved by ensuring the correct polarity, rather than amplitude, of the parameter update increments. Adaptation algorithms based on physical observation of the performance gradient in parameter space are better suited for analog VLSI implementation than algorithms based on a calculated gradient. Among the most generally applicable learning architectures are those that operate on reinforcement signals, and those that blindly extract and classify signals. Model-free adaptive optics leads to efficient and robust analog implementation of the control algorithm using a criterion that can be freely chosen to accommodate different wavefront correctors, and different imaging or laser communication applications.