Chapter 6: Backpropagation Nets

Transcription

1 Chapter 6: Bacpropagation Nets Architecture: at least one laer of non-linear hidden units Learning: supervised, error driven, generalized delta rule Derivation of the weight update formula (with gradient descent approach Practical considerations Variations of BP nets Applications

2 Architecture of BP Nets Multi-laer, feed-forward networ Must have at least one hidden laer Hidden units must be non-linear units (usuall with sigmoid activation functions Full connected between units in two consecutive laers, but no connection between units within one laer. For a net with onl one hidden laer, each hidden unit z_ receives input from all input units x_i and sends output to all output units _ x xn v_p v_n v_ v_np z zp w_ w_2 Y non-linear units

3 Additional notations: (nets with one hidden laer x (x_,... x_n: input vector z (z_,... z_p: hidden vector (after x applied on input laer (_,... _m: output vector (computation result delta_: error term on Y_ Used to update weights w_ Bacpropagated to z_ delta_: error term on Z_ Used to update weights v_i weighted input z_in: v_0 + Sum(x_i * v_i: input to hidden unit Z in: w_0 + Sum(z_ * w_: input to output unit Y_ v_0 w_0 bias x_i v_i z_ w

4 Forward computing: Appl an input vector x to input units Computing activation/output vector z on hidden laer z f ( z 0 + v i x i i Computing the output vector on output laer f ( 0 + w z is the result of the computation. The net is said to be a map from input x to output Theoreticall nets of such architecture are able to approximate an L2 functions (all integral functions, including almost all commonl used math functions to an given degree of accurac, provided there are sufficient man hidden units Question: How to get these weights so that the mapping is what ou want

5 Learning for BP Nets Update of weights in W (between output and hidden laers: delta rule as in a single laer net Delta rule is not applicable to updating weights in V (between input and hidden laers because we don t now the target values for hidden units z_,... z_p Solution: Propagating errors at output units to hidden units, these computed errors on hidden units drives the update of weights in V (again b delta rule, thus called error BACPROPAGATION learning How to compute errors on hidden units is the e Error bacpropagation can be continued downward if the net has more than one hidden laer.

6 BP Learning Algorithm step 0: initialize the weights (W and V, including biases, to small random numbers step : while stop condition is false do steps 2 9 step 2: for each training sample x:t do steps 3 8 /* Feed-forward phase (computing output vector */ step 3: appl vector x to input laer step 4: compute input and output for each hidden unit Z_ z_in : v_0 + Sum(x_i * v_i; z_ : f(z_in; step 5: compute input and output for each output unit Y in : w_0 + Sum(v_ * w_; _ : f(_in;

7 /* Error bacpropagation phase */ step 6: for each output unit Y_ delta_ : (t_ _*f (_in /* error term */ delta_w_ : alpha*delta_*z_ /* weight change */ step 7: For each hidden unit Z_ delta_in : Sum(delta_ * w_ /* erro BP */ delta_ : delta_in * f (z_in /*error term */ delta_v_i : alpha*delta_*x_i /* weight change */ step 8: Update weights (incl. biases w_ : w_ + delta_w_ for all, ; v_i : v_i + delta_v_i for all i, ; step 9: test stop condition

8 Notes on BP learning: The error term for a hidden unit z_ is the weighted sum of error terms delta_ of all output units Y_ delta_in : Sum(delta_ * w_ times the derivative of its own output (f (z_in In other words, delta_in plas the same role for hidden units v_ as (t_ _ for output units _ Sigmoid function can be either binar or bipolar For multiple hidden laers: repeat step 7 (downward Stop condition: Total output error E Sum(t_ _^2 falls into the given acceptable error range E changes ver little for quite awhile Maximum time (or number of epochs is reached.

9 Derivation of BP Learning Rule Obective of BP learning: minimize the mean squared output error over all training samples p For clarit, the derivation is for error of one sample E 0.5 Approach: gradient descent. Gradient f m ( t given the direction and magnitude of change of f w.r.t its arguments d For a function of single argument f (x, f '( x dx Gradient descent requires that x changes in the opposite direction of the gradient, i.e., x f '( x. Then since / x d / dx for small x 2 we have f '( x x f ' ( x 0 monotonicall decreases 2 E P P m ( t ( p ( p x : t 2

10 For a multi-variable function (e.g., our error function E Gradient descent requires each argument opposite direction of the corresponding Then because de dt we have E E E w dw dt E ( w,... E w n changes in the n n T E E 2 E ( w,... wn wi ( w w i Gradient descent guarantees that E monotonicall decreases, and E 0 iff the partial derivatives E / w i 0 i Chain rule of derivatives is used for deriving partial derivatives d d dx If f ( x and x g( z, then dz dx dz i w i E w i n (... ( + E + w n dw dt E dw dt i ( i. e., w dw,... dt i n i E w T 0 i

11 Update W, the weights of the output laer For a particular weight (from units to J J J J J m J J z in f t in w in f t in f w t w t t w t w w E _ '( ( (b chain rule _ ( _ '( ( _ ( ( ( ( (0.5( ( ( w J The last equalit comes from the fact that onl one of the terms in, namel involves p z w in _ J w w J z J J J J z w E w in f t δ α α δ (.Then _ '( ( Let This is the update rule in Step 6 of the algorithm J Z Y

12 Update V, the weights of the hidden laer For a particular weight v (from unit X to Z E v IJ v IJ m m m m m (( t (( t (( t ( δ ( δ (0.5 w J m v IJ ( t v v f '( _ in ( _ in v IJ z IJ IJ J 2 IJ f ( _ in v (ever IJ ( _ in (becauseδ involvesv I J ( t IJ as it is connected to f '( _ in The last equalit comes from the fact that onl one of the p terms in, namel involves (via J _ in w z w J z J v IJ z z J

13 E v IJ m m m ( δ ( δ ( δ w w w J J J f '( z _ in J f '( z _ in J v x x I IJ I z J f '( z _ in m J f '( z _ in δ J _ in v x δ J w IJ I J ( z _ in J (onl v IJ (because x I x in z _ in I J and z _ in involvesv J IJ are indep.of Let δ J E δ _ in J f '( _ inj.then vij α ( α δ v This is the update rule in Step 7 of the algorithm IJ J x I

14 Strengths of BP Nets Great representation power An L2 function can be represented b a BP net (multi-laer feed-forward net with non-linear hidden units Man such functions can be learned b BP learning (gradient descent approach Wide applicabilit of BP learning Onl requires that a good set of training samples is available Does not require substantial prior nowledge or deep understanding of the domain itself (ill structured problems Tolerates noise and missing data in training samples (graceful degrading Eas to implement the core of the learning algorithm Good generalization power Accurate results for inputs outside the training set

15 Deficiencies of BP Nets Learning often taes a long time to converge Complex functions often need hundreds or thousands of epochs The net is essentiall a blac box If ma provide a desired mapping between input and output vectors (x, but does not have the information of wh a particular x is mapped to a particular. It thus cannot provide an intuitive (e.g., causal explanation for the computed result. This is because the hidden units and the learned weights do not have a semantics. What can be learned are operational parameters, not general, abstract nowledge of a domain Gradient descent approach onl guarantees to reduce the total error to a local minimum. (E ma be be reduced to zero Cannot escape from the local minimum error state Not ever function that is representable can be learned

16 How bad: depends on the shape of the error surface. Too man valles/wells will mae it eas to be trapped in local minima Possible remedies: Tr nets with different # of hidden laers and hidden units (the ma lead to different error surfaces, some might be better than others Tr different initial weights (different starting points on the surface Forced escape from local minima b random perturbation (e.g., simulated annealing Generalization is not guaranteed even if the error is reduced to zero Over-fitting/over-training problem: trained net fits the training samples perfectl (E reduced to 0 but it does not give accurate outputs for inputs not in the training set Unlie man statistical methods, there is no theoreticall wellfounded wa to assess the qualit of BP learning What is the confidence level one can have for a trained BP net, with the final E (which not or ma not be close to zero

17 Networ paralsis with sigmoid activation function Saturation regions: x >> x f ( x /( + e, its derivative f '( x f ( x( f ( x 0 when x. When x falls in a saturation region, f ( x hardl changes its value regardless how fast the magnitude of x increases Input to an unit ma fall into a saturation region when some of its incoming weights become ver large during learning. Consequentl, weights stop to change no matter how hard ou tr. E w α ( α ( t f '( _ in z w Possible remedies: Use non-saturating activation functions Periodicall normalize all weights w w : / w. 2

18 The learning (accurac, speed, and generalization is highl dependent of a set of learning parameters Initial weights, learning rate, # of hidden laers and # of units... Most of them can onl be determined empiricall (via experiments

19 A good BP net requires more than the core of the learning algorithms. Man parameters must be carefull selected to ensure a good performance. Although the deficiencies of BP nets cannot be completel cured, some of them can be eased b some practical means. Initial weights (and biases Random, [-0.05, 0.05], [-0., 0.], [-, ] Normalize weights for hidden laer (v_i (Nguen-Widrow Random assign v_i for all hidden units V_ For each V_, normalize its weight b where Practical Considerations v. 2 is the normalization factor where p # of hiddent nodes, n # of Avoid bias in weight initialization: v v β v i i input nodes. β 2 / v. and β 0.7 n p 2 after normalization

20 Training samples: Qualit and quantit of training samples determines the qualit of learning results Samples must be good representatives of the problem space Random sampling Proportional sampling (with prior nowledge of the problem space # of training patterns needed: There is no theoreticall idea number. Following is a rule of thumb W: total # of weights to be trained (depends on net structure e: desired classification error rate If we have P W/e training patterns, and we can train a net to correctl classif ( e/2p of them, Then this net would (in a statistical sense be able to correctl classif a fraction of e input patterns drawn from the same sample space Example: W 80, e 0., P 800. If we can successfull train the networ to correctl classif ( 0./2* of the samples, we would believe that the net will wor correctl 90% of time with other input.

21 Data representation: Binar vs bipolar Bipolar representation uses training samples more efficientl w α δ z vi α δ x i no learning will occur when x i 0 or z with binar rep. 0 # of patterns can be represented n input units: binar: 2^n bipolar: 2^(n- if no biases used, this is due to (antismmetr (if the net outputs for input x, it will output for input x Real value data Input units: real value units (ma subect to normalization Hidden units are sigmoid Activation function for output units: often linear (even identit e.g., _ in w z Training ma be much slower than with binar/bipolar data (some use binar encoding of real values

22 How man hidden laers and hidden units per laer: Theoreticall, one hidden laer (possibl with man hidden units is sufficient for an L2 functions There is no theoretical results on minimum necessar # of hidden units (either problem dependent or independent Practical rule of thumb: n # of input units; p # of hidden units For binar/bipolar data: p 2n For real data: p >> 2n Multiple hidden laers with fewer units ma be trained faster for similar qualit in some applications

23 Over-training/over-fitting Trained net fits ver well with the training samples (total error E 0, but not with new input patterns Over-training ma become serious if Training samples were not obtained properl Training samples have noise Control over-training for better generalization Cross-validation: dividing the samples into two sets - 90% into training set: used to train the networ - 0% into test set: used to validate training results periodicall test the trained net with test samples, stop training when test results start to deteriorating. Stop training earl (before E 0 Add noise to training samples: x:t becomes x+noise:t (for binar/bipolar: flip randoml selected input units

24 Variations of BP nets Adding momentum term (to speedup learning Weights update at time t+ contains the momentum of the previous updates, e.g., w then ( t + α δ z + µ w ( t, where 0 < µ < α << w ( t + t s µ t s α δ an exponentiall weighted sum of all previous updates Avoid sudden change of directions of weight update (smoothing the learning process Error is no longer monotonicall decreasing Batch mode of weight updates Weight update once per each epoch Smoothing the training sample outliers Learning independent of the order of sample presentations Usuall slower than in sequential mode ( s z ( s

25 Variations on learning rate α Give nown underrepresented samples higher rates Find the maximum safe step size at each stage of learning (to avoid overshoot the minimum E when increasing α Adaptive learning rate (delta-bar-delta method Each weight w_ has its own rate α_ w If remains in the same direction, increase α_ (E has a smooth curve in the vicinit of current W w If changes the direction, decrease α_ (E has a rough curve in the vicinit of current W E / w α δ ( t ( β ( t + β ( t ( t + z α ( t + κ ( γ α ( t α ( t if if if ( t ( t ( t ( t ( t ( t > < 0 0 0

26 delta-bar-delta also involves momentum term (of α Experimental comparison Training for XOR problem (batch mode 25 simulations: success if E averaged over 50 consecutive epochs is less than 0.04 results method simulations success Mean epochs BP ,859.8 BP with momentum ,056.3 BP with deltabar-delta

27 Other activation functions Change the range of the logistic function from (0, to (a, b x Let f ( x /( + e, r b a, η a. g( x rf ( x η is a sigmoid function with range ( a, b g'( x rf ( x( f ( x ( g( x + η( ( g( x + η( r g( x η r 2 2 ( + g( x / r In particular, for bipolar sigmoid function, we have a, b, then r 2, η g( x f ( x, and g'( x g( x( η / r g( x

28 Change the slope of the logistic function σx f ( x /( + e, f '( x σf ( x( f ( x Larger slope: quicer to move to saturation regions; faster convergence Smaller slope: slow to move to saturation regions, allows refined weight adustment σ thus has a effect similar to the learning rate α (but more drastic Adaptive slope (each node has a learned slope f σ _ in, z f ( σ z _ in. Then we have ( w αe / w αδ σ z, whereδ ( t f '( σ vi αe / v i αδ σ xi, whereδ δ σ w f '( σ σ αe / σ αδ _ in, σ αe / σ αδ i i _ in z _ in z _ in

29 Another sigmoid function with slower saturation speed f ( x 2 arctan( x, π f '( x 2 π + x For large x, is much smaller than 2 x x + x ( + e ( + e the derivative of logistic function A non-saturating function (also differentiable 2 f ( x log( + x log( x if if x x 0 < 0 if x 0 f '( x + x, then, f '( x 0 when x if x < 0 x

30 Non-sigmoid activation function Radial based function: it has a center c. f f ( x ( x > 0 for all x becomes smaller when x c becomes larger f ( x 0 when x c e.g., Gaussian function : x f ( x e, f '( x 2xf ( x 2

31 Applications of BP Nets A simple example: Learning XOR Initial weights and other parameters weights: random numbers in [-0.5, 0.5] hidden units: single laer of 4 units (A 2-4- net biases used; learning rate: 0.02 Variations tested binar vs. bipolar representation different stop criteria ( targets with ±.0 and with ± 0.8 normalizing initial weights (Nguen-Widrow Bipolar is faster than binar convergence: ~3000 epochs for binar, ~400 for bipolar Wh?

32

33 Relaxing acceptable error range ma speed up convergence ±.0 is an asmptotic limits of sigmoid function, When an output approaches ±.0, it falls in a saturation region Use ± a where 0 < a <.0 (e.g., ± 0.8 Normalizing initial weights ma also help Binar Bipolar Bipolar with targets ±0.8 Random 2, Nguen-Widrow,

34 Data compression Autoassociation of patterns (vectors with themselves using a small number of hidden units: training samples:: x:x (x has dimension n hidden units: m < n (A n-m-n net n V m W n If training is successful, appling an vector x on input units will generate the same x on output units Pattern z on hidden laer becomes a compressed representation of x (with smaller dimension m < n Application: reducing transmission cost x n V m z z m W n x sender Communication channel receiver

35 Example: compressing character bitmaps. Each character is represented b a 7 b 9 pixel bitmap, or a binar vector of dimension 63 0 characters (A J are used in experiment Error range: tight: 0. (off: 0 0.; on: loose: 0.2 (off: 0 0.2; on: Relationship between # hidden units, error range, and convergence rate (Fig. 6.7, p.304 relaxing error range ma speed up increasing # hidden units (to a point ma speed up error range: 0. hidden units: 0 # epochs 400+ error range: 0.2 hidden units: 0 # epochs 200+ error range: 0. hidden units: 20 # epochs 80+ error range: 0.2 hidden units: 20 # epochs 90+ no noticeable speed up when # hidden units increases to beond 22

36 Other applications. Medical diagnosis Input: manifestation (smptoms, lab tests, etc. Output: possible disease(s Problems: no causal relations can be established hard to determine what should be included as inputs Currentl focus on more restricted diagnostic tass e.g., predict prostate cancer or hepatitis B based on standard blood test Process control Input: environmental parameters Output: control parameters Learn ill-structured control functions

37 Stoc maret forecasting Input: financial factors (CPI, interest rate, etc. and stoc quotes of previous das (wees Output: forecast of stoc prices or stoc indices (e.g., S&P 500 Training samples: stoc maret data of past few ears Consumer credit evaluation Input: personal financial information (income, debt, pament histor, etc. Output: credit rating And man more e for successful application Careful design of input vector (including all important features: some domain nowledge Obtain good training samples: time and other cost

38 Summar of BP Nets Architecture Multi-laer, feed-forward (full connection between nodes in adacent laers, no connection within a laer One or more hidden laers with non-linear activation function (most commonl used are sigmoid functions BP learning algorithm Supervised learning (samples s:t Approach: gradient descent to reduce the total error (wh it is also called generalized delta rule Error terms at output units error terms at hidden units (wh it is called error BP Was to speed up the learning process Adding momentum terms Adaptive learning rate (delta-bar-delta Generalization (cross-validation test

39 Strengths of BP learning Great representation power Wide practical applicabilit Eas to implement Good generalization power Problems of BP learning Learning often taes a long time to converge The net is essentiall a blac box Gradient descent approach onl guarantees a local minimum error Not ever function that is representable can be learned Generalization is not guaranteed even if the error is reduced to zero No well-founded wa to assess the qualit of BP learning Networ paralsis ma occur (learning is stopped Selection of learning parameters can onl be done b trial-and-error BP learning is non-incremental (to include new training samples, the networ must be re-trained with all old and new samples