Lecture 26: Backpropagation and its applications. Pushpak Bhattacharyya Computer Science and Engineering Department IIT Bombay

Transcription

1 CS621: Artificial Atifii lintelligence Lecture 26: Backpropagation and its applications Pushpak Bhattacharyya Computer Science and Engineering g Department IIT Bombay

2 Backpropagation algorithm i w i... Output layer (m o/p neurons) Hidden layers. Input layer (n i/p neurons) Fully connected tdfeed dforward network Pure FF network (no umping of connections over layers)

3 Gradient Descent Equations Δ w δe δw i i δe δnet Δw i δe = η δw = δe δnet = δ i ( η = δnet δw i learning rate, 0 η 1) ( net δ net =ηδ = ηδo δw i i = input at the th layer)

4 Backpropagation for outermost layer δ = δe δnet δe = δo δo δnet m 1 2 E = ( t p op ) 2 p= 1 Hence, δ = ( ( t o ) o (1 o )) Δw = η( t o ) o (1 o ) o i i ( net = input at the th layer)

5 Backpropagation for hidden layers i k.... Output layer (m o/p neurons) Hidden layers Input layer (n i/p neurons) δ k is propagated backwards to find value of δ

6 Backpropagation for hidden Δw = ηδ δ E = δo i o i layers δe δe δo δ = = δnet δo δnet = o k next layer Hence, δ = ( w k k next layer ( 1 o δ E ( δnet = δ ) o k k ) k next layer δ netk ) o δo ( δ w (1 o k ) o i k (1 o ) o ) (1 o )

7 General Backpropagation Rule General weight updating rule: Δw = ηδ i o i Where δ = ( t o ) o (1 o ) for outermost layer = ( w δ ) o k next layer k k ) o (1 o ) i for hidden layers

8 How does it work? Input propagation forward and error propagation backward (e.g. XOR) w 1 =1 θ = 0.5 w 2 =1 x 1 x x 1 x x 1 x 2

9 Local Minima Due to the Greedy nature of BP, it can get stuck in local minimum m and will never be able to reach the global minimum i g as the error can only decrease by weight change. Error Surface m g m- local minima, g- global minima Figure- Getting Stuck in local minimum

10 Momentum factor 1. Introduce momentum factor. ( β Δ w i ) nth iteration = ηδoi + ( Δwi)( n 1) th iteration Accelerates the movement out of the trough. Dampens oscillation inside the trough. Choosing β : If β is large, we may ump over the minimum. i

11 Symmetry breaking If mapping demands different weights, but we start with the same weights everywhere, then BP will never converge. θ = 0.5 w 1 =1 w 2 =1 x 1 x x 1 x XOR n/w: if we s started with identical x 1 x 2 weight everywhere, BP will not converge

12 Backpropagation Applications

13 Feed Forward Network Architecture Problem defined O/P layer Decided by trial error Hidden layer Problem defined I/P layer

14 Digit Recognition Problem Digit recognition: 7 segment display Segment being on/off defines a digit

15 9 O 8 O 7 O... 2 O 1 O Full connection Hidden layer Full connection 7 O 6 O 5 O... 2 O 1 O Seg 7 Seg 6 Seg 5 Seg 2 Seg 1

16 Example Character Recognition Output layer 26 neurons (all capital) First output neuron has the responsibility of detecting all forms of A Centralized representation of outputs In distributed representations, all output neurons participate in output

17 An application in Medical Domain

18 Expert System for Skin Diseases Diagnosis Bumpiness and scaliness of skin Mostly for symptom gathering and for developing diagnosis skills Not replacing doctor s diagnosis

19 Architecture of the FF NN input neurons, 20 hidden layer neurons, 10 output neurons Inputs: skin diseasesymptomssymptoms and their parameters Location, distribution, shape, arrangement, pattern, number of lesions, presence of an active norder, amount of scale, elevation of papuls, color, altered pigmentation, itching, pustules, lymphadenopathy, palmer thickening, results of microscopic examination, presence of herald pathc, result of dermatology test called KOH

20 Output 10 neurons indicative of the diseases: psoriasis, pityriasis rubra pilaris, lichen planus, pityriasis rosea, tinea versicolor, dermatophytosis, cutaneous T cell lymphoma, secondery syphilis, chronic contact dermatitis, soberrheic dermatitis

21 Training data Input specs of 10 model diseases from 250 patients 05is 0.5 some specific symptom value is not knoiwn Trained using standard derror backpropagation algorithm

22 Testing Previously unused symptom and disease data of 99 patients Result: Correct diagnosis achieved for 70% of papulosquamous group skin diseases Success rate above 80% for the remaining diseases except for psoriasis psoriasis diagnosed correctly only in 30% of the cases Psoriasis resembles other diseases within the papulosquamous ous group goupof diseases, and is somewhat difficult cut even for specialists to recognise.

23 Explanation capability Rule based systems reveal the explicit path of reasoning through the textual statements Connectionist expert systems reach conclusions through complex, non linear and simultaneous interaction of many units Analysing the effect of a single input or a single group of inputs would be difficult and would yield incor6rect results

24 Explanation contd. The hidden layer re represents represents the data Outputs of hidden neurons are neither symtoms nor decisions

25 Duration Symptoms & parameters of lesions : weeks 0 Duration of lesions : weeks 1 Internal representation 0 Disease diagnosis Minimal itching Positive KOH test ( Psoriasis node ) Lesions located on feet Minimal increase in pigmentation (Dermatophytosis node) Positive test for pseudohyphae And spores (Seborrheic dermatitis node) Bias Bias Figure : Explanation of dermatophytosis diagnosis using the DESKNET expert system.

26 Discussion Symptoms and parameters contributing to the diagnosis found from the n/w Standard deviation, mean and other tests of significance used to arrive at the importance of contributing parameters The n/w acts as apprentice to the expert

27 Exercise Find the weakest condition for symmetry breaking. It is not the case that only when ALL weights are equal, the network faces the symmetry problem.