Human Visual System Neural Network

Transcription

1 Proceedings of Student-Faculty Research Day, CSIS, Pace University, May 7 th, 2010 Human Visual System Neural Network Stanley Alphonso, Imran Afzal, Anand Phadake, Putta Reddy Shankar, and Charles Tappert Pace University Seidenberg School of CSIS, White Plains, NY 10606, USA {sa08400n, ia93345n, ap08653n, ps08615n, ctappert}@pace.edu Abstract The design of most Artificial Neural Networks for visual pattern recognition does not utilize our knowledge of the human visual system. Nobel Prize winners Hubel and Wiesel discovered line and edge detectors in the visual cortex of cats, and the experiments described in this study model such detectors to improve the accuracy and efficiency of neural networks for character recognition. The results obtained showed that accuracy and efficiency of a neural network system using such detectors in the early layers is superior to one without such detectors that uses adjustable weights directly from the retina. 1. Introduction This section briefly describes the visual system, the discovery by Nobel Laureates Hubel and Wiesel of line and edge detectors in the visual cortex, the field of computational neuroscience that has developed biologically detailed models of the visual system, the area of machine learning that uses simplified neural network models called Artificial Neural Networks (ANNs) in pattern recognition tasks, and a review of studies using ANNs that include line and/or edge detectors. The present study compares the use of ANNs with and without line detectors in the classification of the straight line characters of the English alphabet. The Visual System The visual system of mammals is the part of the nervous system that allows organisms to see, interpreting the information from visible light to build a representation of the world. It consists of the eye, the optic nerve, the lateral geniculate nucleus, and the visual cortex. Hubel and Wiesel won the 1981 Nobel Prize in Physiology/Medicine for their seminal work in the early 1060s [1, 2, 3]. They used microelectrodes to measure activity in individual cells in the visual cortex of cats. The cats were anesthetized and there eyes were open with the controlling muscles paralyzed so as to fix the stare in a specific direction. They discovered that a given cell can be specifically sensitive to a line of light at a specific orientation moving in a certain direction across a specific region of the visual field. These observations are explained by assuming certain patterns of synaptic connections to cells in the lateral geniculate body (a relay station in the visual pathway). A key discovery by Hubel and Wiesel is the existence of line and edge detectors in the visual cortex. Biological Simulations of the Visual System The work of Hubel and Wiesel was instrumental in the creation of what is now called computational neuroscience, studying brain function in terms of information processing properties of structures that make up the nervous system. Computational neuroscience is somewhat distinct from psychological connectionism and theories of learning from disciplines such as machine learning, neural networks and statistical learning theory in that it emphasizes descriptions of functional and biologically realistic neurons (and neural systems) and their physiology and dynamics. These models capture the essential features of the biological system at multiple spatial-temporal scales, from membrane currents, protein and chemical coupling to network oscillations, columnar and topographic architecture and learning and memory. These computational models are used to frame hypotheses that can be directly tested by current or future biological and/or psychological experiments. [4] Biologically detailed models of the brain can now be simulated due to increasingly powerful massively parallel supercomputers. Using biophysical model neurons we have simulated a neuronal network model of layers II/III of the neocortex. These simulations, carried out on the Blue Gene/L supercomputer, comprise up to 22 million neurons and 11 billion synapses and are the largest simulations of this type ever performed. Such model sizes correspond to the cortex of a small mammal. The SPLIT library, used for these simulations, runs on several platforms, from single-processor to massively parallel machines. Performance measurements show good scaling A1.1

2 behavior on the Blue Gene/L supercomputer up to 8192 processors. Several key phenomena seen in the living brain appear as emergent features in the simulations. [5] 18 November 2009 Scientists and engineers at IBM s Almaden Research Center, in San Jose, Calif., announced today at the Supercomputing Conference (SC09) in Portland, Ore., that they have created the largest brain simulation to date on a supercomputer. The number of neurons and synapses in the simulation exceed those in a cat s brain; previous simulations have reached only the level of mouse and rat brains. Experts predict that the simulation will have profound effects in two arenas: It will lead to a better understanding of how the brain s architecture leads to cognition, and it should inspire the design of electronics that mimic the brain s as-yetunmatched ability to do complex computation and learn using a small volume of hardware that consumes little power. [6] Machine Learning and Neural Networks Feed-forward Artificial Neural Networks (ANNs) allow signals to travel one way only from input to output (Figure 1). There is no feedback (looping), i.e. the output of any layer does not affect the same or earlier layers. Feed-forward ANNs tend to be straightforward networks that associate inputs with outputs, and they are extensively used in pattern recognition [7, 8]. Figure 1. A simple feedforward network [8]. The commonest type of artificial neural network consists of three groups, or layers, of units: a layer of input units is connected to a layer of hidden units, which is connected to a layer of output units as shown in Figure 1. The activity of the input units represents the raw information that is fed into the network. The activity of each hidden unit is determined by the activities of the input units and the weights on the connections between the input and the hidden units. The behavior of the output units depends on the activity of the hidden units and the weights between the hidden and output units. In order to train a neural network to perform some task, we must adjust the weights of each unit in such a way that the error between the desired output and the actual output is reduced. This process requires that the neural network compute the error derivative of the weights. In other words, it must calculate how the error changes as each weight is increased or decreased slightly. The back propagation algorithm is the most widely used method for determining the error derivative of the weights. ANNs are machine learning programs based on neuronlike building blocks similar to the neurons in the human brain. Most of the research and applications of neural networks involves feed-forward networks trained by the back-propagation algorithm. These ANNs usually undergo a training phase by feeding it a set of inputs with known outcome, and then back-propagating the known results to adjust the weights among the neural elements. After many iterations of training, called epochs, the NN is able to detect subtle patterns in large data sets and make predictions based on what it has learned through past observations. ANNs Using Line and/or Edge Detectors This section reviews pattern recognition systems that have employed the type of simple line and edge detectors used in this study. One study used line and edge detectors in four orientations 0, 45, 90, and 135 in a pattern recognition algorithm used on Geographic Information System (GIS) images and maps [9]. These line detectors were developed in an earlier study [10]. A scheme for detecting edges and lines in polarimetric Synthetic Aperture Radar (SAR) images was proposed [11]. The line detector was constructed from edge detectors. A traditional bright line extraction process vectorized the raw results and the scheme extracted dark linear structures on various full-polarimetric SAR images. Line detection can be done using edge techniques such as Sobel, Prewitt, Laplacian Gaussian, Zero Crossing and Canny edge detector [12]. Edge detection using Sobel yields pixels lying only on edges and these edges maybe incomplete due to factors such as breaks, noise due to non uniform illumination and intensity discontinuity. Line detection using Hough Transform uses edge detection followed by a linking algorithm to assemble pixels into meaningful edges by considering a point and all the lines that passes through it that can satisfy the slope intercept equation. This Study This study deals with line and edge detectors modeled on the human visual system. The design of most Artificial Neural Networks (ANNs) for visual pattern recognition, however, does not correspond with our knowledge of the human brain. The experiments below attempt to design neural networks that simulate line and edge detectors known to exist in the human visual cortex. The main objective of the experiments is to demonstrate good visual pattern recognition using pre-wired line and edge detectors similar to those of the human visual cortex. Specifically, we show that the accuracy and efficiency of A1.2

3 a neural network system using such detectors in the early layers is superior to one using adjustable weights directly from the retina. 2. Methodology Two pattern classification experiments were performed on the straight line characters of the uppercase English alphabet. The six alphabet character patterns were those consisting of horizontal and vertical line segments, that is the uppercase letters E, F, H, I, L, T. Each of these characters was represented by a 5x7 bit pattern as shown in Figure 2. ***** ***** * * * * ***** **** **** ***** * * * ***** * * * * ***** * Figure 2. Alphabetic input patterns. Experiment 1 ANN without line detectors A simple three-layer feed-forward ANN (Figure 3) was trained to recognize the straight-line alphabetic characters. The input layer modeled a 20x20 retina of binary (0 or 1) units for a total of 400 units. The hidden layer consisted of 50 units, fully connected to the first layer for a total of 20,000 (400x50) variable weights. The output layer consisted of six units, one for each of the six straight-line alphabetic characters, and was fully connected to the hidden layer for a total of 300 (50x6) variable weights. The overall network contained a total of 20,300 variable weights that required training. character pattern. The 168 non-retinal-edge retinal images for "E" can be labeled E(i,j) where the vertical (yaxis) position i varies from 2 to 13 and the horizontal (xaxis) position j from 2 to 15. E(i,j) can be generated by moving the upper-left-corner of the E bit-pattern down i pixels and right j pixels. For example, the retinal images for E(2,2) and E(12,5) are shown in Figure Figure 4. Input E(2,2) and E(12,5) patterns on the retina. For training input patterns, 40 random non-identical positions (randomly choosing i and j without replacement) were generated for each of the 6 characters, for a total of 240 (40*6) training input patterns. For testing, an additional 240 input patterns were similarly generated. These patterns were pre-generated to facilitate training and testing. The sequence of training patterns cycled through the sequence E, F, H, I, L, T forty times for one pass (epoch) of training through the 240 patterns. The system was trained to produce a positive output-unit response for the correct character and negative output-unit responses for the incorrect characters. Experiment 2 ANN with line detectors The ANN for this experiment incorporates horizontal and vertical line detectors into the model (Figure 5). It has five layers: input units, simple line detectors, complex line detectors, hidden units, and output units. Figure 3. Neural network for experiment 1. The alphabet character patterns could be placed in any position inside the 20x20 retina not adjacent to an edge. Shifting the 5x7 bit pattern around the retina yields 168 (12*14) possible non-retinal-edge positions of an alphabet The input layer and input character patterns are as in experiment 1, as are the hidden and output layers. The second layer consists of simple line detectors for detecting horizontal and vertical lines (Figure 6). Each of these simple line detectors had a threshold of 3. The input to a detector unit, when superimposed on an input pattern, is determined by adding the +'s and subtracting the -'s of A1.3

4 underlying active input-pattern units. The pluses indicate excitatory and the minuses inhibitory connections. Then, if the unit s input value is equal to or greater than the threshold, the detector is activated (set to 1), otherwise it is inactive (set to 0). Figure 7. The layout of the 24 complex vertical line detectors, and their corresponding 12 simple vertical line detector centers. Figure 5. Neural network for experiment 2. Horizontal Vertical Figure 6. Simple vertical and horizontal line detectors. The simple horizontal and vertical line detectors are overlapping and cover each of the possible retinal positions. There are a total of 288 (16*18) horizontal line detectors and 288 vertical line detectors for a total of 576 simple line detectors. The third layer consists of a relatively small number of complex line detectors, each taking inputs from a nonoverlapping local subset of simple line detectors. Each complex line detector takes inputs from 12 simple line detectors in a 3x4 grid (3 horizontal and 4 vertical for a vertical line detector, and vice versa for a horizontal line detector). The centers of the 12 simple vertical line detectors that feed the various complex vertical line detectors are shown in Figure 7. Each simple line detector feeds only one complex line detector. If any of the 12 simple line detectors feeding a complex line detector is activated, the output of the complex detector is set to 1 and otherwise set to 0. The complex line detector grids are non-overlapping. The number of complex vertical and horizontal line detectors is 24 (288/12) each, for a total of 48 complex line detectors. We arranged the outputs of the complex vertical line detectors in a 4x6 grid (4 vertical positions and 6 horizontal as shown in Figure 7) and the outputs of the complex horizontal line detectors in a 6x4 grid (6 vertical positions and 6 horizontal). The output of the complex detectors for E(1,1) is shown in Figure 8. vertical horizontal Figure 8. Complex line detector outputs for E(1,1). The two activated (output = 1) complex vertical line detectors respond to the upper and lower portions of the vertical bar of the E and the three activated horizontal line detectors respond to the three horizontal bars of the E. We can rearrange these outputs into one vector, first the vertical then the horizontal, of 48 detectors, and these 48- detector outputs can be pre-computed for each of the input letter patterns used for training and testing. Figure 9 shows the 48-detector outputs for E(1,1) plus five examples of the complex line detector outputs for each of the alphabet characters. The overall network consisted of 6,912 ( ) fixed weights and 2,700 ( ) variable weights. The experiment 2 neural network was trained and tested in the same manner as the experiment 1 network. A1.4

5 vertical horizontal E(2,2) E(12,5) E(9,8) E(7,2) E(11,8) E(9,12) F(12,5) F(9,8) F(7,2) F(11,8) F(9,12) H(12,5) H(9,8) H(7,2) H(11,8) H(9,12) I(12,5) I(9,8) I(7,2) I(11,8) I(9,12) L(12,5) L(9,8) L(7,2) L(11,8) L(9,12) T(12,5) T(9,8) T(7,2) T(11,8) T(9,2) Figure 9. Complex detectors for example characters. 3. Experimental Results Experiment 1 ANN without line detectors The system was trained for a specified number of epochs using backpropagation to train the weights between the retina and hidden layer, and between the hidden layer and output layer. Table 1 shows the results with 10 hiddenlayer units, and includes the number of training epochs, the training time, the percent accuracy on the 240 training patterns, and the percent accuracy on the 240 test patterns. In each case training ran for the full number of epochs. Due to the long training times it was not possible to obtain results with 50 hidden-layer units. Epochs Time Testing 50 ~2.5 hr 100% 26.7% 100 ~4 hr 100% 28.3% 200 ~8 hr 100% 28.8% 400 ~16 hr 100% 30.4% 800 ~30 hr 100% 28.3% 1600 ~2 days 100% 23.8% Average 100% 27.7% Table 1. Results with 10 hidden-layer units. Experiment 2 ANN with line detectors The complex line detectors were pre-computed for each input pattern. The system was trained for a specified number of epochs using backpropagation to train the weights between the layer of complex line detector units and the hidden layer, and between the hidden layer and output layer. Table 2 shows the results using 10 and Table 3 the results using 50 hidden-layer units. Epochs Time Testing 50 0:37 min 47.5% 37.5% 100 0:26 min 100.0% 63.3% 200 0:51 min 100.0% 68.8% 400 2:28 min 71.3% 50.8% 800 3:37 min 100.0% 67.9% :42 min 95.8% 56.7% Average 85.8% 57.5% Table 2. Results with 10 hidden-layer units. Epochs Set/ Attained Time Testing 50/8 41 sec 100% 70.0% 100/9 45 sec 100% 69.8% 200/10 48 sec 100% 71.9% 400/10 49 sec 100% 77.1% 800/8 41 sec 100% 72.5% 1600/9 45 sec 100% 71.3% Average 100% 72.1% Table 3. Results with 50 hidden-layer units. Table 4 shows the confusion matrix for the 400 epoch run above. There were 40 different images per alphabet character. The greatest confusions were E versus F ( %) and F versus T (5 and 20%). Out E F H I L T In E F H I L T Table 4. Confusion matrix, overall accuracy of 77.1%. 4. Conclusions The main conclusion is that character recognition accuracy and efficiency of a neural network using Hubel- Wiesel-like line detectors in the early layers is superior to that of a network using adjustable weights directly from the retina. The recognition accuracies of the systems are show in Figure 10. A1.5

6 No line detectors 10 hidden units Line detectors 10 hidden units Figure 10. Comparison of Recognition Accuracies. Line detectors 50 hidden units The numbers of fixed and variable weights in the two experiments is shown in Table 5. The fewer variable weights that require training in the ANN with line detectors resulted in a more efficient network, decreasing training time by several orders of magnitude. Experiment Fixed Weights Variable Weights Total Weights 1 No Line Detectors 0 20,300 20,300 2 Line Detectors 6,912 2,700 9,612 Table 5. Fixed and variable weights in experiments 1 and Future Work The strength of the study was its simplicity; its weakness was also it simplicity and that the line detectors appear to be designed specifically for the pattern to be classified. The above experiments can be elaborated by increasing the alphabet to the full 26 uppercase letters, the first three as shown in Figure 11. * **** *** * * * * * * * * * * * * * **** * ***** * * * * * * * * * * * **** *** Figure 11. First three alphabet character patterns. The second layer of simple line detectors can be doubled to provide detectors at four angles in increments of 45 degrees, at angles of 0, 45, 90, and 135 degrees. For detecting edges we can add the simple edge detectors (potential horizontal ones are shown in Figure 12) and corresponding complex edge detectors. The output layer can increased to 26 units, one for each of the characters to be recognized and --- Figure 12. Edge detectors for horizontal edges. Experiments could be conducted with both clean and noisy input. Random noise can be added to the input layer patterns: 2% (a random 8 of the 400 input units changed, by either change 1 to 0, or 0 to 1), 5%, 10%, 15%, and 20% noise. References [1] Hubel DH and Wiesel TN (1962). "Receptive fields, binocular interaction and functional architecture in the cat's visual cortex". J. Physiol. (Lond.) 160: pmidlookup?view=long&pmid= [2] Wikipedia, [3] Hubel DH and Wiesel TN. Brain and Visual Perception The Story of a 25-Year Collaboration, Oxford University Press, New York, ISBN [4] What is computational neuroscience? Patricia S. Churchland, Christof Koch, Terrence J. Sejnowski. in Computational Neuroscience pp Edited by Eric L. Schwartz MIT Press. [5] Djurfeldt, M. et al., Massively parallel simulation of brain-scale neuronal network models, Project report for Blue Gene Watson Consortium Days, 2006, Past_Results/ kth%20bgwreport06.pdf [6] Adee, S, IBM Unveils a New Brain Simulator, IEEE Spectrum, November 2009, [7] Artificial Neural Networks, Artificial Intelligence Technologies tutorial, [8] Stergiou, C. and Siganos, D., Neural Networks, report.html#why%20use%20neural%20networks. [9] Funtanilla, L.A. GIS Pattern Recognition and Rejection Analysis Using MATLAB, Proc. ESRI, [10] T.S. Chan, R.K.K. Yip, "Line Detection Algorithm," 13th Int. Conf. Pattern Recognition (ICPR'96), Vol. 2, 1996, pp.126. [11] D. Borghys, V. Lacroix, C. Perneel, "Edge and Line Detection in Polarimetric SAR Images," 16th Int. Conf. Pattern Recognition (ICPR'02) Vol. 2, 2002, pp [12] Wikipedia, A1.6