Road Sign Detection from Complex Backgrounds

Transcription

1 Road Sign from Complex Bakground Chiung-Yao Fang, Chiou-Shann Fuh, and Sei-Wang Chen ( 方瓊瑤 ) ( 傅楸善 ) ( 陳世旺 ) Department of Information and Computer Eduation National Taiwan Normal Univerity Taipei, Taiwan, R. O. C. violet@ie.ntnu.edu.tw Department of Computer Siene and Information Engineering National Taiwan Univerity Taipei, Taiwan, R. O. C. Abtrat Thi paper deribe an approah to deteting road ign from omplex bakground. The input data are olor image equene aquired uing a ingle amorder mounted on a moving vehile. Two neural network are developed to extrat olor and hape feature, repetively, from image equene. A proe haraterized by fuzzy diipline i then introdued to determine road ign andidate baed on the extrated olor and hape feature. Experimental reult have manifeted the appliability of the propoed method. Inremental trategie applied to ontantly inoming video data may improve the performane of the method. Keyword: Road ign detetion; HSI olor model; Neural network; Fuzzy integration 1. Introdution Automati reognition of road ign an be of ue in uh appliation a driver aitant ytem, maintenane of traffi failitie, route planning and management, a well a adminitration of tranportation euritie and effiay all toward the ultimate goal of intelligent tranportation ytem (ITS) [10]. Two major tep ontitute road ign reognition: (1) loalization and () reognition. Thi tudy onern the loalization of road ign from omplex ene. Road ign an be broadly ategorized in term of funtionality into three lae: peifiation, warning, and prohibition. Eah la i haraterized by a few partiular olor, e.g., blue and green (peifiation), yellow and orange (warning), and red (prohibition). While olor provide ruial evidene for road ign detetion, the olor of ign generally degrade with time. Moreover, the hromati propertie of olor ould be altered by hading, hadowing, highlighting, and different weather, air, a well a lighting ondition. Road ign detetion ha to rely on other upport in addition to the olor one. In thi tudy, trutural evidene of road ign i inorporated. The ene under our onideration may be taken from freeway, expreway, highway, boulevard, treet, and ountry road, whih involve oniderably ditint bakground and road ign. Reliable detetion of road ign from variou ene ould be rather hallenging. Many tehnique [1, 3, 4, 5, 6, 7] have been developed to detet road ign from ene. Mot of them worked on ingle input image [3, 4] and ome on grey level image. In thi paper, video olor image equene are employed beaue they provide rih information for the purpoe. Lalonde and Li [3] reported a olor indexing approah to identifying road ign. There wa a prebuilt databae in their ytem. In the databae, road ign were deribed in term of olor hitogram. Identifying a road ign extrated from an image wa arried out imply by omparing it olor hitogram with thoe in the databae. Their method did not take into aount the trutural evidene of road ign. Piioli et al. [7] inorporated both olor and edge information to reognize road ign. They laimed that their tehnique ould readily be applied to temporal image equene and would improve the performane. In thi paper, a trutural evidene of road ign i preented, referred to a irular hape feature. The method for deteting olor and hape feature are addreed in the following etion. Beide, a proe haraterized by fuzzy diipline i introdued for integrating information derived from the ontinually inoming video equene. The experimental reult with real image are demontrated in Setion 3, followed by onluding remark and future work in Setion 4.. Road Sign Sytem In general, road ign onvey traffi information through their hape, olor, and ontent. To attrat human attention, road ign are uually deigned uing partiular olor and hape. Thu, olor and hape are two important feature of

2 road ign, whih are ueful to be a priori information for automati road ign detetion. The outline of our road ign detetion ytem (t ) i hown in Figure 1. One image S of an input video equene, S, i fed into the road ign detetion ytem. Firt, the detetion ytem plit the olor (t ) image hannel to alulate the hue image, S, for h extrating the olor feature. The olor feature i defined a the enter of ome peial olor region. Seond, an edge detetion method i applied to (t ) aquire the edge image, S, for extrating the e hape feature. The hape feature i defined a the enter of ome fixed hape region. Third, the olor and hape feature are integrated to loate the enter of road ign, and thi i the reult of road ign detetion. Finally, the ar peed i predited to update the parameter of our feature extration proedure, and it provide the immediate information for later proeing ueive ( t+1) image S. Two neural network are developed to detet olor and hape feature, repetively. Both of the neural network are ontruted by two layer of neuron, one i input layer, the other i output layer. The ynape between input layer and output layer are full onnetion. Figure how the keth of thee two neural network, and they will be explained in detail later..1 Uing Color Although road ign are mainly made by partiular olor, uh a red, green, blue, and orange, thi fat doe not redue the diffiulty of road ign detetion. On aount of the weather ondition, the outdoor illumination varie, and an not be under ontrol. Moreover, the paint on ign alo deteriorate with time. Thu, we hould look for a uitable olor model for road ign detetion. Many olor model are provided to proe olor image [, 8], inluding RGB (Red, Green, Blue) model, normalized RGB model, HSI (Hue, Saturation, Intenity) model, TekHVCT model, XYZ tritimulu oordinate, CIE (the International Commiion on Illumination) (Lab) pae, CIE (Luv) pae, YIQ and YUV olor ytem, and o on. Different olor model an be utilized for different appliation. Sine the HSI olor model i defined baed on human olor pereption [], and the deigning point of road ign i foued on apturing human attention, HSI model may be uitable for the appliation of road ign detetion. In HSI olor model, the value of aturation hannel and intenity hannel hould inevitably be under the influene of light (unny or hady) and hadow. Neverthele, hue hannel i well known to be invariant to lightne [, 8]. In onideration of the unertainty of the weather and the natural and artifiial damage of road ign, we think hue hannel i the bet hoie among olor feature in road ign detetion.. The Neural Network for Color Feature Sine the olor model ued by the amorder i RGB model, we firt tranfrom RGB model to HSI model. In fat, only the hue hannel i needed. Let (R, G, B) indiate the red, green, and blue olor value of the pixel (k, in the olor image and input to neuron q on the input layer. If, for example, the neural network would be ued to extrat the red olor, and h i defined a follow: 1 [( R G ) + ( R )] o 1 π [( R G ) + ( R )( G )] if ( R-G ) > 0, ( R ) > 0, and ( G ) 0 1 h ( k, = [( R G ) + ( R )] o 1 π [( R G ) + ( R )( G )] if ( R-G ) > 0, ( R ) > 0, and ( G ) < 0 0 (1) Then, the output of neuron q hould be defined by 180 h ( k, + R if h ( k, R o 0 x = h ( k, - Ro if h ( k, < R0 () where R 0 i the hue value of the tandard red olor. Firt, funtion h alulate hue value when the red value i larger than green and blue value in the RGB model. Thi i the looet ontraint about the definition of red. Seond, from equation (), neuron q work like a imilarity funtion, the more imilar between the olor of an image pixel and the tandard red olor, the larger output value of the neuron q. Let r ( i, j, k, = ( i k) + ( j, then the weight between neuron q on the input layer and neuron p on the output layer ould be defined a follow: w A/ r = 0 ( i,j,k, if i k, j l, and T1 r, T (3) here A i a ontant, (i, j) ould be regarded a the enter of a road ign; T 1 and T are the inner and outer radii, repetively, of the ring region we are

3 intereted in and they will be adapted to extrat different ize of road ign. Thu, for every neuron p on the output layer, the net input to the neuron p ould be alulated by net x w, = k l (4) where the indie, k and l, run over all onnetion from input layer to the neuron p. If f () denote the tranfer funtion of neuron p, then the output of the neuron p ould be written a follow: y = f ( net ) = net (5) here the tranfer funtion of neuron p only output the ame input value. The output value of neuron p indiate the poibility whether there i any red ring whoe enter i at (i, j)..3 Uing Shape The deign of road ign hape are only imple geometri hape, inluding irle (e.g. railroad advane warning or ivil defene evauation route), retangle (e.g. regulatory ign or guide ign), otagon (e.g. STOP ign), equilateral triangle (e.g. YIELD ign), and o on. Sine hape feature doe not vary in all weather and light ondition, it i a ueful and robut harateriti of road ign. Shape information epeially help to remove the unuitable olor feature point, uh a the pixel belonging to a red building or a red ar, that i, a large red region but not a road ign. Although ometime road ign may be partly oluded or damaged, the hape information i till very reliable in road ign detetion..4 The Neural Network for Shape Feature Edge are the fundamental element forming hape. Thu, the neuron on input layer work a an edge detetion funtion. Let (R, G, B) indiate the red, green, and blue olor value of the pixel (k, in the olor image and input to neuron q on the input layer, and x denote the output of neuron q, then x. (6) = ( I I k 1 l ) + ( I I k l 1) Here I = ( R + G + B) / 3, and equation (6) ould be replaed by any other edge detetion equation, uh a Robert, Sobel, Laplaian, and o on. Let r ( i, j, k, = ( i k) + ( j. If, for example, the neural network would be ued to extrat the irular hape, then the weight between neuron q on the input layer and neuron p on the output layer ould be defined by w, B/ r ( i,j,k, = B/ r ( i,j,k, 0 if r > T 3, and r mod C < C / if r > T 3, and r mod C C / (7) here T 3, B, and C are ontant, and C depend on the method of edge detetion we hooe. Thu, for every neuron p on the output layer, the net input to the neuron p i given by net = xw, k l (8) where the indie, k and l, run over all onnetion from input layer to the neuron p. If f () denote the tranfer funtion of neuron p, then the output of the neuron p i y = f ( net 1 if net > T 4 ) = 0 (9) here T 4 i a threhold, and f () i a tep funtion. If the output of f () i 1, then we believe that there i at leat a irle whoe enter i at (i, j)..5 Fuzzy Feature Integration The hape and olor feature extrated by the foregoing neural network hould be integrated to loate the enter of road ign. Here we provide a fuzzy approah to aomplih thi job. For every pixel (i 1,j 1 ) in the input image, it fuzzy degree belonging to road ign, the memberhip funtion, μ, i defined a follow: yi j m μ ( i1, j1) = y f ( i1, j1, i, j (10) f i1 j ) 1 i j ( i1 i ) + ( j1 j ) 1 m σ ( i1, j1, i, j ) e = (11) πσ where σ i a ontant; y i the output of the i 1 j 1 neuron (i 1, j 1 ) on the output layer for the neural network of hape feature point detetion; and yi j i the output of the neuron (i, j ) on the output layer for the neural network of olor feature point detetion. Sine there may be more than one road ign in an image, we elet the loal maximum in ub-region to loate the road ign. Let the initial ize of road ign be SIZE; the loation of road ign be defined by Loation( i, j) 1 if μ ( i, j) = 0 max i SIZE / i1 i+ SIZE / = j SIZE / j1 j+ SIZE / μ( i, j ) 1 1 (1)

4 If the output of Loation(i, j) i 1, then we think it ha high poibility that there exit a road ign whoe enter i at (i, j). 3. Experimental Reult At preent, the ytem i firt experimented to detet the red irular road ign, and it an be extended to detet other road ign later. Figure 5 and 6 how the two input video equene, S 1 and S, repetively. Sequene S 1 i taken in the high way, and S i taken on the treet. In thee video equene, the ize of eah image i pixel and the time between two ueive image i 0. eond. Figure 3 how the experimental reult of one frame, the fifth frame of the video equene S 1. Firt, the input image i hown in Figure 3(a). There i a peed limit ign in the image. Seond, the output reult of the neural network for olor feature detetion i hown in Figure 3(b) and the white pixel indiate the olor feature point of the input image. In thi tep, we defined R 0 = 0, B = 1, and the initial value of T 1 and T are 5 and 15, repetively. Third, the output reult of the neural network for hape feature detetion are hown in Figure 3(), and the white pixel indiate the hape feature point of the input image. Here we defined A = 100, C = 6, T 3 = 5, and T 4 = 40. Finally, Figure 3(d) i the integrated reult of the road ign detetion ytem (where σ = and SIZE = 0). The ytem ue one white retangle to indiate one loation of road ign. In thi example, our ytem detet the peed limit ign orretly. Figure 4 how another example provided by video equene S with R 0 = 0, T 1 = T 3 = 10, T =, T 4 = 40, A = 100, B = 1, C = 6, σ = and SIZE = 0. Although there i atually only one no right turn ign in the traffi ene, our ytem detet more than one. We believe the fale alarm will be refined by the following image if the ar peed ould be predited preiely. Now, we an make an obervation on the above example. The reult of the example hown in Figure 3 i motly affeted by olor feature. The main aue i that the number of hape feature point i larger than that of olor feature point. On the other hand, ine the number of olor feature point i larger than that of hape feature point, the reult of the example hown in Figure 4 i mainly affeted by hape feature. In hort, olor and hape feature may have different weight in different example. Finally, Figure 5 and 6 illutrate more experimental reult of the different frame in the video equene S 1 and S, repetively. In thee example, we applied imple linear method to predit the ar peed, but we did not ue a priori knowledge to enhane the orretne of road ign detetion. 4. Conluion and Future Work Our paper deribe a method for deteting road ign from omplex bakground. Input a olor equene image aquired by a ingle amorder, two neural network are developed to extrat the olor and hape feature, repetively. A fuzzy approah to integrating olor and hape feature i introdued for eleting the andidate of the enter of road ign. Experimental reult how that our ytem work well. Our ytem predit the ar peed by linear method now, but in fat the veloity variation of the ar i nonlinear. If the ytem ould inlude a preie tehnique of ar peed predition, uh a Kalman filter [9], then it funtionality will be improved. Furthermore, if we ould aurately trak motion trajetorie of the road ign by inorporating the information of the image equene, we an detet the road ign more robutly. Thee are our future work. Aknowledgment Thi work i upported in part by the National Siene Counil, Taiwan, Republi of China, under ontrat NSC E Referene [1] A. Guidui, 3D Road Reontrution from a Single View, Computer Viion and Image Undertanding, Vol. 70, pp. 1-6, [] R. C. Gonzalez and R. E. Wood, Digital Image Proeing, Addion-Weley, Reading, Maahuett, [3] M. Lalonde and Y. Li, of Road Sign uing Color Indexing, Tehnial Report CRIM-IT-95/1-49, Centre de Reherhe Informatique de Montreal, ( publiation.htm [4] M. Lalonde and Y. Li, Road Sign Reognition, Tehnial Report CRIM-IT-95/09-35, Centre de Reherhe Informatique de Montreal, ( /ime/publiation.htm [5] W. Li, X. Jiang, and Y. Wang, Road Reognition for Viion Navigation of an Autonomou Vehile by Fuzzy Reaoning, Fuzzy Set and Sytem, Vol. 93, pp , [6] P. Parodi and G. Piioli, A Feature Baed Reognition Sheme for Traffi Sene, Proeeding of Intelligent Vehile Sympoium, pp. 9-34, Detroit, [7] G. Piioli, E. D. Miheli, P. Parodi, and M. Campani, A Robut Method for Road Sign and Reognition, Image and Viion Computing, Vol. 14, pp. 09-3, [8] W. Skarbek and A. Kohan, Colour Image Segmentation--A Survey, Thhniher Beriht 94-3, Tehnial Univerity of Berlin, 1994.

5 [9] G. Welh and G. Bihop, An Introdution to the Kalman Filter, kalmanintro.html, [10] Intelligent Tranportation Sytem, An Image Sequene S Input Image S (t) Image Channel Split and Edge Edge Image S e (t) Hue Image S h (t) Neural Shape Feature Neural Color Feature The Center of Some Fixed Shape Fuzzy Feature Point Integration The Center of Some Color Region Inhibitory Connetion Output Layer Update the Parameter of Neural Network Uing the Information from Image S (t) The Center of Road Sign (i,j) Fully Exitatory Connetion p W qp= W, m n Neuron Input Layer Car Veloity and Aeleration (k, q m n Neuron Car Speed Predition Road Sign Loation Output the Poition of Road Sign Figure 1: Outline of road ign detetion. Figure : Sketh of the neural network for olor (hape) feature detetion, where (i, j) i the poition of neuron p on the output layer, (k, i the poition of neuron q on the input layer, and W, i the weight between neuron q on the input layer and neuron p on the output layer.

6 (a) () (b) (d) Figure 3: The experimental reult of one frame in video equene S 1, (a) the input image; (b) the output of the neural network for olor feature detetion; () the output of the neural network for hape feature detetion; and (d) the integrated reult of the road ign detetion ytem. White quare i the deteted road ign. (a) () (b) (d) Figure 4: The experimental reult of one frame in video equene S, (a) the input image; (b) the output of the neural network for olor feature detetion; () the output of the neural network for hape feature detetion; and (d) the integrated reult of the road ign detetion ytem. White quare are the deteted road ign. There i a fale alarm on the upper left orner.

7 (a) Frame 1. (e) Frame 5. (b) Frame. (f) Frame 6. () Frame 3. (g) Frame 7. (d) Frame 4. (h) Frame 8. Figure 5: The experimental reult with video equene S 1. The firt eight frame of thi equene are hown in (a) to (h). The ize of eah image i pixel and the time between two ueive image i 0. eond.

8 (a) Frame 1. () Frame 3. (b) Frame. (d) Frame 4. Figure 6: The experimental reult with video equene S. The firt four frame of thi equene are hown in (a) to (d). The ize of eah image i pixel and the time between two ueive image i 0. eond.