Digital Architecture for Real-Time CNN-based Face Detection for Video Processing

Transcription

1 ASPC lab Grant # June , / 26 igital Architecture for Real-Time CNN-based Face etection for Video Processing Smrity Bhattarai 1, Arjuna Madanayake 1 Renato J. Cintra 2 Stefan uffner 3, Christophe Garcia 3 epartment of Electrical and Computer Engineering, University of Akron, Akron, OH, USA 1 epartamento de Estatística UFPE, Recife, Brazil 2 LIRIS, INSA Lyon, France 3 June 27, 2017

2 Outline 1 Introduction and Motivation to the Research 2 Proposed Approach 3 Hardware Implementation 4 Conclusions ASPC lab sb229@zips.uakron.edu June 27, / 26

3 Image Processing and Face etection The method of converting an image into a digital form to get an enhanced image or extract useful information by performing some operations is called image processing Processing an image signal using mathematical operations for which an input can be either image, video or graph and the output is either image or set of characteristics of the corresponding image Image processing includes applying different sets of signal processing methods on images by treating them mostly as two dimensional (2) signals ASPC lab sb229@zips.uakron.edu June 27, / 26

4 Applications of Image Processing 1 Face etection and Recognition 2 efense Surveillance 3 Moving Object Tracking 4 Content based Image Retrieval 5 Image and Video Compression 6 Image Sharpening and Restoration Explored face detection technique that uses concept of feed forward neural network which is one of the most fundamental image processing technique ASPC lab sb229@zips.uakron.edu June 27, / 26

5 Challenges in Face detection Illumination Variation Expression Variation Variation in Image Position and Scale Variation in Quality of Image ASPC lab June 27, / 26

6 Proposed Approach Explore a digital architecture for real-time face detection and recognition system Approximate hardware realization scheme learning algorithm of convolutional neural networks (CNN) [1] to improve the efficiency of the hardware implementation of neural networks and to reduce the hardware complexity Evaluate accuracy of the system under different constraints Analyze hardware complexity of the system ASPC lab sb229@zips.uakron.edu June 27, / 26

7 Block iagram of the Proposed Approach[1] First Stage Second Stage Input Image Convolution and Sub sampling Block A M0 Stage 1 output Main Block Main Block Bias and Scale ecision Main Block Stage 1 output Sub Block Part 1 Stage 1 output Stage 1 output Part 2 Block A Block A Sub Block M1 M2 M3 N0 N1 N2 N3 N4 N5 Neuron Block ecision Output ASPC lab sb229@zips.uakron.edu June 27, / 26

8 Proposed Approach - First Stage Images are matrices of order SS1 is sub-sampling operation K1, K2, K3, K4 are four kernels of order 5 5 b 11, b 12, b 13, b 14 are biases α 1,α 2,α 3,α 4 are scale factors is the decision block 1, 2, 3, 4 are four output images after first stage and are of order H k (z x, z y ) SS1 K1 K2 K3 K4 S1 α 1 α 2 α 3 α 4 Figure: First Stage B1 ec ˆ b 11 ˆ b 12 ˆ b 13 ˆ b ASPC lab sb229@zips.uakron.edu June 27, / 26

9 Proposed Approach Second Stage K={1,2,..,8} p={1,2,..,8} A k H k (z x, z y ) P k H k (z x, z y ) M k O k,1 Q k H k (z x, z y ) O k,2 K={1,2,..,6} p={8,9,..,14} S2 B2 ec Sum B3 ec b 2,p ˆ N 1,p b 3,p β p β p N 2,p ˆ b 2,p N 1,p b 3,p N 2,p B4 ec b 4 O Figure: Second Stage Images are of matrices of order M k, O k,1 and O k,2 are kernels of order 3 3 ˆb 2,p, b 3,p and b 4 are biases β p and are scale factors and decision block respectively N 1,p are neurons of order 6 7 N 2,p is a single neuron ASPC lab sb229@zips.uakron.edu June 27, / 26

10 Hardware Implementation Convolution and Sub-sampling M N J(i k, j k ) = K(m, n).i [(i k 1) + m, (j k 1) + n] (1) m=1 n=1 H k (z x, z y ) = (1 + z 1 x )(1 + zy 1 ) 4 (2) where, J is the convolution operation H is the sub-sampling operation K is kernel matrix of order m n I is image matrix ASPC lab sb229@zips.uakron.edu June 27, / 26

11 Convolution and Sub-sampling From Equations 1 and 2 Q1 Q2 Q5 Q6 Q20 Q21 Q24 Q25 Q C = L+1 i=l S+1 x(i, j) j=s k11 k12 k15 k21 k45 k51 k54 Q1 Q2 Q5 Q6 A 1 Q20 Q21 Q24 Q25 l11 l12 l15 l21 l45 l51 l54 l55 k55 where, L = 2(T \A) + (C 1)\R + 1 S = 2[(T 1)%A] + (C 1)%R + 1 T = Time Stamp R = Order of Kernel = 5 C = Column Number Q 1 = x 11 + x 12 + x 21 + x 22 Q1 Q2 Q5 Q6 A 2 Q20 Q21 Q24 Q25 m11 m12 m15 m21 m45 m51 m54 m55 Q1 Q2 Q5 Q6 A 3 Q20 Q21 Q24 Q25 n11 n12 n15 n21 n45 n51 n54 n55 Q 25 = x 55 + x 56 + x 65 + x 66 A 4 Figure: Combined Convolution and Sub-sampling Block ASPC lab sb229@zips.uakron.edu June 27, / 26

12 Scaling and Bias Block S1 = (α 1, α 2, α 3, α 4 ) A 1 α 1 X 1 A 2 α 2 X 2 A 3 α 3 X 3 A 4 α 4 X 4 Figure: Scaling Block B1 = b 1,k ˆ b 1,k ˆ = b 1,k.α k + b 1,2,k b 1,2,k ˆ b 1,k b 1,k α k ˆ b 1,k X k ˆX k k = 1, 2, 3, 4 Figure: Bias Block ASPC lab sb229@zips.uakron.edu June 27, / 26

13 ecision Block Input 2 a b 2 a > b x 1 Sel MSB Sel d 0 â Output 1 d 0 d 1 1 d 1 Figure: ecision Block The decision block is basically linear-2 approximation[2] and given as: 1 if x < 2 σ(x) = â. x/2 if 2 x < 2 1 if x 2 where, â = 7/4 ASPC lab sb229@zips.uakron.edu June 27, / 26

14 Hardware Implementation etailed Block iagram of Second Stage SS C2 B1 S1 B2 ec N1 SumB3ec N2 b2,1 β1 b2,2,1 N1,1 NB1 N2,1 A M1 M2 b2,2 β2 b2,2,2 N1,2 NB2 N2,2 b2,3 β3 b2,2,3 N1,3 NB3 N2,3 B M3 M4 b2,4 β4 b2,2,4 N1,4 NB4 N2,4 b2,5 β5 b2,2,5 N1,5 NB5 N2,5 C M5 M6 b2,6 β6 b2,2,6 N1,6 NB6 N2,6 b2,7 β7 b2,2,7 N1,7 NB7 N2,7 M7 b2,8 β8 b2,2,8 N1,8 NB8 N2,8 M8 A B R11 R12 b2,9 β9 b2,2,9 N1,9 NB9 N2,9 B4 ec Output A R21 b2,10 β10 b2,2,10 N1,10 NB10 N2,10 C R22 Sum All A R31 R32 b2,11 β11 b2,2,11 N1,11 NB11 N2,11 B C R41 R42 b2,12 β12 b2,2,12 N1,12 NB12 N2,12 B R51 R52 b2,13 β13 b2,2,13 N1,13 NB13 N2,13 C R61 R62 b2,14 β14 b2,2,14 N1,14 NB14 N2,14 ASPC lab sb229@zips.uakron.edu June 27, / 26

15 Convolution and Sub-sampling Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 a11 a12 a13 a21 a22 a23 a31 a32 a33 F 1 Figure: 1st part of 2nd Stage of Combined Convolution Sub-sampling Block Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 i11 i12 i13 i21 i22 i23 i31 i32 i33 F 9 j11 j12 j13 j21 j22 j23 j31 j32 j33 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Figure: 2nd part of 2nd Stage of Combined Convolution Sub-sampling Block Followed by a scale, bias and decision block like in first stage ASPC lab sb229@zips.uakron.edu June 27, / 26

16 Neuron Block including Bias and ecision Block N 1,p NB p Input 42 ec Output z 1 Figure: Neuron Block Total 14 matrices of size 6 7 and is represented by N p where p = 1, 2,..14 Contains bias for each neuron block represented by the set NB p NB p = ˆ b 3,p Followed by an activation function represented by decision block Scalar multiplied with the matrix generated after second stage of decision block element by element and passes through the added decision block ASPC lab sb229@zips.uakron.edu June 27, / 26

17 Neuron Multiplication Block with Final bias and ecision Block Ĵ p is the output of Neuron Block The set Ĵ p is multiplied with corresponding neurons set called N 2,p Ĵ1 Ĵ2 N2,1 N2,2 All the elements of the resulting set ˆV p is added together to get a value W The value W obtained after Neuron Multiplication is biased with bias B4 The output is then passed through the decision block giving rise to obtain final output Final Output If the final output is positive, the image taken at the beginning is a face else the image taken is not a face ˆ J13 ˆ J14 N2,13 N2,14 W Figure: Neuron Multiplication Block B4 W ec Output Figure: Final Bias and ecision Block ASPC lab sb229@zips.uakron.edu June 27, / 26

18 Analysis and Results Tests were carried out using different set of images All the input images were standardized to size and in.pgm (Portable Gray Map) format The algorithm was first modelled and tested in MATLAB Simulink using Xilinx block set Tested on Xilinx Virtex-6 ML605 FPGA Evaluation Kit 1164 images and were processed Figure: Sample test images ASPC lab June 27, / 26

19 Accuracy etails MATLAB Accuracy etails Tested Images 1164 MATLAB Correctly etected 1124 Accuracy of MATLAB 96.56% Change in Accuracy of esign After Changing the Word-Length of Scale and Biases Word Length Part Scale Bias Integer 8 8 Fractional 5 5 Integer 7 7 Fractional 4 4 Integer 6 6 Fractional 3 3 Integer 5 5 Fractional 2 2 Integer 4 4 Fractional 1 1 Correctly etected Xilinx Accuracy % % % % % ASPC lab sb229@zips.uakron.edu June 27, / 26

20 Hardware Complexity Hardware Consumption Along with Timing etails Scale and Bias Hardware Consumption Stage T Integer Fractional LUTs FFs cpd F max (ns) (MHz) ASPC lab sb229@zips.uakron.edu June 27, / 26

21 Hardware Complexity and Accuracy Change in Accuracy After Changing the Word-Length of Kernel from the First stage Word - Length Correctly etected Xilinx Integer Fractional Accuracy % % % % % Hardware Consumption Along with Timing etails Kernel from first stage Hardware Consumption Integer Fractional LUTs FFs T cpd F max (ns) (MHz) ASPC lab sb229@zips.uakron.edu June 27, / 26

22 Hardware Complexity and Accuracy Change in Accuracy After Changing the Word-Length of Kernel from the Second Stage Word - Length Correctly etected Xilinx Integer Fractional Accuracy % % % % % Hardware Consumption Along with Timing etails Kernel from second stage Hardware Consumption Integer Fractional LUTs FFs T cpd F max (ns) (MHz) ASPC lab sb229@zips.uakron.edu June 27, / 26

23 Hardware Complexity and Accuracy Change in Accuracy After Changing the Word-Length of Neuron Biases Word - Length Correctly etected Integer Fractional Xilinx Accuracy % % % % % Hardware Consumption Along with Timing etails Neuron Hardware Consumption Integer Fractional LUTs FFs T cpd F max (ns) (MHz) ASPC lab sb229@zips.uakron.edu June 27, / 26

24 Conclusions A hardware system capable of face detection and recognition was designed The implementation paves a path for an efficient architecture designed to detect highly variable face patterns The system is a fully trained architecture comprising of a pipeline of convolution, sub-sampling and neural networks, such that no pre-processing of the image is required The experimental results show high accuracy ( 96%) The proposed model can be a good solution in various facial recognition and detection application ASPC lab sb229@zips.uakron.edu June 27, / 26

25 References I [1] C. Garcia and M. elakis, Convolutional face finder: A neural architecture for fast and robust face detection, IEEE Transactions on pattern analysis and machine intelligence, vol. 26, no. 11, pp , [2] J. Schmidhuber, eep learning in neural networks: An overview, Neural Networks, vol. 61, pp , ASPC lab sb229@zips.uakron.edu June 27, / 26

26 Thank You