VIDEO KEY FRAME DETECTION BASED ON THE RESTRICTED BOLTZMANN MACHINE

Transcription

1 Journal of Appled Mathematcs and Computatonal Mechancs 2015, 14(3), p-issn DOI: /amcm e-issn VIDEO KEY FRAME DETECTION BASED ON THE RESTRICTED BOLTZMANN MACHINE Mchał Knop, Tomasz Kapuścńsk, Wocech K. Mleczko Insttute of Computatonal Intellgence Czestochowa Unversty of Technology Częstochowa, Poland Abstract. In ths paper we present a new method for key frame detecton. Our approach s based on a well-known algorthm of the Restrcted Boltzmann Machne (RBM), whch s a pvotal step n our method. The frames are compared to the RBM matcher, whch allows one to search for key frame n the vdeo sequence. The Restrcted Boltzmann Machne s one of sophstcated types of neural networks, whch can process the probablty dstrbuton, and s appled to flterng mage recognton and modellng. The learnng procedure s based on the matrx descrpton of RBM, where the learnng samples are grouped nto packages, and represented as matrces. Our research confrms a potental usefulness for vdeo key frame detecton. The proposed method provdes better results for professonal and hgh-resoluton vdeos. The smulatons we conducted proved the effectveness of our approach. The algorthm requres only one nput parameter. Keywords: Restrcted Boltzmann Machne, key frame detecton, vdeo compresson 1. Introducton The popularty of hgh-defnton dgtal moves has ncreased n recent years. The tme needed to process ndvdual frames s ncreased sgnfcantly when hgh resoluton frames are analyzed. For ths reason, the need arses to fnd more effcent methods of content analyss. A wdely nvestgated scheme s to treat a vdeo as a stream of ndvdual mages wth the same dmensons [1, 2]. However, such methods don't take nto account smlartes of adacent frames. More effcent methods dvde vdeo nto scenes wth smlar frames and are commonly based on key frame detecton algorthms [3-5]. Key frame detecton s a challengng ssue that s solved n varous ways, such as hstogram-based methods, entropy analyss, and correlaton of mages [6-8]. Unfortunately, most methods are senstve to movement of obects n the scene and nose. Some researchers attempt to overcome these problems wth SURF and SIFT algorthms to fnd and compare keyponts of frames. In our work, we mplemented a method that uses an artfcal neural network called Restrcted Boltzmann Machne (RBM). In the proposed approach, we used a mappng error to compare frames and retreve key frames.

2 50 M. Knop, T. Kapuścńsk, W.K. Mleczko 2. Related works 2.1. Entropy method The method proposed by Knop an Dobosz n [9] s based on entropy from theory of nformaton whch s used to determne the complexty of nformaton n each frame. For grayscale vdeo materal, entropy can be calculated usng the formula below: where N E = p log( p ), (1) =0 E s a characterstc value calculated for current frame, p s the next pxel from ths frame, and N s a number of all pxels n the mage. For an HSV color model we need to calculate entropy value for each coeffcent n the palette. Components of the HSV model are not equally sgnfcant. The hue H s the most mportant one because even the smallest changes of ths value should lead to scene changes. The saturaton of color S has relatvely lttle mportance. Small to medum changes n saturaton are fully acceptable. Color value V has the lowest mportance. The fnal value of entropy for gven frame n an HSV color space can be calculated usng the followng formula: F ( E) = a E ( H ) + b E ( S) + c E ( V ). (2) n Parameters a, b and c correspond to weghtng factors when calculatng fnal entropy and ther values are set to 0.9, 0.3 and 0.1 respectvely [10]. To detect a new key frame we compare values of entropy between current key frame and subsequent frames. For comparson, the followng formula was used: E, k E E = E where s the number of the current frame and k s the number of the current key frame. The frst key frame s the frst mage taken from the vdeo. The next key frames are determned based on result of comparson n the formula (see Eq. (3)). If the calculated value of E, k s lower than the assumed threshold, the current frame s set as the new key frame. k k, (3) 2.2. SURF key frame detecton method SKFD s a method of detectng key frames usng SURF. SURF s an algorthm used to determne keyponts. We can use keyponts from two mages and attempt to match them to determne smlartes. The SURF algorthm s based on SIFT and works much faster because t uses Integral Images rather than Dfference of Gaussans [11-14].

3 Vdeo key frame detecton based on the Restrcted Boltzmann Machne 51 SURF has four maor steps: 1. Computng Integral Images, 2. Fast-Hessan Detector, 3. Interest Pont Descrptor, 4. Generatng of keyponts. A sngle key pont s represented by two vectors. The frst one contans pont poston, scale, response/strength of detected feature, orentaton, and laplacan value. The second vector contans ntensty dstrbuton of the pxels around pont of nterest and s made of 64 or 128 values. The output of ths step s factor, percentage relaton of the matched keyponts to all keyponts. factor s n range 0 factor 1, where 0 means no smlarty and 1 means complete smlarty. Next, we compare factor wth nput parameter t. If factor s lower than t, then frame next s tagged as a new key frame. next s then set to another vdeo frame. Ths stage s repeated untl we process all frames. Correctness of SURF algorthm s vtal because t determnes factor values. If the values are ncorrect, the method mght mss or fnd too many key frames [8, 15, 16]. The method can be represented as the followng block dagram (see Fg. 1) Fg.1. SURF key frames detecton

4 52 M. Knop, T. Kapuścńsk, W.K. Mleczko 3. Restrcted Boltzmann Machne learnng Fg. 2. Schematc representaton of a Restrcted Boltzmann Machne (RBM) - bdrectonal The Restrcted Boltzmann Machne s a two-layer recurrent neural network. The nput values are presented on the layer called "vsble" as vector v ( t) = [ v ( t),, 0 M0 ] 0 10 v ( t),, v ( t). M s the number of nputs, t ndcates the specfc sample. The data are transmtted to the layer called "hdden", as vector h 0( t) = [ h10 ( t),, h 0( t ),,h N0( t )], where N s the number of outputs, as depcted n Fgure 2. Gven an observed state, the energy of the ont confguraton of the vsble and hdden unts (v, h) s gven by: where v, (, h) = bvv bhh v hw, E v (4) εvsble εhdden h are the bnary states of vsble unt and hdden unt, b v, b h are ther bases and w s the weght between them. The network assgns a probablty to every possble par of a vsble and a hdden vector va ths energy functon: where v, h are the bnary states of vsble unt and hdden unt, b v, b h are ther bases and w s the weght between them. The network assgns a probablty to every possble par of a vsble and a hdden vector va ths energy functon:, p 1 Z E( v,h ( v, h) = e ), (5) where the partton functon Z, s gven by summng over all possble pars of vsble and hdden vectors: Z = E( v, h e ). (6) v, h The probablty that the network assgns to a vsble vector, v, s gven by summng over all possble hdden vectors:

5 Vdeo key frame detecton based on the Restrcted Boltzmann Machne 53 p 1 (7) Z E( v,h ( v) = e ). Gven a random nput confguraton v, the state of the hdden unt s set to 1 wth probablty: ( ) h P h = 1 = v σ bh + v w, (8) 1 where σ ( x) s the logstc sgmod functon σ ( x) = 1 + exp( x). Smlarly, gven a random hdden vector, the state of the vsble unt can be set to 1 wth probablty: P = h v + (9) ( v 1 ) = σ b h w. The probablty that the network assgns to a tranng mage can be rased by adustng the weghts and bases to lower the energy of that mage and to rase the energy of other mages, especally those that have low energes and therefore make a bg contrbuton to the partton functon. The dervatve of the log probablty of a tranng vector wth respect to a weght s surprsngly smple. log pv ( ) = v h w v h 0, (10) where 0 denotes the expectatons for the data dstrbuton ( p 0 ) and denotes the expectatons for the model dstrbuton ( p ) [17]. It can be done by startng at any random state of the vsble unts and performng alternatng Gbbs samplng for a very long tme. One teraton of alternatng Gbbs samplng conssts of updatng all of the hdden unts n parallel usng Equaton (8) followed by updatng all of the vsble unts n parallel usng Equaton (9). To solve ths problem, Hnton proposed a much faster learnng procedure - the Contrastve Dvergence algorthm [18, 19]. The use of ths procedure can be appled n order to correct the weghts and bas of the network: w η ( v h v h ), (11) = 0 b η ( v v ), (12) v = 0 b η ( h h ). (13) h = 0

6 54 M. Knop, T. Kapuścńsk, W.K. Mleczko 4. Proposed method The proposed method for the key frame detecton s based on the RBM algorthm. Our method conssts of several steps. The frst step dvdes nput vdeo nto ndvdual frames. Next, we create an nput matrx for RBM network based on a key frame. Another stage s an attempt to reconstruct the pattern of key frame based on subsequent frames delvered to the nput of the network. Ths stage s crucal. As a result, of the comparson between two frames (key frame and next frame) we obtan the error mappng, whch s n the range ( 0 error 1), where 1 means no smlarty and 0 complete smlarty. Next we compare the obtaned error wth the assumed threshold. If the effect of ths comparson s hgher than the expected threshold, the algorthm labels current frame as a key frame. Then the network s traned accordng to the new key frame. Otherwse, the network s gven next frame. Dagram below (Fg. 3) shows the proposed algorthm. Fg. 3. Dagram of the proposed method 5. Expermental result Algorthm stages were mplemented n the Java programmng language. The RBM mplementaton s based on an algorthm by A. Karpathy [20], adapted to the proposed method. Other steps were mplemented by the authors. The process of tranng the network for each key frame conssted of 50 epochs. Durng the frst step, a sequence of numbered frames s obtaned. Among these frames, some are detected as key frames. Whether a frame s a key frame or not s determned by calculatng an error usng equaton (10) and comparng t wth the threshold. If the error s greater than the threshold, the gven frame s marked as

7 Vdeo key frame detecton based on the Restrcted Boltzmann Machne 55 a key frame. In our frst experment, we set the threshold to 0.06 (see Fg. 5). Ths value was determned emprcally. Fg. 4. Experment 1. Threshold value was set to 0.06 Expermental results presented n Fgure 4 show smlar effcency to the method proposed n the artcle. Detected key frames match the key frames detected n the mentoned work. The dfference n frame numbers s a result of dfferent numberng scheme. In our work frame numbers start wth 0 whle n artcle [8] frame numbers start wth 1. Fg. 5. Key frame detecton Smlar results were obtaned n the second experment where nput vdeo had a resoluton of 624x256 (Fg. 6). We had to ncrease the threshold to 0.12 because acton n the move was more dynamc than n the prevous experment, whch resulted n generally hgher values of error mappng Fgure 7. Fg. 6. Experment 2. Threshold value was set to 0.12

8 56 M. Knop, T. Kapuścńsk, W.K. Mleczko Fg. 7. Key frame detecton for experment 2 In experment 3, the vdeo nput was obtaned from a USB camera wth a resoluton of 640x480. Movement of the camera durng recordng and changes of lghtng caused sgnfcant fluctuatons whch resulted n problems wth comparng adacent frames (Fg. 8). The threshold for ths experment had to be set to 0.17 n order to reduce a number of detected false-postve key frames. Ths value ddn't elmnate all unnecessary key frames, but hgher values would only cause more problems wth key frame detecton (Fg. 9). Fg. 8. Experment 3. Threshold value was set to 0.17 Fg. 9. Key frame detecton for experment 3

9 Vdeo key frame detecton based on the Restrcted Boltzmann Machne Conclusons The presented method s a novel approach for vdeo compresson. Smulatons verfed the correctness of the algorthm. The presented approach requres only one nput parameter - threshold. Durng the experments we dscovered that the proposed method has better results n professonal vdeos because ths type of vdeo has better lghtng and the transton of obects s slower. Amateur vdeos are often recorded wthout a trpod, thus the movement of the camera s more rapd. The algorthm s more effectve n hgh resoluton vdeos, but t requres more executon tme (bgger nput matrx). The method has been tested on varous vdeos, but due to lack of space we had to narrow the presented smulatons. In addton, our algorthm s resstant to any logo located on the vdeo. In future research we wll attempt to mplement a method whch automatcally adapts threshold value. Acknowledgment The work presented n ths paper was supported by a grant BS/MN /14/P New vdeo compresson method usng neural mage compresson algorthm. References [1] Rutkowsk L., Jaworsk M., Petruczuk L., Duda P., Decson trees for mnng data streams based on the Gaussan approxmaton, IEEE Transactons on Knowledge and Data Engneerng 2014, 26(1), [2] Rutkowsk L., Petruczuk L., Duda P., Jaworsk M., Decson trees for mnng data streams based on the Mcdarmd s bound, IEEE Transactons on Knowledge and Data Engneerng 2013, 25(6), [3] Seelng P., Scene change detecton for uncompressed vdeo, [n:] Technologcal Developments n Educaton and Automaton, Sprnger Netherlands, 2010, [4] Xnyng Wang, Zhengke Weng, Scene abrupt change detecton, [n:] Canadan Conference on Electrcal and Computer Engneerng 2000, 2, [5] Gentao Lu, Xangmng Wen, We Zheng, Pezhou He, Shot boundary detecton and keyframe extracton based on scale nvarant feature transform, [n:] Eghth IEEE/ACIS Internatonal Conference on Computer and Informaton Scence, ICIS 2009, [6] Cernak R., Knop M., Vdeo compresson algorthm based on neural networks, [n:] Artfcal Intellgence and Soft Computng, volume 7894 of Lecture Notes n Computer Scence, Sprnger, Berln - Hedelberg 2013, [7] Knop M., Cernak R., Shah N., Vdeo compresson algorthm based on neural network structures, [n:] Artfcal Intellgence and Soft Computng, volume 8467 of Lecture Notes n Computer Scence, Sprnger Internatonal Publshng, 2014, [8] Grycuk R., Knop M., Mandal S., Vdeo key frame detecton based on surf algorthm, [n:] Artfcal Intellgence and Soft Computng, Lecture Notes n Computer Scence, Sprnger, Berln - Hedelberg 2015,

10 58 M. Knop, T. Kapuścńsk, W.K. Mleczko [9] Knop M., Dobosz P., Neural vdeo compresson algorthm, [n:] Image Processng and Communcatons Challenges 6, volume 313 of Advances n Intellgent Systems and Computng, Sprnger Internatonal Publshng, 2015, [10] Zhong Qu, Ldan Ln, Tengfe Gao, Yongkun Wang, An mproved keyframe extracton method based on HSV colour space, Journal of Software 2013, 8(7). [11] Bay H., Tuytelaars T., Van Gool L., Surf: Speeded up robust features, [n:] Computer Vson- ECCV 2006, Sprnger 2006, [12] Bay H., Ess A., Tuytelaars T., Van Gool L., Speeded-up robust features (SURF), Computer Vson and Image Understandng 2008, 110(3), [13] Lowe D.G., Obect recognton from local scale-nvarant features, [n:] The Proceedngs of the Seventh IEEE Internatonal Conference on Computer Vson 1999, 2, [14] Lowe D.G., Dstnctve mage features from scale-nvarant keyponts, Internatonal Journal of Computer Vson 2004, 60(2), [15] Grycuk R., Gabryel M., Korytkowsk M., Scherer R., Voloshynovsky S., From sngle mage to lst of obects based on edge and blob detecton, [n:] Artfcal Intellgence and Soft Computng, volume 8468 of Lecture Notes n Computer Scence, L. Rutkowsk, M. Korytkowsk, R. Scherer, R. Tadeusewcz, L.A. Zadeh, J.M. Zurada eds., Sprnger Internatonal Publshng, 2014, [16] Grycuk R., Gabryel M., Korytkowsk M., Scherer R., Content-based mage ndexng by data clusterng and nverse document frequency, [n:] Beyond Databases, Archtectures, and Structures, volume 424 of Communcatons n Computer and Informaton Scence, S. Kozelsk, D. Mrozek, P. Kasprowsk, B. Małysak-Mrozek, D. Kostrzewa eds., Sprnger Internatonal Publshng, 2014, [17] Le Roux N., Bengo Y., Representatonal power of restrcted boltzmann machnes and deep belef networks, Neural Computaton 2008, 20(6), [18] G. Hnton, Tranng products of experts by mnmzng contrastve dvergence, Neural Computaton 2002, 14(8), [19] Hnton G., A practcal gude to tranng restrcted Boltzmann machnes, Momentum 2010, 9(1), 926. [20] Karpathy A., Cpsc 540 proect: Restrcted Boltzmann machnes.