A Multi-Level Approach for Temporal Video Segmentation based on Adaptive Examples

Transcription

1 June

2 A Mult-Level Approach for Temporal Vdeo Segmentaton based on Adaptve Eamples Robert Babak Yeganeh Submtted to the Department of Electrcal Engneerng and Computer Scence and the Faculty of the Graduate School of the Unversty of Kansas n partal fulfllment of the requrements for the degree of Master of Scence n Computer Scence. Commttee Dr. John Gauch (Char) Dr. Arvn Agah Dr. James Mller June

3 A Mult-Level Approach for Vdeo Temporal Segmentaton based on Adaptve Eamples 1. Overvew 2. Lterature Revew 3. Segmentaton based on Predefned Eamples (S.P.E.) 4. Segmentaton based on Adaptve Eamples (S.A.E.) 5. Epermentaton Results 6. Conclusons June

4 Outlne 1. Overvew 2. Lterature Revew 3. Segmentaton based on Predefned Eamples (S.P.E.) 4. Segmentaton based on Adaptve Eamples (S.A.E.) 5. Epermentaton Results 6. Conclusons June

5 1. Overvew June

6 1. Overvew June

7 1. Overvew June

8 Outlne 1. Overvew 2. Lterature Revew 3. Segmentaton based on Predefned Eamples (S.P.E.) 4. Segmentaton based on Adaptve Eamples (S.A.E.) 5. Epermentaton Results 6. Conclusons June

9 2. Lterature Revew Representaton Detecton Classfcaton False Detecton and Preventon June

10 2. Lterature Revew Fg.P.1. Illustrates a general process flow for temporal vdeo segmentaton algorthms. June

11 Problems & Solutons (P) Qualty of Detecton (S) Parallel Analyzer (S) Uncertanty Groups (S) Etremely Senstve Change Detector (ESCD) (S) False Negatve Preventon (No Threshold) (S) False Postve Detecton (P) Complety vs. Smplcty (S) Eample based Technque (S) Uncertanty Groups (P) Real Tme (S) Adaptve Eamples (P) Generalty vs. Specfcty (S) Eample based Technque (P) Fleblty and Etensblty (S) Mult-level property (S) Eample based Technque June

12 Outlne 1. Overvew 2. Lterature Revew 3. Segmentaton based on Predefned Eamples (S.P.E.) 4. Segmentaton based on Adaptve Eamples (S.A.E.) 5. Epermentaton Results 6. Conclusons June

13 3. Segmentaton based on Predefned Eamples (S.P.E.) 3.1. Representaton 3.2. Detecton 3.3. Classfcaton June

14 3.1. Representaton Predefned Eamples Qualty Eamples of dfferent duratons Dfferent types of eamples Transton Types Vdeo Types Balance Color Varety Combned Transtons Quantty Fg. P.2. Sample cut fade n fade out and dssolve sequences. June

15 3.1. Representaton Color Moments (Statstcs) Table P.1. Organzes the twenty seven moments n an easy to understand fashon Fg. P.3. Illustrates the center of gravtes for each of the three color components n real lfe pcture. June

16 June Representaton ( ) = y c t M c t y I N c t M ) ( ) ( 1 (P.4) ( ) [ ] = y c t M c t y I c t M N c t S 2 ) ( ) ( ) ( 1 ) ( (P.5) ( ) [ ] 3 3 ( ) ( ) ( 1 ) ( = y c t M c t y I c t M N c t K (P.6) (P.1) (P.2) (P.3) = y c t y I N c t M ) ( 1 ) ( [ ] = y c t M c t y I N c t S 2 ) ( ) ( 1 ) ( [ ] 3 3 ) ( ) ( 1 ) ( = y c t M c t y I N c t K

17 June Representaton F I D d F I M m Dff p F f I p f f p F f I p f f + = = = = = 1/ 0 0 1/ 0 0 β α (P.7) where f f f f f f f f f f f f f f f f M of dervatve w D D w M M m of dervatve w d d moment for used weght w w m m M of dervatve D eample current from the frame current of moment M m of dervatve d stream nput from the frame current the of moment m = = = = = = = = = = = Measure of Dfference

18 3.1. Representaton Fg. P.4. Illustrates generated ft values mage. Fg. P.5. Illustrates sorted ft values mage June

19 3.2. Detecton 3.3. Classfcaton Best Eamples Etracton & Labelng Fg. P.6. Represents the best ft values for each wndow for one mnute of nput data. Localzed Adaptve Threshold threshold = m f + K σ f (P.8) June

20 Outlne 1. Overvew 2. Lterature Revew 3. Segmentaton based on Predefned Eamples (S.P.E.) 4. Segmentaton based on Adaptve Eamples (S.A.E.) 5. Epermentaton Results 6. Conclusons June

21 4. Segmentaton based on Adaptve Eamples (S.A.E.) 4.1. Representaton 4.2. Detecton 4.3. Classfcaton 4.4. False Detecton and Preventon June

22 4.1. Representaton Color Moments (Statstcs) Refer to S.P.E. representaton secton. June

23 4.1. Representaton Adaptve Eamples Cut Dssolve Fade Normal Groups Fg. P.7. Illustrates the process of etractng potental canddate and generatng a cut adaptve eample whle T partton of the wndow s centered on a cut transton. Fg. P.8. Illustrates the process of etractng potental canddate and generatng a dssolve adaptve eample whle T partton of the wndow s centered on a dssolve transtons. June

24 4.1. Representaton Dssolve & Fade (Adaptve Eamples) M t R = α At R + ( 1 α ) Bt R (P.9) σ t R 2 α σ A t B R + (1 α ) σ t R (P.10) Fg. P.9. Illustrates the process of etractng potental canddate and generatng a fade n adaptve eample whle T partton of the wndow s centered on a fade n transton. June

25 4.1. Representaton Normal Groups (Adaptve Eamples) No Threshold Etremely Senstve Change Detector Fg. P.10. Illustrates the process of etractng potental canddate and generatng a normal adaptve eample for gradual transtons detector whle T partton of the wndow s over a regon of no actvty (regons contanng mnor object motons). June

26 4.2. Detecton 4.3. Classfcaton 4.4. False Preventon & Detecton Best Eamples Etracton & Labelng No Threshold - Parallel Analyzer - Uncertanty Groups Analyzers - False Detecton Technques Fg. P.11. Illustrates the hgh level process flow for the second algorthm. Fg. P.12. Illustrates cut and dssolve detecton streams. June

27 Outlne 1. Overvew 2. Lterature Revew 3. Segmentaton based on Predefned Eamples (S.P.E.) 4. Segmentaton based on Adaptve Eamples (S.A.E.) 5. Epermentaton Results 6. Conclusons June

28 5. Epermentaton Results 5.1. Evaluaton Technques 5.2. S.P.E. Results 5.3. S.A.E. Results 5.4. Dscusson June

29 5.1. Evaluaton Technques Manual Detecton (Truth Data) True Postves False Postves False Negatves True Negatves Recall Precson Utlty June

30 5.1. Evaluaton Technques Recall Re = N correct call R = 100% N + N correct mssed (P.11) where TP = = Θ Θ = S { 1... k } j { 1... k } N correct { and S S φ} FN = = Θ Θ = S { 1... k } j { 1... k } N mssed a { and S S = φ} a m m j j June

31 5.1. Evaluaton Technques Precson Pr = N correct ecson P = 100% N + N correct false (P.12) where TP = N correct = Θ Θ = S { 1... k } j { 1... k } and FP = = Θ Θ = S j { 1... k } { 1... k } N false { S S φ} { and S S = φ} j a m m a j j June

32 5.1. Evaluaton Technques Utlty General Defnton Utlty = α Re call + (1 α ) Pr ecson (P.13) Varaton Used Utlty (Recall + Pr ecson) 2 = (P.14) June

33 5.2. S.P.E. Results Fg. P.13. Presents the epermentaton results for 45 mnutes of data. June

34 5.2. S.P.E. Results Fg. P.14. Presents the epermentaton results for 30 mnutes of data. June

35 5.3. S.A.E. Results Table P.2. Presents the fnal results of the second algorthm. June

36 5.3. S.A.E. Results Table P.3. Presents number of true postves false negatves false postves as well as recall precson and utlty for dfferent thresholds used n false postve detector of cut detector. June

37 5.3. S.A.E. Results Fg. P.15. Presents the recall and precson values for dfferent thresholds used n false postve detector of cut detector as well as the ROC curve for the second algorthm. June

38 5.3. S.A.E. Results Fg. P.16. Presents the utlty values for dfferent thresholds used n false postve detector of cut detector as well as the utlty curve for the second algorthm. June

39 5.4. Dscusson S.P.E. + Smplcty + Generalty + Fleblty + Etensblty - Real Tme - Hgh Qualty Detecton S.A.E. + Smplcty ~ Generalty + Fleblty ~ Etensblty + Real Tme + Hgh Qualty Detecton Table P.4. Presents the tme performance of the second algorthm for one mnute of data. June

40 Outlne 1. Overvew 2. Lterature Revew 3. Segmentaton based on Predefned Eamples (S.P.E.) 4. Segmentaton based on Adaptve Eamples (S.A.E.) 5. Epermentaton Results 6. Conclusons June

41 6. Conclusons 6.1. Summary 6.2. Future Works 6.3. Acknowledgements 6.4. References 6.5. Q & A Sesson June

42 6.1. Summary Two methods were mplemented and tested The frst one was based on lots of predefned eamples The second one was based on adaptve eamples The latter method outperformed the frst Our solutons drected all the problems of prevous works: Hgh Qualty Detecton Smplcty Real tme Generalty Fleblty Etensblty June

43 6.2. Future Works Future Enhancements Drect Comparson based on Predefned Eamples Drect Comparson based on Adaptve Eamples Net Generaton Algorthm Smultaneous detectons & use of specalzed clusterng methods to ncrease generalty June

44 6.3. Acknowledgements Dr. John Gauch (Thess Commttee Char) Dr. Arvn Agah (Thess Commttee Member) Dr. James Mller (Thess Commttee Member) June

45 6.4. References 1) Yeganeh Robert A Mult-Level Approach for Vdeo Shot Boundary Detecton based on Adaptve Eamples M.S. Thess KU Lawrence KS ) Refer to [1] for lst of references used durng our work. June

46 6.5. Q & A Sesson June