Considering Granularity of Laser Pointer and Speech in Information Integration for Lecture Scene Retrieval

Size: px

Start display at page:

Download "Considering Granularity of Laser Pointer and Speech in Information Integration for Lecture Scene Retrieval"

Winifred Curtis
5 years ago
Views:

1 DEWS2007 E1-3,, wnakano@de.cs.titech.ac.jp, tkobaya@gsic.titech.ac.jp,, naoi.satoshi@jp.fujitsu.com, {yokota,furui}@cs.titech.ac.jp UPRISE e-learning Considering Granularity of Laser Pointer and Speech in Information Integration for Lecture Scene Retrieval Wataru NAKANO, Takashi KOBAYASHI, Satoshi NAOI,, Haruo YOKOTA,, and Sadaoki Abstract FURUI Department of Computer Science, Graduate School of Information Science and Engineering, Tokyo Institute of Technology Ookayama , Meguro-ku, Tokyo , Japan Global Scientific Information and Computing Center, Tokyo Institute of Technology Ookayama , Meguro-ku, Tokyo , Japan FUJITSU LABORATORIES LTD. Kamikodanaka 4 1 1, Nakahara-ku, Kanagawa, Japan wnakano@de.cs.titech.ac.jp, tkobaya@gsic.titech.ac.jp,, naoi.satoshi@jp.fujitsu.com, {yokota,furui}@cs.titech.ac.jp Unified presentation contents are widely used for e-learning and/or e-training. There is a strong need for efficient search mechanism for unified presentation contents. We have proposed a unified presentation contents search mechanism named UPRISE (Unified Presentation Slide Retrieval by Impression Search Engine), and have also proposed a method to use laser pointer and speech information in lecture scene retrieval. In this paper, we propose a method to integrate between laser pointer and speech information by considering the granulity of both two information. We also analyze the relationship between each laser pointer shot and query word for several cases, and improve the proposed method using the result of analysis. We evalutate our approach using actual lecture videos and presentation slides. Key words Information Integration, e-learning, Information Retrieval, Contents Processing

2 1. [1] [4] e-learning e-learning UPRISE(Unified Presentation Slide Retrieval by Impression Search Engine) [5] UPRISE UPRISE [6] [7] [8] [8] 2 [8] UPRISE [9], [10] [4] SEMP LODEM [11] [12] [13] 3. UPRISE UPRISE UPRISE 3. 1 UPRISE UPRISE 1

Movie Stream Metadata Pointing Information Speech change Presentation Slides 1 scene i scene i+1 scene i+2 xxx xx xxxxx xx slide k change xx xx change slide k+1 xxx xxx xx xxxx xxxxxxx slide

2. 3 I c I c I d change I c (s, k, θ, δ, ε 1, ε 2 ) = s+δ γ=s δ E(γ s, ε 1, ε 2 ) I d (γ, k, θ) δ E(x, ε 1, ε 2 ) E(x, ε 1, ε 2 ) exp(ε 1 x) (x < 0) E(x, ε 1, ε 2 ) = exp( ε 2 x) (x > = 0) I c

3 Movie Stream Metadata Pointing Information Speech change Presentation Slides 1 scene i scene i+1 scene i+2 xxx xx xxxxx xx slide k change xx xx change slide k+1 xxx xxx xx xxxx xxxxxxx slide k+2 UPRISE [14] UPRISE 3. 2 UPRISE I c I c 3 I c I p I d I c I p I p L(s) I p (s, k) = P(s, l) C(s, k, l) l=1 s k l P(s, l) s l C(s, k, l) s l k P(s, l) I d I d I p I d (s, k, θ) = T(s) θ I p (s, k) T(s) s θ I c I c I d change I c (s, k, θ, δ, ε 1, ε 2 ) = s+δ γ=s δ E(γ s, ε 1, ε 2 ) I d (γ, k, θ) δ E(x, ε 1, ε 2 ) E(x, ε 1, ε 2 ) exp(ε 1 x) (x < 0) E(x, ε 1, ε 2 ) = exp( ε 2 x) (x > = 0) I c δ ε δ = 4 ε 1 = 5.0 ε 2 = I c (s, k, θ, δ, ε 1, ε 2 ) UPRISE Φ 3. 3 [7] [7] s k skc(s, k) (Spoken Keyword Count) skc(s, k) I c 4. [6] [8] 4. 1 [15]

4 1 H(l, q) H(l, q) q l H(l, q) 1 phd(s, k) (Pointer Hit Duration) s k s k phd(s, k) L(s) phd(s, k) = H(l, q i ) T(q i ) q i s l=1 T(q i ) q i q i 4. 2 I c 4. 1 phd I c T(s) phd(s, k) I c[d+phd] [6] I c[d+phd] I c[d+phd] (Φ, ω d ) = s+δ γ=s δ E(γ s, ε 1, ε 2 ) I d[d+phd] (γ, k, θ, ω d ) I d[d+phd] (s, k, θ, ω d ) = {T(s) + ω d phd(s, k)} θ I p (s, k) ω d phd ω d = [8] [8] 2 phd(s, k) phd /p (s, k) phd(s, k) k K I p (s, k) 0 phd /p (s, k) = 0 k K I p (s, k) = 0 K 1 phd(s, k) phd /s (s, k) phd(s, k) skc(s, k) 0 phd /s (s, k) = 0 skc(s, k) = 0 P- S- [8] P- S- phd(s, k) phd /ps (s, k) I c[d+phd] phd /ps (s, k) I c[d+phd/ps,p+skc /p ] 5. S S- S- NS- NS NS S- 2 r 1, r 2

5 a laser pointer shot t 1-r 1 t 1 t 2 t 2+r 2 time この時間帯にキーワードが発話されているかどうかを調べる. 発話されていないならば, この照射を無視する. 2 S- 2 NS- NS- phd(s, k) phd /ns (s, k) phd /ns (s, k) L(s) phd /ns (s, k) = H(l, q i ) T(q i ) exist(q i, k, r 1, r 2 ) q i s l=1 exist(q i, k, r 1, r 2 ) q i r 1, r 2 k 1 0 NS-4. 3 P- phd(s, k) phd /pns (s, k) phd /pns (s, k) K phd /ns (s, k) phd /pns (s, k) = 0 k K I p (s, k) 0 k K I p (s, k) = 0 phd /ns phd /pns I c[d+phd] [8] skc I c[d+phd/ns,p+skc /p ] I c[d+phd/pns,p+skc /p ] NS- UPRISE ( 11 ) ( 12 ) Julius 1 [16] [16] 1 Φ θ = 0.4 δ = 4 ε 1 = 5.0 ε 2 = 0.5 ψ 1 5 H(l, q) phd ω d UPRISE 1 (Mean reciprocal rank:mrr) MRR [17] MRR N MRR = 1 N N i= i I c[d+phd/ns,p+skc /p ] I c[d+phd/pns,p+skc /p ] I c[d+phd/s,p+skc /p ] I c[d+phd/ps,p+skc /p ] I c NS- r 1, r 2 {r 1, r 2 } = {5, 5} 3 2 MRR NS- ω d S-NS- P- MRR

6 Mean reciprocal rank Mean reciprocal rank Ic Ic[d+phd/s,p+skc/p] Ic[d+phd/ps,p+skc/p] Ic[d+phd/ns,p+skc/p] Ic[d+phd/pns,p+sk/p] ωd ω d Ic {5, 5} {10, 0} {0, 10} ωd I c[d+phd/pns,p+skc /p ] r 1, r 2 I c[d+phd/pns,p+skc /p ] NS- r 1, r 2 {r 1, r 2 } = {10, 0} 4 {r 1, r 2 } = {5, 5} {10, 0}, {0, 10} 3 MRR 4 {r 1, r 2 } = {0, 10} 2 MRR {r 1, r 2 } = {0, 10} NS- S- NS- NS- NS P- S- phd /ps 2 phd /ps 1 DB ps pns(5,5) pns(10,0) ARCH ps pns(5,5) pns(10,0) ( ) ( ) ( ) ( ) 4 ( ) ( ) ( ) S- NS- NS- r 1, r 2 {r 1, r 2 } = {5, 5} {10, 0} 2 1 P-+ S- P- + NS- P- S- S- NS-

7 2 ( ) DB ps pns(5,5) pns(10,0) pns(5,5) pns(10,0) +title +title ARCH ps pns(5,5) pns(10,0) pns(5,5) pns(10,0) +title +title Mean reciprocal rank Ic[d+phd/pns,p+sk/p] (5,5) Ic[d+phd/pns,p+sk/p] (10,0) Ic[d+phd/pns,p+skc/p] (5,5) +title Ic[d+phd/pns,p+skc/p] (10,0) +title ωd 6. 2 NS NS- 3 P-+ NS- P- + S- S/NS- 2 S/NS- 1 ARCH NS- 80% S/NS- 25% Mean reciprocal rank Ic MRR Ic[d+phd/ps,p+skc/p] Ic[d+phd/pns,p+skc/p] (5,5) +title Ic[d+phd/pns,p+skc/p] (10,0) +title ωd I c[d+phd/pns,p+skc /p ] MRR I c[d+phd/pns,p+skc /p ] MRR 6. 2 S/NS- 6 S/NS- I c[d+phd/pns,p+skc /p ] I c I c[d+phd/ps,p+skc /p ] 6 MRR NS UPRISE NS-

8 NS- NS- S- NS- S/NS- UPRISE NS- {r 1, r 2 } 100% [7], [18] Julius ( , , ) CREST 21 COE [1] R. Müller and T. Ottmann. The Authoring on the Fly system for automated recording and replay of (tele)presentations. Multimedia Syst., Vol. 8, No. 3, pp , [2] Alexander G. Hauptmann and Michael J. Witbrock. Informedia: news-on-demand multimedia information acquisition and retrieval. In M. T. Maybury, editor, Intelligent multimedia information retrieval, pp MIT Press, [3] Maria da Graca Pimentel, Yoshihide Ishiguro, Gregory D. Abowd, Bolot Kerimbaev, and Mark Guzdial. Supporting educational activities through dynamic web interfaces. Interacting with Computers, special issue on interacting with the active Web, Vol. 13, No. 3, pp , February [4],,.. D-I, Vol. J88-D-I, No. 10, pp , [5] Haruo Yokota, Takashi Kobayashi, Taichi Muraki, and Satoshi Naoi. UPRISE: Unified Presentation Slide Retrieval by Impression Search Engine. IEICE Trans. on Info. and Syst., Vol. E87-D, No. 2, pp , [6] W. Nakano, Y. Ochi, T. Kobayashi, Y. Katsuyama, S. Naoi, and H. Yokota. Unified Presentation Contents Retrieval Using Laser Pointer Information. In Proc. of SWOD2005, pp , [7],,,,,,.., Vol. J90-D, No. 2, pp , [8] Wataru Nakano, Takashi Kobayashi, Yutaka Katsuyama, Satoshi Naoi, and Haruo Yokota. Treatment of laser pointer and speech information in lecture scene retrieval. In Proc. of Eighth IEEE Intl. Symp. on Multimedia, pp , [9],,.. Technical Report 2005-SLP ,, [10] Alessandro Vinciarelli and Jean-Marc Odobez. Application of information retrieval technologies to presentation slides. IEEE Trans. on Multimedia, Vol. 8, No. 5, pp , [11] Atsushi Fujii, Katunobu Itou, and Tetsuya Ishikawa. Lodem: A system for on-demand video lectures. Speech Communication, Vol. 48, No. 5, pp , [12] MediaDEPO. [13],,,,,.., Vol. 39, No. 10, pp , Oct [14],,,.., Vol. J88-D-I, No. 3, pp , [15] Yutaka Katsuyama, Noriaki Ozawa, Jun Sun, Hiroaki Takebe, Takashi Kobayashi, Haruo Yokota, and Satoshi Naoi. A New Solution for Extracting Laser Pointer Information from Lecture Videos. In Proc. of E-learn2004, pp , [16],,,,.., 2006-SLP-64-39, pp , [17] E. M. Voorhees and D. M. Tice. The trec-8 question answering track evaluation. In Proc. of TREC-8, pp , [18] Haruo Yokota, Takashi Kobayashi, Hiroaki Okamoto, and Wataru Nakano. Unified contents retrieval from an academic repository. In Proc. of International Symposium on Large-scale Knowledge Resources LKR2006, pp , 2006.

Unied Presentation Contents Retrieval using Voice Information

Unied Presentation Contents Retrieval using Voice Information DEWS2006 6c-o1,, E-mail: {okamoto,wnakano}@decstitechacjp, tkobaya@gsictitechacjp,, naoisatoshi@jpfujitsucom, {yokota,iwano,furui}@cstitechacjp UPRISE (Unified Presentation slide Retrieval by Impression