Deep Captioning with Multimodal Recurrent Neural Networks (m-rnn)

Transcription

1 Deep Captining with Multimdal Recurrent Neural Netwrks (m-rnn) Junhua Ma 1,2, Wei Xu 1, Yi Yang 1, Jiang Wang 1, Zhiheng Huang 1, Alan Yuille 2 1 Baidu Research 2 University f Califrnia, Ls Angeles a clse up f a bwl f fd n a table a train is traveling dwn the tracks in a city a pizza sitting n tp f a table next t a bx f pizza

2 Abstract Three Tasks: captin generatin retrieval (given query sentence) Sentence retrieval (given query image) One mdel (m-rnn): A deep Recurrent NN (RNN) fr the sentences A deep Cnvlutinal NN () fr the images A multimdal layer cnnects the first tw cmpnents State-f-the-art Perfrmance: Fr three tasks On fur datasets: IAPR TC-12 [Grubinger et al. 06 +, Flickr 8K [Rashtchian et al. 10 +, Flickr 30K *Yung et al and MS COCO *Lin et al. 14 +

3 The m-rnn Mdel Layers: Embedding I Embedding II Recurrent Multimdal SftMax w start w 1 The m-rnn mdel fr ne wrd w 1 w 2 w L w end w 1, w 2,, w L is the sentence descriptin f the image w start, w end is the start and end sign f the sentence

4 The m-rnn Mdel Layers: Embedding I Embedding II Recurrent Multimdal SftMax w start w 1 The m-rnn mdel fr ne wrd w 1 w 2 w L w end Detailed calculatin fr recurrent r(t) and multimdal layer m(t) r t = f(u r r t 1 + w t ), w t is the activatin f embedding layer II fr the wrd w t m t = g V w w t + V r r t + V I I, I is the image representatin + here means element-wise plus

5 The m-rnn Mdel Layers: Embedding I Embedding II Recurrent Multimdal SftMax w start w 1 The m-rnn mdel fr ne wrd w 1 w 2 w L w end Nn-linear activatin functins: Fr the recurrent layer: ReLU [Nair and Hintn 10 + f x = max 0, x Fr the wrd embedding layers and the multimdal layer: S-tanh [LeCun et.al 12 +: g x = 1.72 tanh( 2 3 x)

6 The m-rnn Mdel Layers: Embedding I Embedding II Recurrent Multimdal SftMax w start w 1 The m-rnn mdel fr ne wrd w 1 w 2 w L w end The utput f the trained mdel: P(w n w 1:n 1, I)

7 Applicatin captin generatin: Begin with the start sign w start Sample next wrd frm P w n w 1:n 1, I Repeat until the mdel generates the end sign w end

8 Applicatin captin generatin: Begin with the start sign w start Sample next wrd frm P w n w 1:n 1, I Repeat until the mdel generates the end sign w end retrieval given query sentence: Ranking scre: P w Q 1:L I D = L P w Q Q n=2 n w 1:n 1, I D Output the tp ranked images

9 Applicatin captin generatin: Begin with the start sign w start Sample next wrd frm P w n w 1:n 1, I Repeat until the mdel generates the end sign w end retrieval given query sentence: Ranking scre: P w Q 1:L I D = L P w Q Q n=2 n w 1:n 1, I D Output the tp ranked images Sentence retrieval given query image: Prblem: Sme sentences have high prbability fr any image query Slutin: Nrmalize the prbability. I are images sampled frm the training set: P w D 1:L I Q D P w D 1:L = P w D 1:L I P(I ) P w 1:L I

10 Applicatin captin generatin: Begin with the start sign w start Sample next wrd frm P w n w 1:n 1, I Repeat until the mdel generates the end sign w end retrieval given query sentence: Ranking scre: P w Q 1:L I D = L P w Q Q n=2 n w 1:n 1, I D Output the tp ranked images Sentence retrieval given query image: Prblem: Sme sentences have high prbability fr any image query Slutin: Nrmalize the prbability. I are images sampled frm the training set: P w D 1:L I Q D P w D 1:L = P w D 1:L I P(I ) P w 1:L I Equivalent t using a ranking scre: P I Q D w 1:L = P w D 1:L I Q P(I Q ) P wd 1:L

11 Experiment: Retrieval Table 1. Retrieval results n Flickr 30K and MS COCO Sentence Retrival ( t Text) Retrival (Text t ) R@1 R@5 R@10 Med r R@1 R@5 R@10 Med r Flickr30K Randm DeepFE-R (Karpathy et al. 14 ) RVR (Chen & Zitnick 14 ) MNLM-AlexNet (Kirs et al. 14 ) MNLM-VggNet (Kirs et al. 14 ) NIC (Vinyals et al. 14 ) / / 7 LRCN (Dnahue et al. 14 ) / / / / DeepVS-R (Karpathy et al. 14 ) Ours-m-RNN-AlexNet Ours-m-RNN-VggNet MS COCO Randm DeepVS-R (Karpathy et al. 14 ) Ours-m-RNN-VggNet R@K: The recall rate f the grundtruth amng the tp K retrieved candidates Med r: Median rank f the tp-ranked retrieved grundtruth (*) Results reprted n 04/10/2015. The deadline fr ur camera ready submissin.

12 Experiment: Captining Table 2. Captin generatin results n Flickr 30K and MS COCO Flickr30K MS COCO PERP B-1 B-2 B-3 B-4 PERP B-1 B-2 B-3 B-4 RVR (Chen & Zitnick 14 ) DeepVS-AlexNet (Karpathy et al. 14 ) DeepVS-VggNet (Karpathy et al. 14 ) NIC (Vinyals et al. 14 ) LRCN (Dnahue et al. 14 ) DMSM (Fang et al. 14 ) Ours-m-RNN-AlexNet Ours-m-RNN-VggNet B-K: BLEU-K scre PERP: Perplexity (*) Results reprted n 04/10/2015. The deadline fr ur camera ready submissin.

13 Experiment: Captining Table 4. Results n the MS COCO test set B1 B2 B3 B4 CIDEr ROUGE L METEOR Human-c5 (**) m-rnn-c m-rnn-beam-c Human-c40 (**) m-rnn-c m-rnn-beam-c c5 and c40: evaluated using 5 and 40 reference sentences respectively. -beam means that we generate a set f candidate sentences, and then selects the best ne. (beam search) (**) Prvided in (***) We evaluate it n the MS COCO evaluatin server:

14 Discussin

15 Discussin Other language: Chinese 一个年轻的男孩坐在长椅上 一列火车在轨道上行驶 一辆双层巴士停在一个城市街道上 We acknwledge Hayuan Ga and Zhiheng Huang frm Baidu Research fr designing the Chinese image captining system

16 Discussin Other language: Chinese 一个年轻的男孩坐在长椅上 A yung by sitting n a bench. 一列火车在轨道上行驶 A train running n the track. 一辆双层巴士停在一个城市街道上 A duble decker bus stp n a city street. We acknwledge Hayuan Ga and Zhiheng Huang frm Baidu Research fr designing the Chinese image captining system

17 Discussin Can we design a system that learns t describe new visual cncepts frm a few examples?

18 Discussin Can we design a system that learns t describe new visual cncepts frm a few examples? Learning like a Child: Fast Nvel Visual Cncept Learning frm Sentence Descriptins f s, arxiv Efficiently enlarge the vcabulary Needs nly a few images with nly a few minutes Datasets fr evaluatin

19 Fr mre details, please visit the prject page: The updated versin f ur paper: J. Ma, W. Xu, Y. Yang, J. Wang, Z. Huang, A. Yuille, "Deep Captining with Multimdal Recurrent Neural Netwrks (m-rnn)", arxiv: The nvel visual cncept learning paper: J. Ma, W. Xu, Y. Yang, J. Wang, Z. Huang, A. Yuille, "Learning like a Child: Fast Nvel Visual Cncept Learning frm Sentence Descriptins f s", arxiv: a yung girl brushing his teeth with a tthbrush a grup f peple flying kites in a field a man is ding a trick n a skatebard

20 Appendix B-1 B-2 B-3 B-4 m-rnn m-rnn-nembinput m-rnn-onelayeremb m-rnn-emboneinput Perfrmance cmparisn with different wrd-embedding cnfiguratin

21 Appendix B-1 B-2 B-3 B-4 m-rnn m-rnn-visinrnn m-rnn-visinrnn-bth m-rnn-visinrnn-bth-shared Perfrmance cmparisn with different image representatin input methds

22 Table 3. Recurrent layer size and whether LSTM is used MNLM NIC LRCN RVR DeepVS Our m-rnn Size (x4) LSTM Yes Yes Yes N N N