ISSUE 2: TEMPORAL DEPENDENCY?

Size: px

Start display at page:

Download "ISSUE 2: TEMPORAL DEPENDENCY?"

Logan Kennedy
5 years ago
Views:

1 DAR DAR 8/2/18 1

2 DAR q Interpretation ISSUE 2: TEMPORAL DEPENDENCY? Only linear activation function, unit variance p(o 1:2 x 1:2 )= 1 2 exp( 1 2 (o µ)> 1 (o µ)) DAR o 1 o 2 SAR o 1 a o 2 o =[o 1,o 2 ] > RMDN o 1 o 2 µ 1 w m h 1 w o W h h 2 w m µ 1 µ 2 h 1 h 2 µ 1 h 1 w µ h 2 W i x 1 x 2 x 1 x 2 x 1 x 2 µ =[µ 1, µ 2 + w > mw o µ 1 ] > µ 2 = w > m(w h h 1 + W i x 2 ) apple 1 w = > mw o w > mw o 1+(w > mw o ) 2 µ =[µ 1,µ 2 + aµ 1 ] > = apple 1 a a 1+a 2 µ =[µ 1, µ 2 + w µ µ 1 ] > µ 2 = w > h 2 + b apple 1 = 1 2

3 DAR q Interpretation ISSUE 2: TEMPORAL DEPENDENCY? Non-linear activation function + longer sequence DAR o o 1 o 2 p(o 2 o :1, x :2 )=N(o 2 ; µ 2, 1) x w o W h h 1 x 1 x 2 µ 2 where µ 2 = w > m (W h h 1 + W i x 2 + w o o 1 ) w m W i o 2 h 1 non-linearly depends on o non-linearly depends on o o 1 3

4 Proposed model P (o 1:T ; M 1:T )= TY P (o 1:T o 1:t 1 ; M 1:T ) syllable tier phoneme tier t=1 o 1 o 2 o 3 o 4 o 5 frame tier Quantized F Hierarchical softmax M 1 M 2 M 3 M 4 M 5 Data dropout x 1 x 2 x 3 x 4 x 5 8/2/18 4

5 Proposed model syllable phoneme phoneme syllable tier phoneme tier o 1 o 2 o 3 o 4 o 5 frame tier M 1 M 2 M 3 M 4 M 5 a p(t 1) = P t 1 i=t p(t 1) tanh(o i ) t t p(t 1) x 1 x 2 x 3 x 4 x 5 5

6 Generation l Generation by picking one-best 1. Given, compute 2. Pick bo 1:t 1 3. Feedback bo t P (o t bo 1:t 1 ; M t ) bo t = arg max o t P (o t bo 1:t 1 ; M t ) bo 1 M 1 M 2 l Generation by random sampling 1. Given, compute 2. Draw sample from 3. Feedback bo 1:t 1 bo t bo t P (o t bo 1:t 1 ; M t ) P (o t bo 1:t 1 ; M t ) x 1 x 2 8/2/18 6

7 Why not interpolated F? 4 NAT DAR continuous F model (English data) F (Hz) Frame index (utterance BC211 nancy APDC2-166-) NAT DAR continous F model F (Hz) Frame index (utterance ATR Ximera F9 AOZORAR 3372 T1) RMSE CORR U/V error Baseline RMDN % DAR quantized F % DAR continuous F %

8 Hierarchical softmax l Plain softmax : P (o t v t )= e(v t,j) P N i= e(v t,i), where o t,j =1,j 2 {,,N} unvoiced... F level 1 F level N l Two-level hierarchical softmax : P (o t v t )= 8 e >< (v t,n+1 ), if o 1+e (v t,n+1 ) t,n+1 =1 >: 1 e(v t,j ) 1+e (v t,1 ) P N, i=1 e(v t,i ) if o t,j =1,j 2 {1,,N} e (v t,n+1) 1+e (v t,n+1) unvoiced 1 1+e (v t,n+1) e(v t,j ) P N i=1 e(v t,i) o t,n+1 =1 denotes being unvoiced F level 1... F level N

9 Generation method l One-best: j = arg max j e h t,j P N k=1 eh t,k 1 1+e h t,n+1 e h t,n+1 1+e h t,n+1 output = F j, the value for F level j voiced unvoiced l Soft-generation output = NX j=1 e h t,j P N k=1 eh t,k F j F symbol 1 e h t,j P N k=1 eh t,k... F symbol N

10 Why hierarchical softmax l Due to implementation P (o t v t )= e(v t,j) P N i= e(v t,i), where o t,j =1,j 2 {,,N} unvoiced... F level 1 F level N If U/V is determined at first (for soft-gen), how to set the threshold? if e(v t, ) P N i= e(v t,i) > Threshold, then it is unvoiced If directly picking the one-best, it is error-prone (see the results later) Only random sampling may work!

11 Why hierarchical softmax l Also due to optimization Plain softmax: Suppose trained by minimizing the cross entropy E t = NX j= (o t == j) log p t (j), where p t (j) = e v t,j P N i= ev t,i We know the t,k = ( p t (k), if o t 6= k p t (k) 1, if o t == t,k unvoiced F level 1... F level K...

12 Why hierarchical softmax l Also due to optimization Hierarchical softmax: Suppose trained by minimizing the cross entropy E t = NX j= (o t == j) log p t (j) = (o t == ) log p t () (o t > ) log [1 p t ()] where p t (j) = NX j=1 e v t,j P N i=1 ev t,i (o t == j) log p t (j) We know the gradient for k > : ><, if o t == t = t (k), if o t > and o t 6= k t,k >: p t (k) 1, if o t == k Unvoiced frames don t affect the parameters for F levels unvoiced F level t,k F level k

13 Why dropout l Related to bits-back coding [1] and information theory o 1 o 2 o 3 o 4 o 5 M 1 M 2 M 3 M 4 M 5 Data dropout x 1 x 2 x 3 x 4 x 5 If p(o 1:T ) is powerful, no need to assume p(o [2,3] 1:T x 1:T ) To use, we can make the model weaker x 1:T Word dropout is also used for language modeling [2] [1] A. Honkela and H. Valpola. Variational learning and bits-back coding: an information-theoretic view to bayesian learning. IEEE Transactions on Neural Networks, 15(4):8 81, 24. [2] S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. Generating sentences from a continuous space. In Proc. CoNLL, pages 1 21, 216. [3] X. Chen, D. P. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, I. Sutskever, and P. Abbeel. Variational lossy autoencoder. arxiv preprint arxiv: , 216.

14 Compare soft-generation F events F events 1 1 Figure for NF Figure for F CL Probability for each Flevel at each frame Frame index (utterance BC211_nancy_APDC2-166-) 1-best generation method F (Hz) Natural BS NF 1 F CL Frame index (utterance BC211_nancy_APDC2-166-)

15 Compare soft-generation F events F events 1 1 Figure for NF Figure for F CL Probability for each Flevel at each frame Frame index (utterance BC211_nancy_APDC2-166-) soft generation method F (Hz) Natural BS NF 1 F CL Frame index (utterance BC211_nancy_APDC2-166-) 3 4 4

16 Results on English data l Using one-best generation Hierarchical softmax layer Plain softmax layer Model Training strategy RMSE CORR. U/V ERR. RMSE CORR. U/V ERR. QN % % QF FT dropout (P s =.75) % % QF FT dropout (P s =.) % % QF FT dropout (P s =5) % % QF FT no dropout (P s =.) % % QF FT schedule sampling (P s =.) % F (Hz) NAT RNN-MLPG QF DAR-QF Frame index (utterance BC211_nancy_APDC2-166-) 8/2/18 16

17 Results (English) l Samples Preference: 33% VS 67% (p=.1) Preference: 37% VS 63% (p=.1) QF RNN-MLPG SAR-RMDN DAR-QF Natural 17

18 Results on English data l Global variance GV of F NAT RNN RMDN RNN-MLPG SAR-RMDN QF DAR-QF 8/2/18 18

19 Results on English data l Probability of F symbols for each frame F symbol (-127) F symbol (-127) F symbol (-127) Figure for QF Figure for DAR_QF (dropout P Frame index (BC211_nancy_APDC2-166-) s =.5) Figure for DAR_QF (no dropout) Frame index (BC211_nancy_APDC2-166-) Frame index (BC211_nancy_APDC2-166-) feedback sharpens the F distribution 8/2/18 19

Results on English data l Probability of F symbols for t = F symbol (-127) F symbol (-127) F symbol (-127) 1 1 1 Figure for QF

probability of F event.5.3 1 2 3 4 6 7 8 9 Figure for DAR_QF (no dropout) Frame index (BC211_nancy_APDC2-166-).

20 Results on English data l Probability of F symbols for t = F symbol (-127) F symbol (-127) F symbol (-127) Figure for QF Figure for DAR_QF (dropout P Frame index (BC211_nancy_APDC2-166-) s =.5) probability of F event Figure for DAR_QF (no dropout) Frame index (BC211_nancy_APDC2-166-).1 QF DAR_QF (dropout P s =.5) DAR_QF (no dropout) Frame index (BC211_nancy_APDC2-166-) F symbol (-127) dropout blunts the distribution 8/2/18 2

21 Results on English data l Generation by random sampling F (Hz) NAT SAMPLE F (Hz) NAT SAMPLE F (Hz) NAT SAMPLE

22 F symbol (-127) F symbol (-127) F symbol (-127) Results on English data l Generation by random sampling Figure for random sampling 1 1. Given, compute 2. Draw sample bo t from 3. Feedback bo 1:t 1 P (o t bo 1:t 1 ; M t ) P (o t bo 1:t 1 ; M t ) bo Figure for random sampling t 2 Frame index (BC211_nancy_APDC2-166-) Figure for random sampling 3 Frame index (BC211_nancy_APDC2-166-) bo 1 M 1 M 2 x 1 x Frame index (BC211_nancy_APDC2-166-) /2/18 22

23 Results on English data l Interesting randomness Where did you find the report? H* L-H% Economist, October eleventh What's the date of that issue? L* H-H% Economist? October eleventh 8/2/18 23

24 Results on Japanese data Note that RNNQ and DAR used soft-generation method. P d is probability for data dropout; Log-likelihood is calculated on development set; GV denotes utterance-level F global variance. GV of natural F is around 61 Hz. P d Log-likelihood (dev. set) RMSE CORR U/V GV RNN % 51.4 RMDN % 54.2 SAR % 71.8 RNNQ % % 58.6 DAR % % % /2/18 24

25 Results on Japanese data Results of objective evaluation on sampled F trajectories from DAR. The model generated F contours by random sampling for three rounds. Linguistic features Sampling rounds RMSE CORR U/V GV 1st round % 6 Full set linguistic features 2nd round % 6 3rd round % 6.9 1st round % 61.4 Without pitch accents 2nd round % rd round % /2/18 25

26 H-softmax DAR q Empirical analysis F H-softmax linear un-lstm bi-lstm F linear bi-lstm bi-lstm input o t 1 v F levels F levels NAT Sample. input Frame index Frame index 256 Probability of quantized Fs Frame index All layers use tanh activation function F (Hz) F (Hz) Randomly sampled F contour 1 NAT Sample Frame index

27 H-softmax DAR q Empirical analysis F H-softmax linear un-lstm bi-lstm F linear bi-lstm bi-lstm input o t 2 o t 1 v F levels F levels NAT Sample. input Frame index Frame index 256 Probability of quantized Fs Frame index All layers use tanh activation function F (Hz) F (Hz) Randomly sampled F contour 1 NAT Sample Frame index

28 H-softmax DAR q Empirical analysis F H-softmax linear un-lstm bi-lstm 1 NAT Sample. input Frame index Frame index F linear bi-lstm bi-lstm input o t 3 o t 2 o t 1 v F levels F levels Probability of quantized Fs Frame index All layers use tanh activation function F (Hz) F (Hz) Randomly sampled F contour 1 NAT Sample Frame index

29 DAR q Empirical analysis Objective measure on randomly generated results F H-softmax linear F H-softmax linear F H-softmax linear F o t 1 H-softmax linear F o t 2 o t 1 H-softmax linear o t 3 o t 2 o t 1 un-lstm bi-lstm bi-lstm bi-lstm bi-lstm bi-lstm bi-lstm bi-lstm bi-lstm bi-lstm input input input input input DAR CORR RMSE (HZ)

30 Results on Japanese data F H-softmax linear o t Probability of quantized Fs 1. 4 Randomly sampled F contour linear bi-lstm bi-lstm input F H-softmax linear bi-lstm bi-lstm input o t 1 v F levels F levels Frame index Frame index All layers use tanh activation function F (Hz) F (Hz) NAT Sample Frame index NAT Sample Frame index 6 7 3

31 Results on Japanese data F H-softmax linear un-lstm bi-lstm 1 NAT Sample. input Frame index Frame index F H-softmax F levels Probability of quantized Fs Randomly sampled F contour F (Hz) WaveNet F levels.5 F (Hz) 3 2 input Frame index. 1 NAT Sample Frame index 6 7 v All layers use tanh activation function 31

32 Results on Japanese data F H-softmax linear uni-lstm bi-lstm input LSTM cell weight s 32

33 Results on Japanese data q Merge information from multiple tiers RMSE Corr. RNN RMDN DAR RMSE Corr. Log-likelihood (dev. set) Frame Frame + Phoneme Frame + Mora Frame + Word Frame + Accent phrase Frame + Breath group

34 ISSUE 2: TEMPORAL DEPENDENCY? Experiment q Mean-based generation Log-Likelihood (validation set) RMSE CORR U/V err Global variance (GV) natural=61 RNN % 51.4 DAR RMDN % 54.2 SAR RMDN RNN SAR % 71.8 esar % 63.4 WaveNetF % 6 DAR % GV of F at utterance-level (Hz) Natural RNN RMDN SAR esar WaveNet-F DAR Natural RNN RMDN SAR esar WaveNet -F DAR 34

35 ISSUE 2: TEMPORAL DEPENDENCY? Experiment q Mean-based generation RMSE CORR U/V err Global variance (GV) natural=61 WaveNetF DAR % 6 DAR 28.3 SAR RMDN.93 RNN 3.46% 61.5 #. Parameter Training (s/epoch) Generation (ms/frame) WaveNetF 3.28 m DAR 1.48 m

36 ISSUE 2: TEMPORAL DEPENDENCY? Experiment q Random sampling Natural WaveNet Sample 1 F (Hz) Natural WaveNet Sample 2 F (Hz) Natural WaveNet Sample 3 F (Hz) Frame index (ATR Ximera F9 AOZORAR 3372 T1) 6

37 Results on Japanese data q Error analysis Transplant the manual annotation to OpenJTalk format Objective measures using the same model OpenJTalk Manually corrected Random sampling Softgeneration CORR RMSE (HZ) CORR RMSE (HZ)

38 BEYOND DAR Error analysis on DAR q Amenable errors: grapheme-to-phoneme やがて不意を衝かれて我を失った状態が過ぎると初めに湧き上がってきたのは喜びの感情だった Original linguistic features 我 -> が Manually corrected 我 -> われ 38

39 BEYOND DAR Error analysis on DAR q Amenable errors: (some) pitch-accents ライフワークとしてふるさと沖縄の戦禍の記憶を木版に刻み続けてきた Original linguistic features * ふるさとおきなわ Manually corrected * 平板ふるさとおきなわ 39

40 BEYOND DAR Error analysis on DAR q Obstinate errors 当時 N 子は長田区の病院付属保育所で保母さんをしていた Original linguistic features Manually corrected Natural q Using manually corrected test set: Original linguistic features Manually corrected CORR v Test on the same DAR 4

41 Summary q SAR and esar ISSUE 2: TEMPORAL DEPENDENCY? Theoretically tractable and appealing SAR is better than RMDN for F modeling on English data RMDN SAR q DAR Better results (F CORR >.9) Random sampling (the only neural F model) 41

42 DAR Improve DAR??? 8/2/18 42

43 CONCLUSION Remaining gaps q Noisy linguistic features Incorrect phonemes Incorrect pitch accent & combination DAR RMSE CORR U/V Noisy linguistic features % Manual annotated features % bo 1 bo 2 bo 3 x 1 x 2 x 3 bo T x T *... * ツツ... ツツギ... ギワ ts ts... u u g... i w が? われ? 不意を衝かれて我を失った状態 New topic? q Duration + F modeling q SAR (segment-variant + log area ratio) for waveform modeling 43

44 Schedule sampling l Flaw of the theoretical analysis Schedule sampling assumes independence across frames o 1 o 2 o 1 o 2 M 1 M 2 M 1 M 2 or x 1 x 2 x 1 x 2 Left: Right: D KL [D(o 1, o 2 ) P (o 1 )P (o 2 o 1 )] D KL [D(o 1, o 2 ) P (o 1 )P (o 2 )] Data distribution Model distribution From

45 Schedule sampling l Experiment Along with the training epochs feedback soft vector during generation F events F events F events Figure for e9 Figure for e13 Figure for e15 F events Figure for e18 1 F events Figure for e2 of schedule sample F events F events Figure for e22 Figure for e epoch F events Figure for e Frame index (BC211_nancy_APDC2-166-)

46 of schedule sample Schedule sampling l Experiment 1 Along with the training epochs feedback sampling result during generation epoch F events F events F events F events F events F events F events F events Figure for e9 Figure for e13 Figure for e15 Figure for e18 Figure for e2 Figure for e22 Figure for e25 Figure for e Frame index (BC211_nancy_APDC2-166-)

47 F MODEL DAR q Reward Augmented Maximum Likelihood Likelihood Z Ground truth y 1:T (y 1:T, y 1:T ) log p(y 1:T x 1:T ; ) = log p(y 1:T x 1:T ; ) Generated candidates Only use ground truth to train Reward augmented Z y 1:T q(y 1:T y 1:T ; ) log p(y 1:T x 1:T ; ) Empirical distribution based on reward for each candidate q(y 1:T y c(y 1:T ; ) = 1:T, y P 1:T ; ) y c(y 1:T 1:T, y 1:T ; ) 47

48 DAR q Reward Augmented Maximum Likelihood Experiments: F MODEL Step1: for the m-th training sentence 1:T, generate random samples {by (m,1) 1:T,, by(m,) 1:T } x (m) Step2: calculate the reward distribution for the m-th training sentence (using the random samples for that sentence) c(by (m,i) 1:T, y (m) 1:T ; ) =exp(corr(by(m,i) 1:T, y (m) 1:T )/ ) q(by (m,i) 1:T, y (m) 1:T ; ) = c(by (m,i) P 1:T, y (m) 1:T ; ) j=1 c(by(m,j) 1:T, y (m) 1:T ; ) Step3: suppose we use training sentences, we train = arg max X X m=1 i=1 q(by (m,i) 1:T y (m) 1:T ; ) log p(by(m,i) 1:T x 1:T ; ) 48

49 F MODEL DAR q Reward Augmented Maximum Likelihood U/V err RMSE CORR GV DAR 3.46% DAR RAML (initializedby DAR above) 3.8%

50 DAR DAR For MGC 8/2/18

51 EXTENSION DAR for MGC l Not work so well (Further tuning?) MGC 1 dim NAT RNN DAR DAR dropout Frame index (utterance BC211 nancy APDC2-166-) NAT RNN DAR DAR dropout.1 MGC 3 dim Frame index (utterance BC211 nancy APDC2-166-)

Autoregressive Neural Models for Statistical Parametric Speech Synthesis

Autoregressive Neural Models for Statistical Parametric Speech Synthesis シンワン Xin WANG 2018-01-11 contact: wangxin@nii.ac.jp we welcome critical comments, suggestions, and discussion 1 https://www.slideshare.net/kotarotanahashi/deep-learning-library-coyotecnn