Multi-Task Minimum Error Rate Training for SMT

Transcription

1 Minimum Error Rate Training for SMT Patrick Katharina Stefan Department of Computational Linguistics University of Heielberg, Germany

2 Learning Multi-task learning aims at learning several ifferent tasks simultaneously, aressing commonalities through share parameters an moeling ifferences through task-specific parameters. Preestine application: Patent translation over classes of patents w.r.t. International Patent Classification (IPC) commonalities: highly specialize legal jargon not foun in everyay language, rigi textual structure incluing highly formulaic language. ifferences: technological terminology specific to IPC class.

3 IPC Sections A Human Necessities B Performing Operations; Transporting C Chemistry; Metallurgy D Textiles; Paper E Fixe Constructions F Mechanical Engineering; Lighting; Heating; Weapons; Blasting G Physics H Electricity

4 Goal an Approach Goal: Learn a translation system that performs well across several ifferent patent sections, thus benefits from share information, an yet is able to aress the specifics of each patent section. Approach: Machine learning approach to traing off optimality of parameter vectors for each task-specific moel an closeness of these moel parameters to average parameter vector across moels.

5 Minimum Error Rate Training Assume specific setting: Not enough ata for training generative SMT pipeline on all tasks, however, enough ata for tuning for each specific task. In other wors: How much gain is there in extening the stanar tuning technique of minimum error rate training () to multi-task for SMT. Also apply techniques for parameter averaging from istribute learning to a version of average.

6 Parallel Patent Data MAREC: 19 million patent applications an grante patents, stanarize format from four patent organizations (European Patent Office (EP), Worl Intellectual Property Organisation (WO), Unite States Patent an Traemark Office (US), Japan Patent Office (JP)), from 1976 to Extract bilingual abstract an claims sections from the EP an WO parts for German-to-English translation. Sentence splitting an tokenizing with Europarl tools 1. Sentence alignment with Gargantua 1.0b

7 Distribution of IPC sections for e-en abstracts an claims A 266, % B 384, % C 372, % D 50, % E 54, % F 149, % G 291, % H 228, %

8 Parallel ata for e-en patent translation train ev evtest test # parallel sents 1M 2K 2K 2K avg. # tokens e 32,329,745 59,376 60,061 59,930 avg. # tokens en 36,005,763 69,584 70,700 70,331 year

9 Multi-task learning objective Objective: Minimize task-specific loss functions l uner regularization of task-specific parameter vectors w towars an average parameter vector w avg. min w 1,...,w D D D l (w ) + λ w w avg p p (1) =1 =1

10 Multi-task preiction Preiction: Task-specific weight vectors w {w 1,..., w D } that have been ajuste to trae off task-specificity (small λ) an commonality (large λ). or: Average weight vector w avg as a global moel.

11 Average Avg(w (0), D, {c } D =1 ): for = 1,..., D parallel o for t = 1,..., T o w (t) = (w (t 1), c (w )) en for en for return w avg = 1 D D =1 w (T ) Apply ieas from istribute learning (Zinkevich et al. NIPS 10) by basing the istribution strategy on task-specific partitions of ata.

12 Multi-task regularization: Set p=1 in equation 1 to obtain an l 1 regularizer. clipping: Weight vector w is move towars the average weight vector w avg by aing or subtracting the penalty λ for each weight component w [k], an clippe when it crosses the average. coe: Script wrapper aroun the implementation of Bertoli et al. 2009; license unter the LGPL; online at

13 Multi-task M(w (0), D, {c } D =1 ): for t = 1,..., T o w avg (t) = 1 D D =1 w (t 1) for = 1,..., D parallel o w (t) = (w (t 1), c (w )) for k = 1,..., K o if w[k] (t) w avg[k] (t) > 0 then w (t) (t) [k] = max(w avg[k], w (t) [k] λ) else if w (t) (t) [k] w avg[k] < 0 then w (t) (t) [k] = min(w avg[k], w (t) [k] + λ) en if en for en for en for return w (T ) 1,..., w (T ) D, w avg (T )

14 Experimental Setup Open-source Moses SMT system (Koehn et al. 2007); implementation of Bertoli et al All systems use same phrase tables an language moels, traine on 1M parallel ata poole from all IPC sections. in. systems are tune on each IPC section separately. poole system is tune on 2K sentences poole from 250 sentences from each IPC section. Avg an M are algorithms escribe above. w avg is global moel prouce as by-prouct in multi-task learning.

15 Experimental Evaluation All systems evaluate on 8 test sets, each consisting of 2K sentences from a separate IPC omain. Statistical significance of pairwise result ifferences assesse by p-values smaller than 0.05 using Approximate Ranomization test ( & Maxwell2005). statistically significant improvement over in inicate by statistically significant improvement over poole inicate by + statistically significant improvement over Avg inicate by #

16 Experimental Results section in. poole Avg M w avg A # # B # C # # D E F # G H

17 Discussion poole shows no s.s. improvement over in. Best results (bol face) achieve by Avg, M, or w avg. Best results are small, but statistically significant improvements over in. an poole. Significant egraation on section C ( chemistry ) by averaging techniques ue to expeptional character of chemical formulae an compoun names. Interpretation of small improvements with a grain of salt, however, hope for larger improvments with larger feature sets.