Neural POS-Tagging with Julia

Size: px

Start display at page:

Download "Neural POS-Tagging with Julia"

Marvin Strickland
5 years ago
Views:

1 Neural POS-Tagging with Julia Hiroyuki Shindo Julia Tokyo

2 Why Julia? NLP の基礎解析 ( 研究レベル ) ベクトル演算以外の計算も多い ( 探索など ) Java, Scala, C++ Python-like な文法で, 高速なプログラミング言語 Julia, Nim, Crystal? 1

3 Part-of-Speech (POS) Tagging 品詞タグ (45 種類 ) DT: 限定詞 (the, a, an,...) N: 名詞 V: 動詞 CD: 数字 JJ: 形容詞 DT N N V CD N JJ N The auto maker sold 1000 cars last year. The auto maker sold 1000 cars last year. 2

4 List of POS-Tags for English CC - Coordinating conjunction CD - Cardinal number DT - Determiner EX - Existential there FW - Foreign word IN - Preposition or subordinating conjunction JJ - Adjective JJR - Adjective, comparative JJS - Adjective, superlative LS - List item marker MD - Modal NN - Noun, singular or mass NNS - Noun, plural NNP - Proper noun, singular NNPS - Proper noun, plural PDT - Predeterminer POS - Possessive ending PRP - Personal pronoun PRP$ - Possessive pronoun RB - Adverb RBR - Adverb, comparative RBS - Adverb, superlative RP - Particle SYM - Symbol TO - to UH - Interjection VB - Verb, base form VBD - Verb, past tense VBG - Verb, gerund or present participle VBN - Verb, past participle VBP - Verb, non-3rd person singular present VBZ - Verb, 3rd person singular present WDT - Wh-determiner WP - Wh-pronoun WP$ - Possessive wh-pronoun WRB - Wh-adverb 3

5 Neural POS-Tagging Learning Character-level Representations for Part-of- Speech Tagging [Santos+, ICML 2014] 45 次元ベクトル DT N N V CD N JJ N ニューラルネットワーク The auto maker sold 1000 cars last year. 4

6 Neural POS Tagger word-vec char-vec it clearly shows

7 Neural POS Tagger 45*3 Linear ( y=wx + b ) ReLU 150*3 Convolution2D Concat 100 it 50 clearly shows

8 Character Vector c l e a r l y

9 Character Vector 10 c l e a r l y

10 Character Vector Convolution 10 c l e a r l y

11 Character Vector Convolution 10 c l e a r l y

12 Character Vector Convolution 10 c l e a r l y

13 Character Vector Convolution 10 c l e a r l y

14 Character Vector 50*7 Convolution 10 c l e a r l y

15 Character Vector 50 MaxPooling 50*7 Convolution 10 c l e a r l y

16 Neural POS Tagger 45*3 Linear ( y=wx + b ) ReLU 150*3 Convolution2D Concat 100 it 50 clearly shows

17 Neural Network Library for Julia Mocha.jl MXNet.jl etc. 1. NLP 用の minimal な library が欲しい 2. 言語解析では, 動的ネットワークを書きやすいことが重要ネットワーク構造が事前に決まらない ( 途中の計算結果に依存する ) GPU < CPU が起こりうる

18 Merlin.jl: Neural Network Library 実装 : Julia,( 一部 ) C++ Variable Functor Variable

19 Variable Type type Variable value::array grad::array args::union{tuple{variable},vector{variable}} diff::function fixed::bool end

20 Functor Type abstract Functor type ReLU <: Functor end type Linear <: Functor weight::variable bias::variable end etc.

21 type ReLU <: Functor end Functor Type Functor は,apply, diff 関数を持つ function apply(f::relu, x::array) y = similar(x)... y, gy -> diff(f, x, gy) end function diff(f::relu, x::array, y::array)... end

22 Merlin.jl: Imperative Mode x = Variable(rand(Float32,100,1)) f = Linear(Float32, 100, 50) y = x > f function call(f, x) y, diff = apply(f, x) Variable(y, diff,...) end y f x

23 Merlin.jl: Imperative Mode x1, x2 = Variable(...), Variable(...),... y = (x1, x2) > Add() y f x

24 Merlin.jl: Imperative Mode x1, x2, x3 = Variable(...), Variable(...),... y = [x1, x2, x3] > Concat(1) y f x...

25 Merlin.jl: Imperative Mode x = Variable(...) l = Linear(Float32, 100, 50) y = x > l > ReLU() or y = x > [l, ReLU()] y ReLU l x

26 Merlin.jl: Symbolic Mode x = Variable() y = x > f1 > f2 > f3 g = compile(y) y x

27 Sentence-level Forward 45*3 Linear ( y=wx + b ) ReLU 150*3 Convolution2D Concat 100 it 50 clearly shows

28 Sentence-level Forward # functors definition word_embed = Lookup(Float32, 100) char_embed = Lookup(Float32, 10) conv = Conv2d(...) linear = Linear(Float32, 100, 45) # forward output = [vec1, vec2, vec3] > Concat(2) > [conv, ReLU(), linear] loss = output > CrossEntropy(...) # backward diff!(output, loss) # update optimize!(sgd, [word_embed, char_embed, conv, linear])

29 Character-level Forward 50 MaxPooling 50*7 Convolution 10 c l e a r l y

30 Character-level Forward # functors definition conv = Conv2d(...) maxpool = MaxPool2d(...) # forward for i = 1:length(words) chars = words[i].chars output[i] = chars > [conv, maxpool] end

31 Merlin.jl: List of Functors Activation (ReLU, Tanh, Sigmoid) Concat Convolution2D Linear (y = Wx + b) Lookup Table Math Operators (Add, Multiply) MaxPooling2D Window2D

32 Experimental Results Training: Penn Treebank WSJ, sec Testing:, sec. 23 Optimizer: SGD (learning rate = ) Method Accuracy Neural POS-Tagger Stanford NLP (MEMM)

33 Future Work GPU (cudnn) Benchmark test Add more examples for NLP 32

Part-of-Speech Tagging + Neural Networks CS 287

Part-of-Speech Tagging + Neural Networks CS 287 Quiz Last class we focused on hinge loss. L hinge = max{0, 1 (ŷ c ŷ c )} Consider now the squared hinge loss, (also called l 2 SVM) L hinge 2 = max{0, 1