Random Generation of Nondeterministic Tree Automata

Transcription

1 Random Generation of Nondeterministic Tree Automata Thomas Hanneforth 1 and Andreas Maletti 2 and Daniel Quernheim 2 1 Department of Linguistics University of Potsdam, Germany 2 Institute for Natural Language Processing University of Stuttgart, Germany maletti@ims.uni-stuttgart.de Hanoi, Vietnam (TTATT 2013)

2 Outline Motivation Nondeterministic Tree Automata Random Generation Analysis

3 Tree Substitution Grammar with Latent Variables Experiment [SHINDO et al., ACL 2012 best paper] F1 score grammar w 40 full CFG = LTL 62.7 TSG [POST, GILDEA, 2009] = xltl 82.6 TSG [COHN et al., 2010] = xltl CFGlv [COLLINS, 1999] = NTA CFGlv [PETROV, KLEIN, 2007] = NTA CFGlv [PETROV, 2010] = NTA 91.8 TSGlv (single) = RTG TSGlv (multiple) = RTG Discriminative Parsers CARRERAS et al., CHARNIAK, JOHNSON, HUANG,

4 Tree Substitution Grammar with Latent Variables Experiment [SHINDO et al., ACL 2012 best paper] F1 score grammar w 40 full CFG = LTL 62.7 TSG [POST, GILDEA, 2009] = xltl 82.6 TSG [COHN et al., 2010] = xltl CFGlv [COLLINS, 1999] = NTA CFGlv [PETROV, KLEIN, 2007] = NTA CFGlv [PETROV, 2010] = NTA 91.8 TSGlv (single) = RTG TSGlv (multiple) = RTG Discriminative Parsers CARRERAS et al., CHARNIAK, JOHNSON, HUANG,

5 Berkeley Parser Example parse S NP VP DT VBZ NP This is DT JJ NN a silly sentence from

6 Berkeley Parser Example productions S-1 ADJP-2 S S-1 ADJP-1 S S-1 VP-5 VP S-2 VP-5 VP S-1 PP-7 VP S-9 NP Formalism Berkeley parser = CFG (local tree grammar) + relabeling (+ weights)

7 Typical NTA Sizes English BERKELEY parser grammar (1,133 states and 4,267,277 transitions) English EGRET parser grammar Chinese EGRET parser grammar 153 MB 107 MB 98 MB EGRET = HUI ZHANG s C++ reimplementation of the BERKELEY parser (Java)

8 Algorithm testing Observations even efficient algorithms run slow on such data often require huge amounts of memory impossible for inefficient algorithms

9 Algorithm testing Observations even efficient algorithms run slow on such data often require huge amounts of memory impossible for inefficient algorithms realistic, but difficult to use as test data

10 Algorithm testing Observations even efficient algorithms run slow on such data often require huge amounts of memory impossible for inefficient algorithms realistic, but difficult to use as test data Testing on random NTA straightforward to implement straightforward to scale

11 Algorithm testing Observations even efficient algorithms run slow on such data often require huge amounts of memory impossible for inefficient algorithms realistic, but difficult to use as test data Testing on random NTA straightforward to implement straightforward to scale but what is the significance of the results?

12 Outline Motivation Nondeterministic Tree Automata Random Generation Analysis

13 Tree automaton Definition (THATCHER AND WRIGHT, 1965) A tree automaton is a tuple A = (Q, Σ, I, R) with alphabet Q ranked alphabet Σ I Q finite set R Σ(Q) Q states terminals final states rules Remark Instead of (l, q) we write l q

14 Regular Tree Grammar Example Q = {q 0, q 1, q 2, q 3, q 4, q 5, q 6 } Σ = {VP, S,... } F = {q 0 } and the following rules: VP q 4 S q 5 q 1 q 3 q 1 q 4 q 0 S q 6 q 2 q 0

15 Regular Tree Grammar Definition (Derivation semantics) Sentential forms: ξ, ζ T Σ (Q) ξ A ζ if there exist position w pos(ξ) and rule l q R ξ = ξ[l] w ζ = ξ[q] w

16 Regular Tree Grammar Definition (Derivation semantics) Sentential forms: ξ, ζ T Σ (Q) ξ A ζ if there exist position w pos(ξ) and rule l q R ξ = ξ[l] w ζ = ξ[q] w Definition (Recognized tree language) L(A) = {t T Σ f F : t A f }

17 Outline Motivation Nondeterministic Tree Automata Random Generation Analysis

18 Previous Approaches HÉAM et al for deterministic tree-walking automata (and deterministic top-down tree automata)

19 Previous Approaches HÉAM et al for deterministic tree-walking automata (and deterministic top-down tree automata) focus on generating automata uniformly at random (for estimating average-case complexity)

20 Previous Approaches HÉAM et al for deterministic tree-walking automata (and deterministic top-down tree automata) focus on generating automata uniformly at random (for estimating average-case complexity) generator used for evaluation of conversion from det. TWA to NTA

21 Previous Approaches HUGOT et al for tree automata with global equality constraints

22 Previous Approaches HUGOT et al for tree automata with global equality constraints focus on avoiding trivial cases (removal of unreachable states, minimum height requirement)

23 Previous Approaches HUGOT et al for tree automata with global equality constraints focus on avoiding trivial cases (removal of unreachable states, minimum height requirement) generator used for evaluation of emptiness checker

24 Our Approach Goals randomly generate non-trivial NTA generator (potentially) usable for all NTA algorithms

25 Our Approach Goals randomly generate non-trivial NTA generator (potentially) usable for all NTA algorithms When is an NTA non-trivial? large number of states large number of rules

26 Our Approach Goals randomly generate non-trivial NTA generator (potentially) usable for all NTA algorithms When is an NTA non-trivial? large number of states large number of rules

27 Our Approach Goals randomly generate non-trivial NTA generator (potentially) usable for all NTA algorithms When is an NTA non-trivial? large number of states large number of rules its language contains large trees

28 Our Approach Goals randomly generate non-trivial NTA generator (potentially) usable for all NTA algorithms When is an NTA non-trivial? large number of states large number of rules its language contains large trees

29 Our Approach Goals randomly generate non-trivial NTA generator (potentially) usable for all NTA algorithms When is an NTA non-trivial? large number of states large number of rules its language contains large trees its language has many MYHILL-NERODE congruence classes canonical NTA has many states (canonical NTA = equivalent minimal deterministic NTA)

30 Our Approach Restrictions binary trees (all RTL can be such encoded with linear overhead)

31 Our Approach Restrictions binary trees (all RTL can be such encoded with linear overhead) each state is final with probability.5

32 Our Approach Restrictions binary trees (all RTL can be such encoded with linear overhead) each state is final with probability.5 uniform probability for binary/nullary rules

33 Our Approach Restrictions binary trees (all RTL can be such encoded with linear overhead) each state is final with probability.5 uniform probability for binary/nullary rules three parameters 1. input binary ranked alphabet Σ = Σ 2 Σ 0 2. number n of states of generated NTA scaling 3. nullary rule probability d 0 for all nullary rules 4. binary rule probability d 2 for all binary rules

34 Our Approach Algorithm 1. Generate n states [n] = {1,..., n}

35 Our Approach Algorithm 1. Generate n states [n] = {1,..., n} 2. Make q final with probability 0.5 q [n]

36 Our Approach Algorithm 1. Generate n states [n] = {1,..., n} 2. Make q final with probability 0.5 q [n] 3. Add rule α q with probability d 0 α Σ 0, q [n]

37 Our Approach Algorithm 1. Generate n states [n] = {1,..., n} 2. Make q final with probability 0.5 q [n] 3. Add rule α q with probability d 0 α Σ 0, q [n] 4. Add rule σ(q 1, q 2 ) q with probability d 2 σ Σ 2, q, q 1, q 2 [n]

38 Our Approach Algorithm 1. Generate n states [n] = {1,..., n} 2. Make q final with probability 0.5 q [n] 3. Add rule α q with probability d 0 α Σ 0, q [n] 4. Add rule σ(q 1, q 2 ) q with probability d 2 σ Σ 2, q, q 1, q 2 [n] 5. Reject if it is not trim

39 Our Approach Algorithm 1. Generate n states [n] = {1,..., n} 2. Make q final with probability 0.5 q [n] 3. Add rule α q with probability d 0 α Σ 0, q [n] 4. Add rule σ(q 1, q 2 ) q with probability d 2 σ Σ 2, q, q 1, q 2 [n] 5. Reject if it is not trim Evaluation 1. Determinize 2. Minimize 3. Number of obtained states = complexity of the original random NTA

40 Outline Motivation Nondeterministic Tree Automata Random Generation Analysis

41 Determinization Definition (Power-set construction) P(A) = (P(Q), Σ, F, R ) with F = {S Q S F } α {q Q α q R} R α Σ 0 σ(s 1, S 2 ) {q Q σ(q 1, q 2 ) q R, q 1 S 1, q 2 S 2 } R σ Σ 2, S 1, S 2 Q

42 Determinization Definition (Power-set construction) P(A) = (P(Q), Σ, F, R ) with F = {S Q S F } α {q Q α q R} R α Σ 0 σ(s 1, S 2 ) {q Q σ(q 1, q 2 ) q R, q 1 S 1, q 2 S 2 } R σ Σ 2, S 1, S 2 Q Note will be the guiding definition for the analytical analysis

43 Analytical Analysis Intuition power-set construction should create each state S Q given states S 1, S 2 selected uniformly at random, each state q Q should occur in target of σ(s 1, S 2 ) with probability.5 (the same intuition is also used for string automata) this intuition will create large NTA after determinization (but that they remain large after minimization is non-trivial) we will confirm the intuition experimentally

44 Analytical Analysis Theorem If d 2 = 4(1 n2.5) and d0 =.5, then the intuition is met. Proof. Let S 1, S 2 Q be selected uniformly at random σ Σ 2, q Q π(q σ(s 1, S 2 )) = 1 π(q / σ(s 1, S 2 )) = 1 ( ) 1 π(q 1 S 1 ) π(q 2 S 2 ) π(σ(q 1, q 2 ) q R) q 1,q 2 Q = 1 ( 1 d 2 4 ) n 2 = 1 ( n 2.5 ) n 2 = 1 ( n 2.5) n 2 = 1 2

45 Analytical Predictions n d 2 d 2 CI n d 2 d 2 CI

46 Empirical Evaluation Setup Σ = {σ (2), α (0) } evaluation for random NTA with various densities d 2 (at least 40 random NTA per data point d 2 ) logarithmic scale for d 2 (enough datapoints on both sides of the spike)

47 Empirical Evaluation mean # of states in DTA 8 states transition density

48 Empirical Evaluation states 3000 mean # of states in DTA transition density

49 Empirical Evaluation Observations (almost perfect) log-normal distributions we can determine the mean (empirical and analytical)

50 Empirical Evaluation Observations (almost perfect) log-normal distributions we can determine the mean (empirical and analytical) hardest instances

51 Empirical Evaluation Observations (almost perfect) log-normal distributions we can determine the mean (empirical and analytical) hardest instances outside hardest instances: all trivial

52 Empirical Evaluation Observations (almost perfect) log-normal distributions we can determine the mean (empirical and analytical) hardest instances outside hardest instances: all trivial only test on random NTA for hardest density

53 Analytical Predictions n d 2 d 2 CI n d 2 d 2 CI

54 Analytical Predictions + Empirical Evaluation n d 2 d 2 CI n d 2 d 2 CI

55 Analytical Predictions + Empirical Evaluation n d 2 d 2 CI n d 2 d 2 CI [.577,.680] [.032,.053] [.209,.316] [.027,.043] [.102,.174] [.023,.034] [.064,.114] [.021,.030] [.048,.085] [.018,.025] [.038,.066] [.016,.022] CI = confidence interval; 95% confidence level

56 Conclusion Use random NTA carefully!