Generalized Queries on Probabilistic Context-Free Grammars

Transcription

1 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 1, JANUARY Generalized Queries on Probabilistic Context-Free Graars David V. Pynadath and Michael P. Wellan Abstract Probabilistic context-free graars (PCFGs) provide a siple way to represent a particular class of distributions over sentences in a context-free language. Efficient parsing algoriths for answering particular queries about a PCFG (i.e., calculating the probability of a given sentence, or finding the ost likely parse) have been developed and applied to a variety of patternrecognition probles. We extend the class of queries that can be answered in several ways: (1) allowing issing tokens in a sentence or sentence fragent, (2) supporting queries about interediate structure, such as the presence of particular nonterinals, and (3) flexible conditioning on a variety of types of evidence. Our ethod works by constructing a Bayesian network to represent the distribution of parse trees induced by a given PCFG. The network structure irrors that of the chart in a standard parser, and is generated using a siilar dynaic-prograing approach. We present an algorith for constructing Bayesian networks fro PCFGs, and show how queries or patterns of queries on the network correspond to interesting queries on PCFGs. The network foralis also supports extensions to encode various context sensitivities within the probabilistic dependency structure. Index Ters Probabilistic context-free graars, Bayesian networks. 1 INTRODUCTION M The authors are with the Artificial Intelligence Laboratory, University of Michigan, 1101 Beal Avenue, Ann Arbor, MI E-ail: {pynadath, wellan}@uich.edu. Manuscript received 18 Nov. 1996; revised 4 Nov Recoended for acceptance by T. Ishida. For inforation on obtaining reprints of this article, please send e-ail to: tpai@coputer.org, and reference IEEECS Log Nuber OST pattern-recognition probles start fro observations generated by soe structured stochastic process. Probabilistic context-free graars (PCFGs) [1], [2] have provided a useful ethod for odeling uncertainty in a wide range of structures, including natural languages [2], prograing languages [3], iages [4], speech signals [5], and RNA sequences [6]. Doains like plan recognition, where nonprobabilistic graars have provided useful odels [7], ay also benefit fro an explicit stochastic odel. Once we have created a PCFG odel of a process, we can apply existing PCFG parsing algoriths to answer a variety of queries. For instance, standard techniques can efficiently copute the probability of a particular observation sequence or find the ost probable parse tree for that sequence. Section 2 provides a brief description of PCFGs and their associated algoriths. However, these techniques are liited in the types of evidence they can exploit and the types of queries they can answer. In particular, the existing PCFG techniques generally require specification of a coplete observation sequence. In any contexts, we ay have only a partial sequence available. It is also possible that we ay have evidence beyond siple observations. For exaple, in natural language processing, we ay be able to exploit contextual inforation about a sentence in deterining our beliefs about certain unobservable variables in its parse tree. In addition, we ay be interested in coputing the probabilities of alternate types of events (e.g., future observations or abstract features of the parse) that the extant techniques do not directly support. The restricted query classes addressed by the existing algoriths liit the applicability of the PCFG odel in doains where we ay require the answers to ore coplex queries. A flexible and expressive representation for the distribution of structures generated by the graar would support broader fors of evidence and queries than supported by the ore specialized algoriths that currently exist. We adopt Bayesian networks for this purpose, and define an algorith to generate a network representing the distribution of possible parse trees (up to a specified string length) generated fro a given PCFG. Section 3 describes this algorith, as well as our algoriths for extending the class of queries to include the conditional probability of a sybol appearing anywhere within any region of the parse tree, conditioned on any evidence about sybols appearing in the parse tree. The restrictive independence assuptions of the PCFG odel also liit its applicability, especially in doains like plan recognition and natural language with coplex dependency structures. The flexible fraework of our Bayesian-network representation supports further extensions to context-sensitive probabilities, as in the probabilistic parse tables of Briscoe and Carroll [8]. Section 4 explores several possible ways to relax the independence assuptions of the PCFG odel within our approach. Modified versions of our PCFG algoriths can support the sae class of queries supported in the context-free case. 2 PROBABILISTIC CONTEXT-FREE GRAMMARS A probabilistic context-free graar is a tuple Â, N, S, P, where the disjoint sets S and N specify the terinal and nonterinal sybols, respectively, with S Œ N being the /98/$ IEEE J:\PRODUCTION\TPAMI\2-INPROD\106036\106036_1.DOC regularpaper97.dot SB 19,968 01/06/98 8:29 AM 1 / 13

2 2 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 1, JANUARY 1998 S Æ np vp (0.8) pp Æ prep np (1.0) S Æ vp (0.2) prep Æ like (1.0) np Æ noun (0.4) verb Æ swat (0.2) np Æ noun pp (0.4) verb Æ flies (0.4) np Æ noun np (0.2) verb Æ like (0.4) vp Æ verb (0.3) noun Æ swat (0.05) vp Æ verb np (0.3) noun Æ flies (0.45) vp Æ verb pp (0.2) noun Æ ants (0.5) vp Æ verb np pp (0.2) Fig. 1. A probabilistic context-free graar (fro Charniak [2]). start sybol. P is the set of productions, which take the for E Æ x(p), with E Œ N, x Œ (S < N) +, and p = Pr(E Æ x), the probability that E will be expanded into the string x. The su of probabilities p over all expansions of a given nonterinal E ust be one. The exaples in this paper will use the saple graar (fro Charniak [2]) shown in Fig. 1. This definition of the PCFG odel prohibits rules of the for E Æ e, where e represents the epty string. However, we can rewrite any PCFG to eliinate such rules and still represent the original distribution [2], as long as we note the probability Pr(S Æ e). For clarity, the algorith descriptions in this paper assue Pr(S Æ e) = 0, but a negligible aount of additional bookkeeping can correct for any nonzero probability. The probability of applying a particular production E Æ x to an interediate string is conditionally independent of what productions generated this string, or what productions will be applied to the other sybols in the string, given the presence of E. Therefore, the probability of a given derivation is siply the product of the probabilities of the individual productions involved. We define the parse tree representation of each such derivation as for nonprobabilistic context-free graars [9]. The probability of a string in the language is the su taken over all its possible derivations. 2.1 Standard PCFG Algoriths Since the nuber of possible derivations grows exponentially with the string s length, direct enueration would not be coputationally viable. Instead, the standard dynaic prograing approach used for both probabilistic and nonprobabilistic CFGs [10] exploits the coon production sequences shared across derivations. The central structure is a table, or chart, storing previous results for each subsequence in the input sentence. Each entry in the chart corresponds to a subsequence x i x i+j-1 of the observation string x 1 x L. For each sybol E, an entry contains the probability that the corresponding subsequence is derived fro that sybol, Pr(x i x i+j-1 E). The index i refers to the position of the subsequence within the entire terinal string, with i = 1 indicating the start of the sequence. The index j refers to the length of the subsequence. The botto row of the table holds the results for subsequences of length one, and the top entry holds the overall result, Pr(x 1 x L S), which is the probability of the observed string. We can copute these probabilities bottoup, since we know that Pr(x i E) = 1, if E is the observed sybol x i. We can define all other probabilities recursively as the su, over all productions E Æ x(p), of the product p Pr(x i x i+j-1 x). Altering this procedure to take the axiu rather than the su yields the ost probable parse tree for the observed string. Both algoriths require tie O(L 3 ) for a string of length L, ignoring the dependency on the size of the graar. To copute the probability of the sentence Swat flies like ants, we would use the algorith to generate the table shown in Fig. 2, after eliinating any unused interediate entries. There are also separate entries for each production, though this is not necessary if we are interested only in the final sentence probability. In the top entry, there are two listings for the production S Æ np vp, with different subsequence lengths for the right-hand side sybols. The su of all probabilities for productions with S on the left-hand side in this entry yields the total sentence probability of This algorith is capable of coputing any inside probability, the probability of a particular string appearing inside the subtree rooted by a particular sybol. We can work top-down in an analogous anner to copute any outside probability [2], the probability of a subtree rooted by a particular sybol appearing aid a particular string. Given these probabilities, we can copute the probability of any particular nonterinal sybol appearing in the parse tree as the root of a subtree covering soe subsequence. For exaple, in the sentence Swat flies like ants, we can copute the probability that like ants is a prepositional phrase, using a cobination of inside and S Æ vp: S Æ np(2) vp(2): j = 4 S Æ np(1) vp(3): vp Æ verb np pp: vp Æ verb np: Fig. 2. Chart for Swat flies like ants. 3 vp Æ verb pp: np Æ noun pp: np Æ noun np: vp Æ verb np: pp Æ prep np: 0.2 np Æ noun: 0.02 np Æ noun: 0.18 np Æ noun: verb Æ swat: 0.2 verb Æ flies: 0.4 prep Æ like: 1.0 noun Æ ants: noun Æ swat: 0.05 noun Æ flies: 0.45 verbæ like: 0.4 i = J:\PRODUCTION\TPAMI\2-INPROD\106036\106036_1.DOC regularpaper97.dot SB 19,968 01/06/98 8:29 AM 2 / 13

3 PYNADATH AND WELLMAN: GENERALIZED QUERIES ON PROBABILISTIC CONTEXT-FREE GRAMMARS 3 Fig. 3. Parse tree for Swat flies like ants, with (i, j, k) indices labeled. outside probabilities. The Left-to-Right Inside (LRI) algorith [10] specifies how we can use inside probabilities to obtain the probability of a given initial subsequence, such as the probability of a sentence (of any length) beginning with the words Swat flies. Furtherore, we can use such initial subsequence probabilities to copute the conditional probability of the next terinal sybol given a prefix string. 2.2 Indexing Parse Trees Yet other conceivable queries are not covered by existing algoriths, or answerable via straightforward anipulations of inside and outside probabilities. For exaple, given observations of arbitrary partial strings, it is unclear how to exploit the standard chart directly. Siilarly, we are unaware of ethods to handle observation of nonterinals only (e.g., that the last two words for a prepositional phrase). We seek, therefore, a echanis that would adit observational evidence of any for as part of a query about a PCFG, without requiring us to enuerate all consistent parse trees. We first require a schee to specify such events as the appearance of sybols at designated points in the parse tree. We can use the indices i and j to deliit the leaf nodes of the subtree, as in the standard chart parsing algoriths. For exaple, the pp node in the parse tree of Fig. 3 is the root of the subtree whose leaf nodes are like and ants, so i = 3 and j = 2. However, we cannot always uniquely specify a node with these two indices alone. In the branch of the parse tree passing through np, n, and flies, all three nodes have i = 2 and j = 1. To differentiate the, we introduce the k index, defined recursively. If a node has no child with the sae i and j indices, then it has k = 1. Otherwise, its k index is one ore than the k index of its child. Thus, the flies node has k = 1, the noun node above it has k = 2, and its parent np has k = 3. We have labeled each node in the parse tree of Fig. 3 with its (i, j, k) indices. We can think of the k index of a node as its level of abstraction, with higher values indicating ore abstract sybols. For instance, the flies sybol is a specialization of the noun concept, which, in turn, is a specialization of the np concept. Each possible specialization corresponds to an abstraction production of the for E Æ E, that is, with only one sybol on the right-hand side. In a parse tree involving such a production, the nodes for E and E have identical i and j values, but the k value for E is one ore than that of E. We denote the set of abstraction productions as P A Õ P. All other productions are decoposition productions, in the set P D = P\P A, and have two or ore sybols on the righthand side. If a node E is expanded by a decoposition production, the su of the j values for its children will equal its own j value, since the length of the original subsequence derived fro E ust equal the total lengths of the subsequences of its children. In addition, since each child ust derive a string of nonzero length, no child has the sae j index as E, which ust then have k = 1. Therefore, abstraction productions connect nodes whose indices atch in the i and j coponents, while decoposition productions connect nodes whose indices differ. 3 BAYESIAN NETWORKS FOR PCFGS A Bayesian network [11], [12], [13] is a directed acyclic graph where nodes represent rando variables, and associated with each node is a specification of the distribution of its variable conditioned on its predecessors in the graph. Such a network defines a joint probability distribution the probability of an assignent to the rando variables is given by the product of the probabilities of each node conditioned on the values of its predecessors according to the assignent. Edges not included in the graph indicate conditional independence; specifically, each node is conditionally independent of its nondescendants given its iediate predecessors. Algoriths for inference in Bayesian networks exploit this independence to siplify the calculation of arbitrary conditional probability expressions involving the rando variables. By expressing a PCFG in ters of suitable rando variables structured as a Bayesian network, we could in principle support a broader class of inferences than the standard PCFG algoriths. As we deonstrate below, by expressing the distribution of parse trees for a given probabilistic graar, we can incorporate partial observations of a sentence as well as other fors of evidence, and deterine the resulting probabilities of various features of the parse trees. 3.1 PCFG Rando Variables We base our Bayesian-network encoding of PCFGs on the schee for indexing parse trees presented in Section 2.2. The rando variable N ijk denotes the sybol in the parse tree at the position indicated by the (i, j, k) indices. Looking back at the exaple parse tree of Fig. 3, a sybol E labeled (i, j, k) indicates that N ijk = E. Index cobinations not appearing in the tree correspond to N variables taking on the null value nil. Assignents to the variables N ijk are sufficient to describe a parse tree. However, if we construct a Bayesian network using only these variables, the dependency structure would be quite coplicated. For exaple, in the exaple PCFG, the fact that N 213 has the value np would influence whether N 321 takes on the value pp, even given that N 141 (their parent in the parse tree) is vp. Thus, we would need an additional link between N 213 and N 321, and, in fact, between all possible sibling nodes whose parents have ultiple expansions. J:\PRODUCTION\TPAMI\2-INPROD\106036\106036_1.DOC regularpaper97.dot SB 19,968 01/06/98 8:29 AM 3 / 13

4 4 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 1, JANUARY 1998 To siplify the dependency structure, we introduce rando variables P ijk to represent the productions that expand the corresponding sybols N ijk. For instance, we add the node P 141, which would take on the value vp Æ verb np pp in the exaple. N 213 and N 321 are conditionally independent given P 141, so no link between siblings is necessary in this case. However, even if we know the production P ijk, the corresponding children in the parse tree ay not be conditionally independent. For instance, in the chart of Fig. 2, entry (1, 4) has two separate probability values for the production S Æ np vp, each corresponding to different subsequence lengths for the sybols on the right-hand side. Given only the production used, there are again ultiple possibilities for the connected N variables: N 113 = np and N 231 = vp, or N 121 = np and N 321 = vp. All four of these sibling nodes are conditionally dependent, since knowing any one deterines the values of the other three. Therefore, we dictate that each variable P ijk take on different values for each breakdown of the right-hand sybols subsequence lengths. The doain of each P ijk variable therefore consists of productions, augented with the j and k indices of each of the sybols on the right-hand side. In the previous exaple, the doain of P 141 would require two possible values, S Æ np[1, 3]vp[3, 1] and S Æ np[2, 1]vp[2, 1], where the nubers in brackets correspond to the j and k values, respectively, of the associated sybol. If we know that P 141 is the forer, then N 113 = np and N 231 = vp with probability one. This deterinistic relationship renders the child N variables conditionally independent of each other given P ijk. We describe the exact nature of this relationship in Section Having identified the rando variables and their doains, we coplete the definition of the Bayesian network by specifying the conditional probability tables representing their interdependencies. The tables for the N variables represent their deterinistic relationship with the parent P variables. However, we also need the conditional probability of each P variable given the value of the corresponding N variable, that is, Pr(P ijk = E Æ E 1 [j 1, k 1 ] E [j, k ] N ijk = E). The PCFG specifies the relative probabilities of different productions for each nonterinal, but we ust copute the probability, b(e, j, k) (analogous to the inside probability [2]), that each sybol E t on the right-hand side is the root node of a subtree, at abstraction level k t, with a terinal subsequence length j t. 3.2 Calculating b Algorith We can calculate the values for b with a odified version of the dynaic prograing algorith sketched in Section 2.1. As in the standard chart-based PCFG algoriths, we can define this function recursively and use dynaic prograing to copute its values. Since terinal sybols always appear as leaves of the parse tree, we have, for any terinal sybol x Œ S, b(x, 1, 1) = 1, and for any j > 1 or k > 1, b(x, j, k) = 0. For any nonterinal sybol E Œ N, b(e, 1, 1) = 0, since nonterinals can never be leaf nodes. For j > 1 or k > 1, b(e, j, k) is the su, over all productions expanding E, of the probability of that production expanding E and producing a subtree constrained by the paraeters j and k. For k > 1, only abstraction productions are possible. For an abstraction production E Æ E, we need the probabilities that E is expanded into E and that E derives a string of length j fro the abstraction level iediately below E. The forer is given by the probability associated with the production, while the latter is siply b(e, j, k - 1). According to the independence assuptions of the PCFG odel, the expansion of E is independent of its derivation, so the joint probability is siply the product. We can copute these probabilities for every abstraction production expanding E. Since the different expansions are utually exclusive events, the value for b(e, j, k) is erely the su of all the separate probabilities. We assue that there are no abstraction cycles in the graar. That is, there is no sequence of productions E 1 Æ E 2, º, E t-1 Æ E t, E t Æ E 1, since, if such a cycle existed, the above recursive calculation would never halt. The sae assuption is necessary for terination of the standard parsing algorith. The assuption does restrict the classes of graars for which such algoriths are applicable, but it will not be restrictive in doains where we interpret productions as specializations, since cycles would render an abstraction hierarchy ipossible. For k = 1, only decoposition productions are possible. For a decoposition production E Æ E 1 E 2 E (p), we need the probability that E is thus expanded, and that each E t derives a subsequence of appropriate length. Again, the forer is given by p, and the latter can be coputed fro values of the b function. We ust consider every possible subsequence length j t for each E t, Ât= 1 such that j = j. In addition, the E t could appear at t any level of abstraction k t, so we ust consider all possible values for a given subsequence length. We can obtain 1 t t6 t the joint probability of any cobination of j, k = B =1 val- ues by coputing b Et, jt, k 1 t, since the derivation t= 1 6 fro each E t is independent of the others. The su of 1 t t6 t these joint probabilities over all possible j, k = B =1 yields the probability of the expansion specified by the production s right-hand side. The product of the resulting probability and p yields the probability of that particular expansion, since the two events are independent. Again, we can su over all relevant decoposition productions to find the value of b(e, j, 1). The algorith in Fig. 4 takes advantage of the division between abstraction and decoposition productions to copute the values b(e, j, k) for strings bounded by length. The array kax keeps track of the depth of the abstraction hierarchy for each subsequence length Exaple Calculations To illustrate the coputation of b values, consider the result of using Charniak s graar fro Fig. 1 as its input. We initialize the entries for j = 1 and k = 1 to have probability one for each terinal sybol, as in Fig. 1. To fill in the entries for j = 1 and k = 2, we look at all of the abstraction productions. The sybols noun, verb, and prep can all be J:\PRODUCTION\TPAMI\2-INPROD\106036\106036_1.DOC regularpaper97.dot SB 19,968 01/06/98 8:29 AM 4 / 13

5 PYNADATH AND WELLMAN: GENERALIZED QUERIES ON PROBABILISTIC CONTEXT-FREE GRAMMARS 5 Fig. 4. Algorith for coputing b values. Fig. 5. Final table for saple graar. k E b(e, 4, k) k E b(e, 3, k) k E b(e, 2, k) k E b(e, 1, k) 2 S S S S S S S np 0.4 np np np 0.08 vp 0.3 vp vp vp prep 1.0 pp pp 0.08 pp 0.4 verb like 1.0 swat 1.0 flies 1.0 ants 1.0 expanded into one or ore terinal sybols, which have nonzero b values at k = 1. We enter these three nonterinals at k = 2, with b values equal to the su, over all relevant abstraction productions, of the product of the probability of the given production and the value for the righthand sybol at k = 1. For instance, we copute the value for noun by adding the product of the probability of noun Æ swat and the value for swat, that of noun Æ flies and flies, and that of noun Æ ants and ants. This yields the value one, since a noun will always derive a string of length one, at a single level abstraction above the terinal string, given this graar. The abstraction phase continues until we find S at k = 4, for which there are no further abstractions, so we go on to j = 2 and begin the decoposition phase. To illustrate the decoposition phase, consider the value for b(s, 3, 1). There is only one possible decoposition production, sæ np vp. However, we ust consider two separate cases: when the noun phrase covers two sybols and the verb phrase one, and when the noun phrase covers one and the verb phrase two. At a subsequence length of two, both np and vp have nonzero probability only at the botto level of abstraction, while, at a length of one, only at the third. So, to copute the probability of the first subsequence length cobination, we ultiply the probability of the production by b(np, 2, 1) and b(vp, 1, 3). The probability of the second cobination is a siilar product, and the su of the two values provides the value to enter for S. The other abstractions and decopositions proceed along siilar lines, with additional suation required when ultiple productions or ultiple levels of abstraction are possible. The final table is shown in Fig. 5, which lists only the nonzero values Coplexity For analysis of the coplexity of coputing the b values for a given PCFG, it is useful to define d to be the axiu length of possible chains of abstraction productions (i.e., the axiu k value), and to be the axiu J:\PRODUCTION\TPAMI\2-INPROD\106036\106036_1.DOC regularpaper97.dot SB 19,968 01/06/98 8:29 AM 5 / 13

6 6 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 1, JANUARY 1998 Fig. 8. Procedure for finding all possible decoposition productions. Fig. 6. Procedure for generating the network. Fig. 7. Procedure for finding all possible abstraction productions. production length (nuber of sybols on the right-hand side). A single run through the abstraction phase requires tie O( P A ), and for each subsequence length, there are O(d) runs. For a specific value of j, the decoposition phase requires tie O( P D j -1 d ), since, for each decoposition production, we ust consider all possible cobinations of subsequence lengths and levels of abstractions for each sybol on the right-hand side. Therefore, the whole algorith would take tie O(n[d P A + P D n -1 d ]) = O( P n d ). 3.3 Network Generation Phase We can use the b function calculated as described above to copute the doains of rando variables N ijk and P ijk and the required conditional probabilities Specification of Rando Variables The procedure CREATE-NETWORK, described in Fig. 6, begins at the top of the abstraction hierarchy for strings of length n starting at position 1. The root sybol variable, N 1n(kax[n]), can be either the start sybol, indicating the parse tree begins here, or nil *, indicating that the parse tree begins below. We ust allow the parse tree to start at any j and k where b(s, j, k) > 0, because these can all possibly derive strings (of any length bounded by n) within the language. CREATE-NETWORK then proceeds downward through the N ijk rando variables and specifies the doain of their corresponding production variables, P ijk. Each such production variable takes on values fro the set of possible expansions for the possible nonterinal sybols in the doain of N ijk. If k > 1, only abstraction productions are possible, so the procedure ABSTRACTION-PHASE, described in Fig. 7, inserts all possible expansions and draws links fro P ijk to the rando variable N ij(k-1), which takes on the value of the righthand side sybol. If k = 1, the procedure DECOMPOSITION- PHASE, described in Fig. 8, perfors the analogous task for Fig. 9. Procedure for handling start of parse tree at next level. Fig. 10. Procedure for coputing the probability of the start of the tree occurring for a particular string length and abstraction level. decoposition productions, except that it ust also consider all possible length breakdowns and abstraction levels for the sybols on the right-hand side. CREATE-NETWORK calls the procedure START-TREE, described in Fig. 9, to handle the possible expansions of nil * : either nil * Æ S, indicating that the tree starts iediately below, or nil * Æ nil *, indicating that the tree starts further below. START-TREE uses the procedure START-PROB, described in Fig. 10, to deterine the probability of the parse tree starting anywhere below the current point of expansion. When we insert a possible value into the doain of a production node, we add it as a parent of each of the nodes corresponding to a sybol on the right-hand side. We also insert each sybol fro the right-hand side into the doain of the corresponding sybol variable. The algorith descriptions assue the existence of procedures INSERT- STATE and ADD-PARENT. The procedure INSERT-STATE(node, label) inserts a new state with nae label into the doain of variable node. The procedure ADD-PARENT(child, parent) draws a link fro node parent to node child. J:\PRODUCTION\TPAMI\2-INPROD\106036\106036_1.DOC regularpaper97.dot SB 19,968 01/06/98 8:29 AM 6 / 13

7 PYNADATH AND WELLMAN: GENERALIZED QUERIES ON PROBABILISTIC CONTEXT-FREE GRAMMARS Specification of Conditional Probability Table After CREATE-NETWORK has specified the doains of all of the rando variables, we can specify the conditional probability tables. We introduce the lexicographic order a over the set {(j, k) 1 j n, 1 k kax[j]}, where if j 1 < j 2 then (j 1, k 1 ) a (j 2, k 2 ) and if k 1 < k 2 then (j, k 1 ) a (j, k 2 ). For the purposes of siplicity, we do not specify an exact value for each probability Pr(X = x Y), but instead specify a weight, Pr(X = x t Y) µ a t. We copute the exact probabilities through noralization, where we divide each weight by the su  t a t. The prior probability table for the top node, which has no parents, can be defined as follows: Pr(N 1n(kax[n]) = S) µ b(s, n, kax[n]) Pr(N 1n(kax[n]) = nil * ) µ  (j,k)a(n,kax[n]) b(s, j, k). For a given state r in the doain of any P ijk node, where r represents a production and corresponding assignent of j and k values to the sybols on the right-hand side, of the for EÆ E 1 [j 1, k 1 ] E [j, k ](p), we can define the conditional probability of that state as: ijk ijk t = t t t Pr P = r N = E µ p b E, j, k For any sybol E π E in the doain of N ijk, Pr(P ijk = r N ijk = E ) = 0. For the productions for starting or delaying the tree, the probabilities are: Pr(P 1jk = nil * Æ S[j, k ] N ijk = nil * ) µ b[s, j, k ] Pr(P 1jk = nil * Æ nil * N ijk = nil * ) µ  (j,k )a(j,k) b[s, j, k ]. The probability tables for the N ijk nodes are uch sipler, since once the productions are specified, the sybols are copletely deterined. Therefore, the entries are either one or zero. For exaple, consider the nodes N i " j " k with the " parent node P i j k (aong others). For the rule r representing E Æ E 1 [j 1, k 1 ] E [j, k ], Pr(N i j k " " " "-1  1 = E, P i j k = r, º) = 1 when i" = i + j t= t, j = j,. For all sybols other than E, in the doain of N i " j " k, this conditional probability is zero. " We can fill in this entry for all configurations of the other parent nodes (represented by the ellipsis in the condition part of the probability), though we know that any conflicting configurations (i.e., two productions both trying to specify the sybol N i " j " k ) are ipossible. Any configuration of the parent nodes that does not specify a " certain sybol indicates that the N i " j " k " nil with probability one. node takes on the value Network Generation Exaple As an illustration, consider the execution of this algorith using the b values fro Fig. 5. We start with the root variable N 142. The start sybol S has a b value greater than zero here, as well as at points below, so the doain ust include both S and nil *. To obtain Pr(N 142 = S), we siply divide b(s, 4, 2) by the su of all b values for S, yielding The doain of P 142 is partially specified by the abstraction phase for the sybol S in the doain of N 142. There is. only one relevant production, S Æ vp, which is a possible expansion since b(vp, 4, 1) > 0. Therefore, we insert the production into the doain of P 142, with conditional probability one given that N 142 = S, since there are no other possible expansions. We also draw a link fro P 142 to N 141, whose doain now includes vp with conditional probability one given that P 142 = SÆ vp. To coplete the specification of P 142, we ust consider the possible start of the tree, since the doain of N 142 includes nil *. The conditional probability of P 142 = nil * Æ S is , the ratio of b(s, 4, 1) and the su of b(s, j, k) for (j, k) d (4, 1). The link fro P 142 to N 141 has already been ade during the abstraction phase, but we ust also insert S and nil * into the doain of N 141, each with conditional probability one given the appropriate value of P 142. We then proceed to N 141, which is at the botto level of abstraction, so we ust perfor a decoposition phase. For the production SÆ np vp, there are three possible cobinations of subsequence lengths which add to the total length of four. If np derives a string of length one and vp a string of length three, then the only possible levels of abstraction for each are three and one, respectively, since all others will have zero b values. Therefore, we insert the production s Æ np[1, 3]vp[3, 1] into the doain of P 141, where the nubers in brackets correspond to the subsequence length and level of abstraction, respectively. The conditional probability of this value, given that N 141 = S, is the product of the probability of the production, b(np, 1, 3), and b(vp, 3, 1), noralized over the probabilities of all possible expansions. We then draw links fro P 141 to N 113 and N 231, into whose doains we insert np and vp, respectively. The i values are obtained by noting that the subsequence for np begins at the sae point as the original string while that for vp begins at a point shifted by the length of the subsequence for np. Each occurs with probability one, given that the value of P 141 is the appropriate production. Siilar actions are taken for the other possible subsequence length cobinations. The operations for the other rando variables are perfored in a siilar fashion, leading to the network structure shown in Fig Coplexity of Network Generation The resulting network has O(n 2 d) nodes. The doain of each N i11 variable has O( S ) states to represent the possible terinal sybols, while all other N ijk variables have O( N ) possible states. There are n variables of the forer, and O(n 2 d) of the latter. For k > 1, the P ijk variables (of which there are O(n 2 d)) have a doain of O( P A ) states. For P ij1 variables, there are states for each possible decoposition production, for each possible cobination of subsequence lengths, and for each possible level of abstraction of the sybols on the right-hand side. Therefore, the P ij1 variables (of which there are O(n 2 )) have a doain of O( P D j -1 d ) states, where we have again defined d to be the axiu value of k, and to be the axiu production length. J:\PRODUCTION\TPAMI\2-INPROD\106036\106036_1.DOC regularpaper97.dot SB 19,968 01/06/98 8:29 AM 7 / 13

8 8 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 1, JANUARY 1998 Fig. 11. Network fro exaple graar at axiu length 4. Unfortunately, even though each particular P variable has only the corresponding N variable as its parent, a given N variable could have potentially O(n) P variables as parents. The size of the conditional probability table for a node is exponential in the nuber of parents, although given that each N can be deterined by at ost one P (i.e., no interactions are possible), we can specify the table in a linear nuber of paraeters. If we define T to be the axiu nuber of entries of any conditional probability table in the network, then the abstraction phase of the algorith requires tie O( P A T), while the decoposition phase requires tie O( P D n -1 d T ). Handling the start of the parse tree and the potential space holders requires tie O(T). The total tie coplexity of the algorith is then O(n 2 P D n -1 d T + ndt + n 2 d P A T + n 2 dt) = O( P n +1 d T ), which dwarfs the tie coplexity of the dynaic prograing algorith for the b function. However, this network is created only once for a particular graar and length bound. 3.4 PCFG Queries We can use the Bayesian network to copute any joint probability that we can express in ters of the N and P rando variables included in the network. The standard Bayesian network algoriths [11], [12], [14] can return joint probabilities of the for Pr X = x,, X = x 3 i1j1k1 1 i jk 8 or conditional probabilities of the for Pr X ijk = x X = x,, X = x i j k 1 i jk , where each X is either N or P. Obviously, if we are interested only in whether a sybol E appeared at a particular i, j, k location in the parse tree, we need only exaine the arginal probability distribution 2 of the corresponding N variable. Thus, a single network query will yield the probability Pr(N ijk = E). The results of the network query are iplicitly conditional on the event that the length of the terinal string does not exceed n. We can obtain the joint probability by ultiplying the result by the probability that a string in the language has a length not exceeding n. For any j, the probability that we expand the start sybol S into a terinal kax[ j] string of length j is  k= 1 b1s, j, k6, which we can then su for 1 j n. To obtain the appropriate unconditional probability for any query, all network queries reported in n kax[ j] this section ust be ultiplied by   b1s, j, k6. j= 1 k= Probability of Conjunctive Events The Bayesian network also supports the coputation of joint probabilities analogous to those coputed by the standard PCFG algoriths. For instance, the probability of a particular terinal string such as Swat flies like ants corresponds to the probability Pr(N 111 = swat, N 211 = flies, N 311 = like, N 411 = ants). The probability of an initial subsequence like Swat flies..., as coputed by the LRI algorith [10], corresponds to the probability Pr(N 111 = swat, N 211 = like). Since the Bayesian network represents the distribution over strings of bounded length, we can find initial subsequence probabilities only over copletions of length bounded by n - L. Although, in this case, our Bayesian network approach requires soe odification to answer the sae query as the standard PCFG algorith, it needs no odification to handle ore coplex types of evidence. The chart parsing and LRI algoriths require coplete sequences as input, so any J:\PRODUCTION\TPAMI\2-INPROD\106036\106036_1.DOC regularpaper97.dot SB 19,968 01/06/98 8:29 AM 8 / 13

9 PYNADATH AND WELLMAN: GENERALIZED QUERIES ON PROBABILISTIC CONTEXT-FREE GRAMMARS 9 gaps or other uncertainty about particular sybols would require direct odification of the dynaic prograing algoriths to copute the desired probabilities. The Bayesian network, on the other hand, supports the coputation of the probability of any evidence, regardless of its structure. For instance, if we have a sentence Swat flies... ants where we do not know the third word, a single network query will provide the conditional probability of possible copletions Pr(N 311 N 111 = swat, N 211 = flies, N 411 = ants), as well as the probability of the specified evidence Pr(N 111 = swat, N 211 = flies, N 411 = ants). This approach can handle ultiple gaps, as well as partial inforation. For exaple, if we again do not know the exact identity of the third word in the sentence Swat flies... ants, but we do know that it is either swat or like, we can use the Bayesian network to fully exploit this partial inforation by augenting our query to specify that any doain values for N 311 other than swat or like have zero probability. Although these types of queries are rare in natural language, doains like speech recognition often require this ability to reason when presented with noisy observations. We can answer queries about nonterinal sybols as well. For instance, if we have the sentence Swat flies like ants, we can query the network to obtain the conditional probability that like ants is a prepositional phrase, Pr(N 321 = pp N 111 = swat, N 211 = like, N 311 = like, N 411 = ants). We can answer queries where we specify evidence about nonterinals within the parse tree. For instance, if we know that like ants is a prepositional phrase, the input to the network query will specify that N 321 = pp, as well as specifying the terinal sybols. Alternate network algoriths can copute the ost probable state of the rando variables given the evidence, instead of a conditional probability [11], [15], [14]. For exaple, consider the case of possible four-word sentences beginning with the phrase Swat flies... The probability axiization network algoriths can deterine that the ost probable state of terinal sybol variables N 311 and N 411 is like flies, given that N 111 = swat, N 211 = flies, and N 511 = nil Probability of Disjunctive Events We can also copute the probability of disjunctive events through ultiple network queries. If we can express an event as the union of utually exclusive events, each of the for X = x Ÿ Ÿ X = x, then we can query the i1j1k1 1 i jk network to copute the probability of each, and su the results to obtain the probability of the union. For instance, if we want to copute the probability that the sentence Swat flies like ants contains any prepositions, we would query the network for the probabilities Pr(N i12 = prep N 111 = swat, N 211 = like, N 311 = like, N 411 = ants), for 1 i 4. In a doain like plan recognition, such a query could correspond to the probability that an agent perfored soe coplex action within a specified tie span. In this exaple, the individual events are already utually exclusive, so we can su the results to produce the overall probability. In general, we ensure utual exclusivity of the individual events by coputing the conditional probability of the conjunction of the original query event and the negation of those events sued previously. For our exaple, the overall probability would be Pr(N 112 = prep e) + Pr(N 212 = prep, N 112 π prep e) + Pr(N 112 = prep, N 112 π prep, N 212 π prep e) + Pr(N 112 = prep, N 112 π prep, N 212 π prep, N 312 π prep e), where e corresponds to the event that the sentence is Swat flies like ants. The Bayesian network provides a unified fraework that supports the coputation of all of the probabilities described here. We can copute the probability of any event e, where e is a set of utually exclusive events h X Œ ; ŸŸ X Œ ; > it1jt1kt1 t1 it j k t t t t t t tc with each X being t= 1 either N or P. We can also copute probabilities of events where we specify relative likelihoods instead of strict subset restrictions. In addition, given any such event, we can deterine the ost probable configuration of the uninstantiated rando variables. Instead of designing a new algorith for each such query, we have only to express the query in ters of the network s rando variables, and use any Bayesian network algorith to copute the desired result Coplexity of Network Queries Unfortunately, the tie required by the standard network algoriths in answering these queries is potentially exponential in the axiu string length n, though the exact coplexity will depend on the connectedness of the network and the particular network algorith chosen. The algorith in our current ipleentation uses a great deal of preprocessing in copiling the networks, in the hope of reducing the coplexity of answering queries. Such an algorith can exploit the regularities of our networks (e.g., the conditional probability tables of each N ijk consist of only zeroes and ones) to provide reasonable response tie in answering queries. Unfortunately, such copilation can itself be prohibitive and will often produce networks of exponential size. There exist Bayesian network algoriths [16], [17] that offer greater flexibility in copilation, possibly allowing us to liit the size of the resulting networks, while still providing acceptable query response ties. Deterining the optial tradeoff will require future research, as will deterining the class of doains where our Bayesian network approach is preferable to existing PCFG algoriths. It is clear that the standard dynaic prograing algoriths are ore efficient for the PCFG queries they address. For doains requiring ore general queries of the types described here, the flexibility of the Bayesian network approach ay justify the greater coplexity. 4 CONTEXT SENSITIVITY For any doains, the independence assuptions of the PCFG odel are overly restrictive. By definition, the probability of applying a particular PCFG production to expand a given nonterinal is independent of what sybols have coe before and of what expansions are to occur after. Even this paper s siplified exaple illustrates soe of the weaknesses of this assuption. Consider the interediate string Swat ants like noun. It is iplausible that the J:\PRODUCTION\TPAMI\2-INPROD\106036\106036_1.DOC regularpaper97.dot SB 19,968 01/06/98 8:29 AM 9 / 13

10 10 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 1, JANUARY 1998 probability that we expand noun into flies instead of ants is independent of the choice of swat as the verb or the choice of ants as the object. Of course, we ay be able to correct the odel by expanding the set of nonterinals to encode contextual inforation, adding productions for each such expansion, and thus preserving the structure of the PCFG odel. However, this can obviously lead to an unsatisfactory increase in coplexity for both the design and use of the odel. Instead, we could use an alternate odel which relaxes the PCFG independence assuptions. Such a odel would need a ore coplex production and/or probability structure to allow coplete specification of the distribution, as well as odified inference algoriths for anipulating this distribution. 4.1 Direct Extensions to Network Structure The Bayesian network representation of the probability distribution provides a basis for exploring such context sensitivities. The networks generated by the algoriths of this paper iplicitly encode the PCFG assuptions through assignent of a single nonterinal node as the parent of each production node. This single link indicates that the expansion is conditionally independent of all other nondescendant nodes, once we know the value of this nonterinal. We could extend the context-sensitivity of these expansions within our network foralis by altering the links associated with these production nodes. We can introduce soe context sensitivity even without adding any links. Since each production node has its own conditional probability table, we can define the production probabilities to be a function of the (i, j, k) index values. For instance, the nuber of words in a group strongly influences the likelihood of that group foring a noun phrase. We could odel such a belief by varying the probability of a np appearing over different string lengths, as encoded by the j index. In such cases, we can odify the standard PCFG representation so that the probability inforation associated with each production is a function of i, j, and k, instead of a constant. The dynaic prograing algorith of Fig. 4 can be easily odified to handle production probabilities that depend on j and k. However, a dependency on the i index as well would require adding it as a paraeter of b and introducing an additional loop over its possible values. Then, we would have to replace any reference to the production probability, in either the dynaic prograing or network generation algorith, with the appropriate function of i, j, and k. Alternatively, we ay introduce additional dependencies on other nodes in the network. A PCFG extension that conditions the production probabilities on the parent of the left-hand side sybol has already proved useful in odeling natural language [18]. In this case, each production has a set of associated probabilities, one for each nonterinal sybol that is a possible parent of the sybol on the lefthand side. This new probability structure requires odifications to both the dynaic prograing and the network generation algoriths. We ust first extend the probability inforation of the b function to include the parent nonterinal as an additional paraeter. It is then straightforward Fig. 12. Subnetwork incorporating parent sybol dependency. to alter the dynaic prograing algorith of Fig. 4 to correctly copute the probabilities in a botto-up fashion. The odifications for the network generation algorith are ore coplicated. Whenever we add P ijk as a parent for soe sybol node N ijk, we also have to add N ijk as a parent of P ijk. For exaple, the dotted arrow in the subnetwork of Fig. 12 represents the additional dependency of P 112 on N 113. We ust add this link because N 112 is a possible child nonterinal, as indicated by the link fro P 113. The conditional probability tables for each P node ust now specify probabilities given the current nonterinal and the parent nonterinal sybols. We can copute these by cobining the odified b values with the conditional production probabilities. Returning to the exaple fro the beginning of this section, we ay want to condition the production probabilities on the terinal string expanded so far. As a first approxiation to such context sensitivity, we can iagine a odel where each production has an associated set of probabilities, one for each terinal sybol in the language. Each represents the conditional probability of the particular expansion given that the corresponding terinal sybol occurs iediately previous to the subsequence derived fro the nonterinal sybol on the left-hand side. Again, our b function requires an additional paraeter, and we need a odified version of the dynaic prograing algorith to copute its values. However, the network generation algorith needs to introduce only one additional link, fro N i11 for each P (i+1)jk node. The dashed arrows in the subnetwork of Fig. 13 reflect the additional dependencies introduced by this context sensitivity, using the network exaple fro Fig. 11. The P 1jk nodes are a special case, with no preceding terinal, so the steps fro the original algorith are sufficient. We can extend this conditioning to cover preceding terinal sequences rather than individual sybols. Each production could have an associated set of probabilities, one for each possible terinal sequence of length bounded by soe paraeter h. The b function now requires an additional paraeter specifying the preceding sequence. The network generation algoriths ust then add links to P ijk fro nodes N (i-h)11, º, N (i-1)11, if i h, or fro N 111, º, N (i-1)11, if i < h. The conditional probability tables then specify the probability of a particular expansion given the sybol on the left-hand side and the preceding terinal sequence. J:\PRODUCTION\TPAMI\2-INPROD\106036\106036_1.DOC regularpaper97.dot SB 19,968 01/06/98 8:29 AM 10 / 13