2. Probabilistic Ontology Model (POM)

Transcription

1 Sring 2010 LTI Reort Probabilti Ontology Model Ni Lao 1. Introdution Th a ritique or Tom M. Mithell s talk about the Read the Web rojet. In the talk, Tom desribed the researh to develo a never-ending language learner that runs 24 hours er day, orever. The system able to extrat more inormation rom the eb to oulate its groing knoledge base o strutured knoledge. One limitation to the system, in my oinion, the need or human to suly ontology. Th less aliable i e ant to aly the method to many ne domains, or to very large domains. Thereore, it very imortant to develo methods that aid the roess o onstruting ontology or a ne domain. Here I am giving a skethy design o a system hih an automatially extrat ontology rom arsed text. I ill irst deine a robabilti ontology model based on aths on the ontology grah, and then I ill desribe eiient inerene, arameter estimation, and struture indution roedures or th model. Finally, I ill give an exeriment design on Penn TreeBank data. 2. Probabilti Ontology Model (POM 2.1. Te xt Rere sentation ith Deende ny Trees We assume that text reresented as a set o deendeny arse trees, ith roer nouns and ommon nouns annotated. For examle the sentene "Li Lei rom Peole's Reubli o China" annotated as Li Lei rom The Peole s Reubli o China Folloing revious orks on named entity extration [] and semanti role labeling [], a attern on the arse tree an be deined as either the ath beteen any to nouns (e.g. XXX rom XXX, or the ath rom a noun to any o its anestors (e.g. rom XXX, rom XXX. Alternatively, e may also onsider a more omlex reresentation, hih break omound nouns into single ords. Th reresentation may hel leverage the relationshis beteen ords in a omound noun: or examle horse leg indiates that leg a art o horse. Li Lei rom Peole the Reubli s o China

2 2.2. Rerese nting Ontology An ontology O={C, E, A} a tule o a set o onets C={ i }, a set o entities E={e j }, and a set o attributes A={a k }. We assume that eah onet aroximately orresonds to a ommon noun hrase (CNP (e.g. mouse, mountain, eah entity e aroximately orresonds to a roer noun hrase ( (e.g. Mikey, Everest, and eah attribute a aroximately orresonds to a ossible relation beteen onets (e.g. CEOo(e 1, e 2 reresenting that e 1 the CEO o e 2. Eah onet assoiated ith a set o attributes.a. We assume that the onets orm hierarhial (ayli strutures, ith eah node inheriting all the attributes o its anestors. Th quality ill be more onrete ater e desribe the random alk based robabilti model. ood animal organm lant arnivor grazer grass tree ol shee girae Rudolh Dolly 2.3. A Pro babili sti M o del B a sed on Rando m Wa l k on Ontology Grah The roblems o eliiting an ontology O ith its arameter θ an be seem as maximizing the log likelihood o a orus D given th ontology. Given a orus D={s }, here eah sentene s i a arse tree annotated ith CNP and tags, e deine our objetive untion as the regularized robability over the training data L( θ log P( s ; θ θ / 2 (1 m1.. M m The robability o eah sentene deined as the robability o its entities. Ps ( m ( T ( m ( m ex( e (, t T ( m ex( e (, t ; (2 e ( ( ( here e m ( e m 1,..., e m n a vetor o entities in the m-th sentene. Here e treat eah entity as a ategorial random variable hih has domain as all the noun hrases (s and CNPs in the orus. The hole learning roess very muh like the roblem o o-lustering the set o all entities (airs and atterns in the sentenes. We introdue three sets o arameters, β, and γ, here orresond to the assoiation rom atterns to attributes, β orresond to the assoiations rom entities (airs to onets (airs, and γ orresond to the assoiation beteen onet (airs and attributes.

3 e1 1 e2 e1 2 σ(γ e3 σ(γ e4 a σ(γ t : rom a rom t : lives in e4 t : in t : rom The ontology A arsed sentene We deine a omosite onet as the ordered tule o any to basi onets in the ontology. A unary ath P(t,e deined as one o the aths starting rom attern t and end at entity e, here t a unary attern. A orresonding unary eature ires only i both.e and.t are mathed on the arse tree o a sentene. Its eight deined as θ, the multiliation o all the ontology arameters ( s, β s, and γ s along the ath. Similarly, a aire ath P (t,e1,e2 deined as one o the hyer-aths hih start rom a aire attern t and end at entities e1 and e2. And its orresonding aire eature has eight θ, the multiliation o all the ontology arameters ( s, β s, and γ s along the ath.,,, ( ( β γ here a, the number o times aears in eature. b,β and,γ are deined similarly. We use the strethed hyerboli tangent untion σ(x=tanh(x/2=(e x -1/ (e x +1 to transorm the range o β s and γ s rom R + to [0,1. In th ay, it easier to interret their semantis: hether a lass inherits another lass, or hether a lass has the attribute a. It also reerable in the sense that the sarsity o arameters are maintained: σ(x=0 i x=0. To simliy our notations, e deine =, β =, γ = σ(γ. Then Eq (3 an be simliied as, (3 (4 3. Learning POM 3.1. Inerene To simly inerene e use Pseudo Log Likelihood (PLL to aroximate the joint robability deined by Eq. (2. Basially, e redit one NP in the sentene onditioned on the rest o the sentene. One an sho that as number o training data goes to ininity, the arameters estimated rom Maximum Conditional Likelihood Estimation (MLE and maximum Conditional Pseudo-Likelihood Estimation (MCPE beome the same[]. The robability o observing the m-th sentene s deined as

4 1 P( s ; P( e s / e ; ex( ( e, s / e ( m ( m ( m ( m T ( m ( m ( m i i ( m i i i i Zi ( Z ( ex( ( e, s / e N E ex( ( e, s / e ( m T ( m ( m ative T ( m ( m i i i i i i e ative i eie i, here e i denote its i-th entity, N the total number o entities in the ontology, and E i ative the set o entities hih, i laed in the i-th slot, an ativate any o the eatures. We use searh roess to ind E i ative and the set o ativated eatures. We deine the er-instane objetive untion as ( m ( m ( m ( m i i i l ( log P( e s / e ; (6 Algorithm 1 illustrates the roedure to ind E i ative. We irst onsider any unary attern t mathed in the sentene ith e i as it s argument. We an ind all aths orresonding to its don stream entities in the ontology by deth irst searh. Then e onsider any aire attern t mathed in the sentene ith e i as one o its argument and e j as another argument. Algorithm 1: Finding Ativated Paths/Entities Inut: sentene s Outut: the set o ativated entities {E i ative } Initialize: {E i ative = } or any unary attern t mathed in s at the i-th slot or any its immediately related onet E i ative = E i ative Deendent( or any aire attern t mathed in s at the i-th and j-th slot or any its immediately related omosite onet i.hild1 Anestor(e i E j ative = E j ative Deendent(.hild2 i.hild2 Anestor(e j E i ative = E i ative Deendent(.hild1 return {E i ative } For a large ontology, the number o aths (eatures an be very large. Exliitly enumerate eah o them an be very ineiient. Here e develo a dynami rogramming roedure to estimate the gradient o s. Let s deine several useul notations irst. Deine as the rodut o arameters beore (ustream, lose to the atterns on ath, and the rodut o arameters ater (donstream, lose to the entities on ath. Then, e an deomose /. Deine a P. b,. ea. b,. ee. e,. bp (. e; P, b,. P, b These quantities have the olloing relations, hih only deend on loalized inormation, and thereore enable dynami rogramming strategy: (5 (7

5 a I( e a ' e. E ' don( ( e; ' P e. E ' don( ' u( ' ' P a' ' ' P, b ' P b\ P ' u(. CC here.e 1 and.e 2 are the entity on the donstream and ustream side o link resetively, and.cc the set o omosite onets that use.e 2 as one o its argument. ( Para meter Estima tion Generally or any Markov random ields, the gradient o log-likelihood an be ritten as here l( (9 ( x ( x; the exetation under dtribution, and the emirial dtribution. Thereore, l( I(. e etgt (. e; Fative 0 else Here e elise the suer srit m **and subsrit i **or larity. The gradients o s, β s, and γ s an be estimated by the hain rule l( θ I(. e etgt (. e; ative F l( θ l( θ l( θ l( θ 2e ( e 1 l( θ l( θ 2e ( e 1 No e an rerite the gradient in Eq.(8 as: 2 2 (10 (11 li ( θ g( ei g( ei, e j ji ga ( a P( a P( a g( a, b a P( a, b, b P( a, b, b We an use seond order otimization roedure like orthant-e L-BFGS (Andre & Gao, 2007 to estimate the arameters Struture Learning The roblem o learning the struture o ontology has three sub roblems: dover ne rules, dover ne onets, and dovery ne attributes. The details o them are to be designed. 4. Exeriment Initial exeriment an be onduted on small but ell ontrolled orus like Penn Tree Bank. (12