NLP In Algorithms Graph-based - MIT OpenCourseWare

Graph-based Algorithms in NLPRegina BarzilayMITNovember, 2005

Graph-Based Algorithms in NLP In many NLP problems entities are connected by arange of relations Graph is a natural way to capture connectionsbetween entities Applications of graph-based algorithms in NLP:– Find entities that satisfy certain structuralproperties defined with respect to other entities– Find globally optimal solutions given relationsbetween entities

Graph-based Representation Let G(V, E) be a weighted undirected graph– V - set of nodes in the graph– E - set of weighted edges Edge weights w(u, v) define a measure of pairwisesimilarity between nodes u,v0.20.40.30.40.70.1

Graph-based Representation33231253155423504551 2233 4533 55550

Examples of Graph-based Citation Netyescitationreference relationTextnosentsemantic connectivity

Hubs and Authorities Algorithm(Kleinberg, 1998) Application context: information retrieval Task: retrieve documents relevant to a given query Naive Solution: text-based search– Some relevant pages omit query terms– Some irrelevant do include query termsWe need to take into account the authority of the page!

Analysis of the Link Structure Assumption: the creator of page p, by including alink to page q, has in some measure conferredauthority in q Issues to consider:– some links are not indicative of authority (e.g.,navigational links)– we need to find an appropriate balance betweenthe criteria of relevance and popularity

Outline of the Algorithm Compute focused subgraphs given a query Iteratively compute hubs and authorities in thesubgraphHubsAuthorities

Focused Subgraph Subgraph G[W ] over W V , where edgescorrespond to all the links between pages in W How to construct G for a string ?– G has to be relatively small– G has to be rich in relevant pages– G must contain most of the strongestauthorities

Constructing a Focused Subgraph:NotationsSubgraph ( , Eng, t, d) : a query stringEng: a text-based search enginet, d: natural numbersLet R denote the top t results of Eng on

Constructing a Focused Subgraph:AlgorithmSet Sc : R For each page p R Let (p) denote the set of all pages p points toLet (p) denote the set of all pages pointing to pAdd all pages in (p) to S If (p) d thenAdd all pages in (p) to S ElseAdd an arbitrary set of d pages from (p) to S EndReturn S

Constructing a Focused Subgraphbaseroot

Computing Hubs and Authorities Authorities should have considerable overlap interms of pages pointing to them Hubs are pages that have links to multipleauthoritative pages Hubs and authorities exhibit a mutually reinforcingrelationshipHubsAuthorities

An Iterative Algorithm For each page p, compute authority weight x(p) andhub weight y (p)– x(p) 0, x(p) 0 (p) 2(p) 2–(x) 1,(y) 1p s p s Report top ranking hubs and authorities

I operationGiven {y (p) }, compute:x(p) y (p)q:(q,p) Eq1q2q3page px[p]: sum of y[q]for all q pointing to p

O operationGiven {x(p) }, compute:y (p) x(p)q:(p,q) Eq1page py[p]: sum of x[q]for all q pointed to by pq2q3

Algorithm:IterateIterate (G,k)G: a collection of n linked pagedk: a natural numberLet z denote the vector (1, 1, 1, . . . , 1) RnSet x0 : zSet y0 : zFor i 1, 2, . . . , kApply the I operation to (xi 1 , yi 1 ), obtaining new x-weights x iApply the O operation to (x i , yi 1 ), obtaining new y-weights yi Normalize x i , obtaining xiNormalize yi , obtaining yiReturn (xk , yk )

Algorithm: FilterFilter (G,k,c)G: a collection of n linked pagedk,c: natural numbers(xk , yk ) : Iterate(G, k)Report the pages with the c largest coordinates in xk as authoritiesReport the pages with the c largest coordinates in yk as hubs

ConvergenceTheorem: The sequence x1 , x2 , x3 and y1 , y2 , y3converge. Let A be the adjacency matrix of g Authorities are computed as the principaleigenvector of AT A Hubs are computed as the principal eigenvector ofAAT

Subgraph obtained from ord.comFord Motor Companyhttp://www.eff.org/blueribbon.htmlCampaign for Free Speechhttp://www.mckinley.comWelcome to Magellan!http://www.netscape.comWelcome to Netscape!http://www.linkexchange.comLinkExchange — Welcomehttp://www.toyota.comWelcome to Toyota

Authorities obtained fromwww.honda.com0.202http://www.toyota.comWelcome to w.ford.comFord Motor Company0.173http://www.bmwusa.comBMW of North America, .saturncars.comSaturn Web Site0.155http://www.nissanmotors.comNISSAN

PageRank Algorithm (Brin&Page,1998)Original Google ranking algorithm Similar idea to Hubs and Authorities Key differences:– Authority of each page is computed off-line– Query relevance is computed on-line Anchor text Text on the page– The prediction is based on the combination ofauthority and relevance

Intuitive JustificationFrom The Anatomy of a Large-Scale Hypertextual WebSearch Engine (Brin&Page, 1998)PageRank can be thought of as a model of used behaviour. Weassume there is a “random surfer” who is given a web page atrandom and keeps clicking on links never hitting “back” buteventually get bored and starts on another random page. Theprobability that the random surfer visists a page is its PageR ank. And, the d damping factor is the probability at each pagethe “random surfer” will get bored and request another ran dom page.Brin, S., and L. Page. "The Anatomy of a Large-Scale Hypertextual Web Search Engine."WWW7 / Computer Networks 30 no. 1-7 (1998): 107-117.Paper available at http://dbpubs.stanford.edu:8090/pub/1998-8.

PageRank ComputationIterate PR(p) computation:pages q1 , . . . , qn that point to page pd is a damping factor (typically assigned to 0.85)C(p) is out-degree of pP R(qn )P R(q1 ))P R(p) (1 d) d ( . C(q1 )C(qn )

Notes on PageRank PageRank forms a probability distribution over webpages PageRank corresponds to the principal eigenvectorof the normalized link matrix of the web

Extractive Text SummarizationTask: Extract important information from a textFigure removed for copyright reasons. Screenshots of several website text paragraphs.

Text as a GraphS1S2S6S3S5S4

Centrality-based Summarization(Radev) Assumption: The centrality of the node is anindication of its importance Representation: Connectivity matrix based onintra-sentence cosine similarity Extraction mechanism:– Compute PageRank score for every sentence u(1 d)P ageRank(u) dNP ageRank(v)deg(v)v adj [u], where N is the number of nodes in the graph– Extract k sentences with the highest PageRanks score

Does it work? Evaluation: Comparison with human createdsummary Rouge Measure: Weighted n-gram overlap (similarto Bleu)MethodRouge 666

Does it work? Evaluation: Comparison with human createdsummary Rouge Measure: Weighted n-gram overlap (similarto Bleu)MethodRouge 666

Graph-Based Algorithms in NLP Applications of graph-based algorithms in NLP:– Find entities that satisfy certain structuralproperties defined with respect to other entities– Find globally optimal solutions given relationsbetween entities

Min-Cut: Definitions Graph cut: partitioning of the graph into twodisjoint sets of nodes A,B Graph cut weight: cut(A, B) u A,v B w(u, v)– i.e. sum of crossing edge weights Minimum Cut: the cut that minimizescross-partition similarity0.20.40.30.40.70.10.20.40.30.40.70.1

Finding Min-Cut The problem is polynomial time solvable for 2-classmin-cut when the weights are positive– Use max-flow algorithm In general case, k way cut is N P -complete.– Use approximation algrorithms (e.g.,randomized algorithm by Karger)MinCut first used for NLP applications byPang&Lee’2004 (sentiment classification)

Min-Cut for Content SelectionTask: Determine a subset of database entries to beincluded in the generated documentTEAM STAT COMPARISONOakland RaidersNew England -1227:40032:201st DownsTotal YardsPassingRushingPenalties3rd Down Conversions4th Down ConversionsTurnoversPossessionINDIVIDUAL LEADERSNew England PassingOakland PassingCollinsJordanCrockettC/ATT YDS18/39265TD3Oakland RushingCAR T0New England RushingLG1419DillonFaulkOakland ReceivingREC53C/ATT YDS24/38 306CAR235YDS6311TD20LG104New England igure by MIT OCW.

Parallel Corpus for Text GenerationPassingPLAYERCP/AT YDS AVGTD INTGarcia.14/21 195 9.3. . .10.Brunell 17/38 192 6.00 0RushingPLAYERSuggs.REC YDS AVG2282 3.7. . .LG TD251.FumblesPLAYERColesPortisDavisLittle.FUM LOST REC110110001001. . .YDS0000.Suggs rushed for 82 yards and scored a touchdownin the fourth quarter, leading the Browns to a 17-13win over the Washington Redskins on Sunday. Jeff Gar cia went 14-of-21 for 195 yards and a TD for theBrowns, who didn’t secure the win until Coles fum bled with 2:08 left. The Redskins (1-3) can pin theirthird straight loss on going just 1-for-11 on third downs,mental mistakes and a costly fumble by Clinton Por tis. “My fumble changed the momentum”, Portissaid. Brunell finished 17-of-38 for 192yards, but was unable to get into any rhythm becauseCleveland’s defense shut down Portis. The Browns fakeda field goal, but holder Derrick Frost was stopped shortof a first down. Brunell then completed a 13-yard passto Coles, who fumbled as he was being taken down andBrowns safety Earl Little recovered.

Content Selection: Problem Formulation Input format: a set of entries from a relational database– “entry” “raw in a database” Training: n sets of database entries with associatedselection labelsJordanCrockettOakland RushingTDCAR YDS181703208LG1419Figure by MIT OCW. Testing: predict selection labels for a new set of entries

Simple SolutionFormulate content selection as a classification task: Prediction: {1,0} Representation of the 0Goal: Learn classification function P (Y X) that canclassify unseen examplesX Smith, 28, 9, 1 Y1 ?

Potential Shortcoming: Lack of Coherence Sentences are classified in isolation Generated sentences may not be connected in ameaningful wayExample: An output of a system that automaticallygenerates scientific papers (Stribling et al., 2005):Active networks and virtual machines have a long history ofcollaborating in this manner. The basic tenet of this solutionis the refinement of Scheme. The disadvantage of this typeof approach, however, is that public-private key pair and redblack trees are rarely incompatible.

Enforcing Output CoherenceSentences in a text are connectedThe New England Patriots squandered a couple big leads. That wasmerely a setup for Tom Brady and Adam Vinatieri, who pulled out oneof their typical last-minute wins.Brady threw for 350 yards and three touchdowns before Vinatieri kickeda 29-yard field goal with 17 seconds left to lead injury-plagued New Eng land past the Atlanta Falcons 31-28 on Sunday.Simple classification approach cannot enforce coherenceconstraints

Constraints for Content SelectionCollective content selection: consider all the entriessimultaneously Individual constraints:3Branch scores TD710 Contextual constraints:3Brady passes to Branch733Branch scores TD710

Individual PreferencesindY0.80.20.5M0.10.50.9NY M N entries

Combining Individual and 20.10.9NY M N entries

Collective Classificationx C selected entitiesind (x)preference to be selectedlinkL (xi , xj )xi and xj are connected by link of type LMinimize penalty:ind (x) x C ind (x) x C linkL (xi , xj )Lxi C xj C Goal: Find globally optimal label assignment

Optimization Frameworkind (x) x C ind (x) x C linkL (xi , xj )Lxi C xj C Energy minimization framework (Besag, 1986,Pang&Lee, 2004) Seemingly intractable Can be solved exactly in polynomial time (scores arepositive) (Greig et al., 1989)

Graph-Based FormulationUse max-flow to compute minimal cut partitionlinkindY0.80.21.00.5M0.10.50.20.10.9NY M N entries

Learning TaskYMN Learning individual preferences Learning link structure

Learning Individual Preferences Map attributes of a database entry to a feature vectorJordanCrockettOakland RushingTDCAR YDS181703208LG1419Figure by MIT OCW.X Jordan, 18, 17, 0, 14 , Y 1X Crockett, 3, 20, 8, 19 , Y 0 Train a classifier to learn D(Y X)