Natural Language Understanding In A Continuous Space

Natural Language Understandingin a Continuous SpaceKarl-Moritz Hermann, Nal Kalchbrenner, Edward Grefenstette,and Phil Blunsomphil.blunsom@cs.ox.ac.uk

Features and NLPTwenty years ago log-linear models freed us from the shackles ofsimple multinomial parametrisations, but imposed the tyranny offeature engineering.2/38

Features and NLPDistributed/neural models allow us to learn shallow features for ourclassifiers, capturing simple correlations between inputs.3/38

Features and NLPFully connectedlayerK-Max pooling(k 3)FoldingWideconvolution(m 2)Dynamick-max pooling(k f(s) 5)Wideconvolution(m 3)Projectedsentencematrix(s 7)game's the same, just got more fierceDeep learning allows us to learn hierarchical generalisations.Something that is proving rather useful for vision, speech, and nowNLP.4/38

Outline1 Distributed Representations in Compositional Semantics2 From Vector Space Compositional Semantics to MTDistributed Representations in Compositional Semantics5/38

How to Represent Meaning in NLPWe can represent words using a number of approaches Characters POS tags Grammatical roles Named Entity Recognition Collocation and distributional representations Task-specific featuresAll of these representations can be encoded in vectors. Some ofthese representations capture meaning.Distributed Representations in Compositional Semantics6/38

A harder problem: paraphrase detectionQ: Do two sentences (roughly) mean the same?“He enjoys Jazz music” “He likes listening to Jazz” ?A: Use a distributional representation to find out?Distributed Representations in Compositional Semantics7/38

A harder problem: paraphrase detectionQ: Do two sentences (roughly) mean the same?“He enjoys Jazz music” “He likes listening to Jazz” ?A: Use a distributional representation to find out?Most representations not sensible on the sentence level Characters ? POS tags ? Grammatical roles ? Named Entity Recognition ? Collocation and distributional representations ? Task-specific features ?Distributed Representations in Compositional Semantics7/38

Why can’t we extract hierarchical features?The curse of dimensionalityAs the dimensionality of a representation increases, learningbecomes less and less viable due to sparsity.Dimensionality for collocation One word per entry: Size of dictionary (small) One sentence per entry: Number of possible sentences(infinite)) We need a di erent method for representing sentencesDistributed Representations in Compositional Semantics8/38

What is Deep LearningDeep Learning for LanguageLearning a hierarchy of features, where higher levels of abstractionare derived from lower levels.Distributed Representations in Compositional Semantics9/38

A door, a roof, a window: It’s a house,Distributed Representations in Compositional Semantics0.20.30.40.10.50.10.40.70.30.50.30.810/38

CompositionLots of possible ways to compose vectors Addition Multiplication Kronecker Product Tensor Magic Matrix-Vector multiplication .RequirementsNot commutativeEncode its parts?More than parts?Mary likes John 6 John likes MaryMagic carpet Magic CarpetMemory lane 6 Memory LaneDistributed Representations in Compositional Semantics11/38

AutoencodersWe want to ensure that the joint representation captures themeaning of its parts. We can achieve this by autoencoding ourdata at each step:For this to work, our autoencoder minimizes an objective functionover inputs xi , i 2 N and their reconstructions xi0 :N1X 0J xi2xi2iDistributed Representations in Compositional Semantics12/38

Recursive Autoencoders (RAE)We still want to learn how to represent a full sentence (or house).To do this, we chain autoencoders to create a recursive structure.We use a composition functiong (W input bias)g is a non-linearity (tanh, sigm)W is a weight matrixb is a biasDistributed Representations in Compositional Semantics13/38

A di erent task: paraphrase detectionQ: Do two sentences (roughly) mean the same?“He enjoys Jazz music” “He likes listening to Jazz” ?A: Use deep learning to find out!Distributed Representations in Compositional Semantics14/38

Other Applications: Stick a label on top1. Combine label andreconstruction errorE (N, l, ) XErec (n, ) Elbl (vn , l, )n2N1[xn kyn ] rn212Elbl (v , l, ) kl v k2Erec (n, ) 22. Strong results for a numberof tasks:Sentiment AnalysisParaphrase DetectionImage Search.Distributed Representations in Compositional Semantics15/38

Convolution Sentence ModelsDeep learning is suppose to learn the features for us, so can we doaway with all this structural engineering and forget about latentparse trees?Distributed Representations in Compositional Semantics16/38

Convolution Sentence ModelsOpentheDistributed Representations in Compositional SemanticspodbaydoorsHAL17/38

Convolution Sentence Modelsm 2OpentheDistributed Representations in Compositional SemanticspodbaydoorsHAL17/38

Convolution Sentence Modelsm 3m 2OpentheDistributed Representations in Compositional SemanticspodbaydoorsHAL17/38

Convolution Sentence Modelsm 3m 2OpentheDistributed Representations in Compositional SemanticspodbaydoorsHAL17/38

A CSM for Dialogue Act TaggingA: My favourite show is Masterpiece Theatre.Statement-Non-OpinionA: Do you like it by any chance?Yes-No-QuestionB: Oh yes!Yes-AnswersA: You do!Declarative Yes-No-QB: Yes, very much.Yes-AnswersA: Well, wouldn't you know.ExclamationB: As a matter of fact, I prefer public television.Statement-non-opinionB: And, uh, I have, particularly enjoy Englishcomedies.Statement-non-opinionDistributed Representations in Compositional Semantics18/38

A CSM for Dialogue Act TaggingDave: Hello HAL, doyou read me HAL?HAL: Affirmative, Dave,I read you.Distributed Representations in Compositional SemanticsDave: Open the pod baydoors, HAL.HAL: I'm sorry, Dave,I'm afraid I can't do that.19/38

A CSM for Dialogue Act TaggingDave: Hello HAL, doyou read me HAL?HAL: Affirmative, Dave,I read you.Distributed Representations in Compositional SemanticsDave: Open the pod baydoors, HAL.HAL: I'm sorry, Dave,I'm afraid I can't do that.19/38

A CSM for Dialogue Act TaggingDave: Hello HAL, doyou read me HAL?HAL: Affirmative, Dave,I read you.Distributed Representations in Compositional SemanticsDave: Open the pod baydoors, HAL.HAL: I'm sorry, Dave,I'm afraid I can't do that.19/38

A CSM for Dialogue Act TaggingDave: Hello HAL, doyou read me HAL?HAL: Affirmative, Dave,I read you.Dave: Open the pod baydoors, HAL.HAL: I'm sorry, Dave,I'm afraid I can't do that.s4HDaveOHALIx3hi g(Ixi1 Hi1hi1h4p4 Ssi )ipi softmax(O hi )Distributed Representations in Compositional Semantics19/38

A CSM for Dialogue Act TaggingDave: Hello HAL, doyou read me HAL?HAL: Affirmative, Dave,I read you.Dave: Open the pod baydoors, HAL.HAL: I'm sorry, Dave,I'm afraid I can't do that.s4HDaveOHALIx3hi g(Ixi1 Hi1hi1h4p4 Ssi )ipi softmax(O hi )State of the art results while allowing online processing of dialogue.Distributed Representations in Compositional Semantics19/38

Convolution Sentence Models: Question Answering?x : have-population-of(vancouver, x)Whatisthe population of Vancouver ?Competitive with a template based approach with lots of handengineered features.Distributed Representations in Compositional Semantics20/38

Convolution Sentence ModelsProjectedsentencematrix(s 7)The cat sat on the red matDistributed Representations in Compositional Semantics21/38

Convolution Sentence ModelsWideconvolution(m 3)Projectedsentencematrix(s 7)The cat sat on the red matDistributed Representations in Compositional Semantics21/38

Convolution Sentence ModelsDynamick-max pooling(k f(s) 5)Wideconvolution(m 3)Projectedsentencematrix(s 7)The cat sat on the red matDistributed Representations in Compositional Semantics21/38

Convolution Sentence ModelsWideconvolution(m 2)Dynamick-max pooling(k f(s) 5)Wideconvolution(m 3)Projectedsentencematrix(s 7)The cat sat on the red matDistributed Representations in Compositional Semantics21/38

Convolution Sentence ModelsFoldingWideconvolution(m 2)Dynamick-max pooling(k f(s) 5)Wideconvolution(m 3)Projectedsentencematrix(s 7)The cat sat on the red matDistributed Representations in Compositional Semantics21/38

Convolution Sentence ModelsK-Max pooling(k 3)FoldingWideconvolution(m 2)Dynamick-max pooling(k f(s) 5)Wideconvolution(m 3)Projectedsentencematrix(s 7)The cat sat on the red matDistributed Representations in Compositional Semantics21/38

Convolution Sentence ModelsFully connectedlayerK-Max pooling(k 3)FoldingWideconvolution(m 2)Dynamick-max pooling(k f(s) 5)Wideconvolution(m 3)Projectedsentencematrix(s 7)The cat sat on the red matDistributed Representations in Compositional Semantics21/38

Small Sentiment TaskSentiment prediction on the Stanford movie reviews dataset.Distributed Representations in Compositional Semantics22/38

Large Sentiment TaskAccuracy on the larger Twitter sentiment dataset.Distributed Representations in Compositional Semantics23/38

Question Classification TaskClassifierFeaturesH IERunigram, POS, head chunksNE, semantic relationsAcc. (%)91.0M AX E NTunigram, bigram, trigramPOS, chunks, NE, supertagsCCG parser, WordNet92.693.6M AX E NTunigram, bigram, trigramPOS, wh-word, head wordword shape, parserhypernyms, WordNet95.0SVMunigram, POS, wh-wordhead word, parserhypernyms, WordNet60 hand-coded rulesClassifierASVMB I NBM AX E NTM AX -TDNNNB OWDCNNM AX -TDNNunsupervised vectors84.4NB OWunsupervised vectors88.2DCNNunsupervised vectors93.0Table 3: Accuracy ondataset. The three non-neuon unigram and bigram feaported from (Go et al., 200function. At training time wpenultimate layer after the(Hinton et al., 2012).We see that the DCNNTable 2: Accuracy of six-way question classificaforms the other neural anSix-way question tionclassificationthe TRECquestionsdataset, e.g.on the TREConquestionsdataset.The secondThe NBoW performs simicolumn details the external features used in then-gram based ous approaches. The first four results are reforms worse than the NBospectively from Li and Roth(2002), Blunsom et al.Output:NUMBERcessive pooling of the max(2006), Huang et al. (2008) and Silva et al. (2011).Distributed Representations in Compositional Semanticslatter discards most of24/38the s

Feature: not only . . . but ransitionsbeonlymerelycrash,movieDistributed Representations in Compositional elyis25/38

Feature: as . . . as . . . leoncrimeathleteandlanefirestraintDistributed Representations in Compositional specialastheitmayandbringsan.theeffectswarmth26/38

Feature: kesquirkyDistributed Representations in Compositional dyouactedcastfrom scinatingimpressiveamusingintelligencecomedy27/38

Outline1 Distributed Representations in Compositional Semantics2 From Vector Space Compositional Semantics to MTFrom Vector Space Compositional Semantics to MT28/38

Generalisation in MT我From Vector Space Compositional Semantics to MT一杯白葡萄酒。29/38

Generalisation in MTLambda CalculusGeneralisation我From Vector Space Compositional Semantics to MT一杯白葡萄酒。29/38

Generalisation in MTi 'd like a glass of white wine , please .GenerationLambda CalculusGeneralisation我From Vector Space Compositional Semantics to MT一杯白葡萄酒。29/38

Generalisation in MTi 'd like a glass of white wine , please ormal logical representations are very hard to learn from data. Letus optimistically assume a vector space and see how we go.From Vector Space Compositional Semantics to MT29/38

GenerationA simple distributed representation language model:R(wn-2)C2 xpn Cnp(wn wnR(wn-1)C12 R(wn 2 )1 , wn 2 )/xpn Cn1 R(wn 1 )exp (R(wn )T pn )This is referred to as a log-bilinear model.From Vector Space Compositional Semantics to MT30/38

GenerationA simple distributed representation language model:R(wn-2)C2xpn Cnp(wn wnR(wn-1) C1x2 R(wn 2 ) Cn1 , wn 2 )/pn 1 R(wn 1 )exp (R(wn )T (pn ))Adding a non-linearity gives a version of what is often called aneural, or continuous space, LM.From Vector Space Compositional Semantics to MT30/38