Natural Language Processing - University Of California, Berkeley

Natural Language ProcessingInfo 159/259Midterm review (Mar 9, 2021)In-class questions:http://bit.ly/nlpqsDavid Bamman, UC Berkeley

Midterm Thursday 3/11, 2:10-3:30pm PST on bCourses. This midterm must be carriedout entirely independently! You are free to use any of your notes, readings, or lecture material; themidterm will cover all material through 3/4 lecture on parsing. Questions will resemble what you have already experienced with the quizzes(e.g., multiple choice) along with some numerical answer/fill-in-the-blankquestions as well. Have your calculator ready! You can expect more attention to the topics we cover in lectures and in thehomeworks, but everything in the lectures and readings is fair game.

Big ideas Classification Naive Bayes, Logisticregression, feedforward neuralnetworks, CNN, BERT Language modelingMarkov assumption,featurized, neuralProbability/statistics in NLPWhere does NLP data come from? Annotation process Interannotator agreementChain rule of probability,independence, Bayes’ rule

Big ideas Lexical semantics and wordrepresentations Distributional hypothesis Distributed representations Subword embedding models Contextualized wordrepresentations (ELMO/BERT)Evaluation metrics (accuracy,precision, recall, F score, perplexity,parseval) Sequence labeling POS, NER Methods: HMM, MEMM, CRF,RNN, BiRNN, BERT Phrase-structure parsing,CFG, PCFG CKY for recognition, parsingTrees

Big ideas What defines the models we’ve seen so far? What formally distinguishes anHMM from an MEMM? How do we train those models? For all of the problems we’ve seen (sentiment analysis, POS tagging, phrasestructure parsing), how do we evaluate the performance of different models? If faced with a new NLP problem, how would you decide between thealternatives you know about? How would you adapt an MEMM, for example,to a new problem?

Can we get a high-level overview of what distinguishes each neural network (ex: CNN, RNN,bidirectional, etc.) and an example/case of when we would use each? Feedforward NN (e.g., multi-layer perceptron). Inputs: fixed-dimensionalfeature vector (e.g., bag of words representation; not original wordsequence). Output: single prediction for that sequence. CNN. Inputs: original word sequence. Performs the same action(convolution) over each window in the original sequence. Learns to identifyimportant ngrams in the original sequence. Can be used for documentclassification (by adding a softmax layer on top), or to generate arepresentation for the entire sequence.

Can we get a high-level overview of what distinguishes each neural network (ex: CNN, RNN,bidirectional, etc.) and an example/case of when we would use each? RNN. Inputs: original word sequence. Generates a representation for eachtoken in a sequence that is aware of words on the left context (forward RNN),right context (backward RNN) or both contexts (bidirectional RNN).Information from word i needs to pass through the hidden states for words[i 1, , j-1] to influence word j. Can be used for document classification (byadding a softmax layer on top of the final word representation), sequencelabeling (by adding a softmax layer on top of each token representation), orgenerating representations for each token in the sequence.

Can we get a high-level overview of what distinguishes each neural network (ex: CNN, RNN,bidirectional, etc.) and an example/case of when we would use each? Transformer. Inputs: original word sequence. Generates a representation foreach token in a sequence, for each layer in the transformer, by usingattention over all words in the previous layer. The representation for word iand layer k has simultaneous access to all word representations at layer k-1(representations not diluted as they are for an RNN). Can be used fordocument classification (by adding a softmax layer on top of the final wordrepresentation), sequence labeling (by adding a softmax layer on top of eachtoken representation), or generating representations for each token in thesequence.

Can we get a high-level overview of what distinguishes each neural network (ex: CNN, RNN,bidirectional, etc.) and an example/case of when we would use each?FeedforwardNN (e.g. MLP)InputGeneratesrepresentation foreach token?Generatesrepresentation forentire input?Common tasksCNNFixed-dimensionalOriginal sequencefeature vectorNoYes (e.g., hiddenlayer in l sequenceOriginal sequenceYesYesYes (e.g.,Yes (e.g., hiddenconcatenation of allstate for last tokenfilters afterin ntationlearningYes (e.g., [CLS]token in BERT)DocumentSequence labeling,classification,representationsequence labeling,learningrepresentationlearning

RBBiLSTM for each word; concatenatefinal state of forward LSTM, backwardLSTM, and word embedding asrepresentation for a word.bigly4 3 -2 -1 4 9 0 0 0 0 0Lample et al. (2016), “Neural Architectures for Named EntityRecognition”000000.7 -1.1 -5.40.7 -1.1 -5.4word embeddingbibcharacter BiLSTM2.7gi3.1-1.4-2.30.72.7lg3.1-1.4-2.30.7 2.7yl3.1-1.4-2.30.72.7y3.1-1.4-2.30.7 2.73.1-1.4-2.30.710

RBCharacter CNN for each word; concatenatecharacter CNN output and wordembedding as representation for a word.biglyChu et al. (2016), “Named Entity Recognition withBidirectional LSTM-CNNs”42.7max -2.30.7 0-2.33.1l3 -2 -1 40.7 2.7-1.4-2.33.1-1.4-2.30.70.7 2.73.1-1.40000word embedding-2.30.7y11

Can you summarize when to use each model (classification, language modeling, etc.), and which arediscriminative vs. generative? Classification: predict a label for each document. Language modeling: applications that need information about fluency (autocorrect, translation,speech recognition, OCR); and a general framework for learning representations of words. Word representations: learn representations of words (symbols generally) that are sensitive totheir context of use, both aggregate context (e.g. word2vec) and local sentence context (e.g.,BERT). Sequence labeling: predict a label for each token. Parsing: predict syntactic structure (generally, any tree structure).

Generative vs.Discriminative modelsGenerative models specify a joint distribution over the labels and the data. With thisyou could generate new data P(X, Y ) P(Y ) P(X Y ) Discriminative models specify the conditional distribution of the label y given thedata x. These models focus on how to discriminate between the classesP(Y X )

Naive BayesBinary logistic regressionP (Y y)P (X x Y y)P (Y y X x) Py P (Y y)P (X x Y y) , ) ( expn 1HMMP (x1 , . . . , xn , y1 , . . . , yn )i 1 nP (yi yi1)i 1P (xi yi )nMEMMi 1CRFP (y x, ) P (yi yi1 , x,)exp( (x, y) )y Y exp( (x, y ))

Can you go over BERT vs ELMO, and how attention works?

ELMoBERTStacked BiRNN trained to predict nextword in language modeling ters et al. 2018Transformer-based model to predict maskedword using bidirectional context next thethethethethetheDevlin et al. 2019

ELMo Peters et al. (2018), “Deep Contextualized Word Representations” (NAACL) Big idea: transform the representation of a word (e.g., from a static wordembedding) to be sensitive to its local context in a sentence and optimizedfor a specific NLP task. Output word representations that can be plugged into just about anyarchitecture a word embedding can be used.

ELMo Train a bidirectional RNN language model with L layers on a bunch of text. Learn parameters to combine the RNN output across all layers for eachword in a sentence for a specific task (NER, semantic role labeling,question answering etc.). Large improvements over SOTA for lots of NLPproblems.

BERT Transformer-based model (Vaswani et al. 2017) to predict masked wordusing bidirectional context next sentence prediction. Generates multiple layers of representations for each token sensitive to itscontext of use.

Each token in the input starts out representedby token and position -0.9-0.7e1,1e1,2e1,3Thedogbarked0.2

The value for time step j at layer i is the resultof attention over all time steps in the previouslayer 2

.7-1.11.6-0.3-0.9-0.7e1,1e1,2e1,3Thedogbarked0.2

1,3Thedogbarked0.2

.7-1.11.6-0.3-0.9e1,1e1,2e1,3Thedogbarked

hedogbarked

1.6-0.3-0.9e1,1e1,2e1,3Thedogbarked

gbarked

At the end of this process, we have onerepresentation for each layer for each Thedogbarked

BERT Learn the parameters of this model with two objectives: Masked language modeling Next sentence prediction

Masked LM Mask one word from the input and try to predict that word as the output More powerful than an RNN LM (or even a BiRNN LM) since it can reasonabout context on both sides of the word being predicted. A BiRNN models context on both sides, but each RNN only has access toinformation from one direction.

0.3 0.2 0.70e3,1-1.7 -0.6 -0.5 -1.6-1.6 -0.6 -0.3 -0.4-1 -0.6 2.3 0.9-0.1 0.2 0.4 -0.61.1 -1.5 0.3 0.4-0.4 -1.1 -0.6 -0.3e3,2e3,3e3,4e3,5e3,60.5 1.9 -1.2 -0.2-0.6 -0.7 -1.4 -2.10.6 0.220.9-1.10-1.6 -0.71.9 0.6 -0.4 -0.3e2,1e2,2e2,3e2,4e2,5e2,6-0.5 -0.5 0.6 0.70.1 0.7 -0.5 0.82.5 -1.7 -0.9 -2.8-0.5 -1.1 -0.6 1.40.6 -1.7 1.6 2.11.1 -0.9 0.5 [SEP]

dog0.3 0.2 0.70e3,1-1.7 -0.6 -0.5 -1.6-1.6 -0.6 -0.3 -0.4-1 -0.6 2.3 0.9-0.1 0.2 0.4 -0.61.1 -1.5 0.3 0.4-0.4 -1.1 -0.6 -0.3e3,2e3,3e3,4e3,5e3,60.5 1.9 -1.2 -0.2-0.6 -0.7 -1.4 -2.10.6 0.220.9-1.10-1.6 -0.71.9 0.6 -0.4 -0.3e2,1e2,2e2,3e2,4e2,5e2,6-0.5 -0.5 0.6 0.70.1 0.7 -0.5 0.82.5 -1.7 -0.9 -2.8-0.5 -1.1 -0.6 1.40.6 -1.7 1.6 2.11.1 -0.9 0.5 [SEP]

bark0.3 0.2 0.70e3,1-1.7 -0.6 -0.5 -1.6-1.6 -0.6 -0.3 -0.4-1 -0.6 2.3 0.9-0.1 0.2 0.4 -0.61.1 -1.5 0.3 0.4-0.4 -1.1 -0.6 -0.3e3,2e3,3e3,4e3,5e3,60.5 1.9 -1.2 -0.2-0.6 -0.7 -1.4 -2.10.6 0.220.9-1.10-1.6 -0.71.9 0.6 -0.4 -0.3e2,1e2,2e2,3e2,4e2,5e2,6-0.5 -0.5 0.6 0.70.1 0.7 -0.5 0.82.5 -1.7 -0.9 -2.8-0.5 -1.1 -0.6 1.40.6 -1.7 1.6 2.11.1 -0.9 0.5 SEP]

BERT Deep layers (12 for BERT base, 24 for BERT large) Large representation sizes (768 per layer) Pretrained on English Wikipedia (2.5B words) and BooksCorpus (800Mwords)

Attention Let’s incorporate structure (and parameters) into a network thatcaptures which elements in the input we should be attending to (andwhich we can ignore).

v ℛH2.73.1 -1.4 -2.3 0.7Define v to be a vector to be learned; think of it as an “importantword” vector. The dot product here measures how similar each inputvector is to that “important word” vector2.7-0.7 -0.8 -1.3 -0.2 -0.92.3 1.5 1.1 1.4 1.3-0.9 -1.5 -0.7 0.9 0.2-0.1 -0.7 -1.6 0.2 0.6Ilovedthemovie!x1x2x3x4x53.1 -1.4 -2.3 0.7

v ℛH2.73.1 -1.4 -2.3 0.7-3.42.4-0.8-1.21.7r1 v x1r2 v x2r3 v x3r4 v x4r5 v x52.7-0.7 -0.8 -1.3 -0.2 -0.92.3 1.5 1.1 1.4 1.3-0.9 -1.5 -0.7 0.9 0.2-0.1 -0.7 -1.6 0.2 0.6Ilovedthemovie!x1x2x3x4x53.1 -1.4 -2.3 0.7

Convert r into a vector of normalized weights that sum to 1.a softmax(r)a00.640.020.020.32r-3.42.4-0.8-1.21.7r1 v x1r2 v x2r3 v x3r4 v x4r5 v x52.7-0.7 -0.8 -1.3 -0.2 -0.92.3 1.5 1.1 1.4 1.3-0.9 -1.5 -0.7 0.9 0.2-0.1 -0.7 -1.6 0.2 0.6Ilovedthemovie!x1x2x3x4x53.1 -1.4 -2.3 0.7

y1.9 -0.2 -1.1 -0.2 -0.7x1a1 x2a2 x3a3 x4a4 x5a5weighted sum2.73.1 -1.4 -2.3 0.7I-0.7 -0.8 -1.3 -0.2 -0.92.3 1.5 1.1 1.4 1.3-0.9 -1.5 -0.7 0.9 0.2-0.1 -0.7 -1.6 0.2 0.6lovedthemovie!

Attention Lots of variations on attention: Linear transformation of x into before dotting with v Non-linearities after each operation. “Multi-head attention”: multiple v vectors to capture different phenomenathat can be attended to in the input. Hierarchical attention (sentence representation with attention over words document representation with attention over sentences).

For CNNs, what is the purpose of max pooling and global pooling? Maxpooling down-samples a layer by selecting a single point fromsome set Max-pooling over time (global max pooling) selects the largest valueover an entire sequence. This provides a mechanism to have fixeddimensional representation for variable-sized inputs. Global max pooling is very common for NLP problems.