CSE 5525: Foundaons Of Speech And Language Processing .

CSE 5525: Founda.ons ofSpeech and Language ProcessingDialogueHuan Sun (CSE@OSU)Slides were largely adapted from Prof. Greg DurreH @ UT Aus.n.Some images/examples were from the two textbooks by (1) Jurafsky and Mar.n and (2) Eisenstein.

This Lecture‣ Review QA & connect QA with dialogue‣ Classic chatbots‣ Neural chatbots‣ Task-oriented dialogue

Recall: Seman.c Parsing‣ Parsing ques.on x intoderiva,on d:Berant et al. (2013)

Recall: Seman.c Parsing‣ Parsing ques.on x intoderiva,on d:1. What approaches cannot be used for this task?(1) Feature extraction log-linear model(2) Seq2seq model(3) Classification of x with fixed categoriesBerant et al. (2013)

Recall: SQuAD dataset2. This dataset is sometimes regarded as *not* realistic. Why?(1) Questions are synthesized by computers(2) Answers are not accurate(3) It cannot be used to develop information retrieval models(4) Humans created questions when looking at the paragraphRajpurkar et al. (2016)

Recall: Mul.-hop dataset HotpotQADoc 1Ques%on: What government posi,on was held by the woman who portrayedCorliss Archer in the film Kiss and Tell ?Example picked from HotpotQA [Yang et al., 2018]Shirley Temple Black was an American actress, businesswoman, and singer As an adult, she served as Chief of Protocol of the United States Same en.tyDoc 2Kiss and Tell is a comedy film in which 17-year-old Shirley Temple acts as Corliss Archer .Doc 3Same en.tyMeet Corliss Archer is an American television sitcom that aired on CBS 3. Which one below is NOT a drawback of this dataset?(1) High lexical word overlap between a question and a single Doc(2) Questions are too long and may involve multiple relations.(3) Answer spans may be found without context considered

Recall: BERT model for QA4. Which QA dataset(s) is the above formulation more often used for?(1) SQuAD(2) HotpotQA(3) GeoQueryDevlin et al. (2019)

Dialogue8

Search/QA as Dialogue‣ Dialogue is a very natural way to find informa.on from a search engineor a QA system

Search/QA as Dialogue

Search/QA as Dialogue‣ “Has Chris PraH won an Oscar?” / “Has he won an Oscar”

QA as Dialogue‣ Dialogue is a very natural way to find informa.on from a search engineor a QA systemIyyer et al. (2017)

QA as Dialogue‣ Conversa.onal machine reading:answer ques.ons based on apassage‣ Interes.ng and poten.ally usefulideaSaeidi et al. (2018)

This Lecture‣ QA as dialogue‣ Classic chatbots‣ Neural chatbots‣ Task-oriented dialogue

Chatbots

ELIZA‣ Created 1964-1966 atMIT, heavily scriptedEliza responds like a psychotherapistWeizenbaum (1966)

ELIZA‣ Created 1964-1966 atMIT, heavily scriptedFeatured in Young Sheldon: https://www.youtube.com/watch?v 2V9skeVray4image source: https://web.njit.edu/ ronkowit/eliza.htmlWeizenbaum (1966)

ELIZA‣ PaHern matching and subs.tu.on, using transforma.on rule(.*) you (.*) meWhy do you think I 2 you?

ELIZA‣ PaHern matching and subs.tu.on, using transforma.on ruleMy (.) (.*)What else comes to mind when youthink of your 1?

ELIZA‣ Iden.fy keyword, iden.fy context, apply transforma.on rule(.*) you (.*) meWhy do you think I 2 you?My (.) (.*)What else comes to mind when youthink of your 1?‣ Very liHle need to generate new content, but can only have one type ofconversa.on; not coherent

Cleverbot‣ Carpenter (1986), online system built in 2006

Cleverbot‣ “Nearest neighbors”: A user says statement A - Cleverbot finds ahuman response to statement A in human-human or human-computerUser: Hi there cleverbot.chats - repeat thatCleverbot: How are you?User: Doing okay. what plans do you have today?Cleverbot: I do revision.User: What are you revising for?Cleverbot: Maths history.User: How old are you?Cleverbot: That's none of your business.User: What is maths history?Cleverbot: I don’t know, farming maybeCheck ta-for-machine-learning/

Cleverbot‣ Carpenter (1986), online system built in 2006‣ “Nearest neighbors”: human says statement A, find a human responsein human-human or human-computer chats to statement A, repeat that‣ Can ouen give sensibleanswers, but the bot doesn’treally impose high-leveldiscourse structureUser: Hi there cleverbot.Cleverbot: How are you?User: Doing okay. what plans do you have today?Cleverbot: I do revision.User: What are you revising for?Cleverbot: Maths history.User: How old are you?Cleverbot: That's none of your business.User: What is maths history?Cleverbot: I don’t know, farming maybe

Data-Driven Approacheson TwitterRiHer et al. (2011)

Data-Driven Approaches‣ Can treat as a machine transla.on problem: “translate” from currentuHerance to next oneObservation: structural relationship between Stimulus and ResponseRiHer et al. (2011)

Data-Driven Approaches‣ Can treat as a machine transla.on problem: “translate” from currentuHerance to next oneObservation: structural relationship between Stimulus and ResponseWhat words/phrases are aligned in this example?RiHer et al. (2011)

Data-Driven Approaches‣ Can treat as a machine transla.on problem: “translate” from currentuHerance to next one‣ How to decide larger phrase (not just word-word) alignment?RiHer et al. (2011)

Data-Driven Approaches‣ Can treat as a machine transla.on problem: “translate” from currentuHerance to next one‣ How to decide larger phrase (not just word-word) alignment?‣ use sta.s.cal measures to filter out phrase pairs with low correla.onRiHer et al. (2011)

Data-Driven Approachesproposed method phrase alignmentis weakerRiHer et al. (2011)

What other approaches can you think of?RiHer et al. (2011)

Neural Chatbots

Seq2seq modelsIWhat areyou doingam going home [STOP] s ‣ Just like the machine transla.on task men.oned earlier‣ can train seq2seq models for this task

Sub.tles Data for Model Developmentdo you want to meet your sponsor for the last 10 years ?of course ! but he doesn’t want to see me !and where had you been just before ?i' d been to the palace of the legion of honor , the art gallery .yeah , we were just going to hit up taco bell .well , it' s my pleasure .

Sub.tles Datado you want to meet your sponsor for the last 10 years ?of course ! but he doesn’t want to see me !and where had you been just before ?i' d been to the palace of the legion of honor , the art gallery .yeah , we were just going to hit up taco bell .well , it' s my pleasure .‣ Why might this model perform poorly? What might it be bad at?

Lack of Diversity‣ Why?Li et al. (2016)

Lack of Diversity‣ Training to maximize likelihood gives a system that prefers commonresponses:max P(response input)Li et al. (2016)

Lack of Diversity‣ Solu.on: mutual informa.on criterion; response R should bepredic.ve of user uHerance U as well‣ Standard condi.onal likelihood: log P (R U )P (R, U )‣ Mutual informa.on: log log P (R U )P (R)P (U )log P (R)‣ log P(R) probabili.es under a language modelLi et al. (2016)

38

Lack of Diversity(maximum mutual info)‣ OpenSub.tles dataLi et al. (2016)

Another idea: Specificity‣ Train a specificity classifier on labeled dataI don’t know SPECIFICITY 1Going to the store SPECIFICITY 3Ko, DurreH, Li (2019)

Specificity‣ Train a specificity classifier on labeled dataI don’t know SPECIFICITY 1Going to the store SPECIFICITY 3‣ When training the decoder, condi.on on the automa&cally predictedspecificity of the responseIWhat aredon’t know [STOP]you doing s SPECIFICITY 1 (nonspecific)Ko, DurreH, Li (2019)

Specificity‣ At test .me, setthe specificitylevel higher toget less genericresponsesIWhat aredon’t know [STOP]you doing s SPECIFICITY 1 (nonspecific)Going toWhat areyou doingthe store [STOP] s SPECIFICITY 4 (specific)Ko, DurreH, Li (2019)

Specificityissues in the generated responses‣ Can use other models to try to fix these issues. But the facts are s.llall made up, even if they make senseKo, DurreH, Li (2019)

PersonaChatZhang et al. (2018)

State of Chatbots‣ Can force chatbots to give consistent answers with a persona, buts.ll probably not very interes.ng‣ “Wizard of Wikipedia:” chatbot that can discuss topics by retrievingfrom Wikipedia [Dinan et al., 2019]‣ XiaoIce: Microsou chatbot in Chinese; according to [1], 660 milliononline users, over 30 billion conversa.ons people do seem to liketalking to them ?[1] ons-andwins-over-millions-of-fans/

Task-Oriented Dialogue

Task-Oriented Dialogue‣ Personal assistants / API front-ends:Siri, find me a good sushirestaurant in ChelseaSushi Seki Chelsea is a sushirestaurant in Chelsea with 4.4 starson Google

Task-Oriented Dialogue‣ Personal assistants / API front-ends:Siri, find me a good sushirestaurant in ChelseaSushi Seki Chelsea is a sushirestaurant in Chelsea with 4.4 starson GoogleHow expensive is it?Entrees are around 30 eachFind me something cheaper

Task-Oriented Dialogue‣ Customer service:Hey Alexa, why isn’t my Amazonorder here?Let me retrieve your order.Your order was scheduled to arriveat 4pm today.It never cameOkay, I can put you through tocustomer service.

Air Travel Informa.on Service (ATIS)‣ Given an uHerance, predict a domain-specific seman.c interpreta.on‣ Can formulate as seman.c parsing, but simple “slot-filling” solu.ons(classifiers) work well tooDARPA (early 1990s), Figure from Tur et al. (2010)

Intents‣ 29 different intentswhich flights go from cleveland to indianapolis on april fiuhIntent: flightdoes tacoma airport offer transporta.on from the airport to thedowntown areaIntent: ground servicewhat days of the week do flights from san jose to nashville fly onIntent: day namewhat meals are served on american flight 811 from tampa to milwaukeeIntent: meal

Joint Intent Classifica.on and Tagging‣ RNN jointlypredicts intentand slot tags?Liu and Lane (2016)

Joint Intent Classifica.on and Tagging‣ RNN jointlypredicts intentand slot tagswhat is c here?

Joint Intent Classifica.on and Tagging‣ RNN jointlypredicts intentand slot tagsLiu and Lane (2016)

Air Travel Informa.on Service (ATIS)as conversa.onal seman.c parsingSuhr et al. (2018)

Air Travel Informa.on Service (ATIS)as conversa.onal seman.c parsing‣ Need to use dialogue context to do the right thing. Here we’reappending American Airlines as a constraint to the previous query‣ seq2seq model mapping to query with copy mechanismSuhr et al. (2018)

Full Dialogue Task‣ Parsing / language understandingis just one piece of a systemYoung et al. (2013)

Full Dialogue Task‣ Parsing / language understandingis just one piece of a system‣ User uHerance - update dialogue state - take an ac.on decided bypolicy (e.g., query the restaurant database) - say somethingYoung et al. (2013)

Full Dialogue Task‣ Parsing / language understandingis just one piece of a system‣ Dialogue state: reflects anyinforma.on about theconversa.on (e.g., searchhistory)‣ User uHerance - update dialogue state - take an ac.on decided bypolicy (e.g., query the restaurant database) - say something‣ Much more complex than chatbots!Young et al. (2013)

‣ Full dialogue: User uHerance - update dialogue state - take an ac.ondecided by policy (e.g., query the restaurant database) - say something“Find me a good sushi restaurant in Chelsea”restaurant type - sushilocation - Chelseacurr result - execute search()“Sushi Seki Chelsea is a sushi restaurant in Chelsea with4.4 stars on Google”“How expensive is it?”get value(cost, curr result)“Entrees are around 30 each”

POMDP-based Dialogue Systems‣ POMDP: user is the “environment,” an uHerance is a noisy signal of state‣ Dialogue model: can look like a parser or any kind of encoder model‣ Generator: use templates or seq2seq modelReinforcement learning type of approachYoung et al. (2013)

POMDP-based Dialogue Systems‣ POMDP: user is the “environment,” an uHerance is a noisy signal of state‣ Dialogue model: can look like a parser or any kind of encoder model‣ Generator: use templates or seq2seq model‣ Where do rewards come from?Young et al. (2013)