EXAMPLE-BASED ERROR RECOVERY STRATEGY FOR SPOKEN DIALOG SYSTEM

EXAMPLE-BASED ERROR RECOVERY STRATEGY FOR SPOKEN DIALOG SYSTEMCheongjae Lee, Sangkeun Jung, Donghyeon Lee, and Gary Geunbae LeePohang University of Science and TechnologyDepartment of Computer Science and EngineeringPohang, South Korea{lcj80, hugman, semko, gblee}@postech.ac.krABSTRACTError handling has become an important issue in spoken dialog systems. We describe an example-based approach todetect and repair errors in an example-based dialog modeling framework. Our approach to error recovery is focusedon the re-phrase strategy with a system and a task guidanceto help the novice users to re-phrase well-recognizable andwell-understandable input. The dialog system gives possibleutterance templates and contents related to the current situation when errors are detected. An empirical evaluation of thecar navigation system shows that our approach is effective tothe novice users for operating the spoken dialog system.Index Terms— Error Handling, Error Detection, ErrorRecovery, Example-based Dialog Modeling1. INTRODUCTIONThe development of spoken dialog systems involves humanlanguage technologies which must cooperate in order to answer user queries. Since the performance in human languagetechnologies such as automatic speech recognition and natural language understanding have been improved, this advancehas made it possible to develop spoken dialog systems formany different application domains.Nevertheless, there are inevitable bottlenecks for practicalspoken dialog systems. One of the critical problems whichmust be considered by the dialog manager is the propagationof errors through prior modules. Errors in spoken dialog systems are prevalent due to speech recognition or language understanding errors. The recognition module must process thespontaneous speech with noisy environments. Consequently,the recognized utterance by this module inherently incorporate some errors. The recognition errors in practical systemsare further aggravated by the large vocabulary and large variability of the user. The understanding module could alsoTHIS WORK WAS SUPPORTED BY GRANT NO. RTI04-02-06FROM THE REGIONAL TECHNOLOGY INNOVATION PROGRAM OFTHE MINISTRY OF COMMERCE, INDUSTRY AND ENERGY (MOCIE).978-1-4244-1746-9/07/ 25.00 2007 IEEE538make its own errors which are mainly due to the lack of coverage of the semantic domain when faced with strange inputs. Finally, the semantic representation provided to the dialog manager might also cause system response errors. Theseerrors appear across all domains and dialog genres.To avoid these errors, a basic solution is to improve theaccuracy and robustness of the recognition and understandingprocess. However, the spoken dialog system should also beable to adopt mechanisms for detecting and repairing potential errors at the conversational level since the development ofthe perfect conventional systems is impossible. The goal oferror handling through human-computer communication is tomaximize the user’s satisfaction of using the system to guidefor the repair of the wrong information by human-computerinteraction. Error handling is a more serious issue for theusers who are not experienced at spoken dialog systems. Empirically, we have observed that the experts (i.e. developersor experienced users) can operate these systems with a lowerror rate. In contrast, novice users suffer from more errorsto handle the spoken dialog system. They show two criticalproblems for using the system. One of them is that they donot know the functionality and the coverage of informationserviced by the system. The other is that they do not knowwhat and how to say for operating the system at the currentsituation. However, the spoken dialog system deployed in realworld should be broadly used by the novices to the experts.In this paper, we introduce example-based error recoverystrategies to be helpful for beginners to operate the spokendialog systems. The basic idea of our approach is for thenovices to receive guidelines what and how to say for achieving their goals. This is an extended idea from the ComputerAssisted Language Learning (CALL) [1]. In the case of CALLsystem, when students cannot proceed the current dialog scenario, a tutor gives hints for the students to speak appropriately at the next turn. Similarly, when the user wants to access information of interest using the spoken dialog systems,he/she can operate the system easily by learning how to usethe system via example-based error recovery strategies. Webegin by giving some related works of error handling in spoken dialog systems. After that we describe an example-basedASRU 2007

dialog modeling for our dialog manager. Then the examplebased error recovery method is explained step by step. Afterthat we show the experimental results for evaluation of ourerror recovery approaches. Finally, we draw conclusions andmake suggestions for future works.users, the system should help to speak well-recognizable andwell-understandable utterances at the current situation. Ideally, one of the best error recovery strategies is that the userscan gradually learn how to operate the dialog system. In thispaper, we propose the example-based error recovery strategyto achieve these goals for the users.2. RELATED WORKS3. EXAMPLE-BASED DIALOG MODELINGError handling for spoken dialog systems involves a numberof stages that include error detection and error recovery [2][3]. Several approaches have been proposed to detect andhandle the errors generated in the recognition and understanding processes. The most commonly used measure for errordetection is a con¿dence score [4][5]. The decision to engage these strategies is typically based on comparing the con¿dence score against the manually preset threshold. However, the con¿dence scores are not entirely reliable and dependent on the noisy environments and the user types. In addition, false acceptance, where the con¿dence score exceedsthe threshold but really an error occurs, is more problematicas it may not be easy for the user to correct the system andput the dialog back on track. Thus, it can bring some problems at the level of dialog management. Recently, the n-besthypotheses of the recognition and understanding modules areconsidered to estimate the belief state with the uncertainty inthe framework of Partially Observable Markov Decision Processes (POMDPs) [6].At the level of the dialog manager, some error recoverystrategies, i.e., explicit/implicit con¿rmation and re-phrasing,can be adopted to repair these errors. An explicit con¿rmationtakes the form of a question that asks explicitly for con¿rmation of the main slots of the task (i.e. ”origin”, ”destination”and ”date” in the Àight reservation system). This may be accompanied by a request to answer with ”yes” or ”no”. Dialogmanager can also use an implicit con¿rmation in which thesystem embeds in its next question a repetition of its understanding of what the user said in the response to the previous question. Explicit and implicit con¿rmations are goodstrategies to repair the information which is not reliable bycomputing the con¿dence scores on various levels includingthe phonetic level, the word level, and the utterance level. Inthese cases, the user says a partial phrase or a short utteranceto acknowledge and con¿rm the dialog state. However, thede¿ciency of context may makes new errors in recognizingand understanding the user’s utterance. In addition, the distribution of user behaviors in coping with errors shows thatusers in the successful error recoveries use signi¿cantly morerephrasing than attempt to repair a chain of errors [7]. Fromthese reasons, we believe that the re-phrase strategy is moresuccessful to repair errors in the dialog manager. However, allof repeating the previous utterance cannot correct the errorsfor the system to manage the user’s utterance. In particular,the novice users have potential problems of out-of-vocabulary(OOV) and out-of-utterance (OOU). In the view of novice539Our error recovery strategy is implemented based on an ExampleBased Dialog Modeling (EBDM) which is one of generic dialog modelings technology [8]. We begin with a brief overviewof the EBDM framework in this section. We have proposedthe EBDM for automatically predicting the next actions thatthe system executes inspired by the Example-Based MachineTranslation (EBMT) [9]. The EBMT is a translation systemin which the source sentence can be translated by the similar example fragments within a large parallel corpus withoutknowledge of the language’s structure. We think that the ideaof EBMT can be extended to determine the next system actions by ¿nding the dialog examples within the dialog corpus.The system action can be selected by searching the similaruser utterance with the dialog state which is de¿ned as therelevant internal variables that affect the next system action.For an EBDM, we should automatically make an example database from the dialog corpus. The Dialog ExampleDataBase (DEDB) is semantically indexed to generalize thedata in which the keys for indexing dialog examples can bedetermined according to state variables chosen by a systemdesigner for domain-speci¿c applications. Figure 1 illustrateshow to map each utterance pairs (user-system utterances) ondialog corpus onto semantic records on DEDB. The DEDBretrieves dialog examples which are similar to the current state.When there is no example, the dialog expert has some relaxation strategies according to the genre and the domain of thedialog. The expert can relax particular variables that havebeen earlier used to search the dialog example. The aim ofeach relaxation strategy is to exclude some constraints for apartial match. The examples from the partial match may beless similar to the current dialog situation. However, this relaxation strategy is required for solving the data sparsenessproblem. Once the relevant example or examples have beenselected using the query keys, we can predict the next actionson the current dialog state. We should choose the best oneby using the utterance similarity which includes the lexicosemantic similarity and the discourse history similarity. Thelexico-semantic similarity is de¿ned as a normalized edit distance between lexico-semantic utterances of the current userand retrieved examples. We also de¿ne the degree of the discourse history similarity which is a cosine measure betweenthe binary vectors that are assigned with the value 1 if the slotis already ¿lled, and 0 otherwise. Given two similarity measures, the utterance similarity can be expanded using interpolation with empirically de¿ned weights for each application.