Speech Recognition: Statistical Methods

4 Speech Recognition: Statistical Methodsof words. For this step we use a statistical model(called the hidden Markov model or HMM) of theacoustic signals of either individual words or subword units (e.g., phonemes) to compute the quantity PA ðXjWÞ. We train the acoustic models from atraining set of speech utterances, which have beenappropriately labeled to establish the statisticalrelationship between X and W. Step 2: language modeling for assigning probabilities, PL ðWÞ, to sequences of words that form validsentences in the language and are consistent withthe recognition task being performed. We trainsuch language models from generic text sequences,or from transcriptions of task-specific dialogues.(Note that a deterministic grammar, as is used inmany simple tasks, can be considered a degenerateform of a statistical language model. The ‘coverage’of a deterministic grammar is the set of permissibleword sequences, i.e., expressions that are deemedlegitimate.). Step 3: hypothesis search whereby we find theword sequence with the maximum a posterioriprobability by searching through all possibleword sequences in the language.In step 1, acoustic modeling (Young, 1996; Rabineret al., 1986), we train a set of acoustic models for thewords or sounds of the language by learning thestatistics of the acoustic features, X, for each wordor sound, from a speech training set, where we compute the variability of the acoustic features during theproduction of the words or sounds, as represented bythe models. For large vocabulary tasks, it is impractical to create a separate acoustic model for everypossible word in the language since it requires fartoo much training data to measure the variability inevery possible context. Instead, we train a set of about50 acoustic-phonetic subword models for the approximately 50 phonemes in the English language,and construct a model for a word by concatenating(stringing together sequentially) the models for theconstituent subword sounds in the word, as definedin a word lexicon or dictionary, where multiple pronunciations are allowed). Similarly, we build sentences (sequences of words) by concatenating wordmodels. Since the actual pronunciation of a phonememay be influenced by neighboring phonemes (thoseoccurring before and after the phoneme), the set ofso-called context-dependent phoneme models areoften used as the speech models, as long as sufficientdata are collected for proper training of these models.In step 2, the language model (Jelinek, 1997;Rosenfeld, 2000) describes the probability of a sequence of words that form a valid sentence in the tasklanguage. A simple statistical method works well,based on a Markovian assumption, namely that theprobability of a word in a sentence is conditioned ononly the previous N–1 words, namely an N-gramlanguage model, of the form:PL ðWÞ ¼ PL ðw1 ; w2 ; . . . ; wM Þ¼MYPL ðwm jwm 1 ; wm 2 ; . . . ; wm Nþ1 Þm¼1where PL ðwm jwm 1 ; wm 2 ; . . . ; wm Nþ1 Þ is estimatedby simply counting up the relative frequencies ofN-tuples in a large corpus of text.In step 3, the search problem (Ney, 1984; Paul,2001) is one of searching the space of all validsound sequences, conditioned on the word grammar,the language syntax, and the task constraints, to findthe word sequence with the maximum likelihood.The size of the search space can be astronomicallylarge and take inordinate amounts of computingpower to solve by heuristic methods. The use ofmethods from the field of Finite State AutomataTheory provide finite state networks (FSNs)(Mohri, 1997), along with the associated searchpolicy based on dynamic programming, that reducethe computational burden by orders of magnitude,thereby enabling exact solutions in computationally feasible times, for large speech recognitionproblems.Development of a Speech RecognitionSystem for a Task or an ApplicationBefore going into more detail on the various aspectsof the process of automatic speech recognition bymachine, we review the three steps that must occurin order to define, train, and build an ASR system(Juang et al., 1995; Kam and Helander, 1997). Thesesteps are the following:. Step 1: choose the recognition task. Specify theword vocabulary for the task, the set of units thatwill be modeled by the acoustic models (e.g., wholewords, phonemes, etc.), the word pronunciationlexicon (or dictionary) that describes the variationsin word pronunciation, the task syntax (grammar),and the task semantics. By way of example, for asimple speech recognition system capable of recognizing a spoken credit card number using isolateddigits (i.e., single digits spoken one at a time), thesounds to be recognized are either whole wordsor the set of subword units that appear in the digits/zero/ to /nine/ plus the word /oh/. The word vocabulary is the set of 11 digits. The task syntaxallows any single digit to be spoken, and the task

Speech Recognition: Statistical Methods 5Figure 4 Framework of ASR system.semantics specify that a sequence of isolated digitsmust form a valid credit card code for identifyingthe user. Step 2: train the models. Create a method for building acoustic word models (or subword models) froma labeled speech training data set of multiple occurrences of each of the vocabulary words by one ormore speakers. We also must use a text trainingdata set to create a word lexicon (dictionary) describing the ways that each word can be pronounced(assuming we are using subword units to characterize individual words), a word grammar (or languagemodel) that describes how words are concatenatedto form valid sentences (i.e., credit card numbers),and finally a task grammar that describes whichvalid word strings are meaningful in the taskapplication (e.g., valid credit card numbers). Step 3: evaluate recognizer performance. We needto determine the word error rate and the task errorrate for the recognizer on the desired task. For anisolated digit recognition task, the word error rateis just the isolated digit error rate, whereas the taskerror rate would be the number of credit carderrors that lead to misidentification of the user.Evaluation of the recognizer performance oftenincludes an analysis of the types of recognitionerrors made by the system. This analysis can leadto revision of the task in a number of ways, rangingfrom changing the vocabulary words or the grammar (i.e., to eliminate highly confusable words) tothe use of word spotting, as opposed to word transcription. As an example, in limited vocabularyapplications, if the recognizer encounters frequentconfusions between words like ‘freight’ and ‘flight,’it may be advisable to change ‘freight’ to ‘cargo’ tomaximize its distinction from ‘flight.’ Revision ofthe task grammar often becomes necessary if therecognizer experiences substantial amounts ofwhat is called ‘out of grammar’ (OOG) utterances,namely the use of words and phrases that are notdirectly included in the task vocabulary (ISCA,2001).The Speech Recognition ProcessIn this section, we provide some technical aspects of atypical speech recognition system. Figure 4 shows ablock diagram of a speech recognizer that follows theBayesian framework discussed above.The recognizer consists of three processing steps,namely feature analysis, pattern matching, and confidence scoring, along with three trained databases, theset of acoustic models, the word lexicon, and thelanguage model. In this section, we briefly describeeach of the processing steps and each of the trainedmodel databases.Feature AnalysisThe goal of feature analysis is to extract a set ofsalient features that characterize the spectral properties of the various speech sounds (the subword units)and that can be efficiently measured. The ‘standard’feature set for speech recognition is a set of melfrequency cepstral coefficients (MFCCs) (which perceptually match some of the characteristics of thespectral analysis done in the human auditory system)(Davis and Mermelstein, 1980), along with the firstand second-order derivatives of these features. Typically about 13 MFCCs and their first and secondderivatives (Furai, 1981) are calculated every 10 ms,leading to a spectral vector with 39 coefficients every10 ms. A block diagram of a typical feature analysisprocess is shown in Figure 5.The speech signal is sampled and quantized, preemphasized by a first-order (highpass) digital filterwith pre-emphasis factor a (to reduce the influenceof glottal coupling and lip radiation on the estimated

6 Speech Recognition: Statistical MethodsFigure 6 Three-state HMM for the sound /s/.Figure 5 Block diagram of feature analysis computation.vocal tract characteristics), segmented into frames,windowed, and then a spectral analysis is performedusing a fast Fourier transform (FFT) (Rabiner andGold, 1975) or linear predictive coding (LPC) method(Atal and Hanauer, 1971; Markel and Gray, 1976).The frequency conversion from a linear frequencyscale to a mel frequency scale is performed in thefiltering block, followed by cepstral analysis yieldingthe MFCCs (Davis and Mermelstein, 1980), equalization to remove any bias and to normalize the cepstral coefficients (Rahim and Juang, 1996), andfinally the computation of first- and second-order(via temporal derivative) MFCCs is made, completingthe feature extraction process.Figure 7 Concatenated model for the word ‘is.’Figure 8 HMM for whole word model with five states.Acoustic ModelsThe goal of acoustic modeling is to characterize thestatistical variability of the feature set determinedabove for each of the basic sounds (or words) of thelanguage. Acoustic modeling uses probability measures to characterize sound realization using statistical models. A statistical method, known as the hiddenMarkov model (HMM) (Levinson et al., 1983;Ferguson, 1980; Rabiner, 1989; Rabiner and Juang,1985), is used to model the spectral variability of eachof the basic sounds of the language using a mixturedensity Gaussian distribution (Juang et al., 1986;Juang, 1985), which is optimally aligned with aspeech training set and iteratively updated and improved (the means, variances, and mixture gains areiteratively updated) until an optimal alignment andmatch is achieved.Figure 6 shows a simple three-state HMM for modeling the subword unit /s/ as spoken at the beginningof the word /six/. Each HMM state is characterized bya probability density function (usually a mixtureGaussian density) that characterizes the statisticalbehavior of the feature vectors at the beginning(state s1), middle (state s2), and end (state s3) of thesound /s/. In order to train the HMM for each subwordunit, we use a labeled training set of words and sentences and utilize an efficient training procedureknown as the Baum-Welch algorithm (Rabiner, 1989;Baum, 1972; Baum et al., 1970) to align each of thevarious subword units with the spoken inputs, and thenestimate the appropriate means, covariances, and mixture gains for the distributions in each subword unitstate. The algorithm is a hill-climbing algorithm and isiterated until a stable alignment of subword unit models and speech is obtained, enabling the creation ofstable models for each subword unit.Figure 7 shows how a simple two-sound word, ‘is,’which consists of the sounds /ih/ and /z/, is created byconcatenating the models (Lee, 1989) for the /ih/sound with the model for the /z/ sound, therebycreating a six-state model for the word ‘is.’Figure 8 shows how an HMM can be used to characterize a whole-word model (Lee et al., 1989). In this

Speech Recognition: Statistical Methods 7case, the word is modeled as a sequence of M ¼ 5HMM states, where each state is characterized by amixture density, denoted as bj ðxt Þ where the modelstate is the index j, the feature vector at time t isdenoted as xt , and the mixture density is of the form:bj ðxt Þ ¼KXcjk N ½xt ; mjk ; Ujk k¼1xt ¼ ðxt1 ; xt2 ; . . . ; xtD Þ; D ¼ 39K ¼ number of mixture components inthe density functioncjk ¼ weight of kth mixture component instate j; cjk 0N ¼ Gaussian density functionmjk ¼ mean vector for mixture k; state jUjk ¼ covariance matrix for mixture k;state jKXcjk ¼ 1;1jMk¼1Z1bj ðxt Þdxt ¼ 1;1jM 1Included in Figure 8 is an explicit set of state transitions, aij , which specify the probability of making atransition from state i to state j at each frame, therebydefining the time sequence of the feature vectors overthe duration of the word. Usually the self-transitions,aii, are large (close to 1.0), and the skip-state transitions, a13, a24, a35, are small (close to 0).Once the set of state transitions and state probability densities are specified, we say that a model l(which is also used to denote the set of parametersthat define the probability measure) has been createdfor the word or subword unit. (The model l is oftenwritten as lðA; B; pÞ to explicitly denote the modelparameters, namely A ¼ faij ; 1 i; j Mg, which isthe state transition matrix, B ¼ fbj ðxt Þ; 1 j Mg,which is the state observation probability density,and p ¼ fpi ; 1 i Mg, which is the initial state distribution). In order to optimally train the variousmodels (for each word unit [Lee et al., 1989] or subword unit [Lee, 1989]), we need to have algorithmsthat perform the following three steps or tasks (Rabiner and Juang, 1985) using the acoustic observationsequence, X, and the model l:a. likelihood evaluation: compute PðXjlÞb. decoding: choose the optimal state sequence for agiven speech utterancec. re-estimation: adjust the parameters of l to maximize PðXjlÞ.Each of these three steps is essential to definingthe optimal HMM models for speech recognitionbased on the available training data and each taskif approached in a brute force manner would becomputationally costly. Fortunately, efficient algorithms have been developed to enable efficient andaccurate solutions to each of the three steps thatmust be performed to train and utilize HMM modelsin a speech recognition system. These are generallyreferred to as the forward-backward algorithm orthe Baum-Welch re-estimation method (Levinsonet al., 1983). Details of the Baum-Welch procedureare beyond the scope of this article. The heart of thetraining procedure for re-estimating model parameters using the Baum-Welch procedure is shown inFigure 9.Recently, the fundamental statistical method, whilesuccessful for a range of conditions, has beenaugmented with a number of techniques that attemptto further enhance the recognition accuracy and makethe recognizer more robust to different talkers, background noise conditions, and channel effects. Onefamily of such techniques focuses on transformationof the observed or measured features. The transformation is motivated by the need for vocal tract lengthnormalization (e.g., reducing the impact of differencesin vocal tract length of various speakers). Anothersuch transformation (called the maximum likelihoodlinear regression method) can be embedded in thestatistical model to account for a potential mismatchbetween the statistical characteristics of the training data and the actual unknown utterances tobe recognized. Yet another family of techniques(e.g., the discriminative training method based onminimum classification error [MCE] or maximummutual information [MMI]) aims at direct minimization of the recognition error during the parameteroptimization stage.Word LexiconThe purpose of the word lexicon, or dictionary, is todefine the range of pronunciation of words in the taskvocabulary (Jurafsky and Martin, 2000; Riley et al.,1999). The reason that such a word lexicon is necessary is because the same orthography can be pronounced differently by people with different accents,or because the word has multiple meanings thatchange the pronunciation by the context of its use.For example, the word ‘data’ can be pronounced as:/d/ /ae/ /t/ /ax/ or as /d/ /ey/ /t/ /ax/, and we would need

8 Speech Recognition: Statistical MethodsFigure 9 The Baum-Welch training procedure.both pronunciations in the dictionary to properlytrain the recognizer models and to properly recognizethe word when spoken by different individuals. Another example of variability in pronunciation fromorthography is the word ‘record,’ which can be eithera disk that goes on a player, or the process of capturing and storing a signal (e.g., audio or video). Thedifferent meanings have significantly different pronunciations. As in the statistical language model, theword lexicon (consisting of sequences of symbols) canbe associated with probability assignments, resultingin a probabilistic word lexicon.Language ModelThe purpose of the language model (Rosenfeld, 2000;Jelinek et al., 1991), or grammar, is to provide a tasksyntax that defines acceptable spoken input sentencesand enables the computation of the probability of theword string, W, given the language model, i.e.,PL ðWÞ. There are several methods of creating wordgrammars, including the use of rule-based systems(i.e., deterministic grammars that are knowledgedriven), and statistical methods that compute an estimate of word probabilities from large training sets oftextual material. We describe the way in which astatistical N-gram word grammar is constructedfrom a large training set of text.Assume we have a large text training set of labeledwords. Thus for every sentence in the training set,we have a text file that identifies the words in thatsentence. If we consider the class of N-gram wordgrammars, then we can estimate the word probabilities from the labeled text training set using countingmethods. Thus to estimate word trigram probabilities(that is the probability that a word wi was precededby the pair of words ðwi 1 ; wi 2 Þ), we compute thisquantity as:Pðwi jwi 1 ; wi 2 Þ ¼Cðwi 2 ; wi 1 ; wi ÞCðwi 2 ; wi 1 Þwhere Cðwi 2 ; wi 1 ; wi Þ is the frequency count ofthe word triplet (i.e., trigram) consisting ofðwi 2 ; wi 1 ; wi Þ that occurred in the training set, andCðwi 2 ; wi 1 Þ is the frequency count of the wordduplet (i.e., bigram) ðwi 2 ; wi 1 Þ that occurred in thetraining set.Although the method of training N-gram wordgrammars, as described above, generally works quitewell, it suffers from the problem that the counts ofN-grams are often highly in error due to problems ofdata sparseness in the training set. Hence for a texttraining set of millions of words, and a word vocabulary of several thousand words, more than 50% ofword trigrams are likely to occur either once or not atall in the training set. This leads to gross distortions inthe computation of the probability of a word string,as required by the basic Bayesian recognition algorithm. In the cases when a word trigram does notoccur at all in the training set, it is unacceptable todefine the trigram probability as 0 (as would be required by the direct definition above), since this leadsto effectively invalidating all strings with that particular trigram from occurring in recogn

ferred to as the speech dialog circle, using an example in the telecommunications context. The customer initially makes a request by speaking an utterance that is sent to a machine, which attempts to recognize, on a word-by-word basis, the spoken speech. The process of recognizing the words in the speech is called automatic speech recognition (ASR)