A SpectralÕtemporal Method For Robust Fundamental .

A spectralÕtemporal method for robust fundamental frequencytrackingStephen A. Zahoriana兲 and Hongbing HuDepartment of Electrical and Computer Engineering, State University of New York at Binghamton,Binghamton, New York 13902, USA共Received 14 December 2006; revised 2 April 2008; accepted 7 April 2008兲In this paper, a fundamental frequency 共F0兲 tracking algorithm is presented that is extremely robustfor both high quality and telephone speech, at signal to noise ratios ranging from clean speech tovery noisy speech. The algorithm is named “YAAPT,” for “yet another algorithm for pitch tracking.”The algorithm is based on a combination of time domain processing, using the normalized crosscorrelation, and frequency domain processing. Major steps include processing of the originalacoustic signal and a nonlinearly processed version of the signal, the use of a new method forcomputing a modified autocorrelation function that incorporates information from multiple spectralharmonic peaks, peak picking to select multiple F0 candidates and associated figures of merit, andextensive use of dynamic programming to find the “best” track among the multiple F0 candidates.The algorithm was evaluated by using three databases and compared to three other published F0tracking algorithms by using both high quality and telephone speech for various noise conditions.For clean speech, the error rates obtained are comparable to those obtained with the best resultsreported for any other algorithm; for noisy telephone speech, the error rates obtained are lower thanthose obtained with other methods. 2008 Acoustical Society of America. 关DOI: 10.1121/1.2916590兴PACS number共s兲: 43.72.Ar, 43.72.Dv 关DOS兴I. INTRODUCTIONNumerous studies show the importance of prosody forhuman speech recognition, but only a few automatic systemsactually combine and use fundamental frequency 共F0兲,1 withother acoustic features in the recognition process to significantly increase the performance of automatic speech recognition 共ASR兲 systems 共Ostendorf and Ross, 1997; Shriberg etal., 1997; Ramana and Srichland, 1996; Wang and Seneff,2000; Bagshaw et al., 1993兲. F0 tracking is especially important for ASR in tonal languages, such as Mandarin speech,for which pitch patterns are phonemically important 共Wangand Seneff, 1998; Chang et al., 2000兲. Other applications foraccurate F0 tracking include devices for speech analysis,transmission, synthesis, speaker recognition, speech articulation training aids for the deaf 共Zahorian et al., 1998兲, andforeign language training. Despite decades of research, automatic F0 tracking is still not adequate for routine applicationsin ASR or for scientific speech measurements.An important consideration for any speech processingalgorithm is performance using telephone speech, due to themany applications of ASR in this domain. However, sincethe fundamental frequency is often weak or missing for telephone speech and the signal is distorted, noisy, and degradedin quality overall, pitch detection for telephone speech isespecially difficult 共Wang and Seneff, 2000兲.A number of pitch detection algorithms have been reported by using time domain and frequency domain methodswith varying degrees of accuracy 共Talkin, 1995; Liu and Lin,Author to whom correspondence should be addressed. Tel.: 共607兲 7774846. FAX: 共607兲 777-4464. Electronic mail: zahorian@binghamton.edu.a兲J. Acoust. Soc. Am. 123 共6兲, June 2008Pages: 4559–45712001; Boersma and Weenink, 2005; de Cheveigne andKawahara, 2002; Nakatani and Irino, 2004兲. Many studieshave compared the robustness of pitch tracking for a varietyof speech conditions 共Rabiner et al., 1976; Mousset et al.,1996; Parsa and Jamieson, 1999兲. However, robust pitchtracking methods, which can easily be integrated with otherspeech processing steps in ASR, are not widely available. Tomake available a public domain algorithm for accurate androbust pitch tracking, the methods presented in this in thispaper were developed.A key component in “yet another algorithm for pitchtracking” 共YAAPT兲 is the normalized cross correlation function 共NCCF兲 as used in the “robust algorithm for pitch tracking” 共RAPT兲 共Talkin, 1995兲. However, in early pilot testing,the NCCF alone did not reliably give good F0 tracks, especially for noisy and/or telephone speech. Frequently, theNCCF method alone resulted in gross F0 errors 共especiallyF0 doubling for telephone speech兲 that could easily be spotted by overlaying obtained F0 tracks with the low frequencypart of a spectrogram. YAAPT is the result of efforts to incorporate this observation in a formal algorithm.In this paper, we describe methods for enhancing andextracting spectrographic information and combining it withF0 estimates from correlation methods to create a more robust overall F0 track. Another innovation is to separatelycompute F0 candidates from both the original speech signaland a nonlinearly processed version of the signal and then tofind the “lowest cost” track among the candidates by usingdynamic programming. The basic elements of YAAPT werefirst given in the work of Kasi and Zahorian 共2002兲 andmodifications were described in the work of Zahorian et al.共2006兲. In this paper, we give a comprehensive description of0001-4966/2008/123共6兲/4559/13/ 23.00 2008 Acoustical Society of America4559

this paper are given in Table II. The algorithm is frame basedby using overlapping frames with frame lengths and framespacings as given in Table I.B. PreprocessingPreprocessing consists of creating multiple versions ofthe signal, as shown in the block diagram of Fig. 1. The keyidea is to create two versions of the signal: bandpass filteredversions of both the original and nonlinearly processed signals. The bandwidths 共50– 1500 Hz兲 and orders 共150 points兲of the bandpass finite impulse response 共FIR兲 filters wereempirically determined by inspection of many signals in timeand frequency and also by overall F0 tracking accuracy.These two signals are then independently processed to obtainF0 candidates by using the time domain NCCF algorithm, asdiscussed in Sec. II D.FIG. 1. 共Color online兲 Flow chart of YAAPT. Numbers in parentheses correspond to the steps listed in Sec. II A.the complete algorithm and extensive formal evaluation results.II. THE ALGORITHMA. Algorithm overviewThe F0 tracking algorithm presented in this paper performs F0 tracking in both the time domain and frequencydomain. As summarized in the flow chart in Fig. 1, the algorithm can be loosely divided into four main steps:共1兲 Preprocessing: Multiple versions of the signal are created via nonlinear processing 共Sec. II B兲.共2兲 F0 track calculation from the spectrogram of the nonlinearly processed signal: An approximate F0 track is estimated by using a spectral harmonics correlation 共SHC兲technique and dynamic programming. The normalizedlow frequency energy ratio 共NLFER兲 is also computedfrom the spectrogram as an aid for F0 tracking 共Sec.II C兲.共3兲 F0 candidate estimation based on the NCCF: Candidatesare extracted from both the original and nonlinearly processed signals with further candidate refinement basedon the spectral F0 track estimated in step 2 共Sec. II D兲.共4兲 Final F0 determination: Dynamic programming is applied to the information from steps 2 and 3 to arrive at afinal F0 track, including voiced/unvoiced decisions 共Sec.II E兲.The algorithm incorporates several experimentally determined parameters, such as F0 search ranges, thresholds forpeak picking, filter bandwidths, and dynamic programmingweights. These parameters are listed in Table I along withvalues used for experimental results reported in this paper.Similarly, to aid in the explanation of the algorithm and theerror measures used for evaluation, primary variables used in4560J. Acoust. Soc. Am., Vol. 123, No. 6, June 20081. Nonlinear processingNonlinear processing of a signal creates sum and difference frequencies, which can be used to partially restore amissing fundamental. Two types of nonlinear processing, theabsolute value of the signal and squared value of the signal,were considered. Since experimental evaluations indicatedslightly better F0 tracking accuracy by using the squaredvalue, the squared value was used for the primary experimental results reported in this paper. The general idea ofusing nonlinearities such as center clipping to emphasize F0has long been known 共see the work of Hess, 1983 for anextensive discussion兲 but appears not to be used in most ofthe pitch detectors developed since about 1990. For example,the pitch detectors YIN 共de Cheveigne and Kawahara, 2002兲and DASH 共Nakatani and Irino, 2004兲 do not make use ofnonlinearities. Of the seven pitch detectors evaluated byParsa and Jamieson 共1999兲, only one used a nonlinearity共center clipping兲. Most previous use of nonlinearities in F0detection algorithms was aimed at spectral flattening or reducing formant strength, rather than restoring a missing fundamental 共for example, the work of Rabiner and Schafer,1978兲.As shown in the work of Zahorian et al. 共2006兲, thefundamental frequency 共F0兲 reappears by squaring the signalin which the fundamental is either very weak or absent, suchas telephone speech. The restoration of the fundamental byusing the squaring operation is also illustrated by using spectrograms in Fig. 2. The top panel depicts the spectrogram ofa studio quality version of a speech signal, for which thefundamental frequency is clearly apparent. The middle panelshows the spectrogram of the telephone version of the samespeech sample, for which the fundamental frequency below200 Hz is largely missing. In contrast, the fundamental frequency is more clearly apparent in the spectrogram of thenonlinearly processed telephone signal shown in the bottompanel. A bandpass filter 共50– 1500 Hz兲 was used after thenonlinearity to reduce the magnitude of the dc component.This same effect was observed for many other examples.S. A. Zahorian and H. Hu: Spectral/temporal F0 tracking