Time Domain Methods In Speech Processing

Digital Speech Processing—Lectures 7-8Time Domain Methodsin Speech Processing1

General Synthesis Modelvoiced soundamplitudeT1Log Areas, ReflectionCoefficients, Formants, VocalTract Polynomial, ArticulatoryParameters, T2R( z ) 1 α z 1unvoiced soundamplitudePitch Detection, Voiced/Unvoiced/Silence Detection, Gain Estimation, Vocal Tract2Parameter Estimation, Glottal Pulse Shape, Radiation Model

General Analysis Models[n]SpeechAnalysisModelPitch Period, T[n]Glottal Pulse Shape, g[n]Voiced Amplitude, AV[n]V/U/S[n] SwitchUnvoiced Amplitude, AU[n]Vocal Tract IR, v[n]Radiation Characteristic, r[n] All analysis parameters are time-varying at ratescommensurate with information in the parameters; We need algorithms for estimating the analysisparameters and their variations over time3

Overviewspeech, x[n]SignalProcessingrepresentationof speechspeech or musicA(x,t)formantsreflection coefficientsvoiced-unvoiced-silencepitchsounds of languagespeaker identificationemotions time domain processing direct operations on the speech waveform frequency domain processing direct operations on a spectralrepresentation of the signalx[n]systemzero crossing ratelevel crossing rateenergyautocorrelation simple processing enables various types of feature estimation4

BasicsVP1P2 8 kHz sampled speech (bandwidth 4 kHz) properties of speech change withtimeU/S excitation goes from voiced tounvoiced peak amplitude varies with thesound being produced pitch varies within and acrossvoiced sounds periods of silence wherebackground signals are seen the key issue is whether we cancreate simple time-domain processingmethods that enable us tomeasure/estimate speechrepresentations reliably and accurately5

Fundamental Assumptions properties of the speech signal change relativelyslowly with time (5-10 sounds per second)– over very short (5-20 msec) intervals uncertaintydue to small amount of data, varying pitch, varyingamplitude– over medium length (20-100 msec) intervals uncertainty due to changes in sound quality,transitions between sounds, rapid transients inspeech– over long (100-500 msec) intervals uncertaintydue to large amount of sound changes there is always uncertainty in short timemeasurements and estimates from speechsignals6

Compromise Solution “short-time” processing methods short segments ofthe speech signal are “isolated” and “processed” as ifthey were short segments from a “sustained” sound withfixed (non-time-varying) properties– this short-time processing is periodically repeated for theduration of the waveform– these short analysis segments, or “analysis frames” almostalways overlap one another– the results of short-time processing can be a single number (e.g.,an estimate of the pitch period within the frame), or a set ofnumbers (an estimate of the formant frequencies for the analysisframe)– the end result of the processing is a new, time-varying sequencethat serves as a new representation of the speech signal7

Frame-by-Frame Processingin Successive WindowsFrame 1Frame 2Frame 3Frame 4Frame 575% frame overlap frame length L, frame shift R L/4Frame1 {x[0],x[1], ,x[L-1]}Frame2 {x[R],x[R 1], ,x[R L-1]}Frame3 {x[2R],x[2R 1], ,x[2R L-1]} 8

Frame 1: samples 0,1,., L 1Frame 2: samples R, R 1,., R L 1Frame 3: samples 2 R, 2 R 1,., 2 R L 1Frame 4: samples 3R,3R 1,.,3R L 1

Frame-by-Frame Processing inSuccessive WindowsFrame 1Frame 250% frame overlapFrame 3Frame 4 Speech is processed frame-by-frame in overlapping intervals until entireregion of speech is covered by at least one such frame Results of analysis of individual frames used to derive model parameters insome manner Representation goes from time sample x[n], n L ,0, 1, 2, L to parametervector f [m], m 0, 1, 2, L where n is the time index and m is the frame index.10

Frames and WindowsFS 16, 000 samples/secondL 641 samples (equivalent to 40 msec frame (window) length)R 240 samples (equivalent to 15 msec frame (window) shift)Frame rate of 66.7 frames/second11

Short-Time Processingspeechwaveform, x[n]short-timeprocessingspeech representation,f[m]x [n ] samples at 8000/sec rate; (e.g. 2 seconds of 4 kHz bandlimitedspeech, x [n ], 0 n 16000)rf [m] {f 1 [m], f2 [m],., fL [m]} vectors at 100/sec rate, 1 m 200,L is the size of the analysis vector (e.g., 1 for pitch period estimate, 12 forautocorrelation estimates, etc)12

Generic Short-Time Processing Qnˆ T ( x[m]) w% [n m] m n nˆx[n]T( )linear or non-lineartransformationT(x[n])w[n] Qn̂window sequence(usually finite length) Qn̂ is a sequence of local weighted averagevalues of the sequence T(x[n]) at time n nˆ13

Short-Time EnergyE x 2 [m ]m -- this is the long term definition of signal energy-- there is little or no utility of this definition for time-varying signalsEnˆ nˆ x 2 [m] x 2 [nˆ L 1] . x 2 [nˆ ]m nˆ L 1-- short-time energy in vicinity of time nˆT (x) x2w% [n ] 1 00 n L 1otherwise14

Computation of Short-Time Energyw% [n m]n L 1n window jumps/slides across sequence of squared values, selecting intervalfor processing what happens to En̂ as sequence jumps by 2,4,8,.,L samples ( En̂ is a lowpassfunction—so it can be decimated without lost of information; why is En̂ lowpass?) effects of decimation depend on L; if L is small, then En̂ is a lot more variablethan if L is large (window bandwidth changes with L!)15

Effects of WindowQnˆ T ( x [n ]) w% [n ] x ′[n ] w% [n ]n nˆn nˆ w% [n] serves as a lowpass filter on T ( x[n]) which often has a lot ofhigh frequencies (most non-linearities introduce significant highfrequency energy—think of what ( x[n] x[n] ) does in frequency) often we extend the definition of Qn̂ to include a pre-filtering termso that x[n] itself is filtered to a region of interestxˆ[n]LinearFilterx[n]T ( x[n])T( )Qnˆ Qnw% [n]n nˆ16

Short-Time Energy serves to differentiate voiced and unvoiced sounds in speechfrom silence (background signal)natural definition of energy of weighted signal is:Enˆ 2m x [m] w% [nˆ m] (sum or squares of portion of signal)-- concentrates measurement at sample nˆ, using weighting w% [nˆ - m]Enˆ m x [m] w% [nˆ m] 22 x 2 [m] h[nˆ m ]m h[n ] w% 2 [n ]short time energyx[n]FS()2x2[n]FSh[n]Enˆ Enn nˆFS / R17

Short-Time Energy Properties depends on choice of h[n], or equivalently, window w[n]– if w[n] duration very long and constant amplitude (w[n] 1,n 0,1,.,L-1), En would not change much overtime, and would not reflect the short-time amplitudes ofthe sounds of the speech– very long duration windows correspond to narrowbandlowpass filters– want En to change at a rate comparable to the changingsounds of the speech this is the essential conflict inall speech processing, namely we need short durationwindow to be responsive to rapid sound changes, butshort windows will not provide sufficient averaging togive smooth and reliable energy function18

Windows consider two windows, w[n]– rectangular window: h[n] 1, 0 n L-1 and 0 otherwise– Hamming window (raised cosine window): h[n] 0.54-0.46 cos(2πn/(L-1)), 0 n L-1 and 0 otherwise– rectangular window gives equal weight to all Lsamples in the window (n,.,n-L 1)– Hamming window gives most weight to middlesamples and tapers off strongly at the beginning andthe end of the window19

Rectangular and Hamming WindowsL 21 samples20

Window Frequency Responses rectangular windowH (ej ΩTsin(ΩLT / 2) j ΩT ( L 1)/ 2) esin(ΩT / 2) first zero occurs at f Fs/L 1/(LT) (or Ω (2π)/(LT)) nominal cutoff frequency of the equivalent “lowpass” filter Hamming windoww% H [n] 0.54w% R [n] 0.46*cos(2π n / ( L 1)) w% R [n] can decompose Hamming Window FR into combinationof three terms21

RW and HW Frequency Responses log magnitude response of RW and HW bandwidth of HW is approximately twicethe bandwidth of RW attenuation of more than 40 dB for HWoutside passband, versus 14 dB for RW stopband attenuation is essentiallyindependent of L, the window duration increasing L simply decreases windowbandwidth L needs to be larger than a pitch period(or severe fluctuations will occur in En), butsmaller than a sound duration (or En willnot adequately reflect the changes in thespeech signal)There is no perfect value of L, since a pitch period can be as short as 20 samples (500 Hz at a 10 kHzsampling rate) for a high pitch child or female, and up to 250 samples (40 Hz pitch at a 10 kHz samplingrate) for a low pitch male; a compromise value of L on the order of 100-200 samples for a 10 kHz sampling22rate is often used in practice

Window Frequency ResponsesRectangular Windows,L 21,41,61,81,101Hamming Windows,L 21,41,61,81,10123

Short-Time EnergyShort-time energy computation:Enˆ ( x[m]w[nˆ m])2m ( x[m]) m 2w% [nˆ m]For L-point rectangular window,w% [m] 1, m 0,1,., L 1givingEnˆ nˆ m nˆ L 1( x[m]) 224

Short-Time Energy using RW/HWEn̂En̂L 51L 51L 101L 101L 201L 201L 401L 401 as L increases, the plots tend to converge (however you are smoothing sound energies) short-time energy provides the basis for distinguishing voiced from unvoiced speechregions, and for medium-to-high SNR recordings, can even be used to find regions ofsilence/background signal25

Short-Time Energy for AGCCan use an IIR filter to define short-time energy, e.g., time-dependent energy definitionσ [n ] 2 x [m]h[n m] / h[m]2m m 0 consider impulse response of filter of formh[n ] α n 1u[n 1] α n 1 n 1 0σ [n ] 2 n 1(1 α ) x 2 [m]α n m 1u[n m 1]m 26

Recursive Short-Time Energyu[n m 1] implies the condition n m 1 0or m n 1 givingσ [n ] 2n 1 (1 α ) x 2 [m] α n m 1 (1 α )( x 2 [n 1] α x 2 [n 2] .)m for the index n 1 we haveσ [n 1] 2n 2 (1 α ) x 2 [m] α n m 2 (1 α )( x 2 [n 2] α x 2 [n 3] .)m thus giving the relationshipσ 2 [n ] α σ 2 [n 1] x 2 [n 1](1 α )and defines an Automatic Gain Control (AGC) of the formG0G[ n ] σ [n ]27

Recursive Short-Time Energyσ 2 [n ] x 2 [n ] h[n ]h[n ] (1 α )α n 1u[n 1]σ 2 (z) X 2 (z) H (z)H (z) h[n ] z n n 1 n(1 α)αu[n 1]z n n (1 α )α n 1 z nn 1m n 1H (z) (1 α )αmz ( m 1)m 0 (1 α ) z 1 (1 α ) z 1α m z mm 0 α zm m (1 α ) zm 0σ 2 [n ] ασ 2 [n 1] (1 α ) x 2 (n 1) 111 α z 1 σ 2 (z) / X 2 (z)28

Recursive Short-Time Energyx[n]( )2 x 2 [ n]z 1(1 α )σ 2 [ n] σ 2 [n 1]z 1ασ [n ] α σ [n 1] x [n 1](1 α )22229

Recursive Short-Time Energy30

Use of Short-Time Energy for AGC31

Use of Short-Time Energy for AGCα 0.9α 0.9932

Short-Time Magnitude short-time energy is very sensitive to largesignal levels due to x2[n] terms– consider a new definition of ‘pseudo-energy’ basedon average signal magnitude (rather than energy)M nˆ x[m] w% [nˆ m]m – weighted sum of magnitudes, rather than weightedsum of squaresx[n]FS x[n] FSw% [n]Mnˆ Mn n nˆFS / R computation avoids multiplications of signal with itself (the squared term)33

Short-Time MagnitudesM n̂M n̂L 51L 51L 101L 101L 201L 201L 401L 401 differences between En and Mn noticeable in unvoiced regions dynamic range of Mn square root (dynamic range of En) level differences between voiced andunvoiced segments are smaller En and Mn can be sampled at a rate of 100/sec for window durations of 20 msec or so efficientrepresentation of signal energy/magnitude34

Short Time Energy and Magnitude—Rectangular WindowEn̂M n̂L 51L 51L 101L 101L 201L 201L 401L 40135

Short Time Energy and Magnitude—Hamming WindowEn̂M n̂L 51L 51L 101L 101L 201L 201L 401L 40136

Other Lowpass Windows can replace RW or HW with any lowpass filerwindow should be positive since this guarantees En and Mn will bepositiveFIR windows are efficient computationally since they can slide by Rsamples for efficiency with no loss of information (what should Rbe?)can even use an infinite duration window if its z-transform is arational function, i.e.,h[n ] a n , n 0, 0 a 1 0n 01H (z) 1 az 1 z a 37