Deep Neural Network Based Speech Enhancement - ViVoLab

5/26/2014Speech Enhancement Based onDeep Neural NetworksChin-Hui LeeSchool of ECE, Georgia Techchl@ece.gatech.eduJoint work with Yong Xu and Jun Du at USTC1Outline and Talk Agenda In Signal Processing Letter, Jan. 2014 Speech enhancement Background Conventional speech enhancement methods Speech enhancement based on deep neural network s SE-DNN: background DNN baseline and enhancement Noise-universal SE-DNNZaragoza, 27/05/1421

5/26/2014Outline Speech enhancement Background Conventional speech enhancement methods Speech enhancement based on deep neural network s SE-DNN: background DNN baseline and enhancement Noise-universal SE-DNNZaragoza, 27/05/143Speech Enhancement Speech enhancement aims at improving the intelligibility and/oroverall perceptual quality of degraded speech signals using audiosignal processing techniques One of the most addressed classical SP problems in recent yearsEnhancingNoisy speech,Exhibition, SNR 5dBClean speech, 8kHzZaragoza, 27/05/1442

5/26/2014Speech Enhancement ApplicationsMobile phone/communicationHearing aidsSecurity monitoring/intelligenceRobust speech/speaker/languagerecognition, etc.Zaragoza, 27/05/145Noise in Speech Enhancement1. Additive noise:𝑦 𝑡 𝑥 𝑡 𝑛 𝑡STFT𝑌 𝑛, 𝑑 𝑋 𝑛, 𝑑 𝑁(𝑛, 𝑑)Focused2. Convolutional noise:𝑦 𝑡 𝑥 𝑡 ℎ 𝑡3. Mixed noise:𝑦 𝑡 𝑥 𝑡 ℎ 𝑡 𝑛 𝑡𝑦 𝑡 [𝑥 𝑡 𝑣 𝑡 ] ℎ 𝑡 𝑛 𝑡Zaragoza, 27/05/1463

5/26/2014Outline Speech enhancement Background Conventional speech enhancement methods Speech enhancement based on deep neural network s SE-DNN: background DNN baseline and enhancement Noise-universal SE-DNNZaragoza, 27/05/147Conventional Speech Enhancement Classified by the number of microphones1. Single channel speech enhancement methods Time and frequency information2.FocusedMicrophone based speech enhancement methods Time and frequency information Spatial information Microphone arraysZaragoza, 27/05/1484

5/26/2014Conventional Single Channel SE1. Spectrum Subtraction, SS2. Wiener Filtering3. Minimum Mean Square Error Short-Time SpectralAmplitude, MMSE-STSA4. MMSE Log Spectral Amplitude, MMSE-LSA5. Optimally Modified LSA, OM-LSA6. Zaragoza, 27/05/149Conventional Single Channel SEy (t )xˆ(t )STFTISTFTY (l , k ) AmplitudeSquaredG (l , k )Y (l , k )2GainFunctionCalculationNoiseEstimation ˆd (l , k )1. STFT on the noisy signal y, get the time-frequency signal 𝑌2. estimate the variance of noise 𝜆𝑑3. estimate all of the parameters (prior SNR 𝛾, posterior SNR 𝜉 and thespeech presence probability, etc.) needed by the gain function4. calculate the gain function G5. multiply 𝑌 with G, then ISTFT to obtain the enhanced signal (usingthe phase of noisy speech)Zaragoza, 27/05/14105

5/26/2014Conventional Single Channel SE: Issues1. Musical noise:Enhanced by SSNoisy speech, exhibitionnoise, SNR 5dBZaragoza, 27/05/1411Conventional Single Channel SE: Issues2. Difficult to deal with the highly non-stationary noise:Enhanced by OM-LSANoisy, Machine Gun,SNR -5dBZaragoza, 27/05/14126

5/26/2014Conventional Single Channel SE: Issues3. Difficult to deal with the low SNR cases:Enhanced by OM-LSANoisy, AGWN, SNR -5dBEasily distorting thespeech components andhas much residual noiseZaragoza, 27/05/1413Conventional Single Channel SE: Issues4. Introducing some non-linear distortion which isfatal for the back-end recognition, coding, etc.5. Learning from human listening experience:many years of exposure to clean speech and noiseZaragoza, 27/05/14147

5/26/2014Conventional Single Channel SE: Issues Analysis of these disadvantagesSTFT𝑦 𝑡 𝑥 𝑡 𝑛 𝑡𝑌 𝑛, 𝑑 𝑋 𝑛, 𝑑 𝑁(𝑛, 𝑑)Gaussian assumptionsUn-correlated assumptionsE{ Y (n, d) } E{ X (n, d) } E{ N (n, d) }222 (n, d) x (n, d) d (n, d)Binary model assumptions:H 0 (n, d) : Y (n, d) N (n, d)H1 (n, d) : Y (n, d) X (n, d) N (n, d)With these inaccurate assumptions, it is hard for conventionalmethods to deliver a satisfactory performance!Zaragoza, 27/05/1415Outline Speech enhancement Background Conventional speech enhancement methods Speech enhancement based on deep neural networks 2.1 Background DNN baseline and enhancement Noise-universal SE-DNNZaragoza, 27/05/14168

5/26/2014DNN-Based Speech Enhancement The signal model of the additive noise:𝑦 𝑡 𝑥 𝑡 𝑛 𝑡STFT𝑌 𝑛, 𝑑 𝑋 𝑛, 𝑑 𝑁(𝑛, 𝑑) Many enhancement methods are derived from this signal model, however,most of them assume that 𝑋 n, 𝑑 is described by a Gaussian mixture model(GMM) and 𝑁(n, d) is a single Gaussian. The relationship between thespeech and noise is complicated in some non-linear fashion. DNN assumes a nonlinear mapping function F:X 𝐹(𝑌) Construct the stereo data based on the additive noise model No special assumptions were made in the DNN based SE methodZaragoza, 27/05/1417Deep Neural Network: Overview1. Hinton proposed the unsupervised Restricted BoltzmannMachine (RBM) based pre-training in 20062. In 2012, MSR, Google and IBM got a great success in largevocabulary continuous speech recognition using DNNs3. Later, DNNs were adopted in many speech-related tasksInputlayerHidden layerOutputlayerMathematicalapproximationZaragoza, 27/05/14189

5/26/2014DNN Based SE: Related Work4. In 2013, Xugang Luproposed deep denoising auto-encoderbased speechenhancementX.-G. Lu, Y. Tsao, S. Matsuda and C. Hori, “Speech enhancement based on deepdenoising Auto-Encoder,” Proc. Interspeech, pp. 436-440, 2013.23DNN Based SE: Related Work6. In 2013, Deliang Wang proposed using DNN to classifythe time-Frequency bins into 1/0 units (ideal binary mask)IBM-DNN enhancedNoisyspeechCochlearFilteringT-F unitslevel featureextractionDNN-based ed IBMEnhancedspeechY. X. Wang and D. L. Wang, “Towards scaling up classification based speech separation,” IEEETrans. on Audio, Speech and Language Processing, Vol. 21, No. 7, pp. 1381-1390, 2013.10

5/26/2014DNN Based SE: Issues Advantages of SE-DNN1. The complicated relationship between noisy andclean speech could be automatically learnt2. The deep architecture could well fit the non-linearrelationship for regression function approximation3. The highly non-stationary noise could be wellsuppressed in the off-line learning framework4. Nearly no Gaussian or independent assumptions5. Nearly no empirical thresholds to avoid the non-lineardistortion in SS-based speech enhancementZaragoza, 27/05/1426DNN Based SE: Issues Difficulties in using SE-DNN1. Which domain is suitable for DNN-based mapping?2. The generalization capacity to unknown environments,especially for unseen noise types?3. How to perform noise adaptation? – robustness issueZaragoza, 27/05/142711

5/26/2014Outline Speech enhancement Background Conventional speech enhancement methods Speech enhancement based on deep neural networks SE-DNN: background DNN baseline and enhancement Noise-universal SE-DNNZaragoza, 27/05/1428System OverviewTraining gEnhancement ̂ lWaveformReconstructionX̂ t Y f1. Feature extraction: log-power spectra2. Waveform reconstruction: overlap-add algorithm3. DNN Training: RBM pre-training back-propagation fine-tuningZaragoza, 27/05/142912

5/26/2014DNN Training(Output with a single frame of clean speech features)2. Fine-tuning 𝑾𝟒1.Pre-training h3𝑾𝟑 𝜺 𝟑h2 h1 𝑾𝟐 𝜺 𝟐𝑾𝟏 𝜺 𝟏 (Input with multiple frames of noisy speech features)1.MMSE-based object function: 𝐸 1𝑁𝑁𝑛 1𝐷𝑑𝑑 1(𝑋𝑛𝑾, 𝒃 𝑋𝑛𝑑 )2 𝜆 𝑾2 302Experimental Setup1. Clean speech set: TIMIT corpus, 8kHz2. Noise set: Additive Gaussian White Noise (AGWN), Babble,Restaurant, Street3. Signal to Noise ratios: Clean, 20dB, 15dB, 10dB, 5dB, 0dB, -5dB4. Construct 100 hours multi-condition training data5. Test set: 200 randomly selected utterances from TIMIT test set,and two unseen noise types: Car and Exhibition6. Three objective quality measures: segmental SNR (SegSNR indB), log-spectral distortion (LSD in dB), perceptual evaluation ofspeech quality (PESQ)7. Standard DNN configurations: 11 frames expansion, 3 hiddenlayers and 2048 hidden units for each8. Competing methods: improved version of the optimallymodified log-spectral amplitude (OM-LSA), denoted as logMMSE (L-MMSE)Zaragoza, 27/05/143113

5/26/2014Baseline Experimental Results: I1. Average LSD using input with different acoustic context on the testset at different SNRs across four noise types: A good choice: 11 framesZaragoza, 27/05/1432Baseline Experimental Results: II2. Average SegSNRs using different training set size on the test set atdifferent SNRs across four noise types: still improving with 100 hoursZaragoza, 27/05/143314

5/26/2014Baseline Experimental Results: III3. Average PESQs among methods on the test set at different SNRswith four noise types. The subscript of 𝐷𝑁𝑁𝑙 represents 𝑙 hidden -MMSE SNN**Shallow Neural Network (SNN) has the same computation complexity with 𝐷𝑁𝑁3 Deep structure can get better performance compared with SNN. 𝐷𝑁𝑁3 outperforms the L-MMSE method, especially at low SNRs.Zaragoza, 27/05/1434Baseline Experimental Results: IV4. PESQs among Noisy, L-MMSE, SNN and 𝐷𝑁𝑁3 at different SNRs inmismatch environments under Car (A) and Exhibition (B) 303.313.013.032.692.712.332.351.931.961.542.832.47 SE-DNN has a generalization capacity to unseen noise types. It canbe further strengthened by adding more noise types in trainingZaragoza, 27/05/143515

5/26/2014Baseline Experimental Results VDNN enhancedPESQ 3.39L-MMSE enhancedPESQ 2.64DNN can dealwith nonstationarynoise givensome noisecharacteristicsto learn.Clean, PESQ 4.5Noisy, street, SNR 10dB,PESQ 2.2Zaragoza, 27/05/1436More demos could be found at: http://home.ustc.edu.cn/ xuyong62/demo/SE DNN.htmlOver-smoothing with SE-DNN (1/2)1. The global variances of the training set were shown. 𝐺𝑉𝑟𝑒𝑓 (𝑑) and 𝐺𝑉𝑒𝑠𝑡 (𝑑)represented the d-th dimension of the global variance of the reference featuresand the estimation features, respectively. And the corresponding dimension37independent variances were denoted as 𝐺𝑉𝑟𝑒𝑓 and 𝐺𝑉𝑒𝑠𝑡16

5/26/2014Over-smoothing with SE-DNN (2/2)DNN enhancedCleanAGWN, SNR 0dB2. The formant peaks were suppressed, especially in the highfrequency band which leads to muffled speechZaragoza, 27/05/1438Methods to Address Over-smoothing The definition of global variance equalization factors:𝛽 ��𝑒𝑓 (𝑑)𝐺𝑉𝑒𝑠𝑡 (𝑑)𝛼(𝑑) 1𝛼 𝐷𝐷𝛼(𝑑)𝑑 1 Proposed method 1: post-processing𝑋 ′′ 𝑑 𝑋 𝑑 𝜂 𝑣(𝑑) 𝑚(𝑑)where 𝑚(𝑑) and 𝑣(𝑑) are the d-th component of the mean and varianceof input noisy features, respectively. And η could be 𝛽 , 𝛼(𝑑) or 𝛼 Proposed method 2: post-training𝐸 1𝑁𝑁𝐷(𝑋𝑛𝑑 𝑾, 𝒃 η 𝑋𝑛𝑑 )2 𝜆 𝑾22𝑛 1 𝑑 1Zaragoza, 27/05/143917

5/26/2014GV Experimental Results (1/2)PESQ results of the L-MMSE method and DNN baseline, comparedwith different post-processing and post-training schemes using 𝛽,𝛼(𝑑) and 𝛼 on the test set at different SNRs across four noise 4𝜶3.723.493.202.862.472.022.961. 𝛼 is better than 𝛽, and 𝛼(𝑑) is the worst, indicating that thedegree of over-smoothing on different dimensions was similar2. Equalization operations were much more beneficial for high SNRs3. Post-training was a little better than Post-processing40GV Experimental Results (2/2)PESQ results in unseen environments under Car and Exhibition noises,labeled as case A and B, respectively. The DNN baseline wascompared with the L-MMSE method and the proposed two globalvariance equalization approaches using the factor 𝛼 81. GV equalization is slightly more effective for unseen noise types2. Post-training was a little better than Post-processingZaragoza, 27/05/144118