Yichen Qin Estimation Of Mixture Models & Carey E. Priebe .

Qin and Priebe: Maximum Lq -Likelihood Estimation via the Expectation-Maximization Algorithmare presented in Section 6. We address the issue of tuning parameter q in Section 7. We conclude with a discussion and directionsfor future research in Section 8, and relegate the proofs toSection Appendix.2. MAXIMUM Lq -LIKELIHOOD ESTIMATION2.1 Definitions and Basic PropertiesDownloaded by [Johns Hopkins University] at 14:19 23 October 2013First, let us start with the traditional maximum likelihood estimation. Suppose data X follows a distribution with probabilitydensity function fθ parameterized by θ Rd . Given thelikelihood estiobserved data x (x1 , . . . , xn ), the maximum mate is defined as θ̂MLE arg maxθ { ni 1 log f (xi ; θ )}. Similarly, the maximum Lq-likelihood estimate (Ferrari and Yang2010) is defined asθ̂MLqE arg maxθ n Lq (f (xi ; θ )),i 1where Lq (u) (u1 q 1)/(1 q) and q 0. By L’Hopital’srule, when q 1, Lq (u) log(u). The tuning parameter q iscalled the distortion parameter, which governs how distortedLq is away from the log function. Based on this property, weconclude that the MLqE is a generalization of the MLE.Define U (x; θ ) θ log f (x; θ ) fθ (x; θ )/f (x; θ ) and U (x; θ, q) θ Lq (f (x; θ )) U (x; θ )f (x; θ )1 q , we know thatθ̂MLE is a solution of the likelihood equation 0 ni 1 U (xi ; θ ).Similarly, θ̂MLqE is a solution of the Lq-likelihood equation0 n U (xi ; θ, q) i 1n U (xi ; θ )f (xi ; θ )1 q .(1)i 1It is easy to see that θ̂MLqE is a solution to a weighted versionof the likelihood equation that θ̂MLE solves. The weights are proportional to the power transformation of the probability densityfunction, f (xi ; θ )1 q . When q 1, the MLqE puts more weighton the data points with high likelihoods, and less weight on thedata points with low likelihoods. The tuning parameter q adjustshow aggressively the MLqE distorts the weight allocation. TheMLE can be considered as a special case of the MLqE withequal weights.In particular, when f is a normal distribution, our μ̂MLqE andˆ2σ MLqE satisfyμ̂MLqE nn 1i 11σˆ2 MLqE nwii 1 wii 1n wi xi ,(2)wi (xi μ̂MLqE )2 ,(3)915for solving Equations (2) and (3). Details of the algorithm aredescribed in Section 8.2.2 Consistency and Bias-Variance TradeoffBefore discussing the consistency of the MLqE, let us look atthe MLE first. It is well studied that the MLE is quite generally a consistent estimator. Suppose, the true distributionf0 F, where F is a family of distributions; we know thatf0 arg maxg F Ef0 log g(X), which shows the consistency ofthe MLE. However, when we replace the log function with theLq function, we do not have the same property.We first define f (r) , a transformed distribution of f called theescort distribution, asf (r) f (x; θ )r.f (x; θ )r dx(4)We also define F to be a family of distributions that is closedunder such a transformation (i.e., f F, f (r) F). Equippedwith these definitions, we have the following property:(1/q)f0 arg max Ef0 Lq (g(X)).g FThus we see that the maximizer of the expectation of Lqlikelihood is the escort distribution (r 1/q) of the true densityf0 . To also achieve consistency for the MLqE, Ferrari and Yang(2010) let q tend to 1 as n approaches infinity.For a parametric distribution family G {f (x; θ ) : θ },suppose it is closed under the escort transformation (i.e., θ , θ , s.t. f (x; θ ) f (x; θ )(1/q) ). We have a similar property, θ̃ arg maxθ Eθ0 Lq (f (X; θ )), where θ̃ satisfiesf (x; θ̃ ) f (x; θ0 )(1/q)We now understand that, when maximizing the Lq-likelihood,we are essentially finding the escort distribution of the true density, not the true density itself, so our MLqE is asymptoticallybiased. However, this bias can be compensated by variance reduction if the distortion parameter q is properly selected. Takethe MLqE for the normal distribution for example. With an appropriate q 1, the MLqE will partially ignore the data pointson the tails while focusing more on fitting data points aroundthe center. The MLqE obtained this way is possibly biased (especially for the scale parameter), but will be less volatile to asignificant change of data on the tails, hence, a good example ofbias-variance tradeoff. q can be considered as a tuning parameterthat adjusts the magnitude of the bias-variance tradeoff.2.3 Confidence Intervalsi 1where wi ϕ(xi ; μ̂MLqE , σˆ2 MLqE )1 q and ϕ is a normal probability density function.From Equations (2) and (3), we conclude that the MLqE ofthe mean and the variance of a normal distribution are just theweighted mean and weighted variance. When q 1, the MLqEgives smaller weights for data points lying in the tail of thenormal distribution, and puts more weights on data points nearthe center. By doing so, the MLqE becomes less sensitive tooutliers than the MLE at the cost of introducing bias into theestimation. A simple and fast reweighting algorithm is availableThere are generally two ways to construct confidence intervals for the MLqE. One is parametric, the other is nonparametric.In this section, we discuss the univariate case. The multivariatecase can be extended naturally.For the parametric way, we know that the MLqE is an Mestimator, whose asymptotic variance is available. To have theasymptotic variance be valid, we need the sample size to bereasonably large so that the central limit theorem works. However, in our application, the MLqE deals with small or moderatesample sizes in most cases. So the parametric way is not ideal,but it does provide a guideline to evaluate the estimator.The second way is the nonparametric bootstrap method. Wecreate bootstrap samples from the original sample, calculate

916Journal of the American Statistical Association, September 2013their MLqEs for all bootstrap samples. We further calculatethe lower and upper quantiles of these MLqEs, and call thesequantiles the lower and upper bounds of the confidence interval.This method is model agnostic, and works well with the MLqE.Downloaded by [Johns Hopkins University] at 14:19 23 October 20133. MLq E OF MIXTURE MODELSWe now look at the problem of estimating mixture models. A mixture model is defined as f (x) kj 1 πj fj (x; θj ).Unlike the exponential family, which is proved to be closedunder the escort transformation (Equation (4)), the mixturemodel family is not closed under such a transformation.For example, consider a mixture model with the complexity k 2. The escort transformation with 1/q 2 of thisdistribution is f (x)(1/q) (π1 ϕ1 (x) π2 ϕ2 (x))2 π12 ϕ1 (x)2 π22 ϕ2 (x)2 2π1 π2 ϕ1 (x)ϕ2 (x), which is a mixture model withthree components.More generally, suppose f0 F, where F is a mixture model(1/q) / F, we know thatfamily with complexity k. Since f0(1/q)f0 g̃ : arg max Ef0 Lq (g(X)),g F(1/q)where g̃ can be considered as the projection of f0onto F.Again, the MLqE of mixture models brings more bias to theestimate. This time, the new bias is a model bias as opposedto the estimation bias which we have discussed in the previoussection. When estimating mixture models using the MLqE, wecarry two types of bias: estimation bias and model bias. Thedistortion parameter q now adjusts both of them. This idea isillustrated in Figure 1(a).There is a simple way to partially correct the bias. Sincewe know that the MLqE is unbiased for the escort distribution of the true distribution. After we obtain the MLqE fromdata, fˆMLqE , we can blow it up by a power transformationqqg fˆMLqE / fˆMLqE dx to get a less biased estimate. However,this only partially corrects the bias since the projection fromthe escort distribution onto the mixture model family cannot berecovered by this transformation.Because the MLqE has the desirable property of being robustagainst outliers, we introduce the gross error model to evaluatethe MLqE’s performance. A gross error model is defined asf0 (x) (1 )f0 (x) ferr (x), where f0 is a mixture modelwith complexity k, ferr can be considered as a measurementerror component, and is the contamination ratio. Hence, f0 is also a mixture model with complexity k 1. The gross errordensity f0 can be considered as a small deviation from the targetdensity f0 . To build an estimator for f0 that is robust againstferr , we apply the MLqE. Generally, there are two ways to applythe MLqE in this situation.First, we can directly use a mixture model with complexityk to estimate f0 based on data from f0 . We call this approachthe direct approach. This time the model is more complex thanbefore. The idea is illustrated in Figure 1(b). Suppose, F is a/ F,mixture model family with complexity k, and f0 F, f0 (1/q)f0 / F. We obtain the MLqE of f0 (x), g̃, by (1/q)f0 g̃ : arg max Ef0 Lq (g(X)).g FFigure 1. Illustration of the MLqE of mixture models: (a) showsthe usual case, which is the MLqE of mixture models with correctlyspecified models, (b) shows the MLqE of nonmeasurement error components f0 within the gross error model f0 using the misspecifiedmodel, and (c) shows the MLqE of nonmeasurement error componentsf0 within the gross error model f0 using the correctly specified model.Here we use the estimation bias and the model bias to offset themeasurement error effect on f0 . Please note that this approachis essentially an estimation under the misspecified model.The second approach is that we use a mixture model withcomplexity k 1 to estimate f0 and project the estimate tothe k component mixture model family by removing the largestvariance component (i.e., the measurement error component)and normalizing the weights. We call this approach the indirectapproach. The projected model is our estimate for f0 . In thiscase, we essentially treat the parameters of the measurementerror component as nuisance parameters. This idea is illustratedin Figure 1(c). In Figure 1(c), g̃ is our estimate of f0 . And g̃0 ,the projection of g̃ onto F0 , is our estimate of f0 . This approachis an estimation conducted under the correctly specified model.Although the model is correctly specified, we may have higherestimation variance as we estimate more parameters.In this article, we will study the MLqE using the above twoapproaches.Please note that, when q 1, the MLqE is an inconsistentestimator. Ferrari and Yang (2010) let q 1 as n to forcethe consistency. In our case, we allow the MLqE to be inconsistent because our data are contaminated. We are no longerafter the true underlying distribution f0 that generates the data,but are more interested in estimating the nonmeasurement errorcomponents f0 using the contaminated data. Since the goal isnot to estimate f0 , being consistent will not help the estimatorin terms of robustness.

Qin and Priebe: Maximum Lq -Likelihood Estimation via the Expectation-Maximization Algorithmn 4. EM ALGORITHM WITH Lq -LIKELIHOODWe now propose a variation of the EM algorithm—the expectation maximization algorithm with Lq-likelihood (EM-Lq),which gives the local maximum Lq-likelihood. Before introducing our EM-Lq, let us briefly review the rationale of the EM.Throughout this article, we use X, Z, Z for random variablesand vectors, and x, z, z for realizations.4.1 Why Does the EM Algorithm Work?The EM algorithm is an iterative method for finding a localmaximum likelihood by making use of observed data X andmissing data Z. The rationale behind the EM is thatn log p(xi ; ) Downloaded by [Johns Hopkins University] at 14:19 23 October 2013i 1n i 1 E oldi 1 p(Z X; old )p(Z X; )Lq (p(Z X; ))p(Z X; old )1 q A( , old ) n E oldn i 1where J ( , old ) is the expected complete log-likelihood,and K( , old ) takes its minimum at old and K( , old ) old 0. Standing at the current estimate old , to climb uphill on ni 1 log p(xi ; ) only requires usto climb J, and K will automatically increase. Meanwhile,the in

Maximum Lq-Likelihood Estimation via the Expectation-Maximization Algorithm: A Robust Estimation of Mixture Models Yichen QIN and Carey E. PRIEBE We introduce a maximum Lq-likelihood estimation (MLqE) of mixture models using our proposed expectation-maximization (EM) al- gorithm, namely the EM algorithm with Lq-likelihood (EM-Lq).