Combining Information Theoretic Kernels With Generative .

162M. Bicego et al. / Neurocomputing 101 (2013) 161–169Typically, the feature vectors resulting from the generativeembedding are used to feed some kernel-based classifier, such as asupport vector machine (SVM) with standard linear or radial basisfunction (RBF) kernels. In this paper, we follow an alternative route:instead of relying on standard kernels, we investigate the use of arecently introduced family of information theoretic (IT) kernels [13].The main idea is that the IT kernels we can exploit the probabilisticnature of the generative embeddings, improving even more theclassification results of the hybrid approaches. In particular weinvestigate a particular class of IT kernels, based on a nonextensive generalization of the classical Shannon information theory,and defined on unnormalized or normalized (i.e., probability) measures. In [13], they were successfully used in text categorizationtasks, based on multinomial text representations (e.g., bags-of-words,character n-grams). Here, the idea is to consider the points of thegenerative embedding as multinomial probability distributions, thusvalid arguments for the information theoretic kernels.We illustrate the performance of combining different generative embeddings with the IT kernels on different medical applications: colon cancer detection on gene expression data,schizophrenia detection on brain MRI images, and renal cellcancer classification on tissue microarray data. Following recentwork, we adopt the so-called pLSA (probabilistic latent semanticanalysis) as a generative model, the usefulness of which has beenrecently shown in different applications [11,14–16]. The experimental results reported in this paper testify for the adequacy andstate-of-the-art performance of the combination of IT kernelswith generative embeddings.Summarizing, the main contributions of the paper are: The investigation of the use of a novel class of information theoretic (IT) kernels [13] as a similarity measure betweenobjects in a generative embedding space.A thorough investigation of different generative embedding(GE) schemes, some of them being very recent. Such a largeand extensive comparison, involving eight different generativeembeddings, was missing from the literature.The application of this hybrid scheme (GE þIT kernels) to themedical domain. Actually it is worth to notice that we exploitthe same scheme for three very different medical applications,which start from very different representations: 3D surfaces(brain classification), images (renal cancer), and microarrayexpression matrices (colon cancer).The remaining sections of the paper are organized as follows. InSection 2, the fundamental ideas of generative embeddings arereviewed, together with the basics of the schemes here investigated, while Section 3 describes the IT kernels. The proposed way ofusing the IT kernels with the generative embeddings is formalizedin Section 4. Details on applications and experimental results arereported in Section 5, and Section 6 concludes the paper.2. Generative embeddingsPursuing principled hybrid discriminative–generative classifierlearning methods is, arguably, one of the currently most interestingchallenges in machine learning research. The underlying motivationis the clear complementarity of discriminative and generativestrategies: asymptotically (in the number of labeled training examples), classification error of discriminative methods is lower than forgenerative ones [1]. On the other side, generative schemes areeffective with less data; furthermore, they allow for easier/simplerhandling of missing data and inclusion of prior knowledge about thedata. Among these hybrid generative–discriminative methods, theinterest in ‘‘generative embeddings’’ (also called generative scorespaces) has been increasing in recent years, as is testified by anincreasing literature on the class of methods (see, among other,[4,11,14,17–21]).Generative embeddings involve three key building blocks: (i) agenerative model (or a family thereof) is adopted and learnedfrom the data; (ii) this learned model is used to obtain a mappingbetween the original object space and a fixed-dimension vectorspace (usually called a score space); (iii) the objects in the trainingset are mapped into the score space and used by some discriminative learning technique. The key idea is the mapping of objectsof possibly different dimension into fixed-dimensional featurevectors, using a model of how this objects are generated. Thisopens the door to the use of standard discriminative techniques(such as support vector machines or logistic regression) and hasbeen shown to achieve higher classification accuracy than purelygenerative or discriminative approaches.Once a generative embedding is obtained, in order to use akernel-based discriminative learning approach, it is necessary toadopt a kernel that expresses similarity between pairs of points inthe score space, maybe also derived from the adopted generativemodel. The most famous example of one such kernel is the Fisherkernel [4], which is simply a Riemennian inner product, using theinverse Fisher matrix of the generative model as the underlyingmetric. In this paper, we will use kernels defined on the scorespace that are independent of the generative model.In the following subsections, we will describe the generativeembeddings used in this paper, after reviewing the pLSA generative model based on which they are built.2.1. Probabilistic latent semantic analysis (pLSA)Consider a set of documents D ¼ fd1 , . . . ,d9D9 g, each containingan arbitrary number of words, all taken from a vocabulary ofW ¼ fw1 , . . . ,w9W9 g; of course, without loss of generality, we maysimply refer to the documents and words by their indices, thus wesimplify the notation by writing D ¼ f1, . . . ,9D9g and W ¼ f1, . . . ,9W9g. This collection of documents is summarized in a bag-of-wordsfashion (i.e., ignoring the word order) into a 9W9 9D9 occurrencematrix C ¼ ½C ij , i ¼ 1, . . . ,9W9, j ¼ 1, . . . ,9D9 , where element Cij indicates the number of occurrences of the i-th word in the j-thdocument.pLSA [22] is a generative mixture model for matrix C wherethe presence of each word in each document is mediated by alatent random variable, Z A Z ¼ f1, . . . ,9Z9g (known as the topic oraspect variable). More specifically, pLSA is a mixture model for thejoint distribution of the pair of random variables D A D andW A W, where the event ðW ¼ i,D ¼ jÞ means that there is anoccurrence of the i-th word in the j-th document. pLSA expressesthe joint probability distribution PðW ¼ i,D ¼ jÞ as a mixture ofdistributions such that, in each component of the mixture (i.e., foreach topic), the random variables W and D are independent (i.e.,PðW ¼ i,D ¼ j9Z ¼ zÞ ¼ PðW ¼ i9Z ¼ zÞPðD ¼ j9Z ¼ zÞ); formally,PðW ¼ i,D ¼ jÞ ¼9Z9XPðZ ¼ zÞPðW ¼ i9Z ¼ zÞPðD ¼ j9Z ¼ zÞ:ð1Þz¼1This model is parameterized by a set of 1 þ 29Z9 multinomial distributions: the distribution of the latent topic variable fPðZ ¼ 1Þ, . . . ,PðZ ¼ 9Z9Þg; the distributions of words fPðW ¼ 19Z ¼ zÞ, . . . ,PðW ¼ 9W99Z ¼ zÞg for each z A f1, . . . ,9Z9g; the distributions ofdocuments fPðD ¼ 19Z ¼ zÞ, . . . , PðD ¼ 9D99 Z ¼ zÞg for eachz A f1, . . . ,9Z9g. Let us write these parameters compactly in a vectorp ¼ ½p1 , . . . ,p9Z9 , where pz PðZ ¼ zÞ and a pair of matrices Q and R,where Q zw PðW ¼ w9Z ¼ zÞ and Rzd PðD ¼ d9Z ¼ zÞ. Of course,both Q and R are stochastic matrices: Q zw Z0, Rzw Z 0,P9W9P9D9R ¼ 1.w ¼ 1 Q zw ¼ 1, andd ¼ 1 zd

M. Bicego et al. / Neurocomputing 101 (2013) 161–169Given a set of N independent samples fðwn ,dn Þ A W D, n ¼1, . . . ,Ng from this generative model, the log-likelihood function(from which the parameters p, Q, and R are to be estimated) canbe easily shown to beLðp,Q ,RÞ ¼9W9 X9D9XC wd logðPðW ¼ w,D ¼ dÞÞ,ð2Þw¼1d¼1Lðp,Q ,RÞ ¼9W9 X9D9X0C wd log@w¼1d¼19Z9X1pz Q zw Rzd A,ð3Þz¼1where Cwd is the number of times the pair (w, d) occurs in the set ofobservations, that is, the number of times that the w-th wordoccurs in the d-th document (as defined above). This shows thatmatrix C contains the sufficient statistics to estimate the parameters of the pLSA model. Of course, maximizing (3) w.r.t. p, Q, andR cannot be done in closed form, but can naturally be addressed viathe EM algorithm [22].It is important to note that the (multinomial) random variableD takes values in the list of documents in the training set. For thisreason, pLSA is not a full generative model of documents in thesense that it has no way to assign a probability to a previouslyunseen document.b , andb, QIn possession of estimates of the model parameters, pbR, it is possible to estimate quantities such as the probability thata given topic is present in a given documentb pbRPðZ ¼ z9D ¼ dÞ ¼ P9Z9zd zbs¼1bsR sd p:ð4Þ2.2. pLSA-based generative embeddingsGenerative embeddings can be divided into two families:those based on the generative model parameters and those basedon hidden variables of those models. The former class derives thefeatures by using differential operations with respect to themodel parameters, while the latter derive feature maps usingthe log-likelihood function of the model, focusing on the randomvariables rather than on the parameters.2.2.1. Parameter-based generative embeddingsIn this subsection, we review three of the best-known generative embeddings based on the generative model parametersThe Fisher score (FS) was the first example of generative embedding [4], and it consists of using as feature vector thetangent vector of the data log likelihood with respect tothe model parameters. In the case of the pLSA model[23], each document d A f1, . . . ,9D9g is mapped into thegradient of its log-probability w.r.t. the model parameters, which we collect into a vector h ðp,Q ,RÞ. Thelog-probability of a document d A f1, . . . ,9D9g, denoted asl(d), is obtained by marginalization,lðdi Þ ¼ log PðD ¼ dÞ ¼ log9W9XPðW ¼ w,D ¼ dÞw¼1¼ log9W9 X9Z9Xpz Q zw Rzd :ð5Þw¼1z¼1The pLSA-based Fisher score maps each document d into avector of dimension containing the derivatives of l(d) w.r.t.to the elements of h. In this score space, we define thekernel simply as the Euclidean inner space. Alternatively(although we do not consider that choice here), the kernel163could be defined as the Riemennian inner product, usingthe inverse Fisher matrix as the metric [4].The TOP kernel (where TOP is an acronym for Tangent Of Posteriorlog-odds [17]) was designed for two-class problems andis based on the gradient of the posterior log-odds ratio.Formally, given parameter estimates of two pLSA generative models for the two classes, hð 1Þ and hð þ 1Þ , agiven document d is mapped into the gradient of theposterior log-odds ratio log PðC ¼ þ 19d, hÞ log PðC ¼ 19d, hÞ w.r.t. h ¼ ðhð 1Þ , hð þ 1Þ Þ. Finally, the TOP kernel isdefined simply as the Euclidean inner product in theresulting vector space.The log-likelihood ratio (LLR) embedding [20] is similar to theFisher score, except that it uses one generative modelper class, rather than a single model. Formally, for aC-class problem, the LLR embedding maps a givendocument d into the concatenation of the gradients oflog Pðd9hð1Þ Þ, . . . ,log Pðd9hðCÞ Þ, w.r.t. the respective parameters. Consequently, the dimensionality of the LLRembedding is C times larger than that of the Fisherembedding.2.2.2. Latent-variable-based embeddingsThese methods, arising from considerations in [14], derivegenerative feature mappings from the log-likelihood, using thehidden variables of the model, rather than on its parameters.The free energy score space (FESS) is based on the observation thatthe free energy bound on the complete log-likelihooddecomposes into a sum of terms [14]; the mapping of agiven document is the vector containing the terms inthis decomposition. The details of the free energy boundand the resulting embedding (the FESS) are too long toinclude here, so the reader is referred to [14].The posterior divergence (PD) embedding is a modification of theFESS embedding [19] which also takes into account howmuch each sample affects the model. Details on thepLSA-based PD embedding and on its relationship withFESS case can be found in [19].The mixture of topics (MT) embedding simply maps a givendocument d into the 9Z9-dimensional vector containingthe conditional probabilities PðZ ¼ 19D ¼ dÞ, . . . , PðZ ¼9Z99D ¼ dÞ. Recall that these probabilities (given theparameter estimates) are computed according to (4).2.2.3. Some remarksRecently, pLSA has been used, not only for text problems, butin several other application areas, including computer vision,bioinformatics (gene expression data), and medical image analysis [11,24,25]. In imaging problems, the idea is use pLSA to modelthe co-occurrence of image features (visual words) [11,25].One obvious question that arises when using pLSA models is theselection of the number of topics 9Z9. In all our application, we haveestimated this number by using the well-known Bayesian information criterion (BIC) [26], which penalizes the likelihood with a termthat depends on number of model parameters. In the pLSA model,the number of free parameters is 9Z9 1 þ9Z9ð9 D9 þ9W9 2Þ. Thus,the number of topics is chosen as the minimizer w.r.t. 9Z9 of thepenalized log-likelihood 9W9 X9D9Xw¼1d¼10C wdlog@9Z9X1pz Q zw Rzd Az¼1pffiffiffiffiþ ½9Z9 1 þ 9Z9ð9D9 þ 9W9 2Þ logð N Þ:

164M. Bicego et al. / Neurocomputing 101 (2013) 161–169In our experiments, we consider two versions of the FESS andMT embeddings. In the first version, we train one pLSA model perclass and concatenate the resulting feature vectorss (we will referthese as FESS-1 and MT-1); in the second version, we train a pLSAmodel for the whole data, ignoring the class label (we will referthese as FESS-2 and MT-2). In summary, we will consider eightdifferent generative embeddings: MT-1, MT-2, FESS-1, FESS-2,LLR, FS, TOP, and PD.We consider two types of kernel-based classifiers: K-NN andSVM. Recall that positive definiteness is a key condition for theapplicability of a kernel in SVM learning. It was shown in [13] thatABkq is a positive definite kernel for q A ½0,1 , while kq is a positivedefinite kernel for q A ½0,2 . Standard results from kernel theory[29, Proposition 3.22] guarantee that the kernel k defined in (12)iinherits the positive definiteness of kq , thus can be safely used inSVM learning algorithms.3. Information theoretic kernels5. Experimental evaluationKernels on probability measures have been shown veryeffective in classification problems involving text, images, andother types of data [13,27,28]. Given two probability measuresp1 and p2, representing two objects, several information theoretickernels (ITKs) can be defined [13]. The Jensen–Shannon kernel isdefined asWe have applied the proposed approach to three (binary)classification problems in the medical domain, which will bedescribed in detail below: binary classification of brain MRIimages into schizophrenia/non-schizophrenia; cancer detecti

Combining information theoretic kernels with generative embeddings . images, sequences) use generative models in a standard Bayesian framework. To exploit the state-of-the-art performance of discriminative learning, while also taking advantage of generative models of the data, generative