Supervised Hierarchical Cross-Modal Hashing - GitHub Pages

HierarchicalDiscriminative LearningLabel Hierarchy1st layer1st layer2nd layerC5 Fc6 Fc7Kth layerRegularizedCross-modal HashingKth layer.2nd terKth layerLayer-wise SupervisionC4.Input C1layer.Image.C3.C22nd00011st layerFc1Layer-wiseSimilarity MatrixHierarchicalDiscriminative LearningLayer-wiseGround TruthFigure 2: Illustration of the proposed scheme. HiCHNet characterizes each modality of the instance with a set of layer-wisehash representations via the corresponding neural network, which is regularized to retain the hierarchical discriminativecapability and hence preserve the layer-wise semantic similarities derived from the ground truth labels.correlation maximization (SCM) method to seamlessly integratethe semantic labels into the hashing learning. In addition, tocapture the underlying semantic information, Yu et al. [38]introduced a two-stage discriminative coupled dictionary hashing(DCDH) model to jointly learn the coupled dictionaries and hashfunctions for both modalities. Furthermore, arguing that thesemantic affinities can be used to guide the hashing, Lin et al. [17]formulated a semantics-preserving hashing (SePH) paradigm wherethe probability distribution generated from semantic affinities isapproximated via minimizing the Kullback-Leibler divergence. Itis worth noting that the above methods mainly rely on the handcrafted features, which inevitably leads to the separate featureextraction and hash codes learning procedures. To overcome thisdrawback, Jiang et al.[13] established an end-to-end deep crossmodal hashing (DCMH) framework with deep neural networks,one for each modality to perform feature learning from scratch.In spite of the compelling success achieved by these methods ingeneral cases, far too little attention has been paid to the real-worlddomains with hierarchical labels like the fashion domain. In fact, itis inappropriate to directly apply existing supervised methods thattreat all labels equally and overlook the hierarchical relatednessamong them.In fact, the concept of hierarchy has been noticed by manyresearchers [16, 27, 32, 35]. For example, Song et al. [32] exploredthe hierarchical relatedness among user interests and proposeda structure-constrained multi-source multi-task learning schemefor the user interest inference. For the hashing domain, Wang etal. [35] presented a supervised hierarchical deep hashing methodin the context of unimodal hashing. Nevertheless, the potentialof hierarchical labels in cross-modal hashing has not been wellvalidated, which is the major concern of this paper.3PRELIMINARIESWe first introduce the necessary notations throughout the paper,and then define the studied task.3.1NotationN labeledSuppose that we have a set of N instances E {ei }i 1by a set of categories that are not independent but correlatedwith a hierarchy of (K 1) layers. We compile the (K 1) layersfrom top to bottom with the index set {0, 1, · · · , K }, where the 0-thlayer corresponds to the root node. Let c k denote the number ofnodes at the k-th layer. As for the i-th instance ei (vi , ti , Yi ),vi Rdv and ti Rdt stand for the original image and textfeature vectors, where dv and dt represent the respective featuren oKdimensions. Yi ykidenotes the set of label vectors fork 1k , y k , · · · , y k ]T {0, 1}c k is the label vectorei , where yki [yi1ici2kpertaining to the categories of the k-th layer3 . In particular, yikj 1,if the i-th instance ei is labeled with the j-th category at thek-th layer, otherwise yikj 0. For simplicity, we define Yk [yk1 , yk2 , · · · , ykN ] {0, 1}c N as the label matrix for the k-th layerof all instances in E. Moreover, according to the label hierarchy,we introduce a set of K layer-wise inter-modal similarity matricesn oKS Sk, where Sk {0, 1} N N corresponds to the similaritieskk 1among all instances regarding categories in the k-th layer. Inparticular, the (i, j)-th entry Sikj 1 if the image of instance eiand text of instance e j share the identical label for the k-th layer(i.e., yki ykj ), otherwise Sikj 0. Table 1 summarizes the mainnotations used in this paper.3 Here,we do not consider the 0-th layer of the root node.

Table 1: Summary of the main notations.NotationExplanationKLeiykiSkf v (f t )Θv (Θt )vi (ti )bvi (bti )hvi (hti )Num. of layers in the hierarchy except the root.The length of the hash codes.The i-th instance.Label vector of ei pertaining to the k-th layer.Inter-modal similarity matrix of the k-th layer.Hash function for the image (text) modality.Parameters of f v (f t ).Original image (text) feature vector of ei .Image (text) hash codes of ei .Image (text) hash representation of ei .3.2Problem FormulationIn this work, we aim to devise an end-to-end supervised hierarchicalcross-modal hashing learning scheme to obtain the accurate imageand text L-bit hash codes for the i-th instance, namely, bvi { 1, 1} L and bti { 1, 1} L . Based on the hash codes, we canmeasure the inter-modal similarities using the Hamming distanceas dis H (bvi , bt j ) 12 (L bTvi bt j ) and hence perform the cross-modalretrieval.To simplify the presentation, we focus on the cross-modalretrieval for the bimodal data (i.e., the image and text). Withoutlosing the generality, our task can be easily extended to thescenarios with multiple modalities. In particular, we aim tolearn hash codes for the image and text modalities (i.e., bvi sдn(f v (vi ; Θv )) and bti sдn(f t (ti ; Θt ))), respectively. sдn(·) isthe element-wise sign function, which outputs “ 1” for positivereal numbers and “ 1” for negative ones. Here, f v and f t refer tothe hashing networks with parameters Θv and Θt to be learned.4THE PROPOSED MODELIn this section, we present the proposed HiCHNet, as the majornovelty, which is able to effectively leverage the label hierarchyinformation for improving the learning of cross-modal hash codes.In particular, we first set up layer-wise hash representations forcapturing semantic characteristics in different granularity and thenenhance their discriminative power with hierarchical discriminativelearning, and finally instruct the hashing learning with regularizedcross-modal hashing.4.1Layer-wise Hash RepresentationIntuitively, as different modalities of one instance are semanticallycorrelated, an effective hashing model should be able to preservethe similarity between different modalities for the same instance.Nevertheless, it is inadvisable to directly measure the inter-modalsimilarity from the original heterogeneous feature spaces.Inspired by the huge success of the representation learning, weadopt deep neural networks to obtain more powerful image andtext representations. Regarding the image modality, we utilize theconvolution neural network (CNN) adapted from [4] consistingof five convolution layers followed by two fully-connected layers.In particular, given the i-th instance, we feed its original imagefeature vi (i.e., the pixel vector) to the CNNs, and adopt the fc7layer output as the image representation ṽi . As for the text modality,in the similar manner, we employ a neural network comprising onefully-connected layer [20] to transform the original text featurevector ti into the text representation t̃i .Having obtained the image and text representations of instances,we can perform the respective projection from the representationspace to the Hamming space and derive the hash codes for eachmodality. To fully exploit the hierarchy, our idea is to set layer-wiserepresentations for each modality corresponding to the categorylayers of the hierarchy with different granularity. Formally, weequally divide the general L-bit hash codes into K layer-wise hashcodes, namely, bvi [bv1 i , bv2 i , · · · , bvKi ] and bti [bt1i , bt2i , · · · , btKi ],where bvk i and bkti refer to the image and text hash codes of instanceei regarding the k-th layer.For the image modality, we feed the image representation ṽi toK separate networks simultaneously, each of which comprises onefully-connected layer as follows,hvk i s(Wvk ṽi gvk ), k 1, · · · , K,hvk i(1)Rzkwhere refers to the image hash representation for thek-th layer with the dimension of zk , and Wvk and gvk are the weightmatrix and bias vector, respectively. And s : R 7 R is a non-linearfunction applied element wise4 . Then, based on the set of imagen oKhash representations for the i-th instance hvk i, we can get thek 1binary layer-wise image hash codes as follows,bvk i sдn(hvk i ), k 1, · · · , K,wherebvk i(2){ 1, 1}zk .In a similar manner, we can derive then oKlayer-wise text hash representations hktiand binary text hashk 1n oKcodes bktifor the i-th instance. k 14.2Hierarchical Discriminative LearningIn a sense, as to comprehensively encode the necessary semanticinformation from the hierarchy, the layer-wise hash codes, whichcan be regarded as the projected representations for instances inthe Hamming space, should be discriminative towards the semanticclassification in different granularity over the hierarchy. Towardsthis end, we introduce K layer-wise multiple classification taskssimultaneously. For the k-th multi-classification, we particularlytake the k-th layer hash representations as the input and labelsregarding the k-th layer of the hierarchy as the ground truth.For simplicity, we take the discriminative learning of the imagemodality as an example and that of the text modality can beeffortlessly achieved in the same manner.In particular, we feed K layer-wise image hash representationsof the i-th instance to K multi-layer perceptrons as follows,pvk i so f tmax(Uvk hvk i qvk ), k 1, · · · , K,where pvk iRc k(3) refers to the output class distribution pertaining tothe k-th layer of the hierarchy, Uvk and qvk are the weight matrix andbias vector, respectively. Considering that categories in differentgranularity may contribute differently to the discriminativeregularization, we incorporate the layer confidence for each layer.4 Inthis work, we use the hyperbolic tangent function.

Ultimately, adopting the negative log-likelihood loss for the K layerwise discriminative classifications, we have,iKN hXXρk(yki )T log(pvk i ) (yki )T log(pkti ) ,Ψh (4)k 1i 1where ρ k refers to the confidence of the k-th layer and loд(·) is theelement-wise logarithm function.4.3Regularized Cross-modal HashingAbove, we have considered the layer-wise correspondence betweenthe hash codes and the category hierarchy. In this part, wefirst employ the layer-wise regularizations for comprehensivelypreserving the semantic similarities between different modalities.Then, we incorporate the binarization difference penalizing tofurther enhance the cross-modal hashing learning.Semantic Similarity Preserving. To ensure the performanceof cross-modal hashing, one major concern is to preserve the intermodal semantic similarity between two instances when they aremapped from the original representation space to the Hammingspace. Consequently, it is desirable to maximize the Hammingdistance between two instances whose semantic similarity is0, while minimizing that with the similarity of 1. Traditionally,existing researches treat all categories independently and onlydefine the universal semantic similarity to preserve, where thecategory hierarchy has not been utilized. In fact, the hash codes ofinstances from correlated categories, (e.g., “Long Skirt” and “MiniSkirt” are correlated by sharing the same ancestor category “Skirt”)tend to be more similar than that from uncorrelated ones (e.g.,“Wide Leg Jeans” and “Mini Skirt”). Accordingly, we also define thesemantic similarity in the layer-wise manner as follows,1 k T k(5)(h ) ht j ,2 viwhere ϕ ikj denotes the semantic similarity between image ofinstance ei and text of instance ej regarding the k-th layer. Thehash representations hvk i and hktj can be treated as the continuousϕ ikj surrogates of the binary hash codes bvk i and bktj , k 1, 2, · · · , K, i, j 1, 2, · · · , N , respectively.Similar to [13], we encourage ϕ ikj to approximate the binaryground truth Sikj as follows,L(ϕ ikj Sikj ) σ (ϕ ikj )Si j (1 σ (ϕ ikj ))(1 Si j ),kk(6)where σ (·) is the sigmoid function. Besides, considering that labelsin different granularity at different layers may possess differentcapabilities regarding the semantic similarity regularization, wefurther introduce the layer confidence. Simple algebra computationsenable us to reach the following objective function,Γ1 KXk 1τkNXi,j 1(Sikj ϕ ikj log(1 e ϕi j )),k(7)where τk denotes the layer confidence for the k-th layer.Binarization Difference Penalizing. Apart from the semanticpreserving regularization on hvk i ’s and hktj ’s, we further regularizethe binarization differences between hvk i and bvk i , hkti and bkti ,respectively, as to derive the optimal continuous surrogates ofAlgorithm 1 Supervised Hierarchical Cross-Modal HashingInput: Instance set E, similarity matrix set S.n oKOutput: Parameters Θv and Θt , hash code matrices Bk.k 1InitializationInitialize parameters: α, β, γ , τk , ρ k , Θv , Θt , mini-batch size: m,iteration number: M ⌈N /m⌉.repeatfor iter 1, 2, · · · , M doRandomly sample a batch of m instances from E.n oKFeed them into f v and compute Hvk.k 1Update Θv according to Eqn. (11) and (12).end forfor iter 1, 2, · · · , M doRandomly sample a batch of m instances from E.n oKFeed them into f t and compute Hkt.k 1Update Θt according to Eqn. (11) and (12).end for n oKCompute Bkaccording to Eqn. (13).k 1until Convergencethe binary hash codes. For simplicity, we introduce two sets ofn oKn oKlayer-wise hash representation matrices Hvkand Hktk 1k 1for the image and text modalities, respectively, where Hvk [hvk 1 , hvk 2 , · · · , hvk N ] Rzk N and Hkt [hkt1 , hkt2 , · · · , hktN ] Rzk N . Moreover, we can also define two sets of binary layer-wisen oKn oKhash code matrices Bv Bvkand Bt Bkt, where Bvk k 1k 1[bvk 1 , bvk 2 , · · · , bvk N ] { 1, 1}zk N and Bkt [bkt1 , bkt2 , · · · , bktN ] { 1, 1}zk N . The binarization difference regularization thus canbe written as follows,Γ2 K Xk 1 22Bvk Hvk F Bkt Hkt F ,(8)where · F denotes the Frobenius norm.Consequently, we have the following objective function towardsthe hierarchical cross-modal hashing, N XkSikj ϕ ikj log(1 e ϕi j )i,j 1k 1 22 α Bvk Hvk F Bkt Hkt F 22 β Hvk a 2 Hkt a 2 ,Ψr KXτk(9)where α and β are the nonnegative tradeoff parameters and a [1, 1, · · · , 1]T RN , and · 2 denotes the Euclidean norm. Thelast term is to balance the learned hash codes and maximize theinformation conveyed by each bit of the codes [13].Notably, to bridge the semantic gap between different modalitiesmore effectively and boost the performance of the cross-modalhashing, we adopt the unified binary hash codes (i.e., Bvk Bkt Bk )in the training procedure. Towards this end, we slightly adapt the

(a) FashionVC.(b) Ssense.Figure 3: Label hierarchy of datasets FashionVC and Ssense.derivation of the binary hash code matrix Bk as follows, Bk sдn Hvk Hkt .4.4Table 2: Statistics of our datasets.(10)Training SetRetrieval SetQuery SetTotal LabelsThe First Layer LabelsThe Second Layer LabelsJoint Model and OptimizationIntegrating the two key components of the hierarchical discriminativelearning and regularized cross-modal hashing, we reach the finalobjective formulation Ψ as follows,minBk ,Θv ,Θtγ Ψh (1 γ )Ψr ,(11)where γ is the nonnegative tradeoff parameter. Overall, we expectthe layer-wise hash codes to be discriminative for the hierarchicalsemantic classification as well as effective towards the cross-modalhashing. It is worth noting that although we assume that bothmodalities of each instance are observed in the training phase, ourscheme can also be easily extended to handle other scenarios, wheresome training instances miss certain modality. Moreover, once themodel has been trained, we can directly use f v and f t to generatehash codes for any instance with either one or two modalities andfulfill the cross-modal retrieval task.We adopt the alternating optimization strategy to solve Bk , Θvand Θt , where we optimize one variable while fixing the othertwo in each iteration and keep the iterative procedure until theobjective function converges. Due to that Θv and Θt share thesimilar optimization, here we take Θv as an example. We first Ψ calculate the derivative of Ψ with respect to hvk i as hvk iN1X(σ (ϕ ikj )hktj Sikj hktj ) 2α(hvk i bvk i ) 2βHvk a,2 j 1(12) Ψ Ψcan be derived fromusing the Θv hvk ichain rule. As for the binary hash code matrix Bk , we havewhere k 1, · · · , K, and Ψ(13) 2α(2Bk Hvk Hkt ), Bk Ψ Ψ Ψwhere k 1, · · · , K. Indeed,,andenable us tokkk B hvi ht jsolve all the parameters via the stochastic gradient descent (SGD)with back-propagation. The overall procedure of the alternatingoptimization is briefly summarized in Algorithm 1. As each iterationFashionVCSsense16, 86216, 8623, 0003582713, 69613, 6962, 00032428can decrease Ψ, whose lower bound is zero, we can guarantee theconvergence of Algorithm 1 [13, 15, 34].5EXPERIMENTTo evaluate the proposed method, we conducted extensiveexperiments on two real-world datasets by answering the followingresearch questions: Does the proposed HiCHNet outperform the state-of-the-artmethods? What is the component level contribution of HiCHNet? What is the effect of the label hierarchy?In this section, we first introduce the datasets as well as theexperimental settings, and then provide the experimental resultswith detailed discussions over each above research question.5.1DatasetsFor the evaluation, we utilized two datasets: FashionVC and Ssense,where the former is adapted from an existing dataset and the latteris created by our own.FashionVC. On one hand, we adopted the public datasetFashionVC [31] originally collected from the online fashioncommunity Polyvore5 in the context of clothing matching.FashionVC consists of 20, 726 multi-modal fashion items (e.g.,tops and bottoms), where each fashion item is composed of avisual image with a cle

hierarchical labels, which has especially great demand in the fashion domain. We propose a novel supervised hierarchical cross-modal hashing framework, which is able to seamlessly integrate the hierarchical discriminative learning and the regularized cross-modal hashing. We build a large-scale benchmark dataset from the global